- ceph-users - lists.ceph.io

why performance difference between 'rados bench seq' and 'rados bench rand' quite significant
by Louisa 30 Oct '24

30 Oct '24

Hi all, We used 'rados bench' to test 4k object read and write operations. Our cluster is pacific, one node, 11 bluestore osd ,db and wal share the block device. Block device is HDD. 1. testing 4k write with command 'rados bench 120 write -t 16 -b 4K -p rep3datapool --run-name 4kreadwrite --no-cleanup' 2. Before tesing 4k reads, we restarted all OSD daemons. The perfomance of 'rados bench 120 seq -t 16 -p rep3datapool --run-name 4kreadwrite' was very good, which Average IOPS: 17735; using 'ceph daemon osd.1 perf dump rocksdb' , we found the rocksdb:get_latency avgcount: 15189, avgtime: 0.000012947 (12.9us) 3. Before tesing 4k rand reads, we restarted all OSD daemons. 'rados bench 60 rand -t 16 -p rep3datapool --run-name 4kreadwrite' average IOPS: 2071 rocksdb:get_latency avgcount: 8756, avgtime: 0.001761293 (1.7ms) Q1: Why performance difference between 'rados bench seq' and 'rados bench rand' quite significant? How to explain the rocksdb get_latency perfomance between this two scenario? 4. We write 40w 4k object to the pool, restarted all OSD daemons. running 'rados bench 120 seq -t 16 -p rep3datapool --run-name 4kreadwrite' again. Average IOPS~= 2000. rocsdb:get_latency avgtime also reached milliseconds level Q2: Why 'rados bench seq' performance decresing extremly after writing some more 4k object to the pool? Q3: Is there any methods and suggestions to optimized the read performance of this scenario under this hardware configuration.

2 2

Ceph Crash Module "RADOS permission denied"
by mailing-lists 29 Oct '24

29 Oct '24

Hey Cephers, i was investigating some other issue, when I stumbled across this. I am not sure, if this is "as intended" or faulty. This is a cephadm cluster on reef 18.2.4, containerized with docker. The ceph-crash module states that it cant find its key and that it cant access RADOS. Pre-face, this is how its key looks like and what its caps are (dont worry about leaking the key, its a Test-Cluster): client.crash.bi-ubu-srv-ceph2-01 key: AQBi5CBnedEKORAAgaTwkiqO7KJ+wzu8+EGXEQ== caps: [mgr] profile crash caps: [mon] profile crash Looks like it should. As per documentation: https://docs.ceph.com/en/reef/mgr/crash/#automated-collection The key on the node/directory of this ceph-crash container is the same. This is the docker logs of the container running ceph-crash. INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 AuthRegistry(0x7f7ab4067810) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 AuthRegistry(0x7f7abb1ae0c0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7ab99ac640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba1ad640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba9ae640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s As we can see here, it says that it couldn't find its key. After some online research, I found that some names are not good/interpreted correctly. Well, just to test, i've changed the unit.run file of ceph-crash-container and restarted it. This is the part I've changed: From /var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/ceph.client.crash.bi-ubu-srv-ceph2-01.keyring To /var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/keyring Then the logs look different, but the permissions denied error is still there. INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:36:42.273+0000 7f1d04b49640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.277+0000 7f1cff7fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.281+0000 7f1cfffff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s Can anybody tell me if this is "normal"? I think this is suspicious. Because for example, if i do ceph crash ls, it does not yield anything, while there is definitely something on a node inside the folder /var/lib/ceph/xyz/crash/ Thanks in advance and best wishes!

2 1

Ceph Steering Committee Meeting 2024-10-28
by Laura Flores 29 Oct '24

29 Oct '24

Hi all, Topics we discussed today included an explanation of the CVE fix process, an evaluation of the OS recommendation documentation, a discussion on some options about releasing el8 packages for the upcoming quincy point release, and CEC election results. Here are the highlights from today's meeting: - Existing CVEs and fix/release process [Ernesto + Sage + Yaarit]: - Sage: Better to build things into the process instead of a hotfix (exception for serious situation) - Revising CVE reporting process to use github - https://docs.ceph.com/en/latest/security/process/ - [zdover] os-recommendations.rst possibly out of date -- who is the authority on which distros are supported? - add squid: centos 9 (AH), ubuntu 22 (AH), debian 12 (C) - Is Debian 12 supported in Reef? - i think so, based on https://download.ceph.com/debian-reef/dists/bookworm/ --[zdover again]: that was exactly the source upon which I based the question. So I think I have all the information I need. - [cbodley] should quincy release block on centos 8 or no? - Blocked on adding back a centos 8 builder to our matrix - Community members have managed to build packages- could we point to these? - CERN archives is another option - If we drop in packages/containers built outside of the usual upstream build process, how confident are we that these artifacts will be functionally identical to what we would have built ourselves? cf. the issues seen in the past with Debian/Ubuntu-channel builds. A build done by a community member might have different compilation options, directory paths, non-stock libraries etc. - Will meet later on this with Dan Mick - Election: - Results: https://vote.heliosvoting.org/helios/elections/e03494ce-e04c-41d0-bb05-ec5c… - Governance Update: https://github.com/ceph/ceph/pull/60518 - https://pad.ceph.com/p/css-vote-2024q4 Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Influencing the osd.id when creating or replacing an osd
by Shain Miley 28 Oct '24

28 Oct '24

Hello, I am still using ceph-deploy to add osd’s to my cluster. From what I have read ceph-deploy does not allow you to specify the osd.id when creating new osds, however I am wondering if there is a way to influence the number that ceph will assign for the next osd that is created. I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes. Thank you, Shain

7 11

RGW lifecycle wrongly removes NOT expired delete-markers which have a objects
by Александр Руденко 28 Oct '24

28 Oct '24

Hi, We have very strange behavior, when LC "restores" previous versions of objects. It looks like: 1. We have latest object: *object1* 2. We remove object and have delete-marker on top of it. We can't see object1 in bucket listing: *marker (latest)* *object1 (not latests, version1)* 3. LC decides to remove marker as Expired: *RGW log: DELETED::REDACTED-bucket-REDACTED[default.1471649938.1]):REDACTED-object-REDACTED[N3gFGHIZLylFcO9RdnlChEfIAL076VC] (delete marker expiration) wp_thrd: 0, 0* 4. and we can see object1 in bucket listing: *object1 (latest, version1)* But, it is wrong behavior because ExpiredObjectDeleteMarker - it's marker that doesn't have ANY version under it. Unfortunately we can't reproduce this behavior in our test environment (proof that it's not a normal behavior). But it occurs regularly in our production in one customer's bucket. It crashes the customer's application because for it, it looks like an "unknown file". We are 100% sure that it's LC. Because we have logs when an object was uploaded, removed (no version specified), and we have LC log which removes DM but object1 exists. In this case we have unique object names like sha256. It means that we cant have name duplication in this bucket, it can't be a new object with the same name. And we have mtime of "restored" object1 which in the past, for example 20 days past. All our findings are matched together. LC policy: { "Rules": [ { "Status": "Enabled", "Prefix": "", "NoncurrentVersionExpiration": { "NoncurrentDays": 30 }, "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "Remove all NOT latest version" } ] } Ceph 16.2.13 We have about 20 RGW instances and only two of them have LC thread enabled. The bucket has about 1.5M objects and 701 shards. I know that it's too many shards for this bucket, but it's another story) Where can be a problem?

1 0

Re: Procedure for temporary evacuation and replacement
by Frank Schilder 28 Oct '24

28 Oct '24

Hi everybody, apparently, I forgot to report back. The evacuation completed without problems and we are replacing disks at the moment. This procedure worked like a charm (please read the thread to see why we didn't just shut down OSDs and used recovery for rebuild): 1.) For all OSDs: ceph osd out ID # just set them out, this is sticky and does what you want 2.) Wait for rebalance to finish 3.) Replace disks. 4.) Deploy OSDs with the same IDs as before per host. 5.) Start OSDs and let rebalance back. During the evacuation you might want to consider setting "osd_delete_sleep" to a high value to avoid issues due to PG removal reported in this thread; see messages by Joshua Baergen in this thread. The only wish I have is that after setting the OSDs "out" it would be great to have an option to have recovery kick in in addition to speed up movement of data. Instead of just reading shard by shard from the out-OSDs, shards should also be reconstructed by recovery from all other OSDs. Our evacuation lasted for about 2 weeks. If recovery would kick in, this time would go down to 2-3 days. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Sent: Monday, October 28, 2024 4:41 AM To: Frank Schilder Subject: Re: [ceph-users] Re: Procedure for temporary evacuation and replacement Hi Frank, Finally what was the best way to do this evacuation replacement? I want to destroy all my osds node by node in my cluster due to high fragmentation so might follow your method. Thank you

1 0

Re: Install on Ubuntu Noble on Arm64?
by Alex Closs 28 Oct '24

28 Oct '24

Ubuntu noble *is* an LTS release, 24.04 > On Oct 28, 2024, at 06:40, Robert Sander <r.sander(a)heinlein-support.de> wrote: > > Hi > >> On 10/25/24 19:57, Daniel Brown wrote: >> Think I’ve asked this before but — has anyone attempted to use a cephadm type install with Debian Nobel running on Arm64? Have tried both Reef and Squid, neither gets very far. Do I need to file a request for it? > > You mean Ubuntu Noble, right? > > For Ceph Squid there are packages available for Ubuntu Jammy and Debian Bookworm: https://download.ceph.com/debian-squid/dists/ > > For Ceph Reef there are packages available for Ubuntu Focal, Ubuntu Jammy and Debian Bookworm: https://download.ceph.com/debian-reef/dists/ > > I do not think that the Ceph project builds packages for non-LTS distribution versions. > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Install on Debian Nobel on Arm64?
by Daniel Brown 28 Oct '24

28 Oct '24

Think I’ve asked this before but — has anyone attempted to use a cephadm type install with Debian Nobel running on Arm64? Have tried both Reef and Squid, neither gets very far. Do I need to file a request for it? myhost-01:~/ceph$ uname -a Linux cube-man-01 6.8.0-1010-raspi #11-Ubuntu SMP PREEMPT_DYNAMIC Thu Aug 8 23:22:41 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux myhost-01:~/ceph$ sudo ./cephadm add-repo --release reef Installing repo GPG key from https://download.ceph.com/keys/release.gpg... Installing repo file at /etc/apt/sources.list.d/ceph.list... Updating package list... Non-zero exit code 100 from apt-get update apt-get: stdout Hit:1 http://ports.ubuntu.com/ubuntu-ports noble InRelease apt-get: stdout Hit:2 http://ppa.launchpad.net/ansible/ansible/ubuntu noble InRelease apt-get: stdout Hit:3 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease apt-get: stdout Hit:4 http://ports.ubuntu.com/ubuntu-ports noble-backports InRelease apt-get: stdout Hit:5 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease apt-get: stdout Hit:6 https://ppa.launchpadcontent.net/ansible/ansible/ubuntu noble InRelease apt-get: stdout Hit:7 https://download.ceph.com/debian-reef jammy InRelease apt-get: stdout Ign:8 https://download.ceph.com/debian-reef noble InRelease apt-get: stdout Err:9 https://download.ceph.com/debian-reef noble Release apt-get: stdout 404 Not Found [IP: 10.20.1.1 3142] apt-get: stdout Reading package lists... apt-get: stderr E: The repository 'https://download.ceph.com/debian-reef noble Release' does not have a Release file. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/dhbrown/ceph/./cephadm/__main__.py", line 5579, in <module> File "/home/dhbrown/ceph/./cephadm/__main__.py", line 5567, in main File "/home/dhbrown/ceph/./cephadm/__main__.py", line 4575, in command_add_repo File "/home/dhbrown/ceph/./cephadm/cephadmlib/packagers.py", line 192, in add_repo File "/home/dhbrown/ceph/./cephadm/cephadmlib/packagers.py", line 213, in update File "/home/dhbrown/ceph/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: apt-get update: E: The repository 'https://download.ceph.com/debian-reef noble Release' does not have a Release file.

2 2

Ceph native clients
by Tim Holloway 28 Oct '24

28 Oct '24

On the whole, I prefer to use NFS for my clients to use Ceph filesystem. It has the advantage that NFS client/mount is practically guaranteed to be pre-installed on all my client systems. On the other hand, there are downsides. NFS (Ceph/NFS-Ganesha) has been known to be cranky on my network when it comes to serving resources on occasion, and should the NFS host go down, the NFS client cannot fallback to the alternate NFS host without assistance. Which I should probably add, since I run keepalive anyway, but that's another matter. My issue, then is that if I do a native/FUSE mount of Ceph FS on my desktop it hurts Ceph. The desktop hibernates when not in use and that causes the network communications between it and the Ceph servers to break until it wakes up again. In the mean time, I get "laggy OSD" errors. In extreme cases, where the desktop doesn't wake up clean and requires a hardware reset, I think it even ends up with orphan connections, since at one point I recall having to forcible terminate close to 20 connections. Ideally, then, since systemd is handling hibernation, there should be a hook in the Ceph client to ensure it and the Ceph servers are at peace with each other while it's down. Probably better reporting on the Ceph admin side as to the hostnames of clients with connection issues, but that's just icing. Tim

2 2

How Ceph cleans stale object on primary OSD failure?
by Vigneshwar S 28 Oct '24

28 Oct '24

Hi everyone, I'm studying Ceph replication and peering and have a doubt in how Ceph handles cleaning of stale objects on primary OSD failure. Scenario: Consider a placement group: PG1, whose acting set is [A, B, C] and A is the primary. t1: Client writes an object to A t2: A initiates local write t3: A sends replication request to B t4: A sends replication request to C t5: A completes local write t6: A gets replication response from B t7: A gets replication response from C t8: A is crashed and haven't sent acknowledgement to the client In case if A never comes back, the object written in B and C would be stale as it is not acknowledged to the client. How does Ceph cleans this stale object? Also, it would be helpful if you can share any documentation or articles explaining Ceph replication and peering in low-level. Regards, Vigneshwar.

4 8