- ceph-users - lists.ceph.io

PSA: Long Standing Debian/Ubuntu build performance issue (fixed, backports in progress)
by Mark Nelson 09 Feb '24

09 Feb '24

Hi Folks, Recently we discovered a flaw in how the upstream Ubuntu and Debian builds of Ceph compile RocksDB. It causes a variety of performance issues including slower than expected write performance, 3X longer compaction times, and significantly higher than expected CPU utilization when RocksDB is heavily utilized. The issue has now been fixed in main. Igor Fedotov, however, observed during the performance meeting today that there were no backports for the fix in place. He also rightly pointed out that it would be helpful to make an announcement about the issue given the severity for the affected users. I wanted to give a bit more background and make sure people are aware and understand what's going on. 1) Who's affected? Anyone running an upstream Ubuntu/Debian build of Ceph from the last several years. External builds from Canonical and Gentoo suffered from this issue as well, but were fixed independently. 2) How can you check? There's no easy way to tell at the moment. We are investigating if running "strings" on the OSD executable may provide a clue. For now, assume that if you are using our Debian/Ubuntu builds in a non-container configuration you are affected. Proxmox for instance was affected prior to adopting the fix. 3) Are Cephadm deployments affected? Not as far as we know. Ceph container builds are compiled slightly differently from stand-alone Debian builds. They do not appear to suffer from the bug. 4) What versions of Ceph will get the fix? Casey Bodley kindly offered to backport the fix to both Reef and Quincy. He also verified that the fix builds properly with Pacific. We now have 3 separate backport PRs for the releases here: https://github.com/ceph/ceph/pull/55500 https://github.com/ceph/ceph/pull/55501 https://github.com/ceph/ceph/pull/55502 Please feel free to reply if you have any questions! Thanks, Mark -- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

4 4

How to solve data fixity
by Michal Strnad 09 Feb '24

09 Feb '24

Hi all! In the context of a repository-type project, we need to address a situation where we cannot use periodic checks in Ceph (scrubbing) due to the project's nature. Instead, we need the ability to write a checksum into the metadata of the uploaded file via API. In this context, we are not concerned about individual file parts, but rather the file as a whole. Users will calculate the checksum and write it. Based on this hash, we should be able to trigger a check of the given files. We are aware that tools like s3cmd can write MD5 hashes to file metadata, but is there a more general approach? Does anyone have experience with this, or can you suggest a tool that can accomplish this? Thx Michal

4 5

RGW Index pool(separated SSD) tuning factor
by Jaemin Joo 09 Feb '24

09 Feb '24

Hi everyone, I confirmed that write performance has increased too much even if I apply just SSD for the index pool of rgw. I know that ~200 Bytes per object in the index pool is created. When I checked the index pool size, it's around 300 bytes ~ 400 bytes calculated. like me, If it uses index pool applying separated SSD devices, I guess there is the tuning factor like block size to reduce index pool size. Is there the tuning factor or recommendation about separated SSD for the index pool?

1 0

Does it impact write performance when SSD applies into block.wal (not block.db)
by Jaemin Joo 08 Feb '24

08 Feb '24

Hi everyone, I saw the bluestore can separate block.db, block.wal. In my case, I'd like to apply hybrid device which uses SSD, HDD to improve the small data write performance. but I don't have enough SSD to cover block.db and block.wal. so I think it can impact performance even though SSD applies into just block.wal. I just know that block.wal depends on rocksdb cache size as parameters. SSD might not need too much. 1. When I use SSD just into block.wal, Does it impact the write performance of the small data? 2. Should I make lv of ssd per osd for block.wal? 3. How much SSD do I need for block.wal relative to HDD(if I have 100TB)?

3 2

Adding a new monitor fails
by Tim Holloway 08 Feb '24

08 Feb '24

I just jacked in a completely new, clean server and I've been trying to get a Ceph (Pacific) monitor running on it. The "ceph orch daemon add" appears to install all/most of what's necessary, but when the monitor starts, it shuts down immediately, and in the manner of Ceph containers immediately erases itself and the container log, so it's not possible to see what its problem is. I looked at manual installation, but the docs appear to be oriented towards old-style non-container implementation and don't account for the newer /var/lib/ceph/*fsid*/ approach. Any tips? Last few lines in the system journal are like this: Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+0000 7f26810ae700 4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938432) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2] max score 0.00, MB/sec: 351.7 rd, 351.7 wr, level 6, files in(4, 0) out(2) MB in(92.8, 0.0) out(92.8), read-write-amplify(2.0) write- amplify(1.0) OK, records in: 2858, records dropped: 0 output_compression: NoCompression Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+0000 7f26810ae700 4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938452) EVENT_LOG_v1 {"time_micros": 1707235798938446, "job": 6, "event": "compaction_finished", "compaction_time_micros": 276718, "compaction_time_cpu_micros": 73663, "output_level": 6, "num_output_files": 2, "total_output_size": 97309398, "num_input_records": 2858, "num_output_records": 2858, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 2]} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.940+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798941291, "job": 6, "event": "table_file_deletion", "file_number": 14} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.943+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798943980, "job": 6, "event": "table_file_deletion", "file_number": 12} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946734, "job": 6, "event": "table_file_deletion", "file_number": 10} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946789, "job": 6, "event": "table_file_deletion", "file_number": 4} Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000 7f26818af700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000 7f26818af700 -1 mon.dell02@-1(synchronizing) e161 *** Got Signal Terminated *** Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000 7f26818af700 1 mon.dell02@-1(synchronizing) e161 shutdown Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+0000 7f2691a95880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+0000 7f2691a95880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete Feb 06 11:09:59 dell02.mousetech.com bash[1357898]: ceph-278fcd86-0861- 11ee-a7df-9c5c8e86cf8f-mon-dell02

2 6

Performance issues with writing files to Ceph via S3 API
by Renann Prado 08 Feb '24

08 Feb '24

Hello, I have an issue at my company where we have an underperforming Ceph instance. The issue that we have is that sometimes writing files to Ceph via S3 API (our only option) takes up to 40s, which is too long for us. We are a bit limited on what we can do to investigate why it's performing so badly, because we have a service provider in between, so getting to the bottom of this really is not that easy. That being said, the way we use the S3 APi (again, Ceph under the hood) is by writing all files (multiple millions) to the root, so we don't use *no* folder-like structure e.g. we write */<uuid>* instead of */this/that/<uuid>* . The question is: Does anybody know whether Ceph has performance gains when you create a folder structure vs when you don't? Looking at Ceph's documentation I could not find such information. Best regards, *Renann Prado*

2 3

ceph error connecting to the cluster
by arimbidhea3＠gmail.com 08 Feb '24

08 Feb '24

hello, i was tried to create osd but when i run ceph command the output is like this : root@pod-deyyaa-ceph1:~# sudo ceph -s 2024-02-02T16:01:23.627+0700 7fc762f37640 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) can anyone give me a hint or help me to solve this

2 1

ceph orch upgrade to 18.2.1 seems stuck on MDS?
by Nigel Williams 07 Feb '24

07 Feb '24

kicked off ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.1 and just MDS left to do but upgrade has been sitting for hours on this root@rdx-00:~# ceph orch upgrade status { "target_image": " quay.io/ceph/ceph@sha256:a4e86c750cc11a8c93453ef5682acfa543e3ca08410efefa30f520b54f41831f ", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "osd", "mgr", "mon", "crash" ], "progress": "1267/1306 daemons upgraded", "message": "Currently upgrading osd daemons", "is_paused": false } root@rdx-00:~# message seems erroneous as all the OSDs seem to be done.

1 1

Snapshot automation/scheduling for rbd?
by Jeremy Hansen 07 Feb '24

07 Feb '24

Can rbd image snapshotting be scheduled like CephFS snapshots? Maybe I missed it in the documentation but it looked like scheduling snapshots wasn’t a feature for block images. I’m still running Pacific. We’re trying to devise a sufficient backup plan for Cloudstack and other things residing in Ceph. Thanks. -jeremy

3 6

Help: Balancing Ceph OSDs with different capacity
by Jasper Tan 07 Feb '24

07 Feb '24

Hi I have recently onboarded new OSDs into my Ceph Cluster. Previously, I had 44 OSDs of 1.7TiB each and was using it for about a year. About 1 year ago, we onboarded an additional 20 OSDs of 14TiB each. However I observed that many of the data were still being written onto the original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a bottleneck as the 1.7TiB OSDs reached nearfull capacity. I have tried to perform a reweight (both crush reweight and reweight) to reduce the number of PGs on each 1.7TiB. This worked temporarily but resulted in many objects being misplaced and PGs being in a Warning state. Subsequently I have also tried using crush-compat balancer mode instead of upmap but did not see significant improvement. The latest changes I made was to change backfill-threshold to 0.85, hoping that PGs will no longer be assigned to OSDs that are >85% utilization. However, this did not change the situation much as I see many OSDs above >85% utilization today. Attached is a report from ceph report command. For now I have OUT-ed two of my OSDs which have reached 95% capacity. I would greatly appreciate it if someone can provide advice on this matter. Thanks Jasper Tan -- -- -- *The contents of this e-mail message and any attachments are confidential and are intended solely for addressee. The information may also be legally privileged. This transmission is sent in trust, for the sole purpose of delivery to the intended recipient. If you have received this transmission in error, any use, reproduction or dissemination of this transmission is strictly prohibited. If you are not the intended recipient, please immediately NOTIFY the sender by reply e-mail or phone and DELETE this message and its attachments, if any.*

3 3