- ceph-users - lists.ceph.io

Ceph Steering Committee 2024-11-25
by Gregory Farnum 25 Nov '24

25 Nov '24

Another light meeting (we're appreciating it after our heavy governance discussions!): * Cancel next week due to Cephalocon travel * We were blocked on the quincy release due to some build issues with Ganesha (apparently we were pointed at the wrong kind of CentOS repo, and then the team was asking why we are still shipping an old version — consensus was not to change mid-stream without a reason and targeted testing), but those have been sorted out. * Report on the Ceph Ambassadors monthly meeting -Greg

1 0

Encrypt OSDs on running System. A good Idea?
by Giovanna Ratini 25 Nov '24

25 Nov '24

Hello all :-), We use Ceph both as storage in Proxmox and as storage in K8S. I would like to encrypt the OSDs. I have backups of the Proxmox machines, but honestly, I would prefer not to have to use them, as it would take two days to rebuild everything from scratch. I ran some tests on a small Proxmox machine, and telling the OSDs to encrypt themselves doesn’t seem too difficult. However, nothing is running on that storage. What issues should I expect if I take an OSD (15TB) out one at a time, encrypt it, and put it back into the cluster? I would have a long period where some OSDs are encrypted and others are not. How dangerous is this? Has anyone done it before ? Best wishes, Gio

2 2

UPGRADE_REDEPLOY_DAEMON: Upgrading daemon failed
by Stolte, Felix 25 Nov '24

25 Nov '24

Hi folks, we did upgrade one of our clusters from pacific to Quincy. Everything worked fine, but cephadm complains about one osd not being upgraded: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.15 on host osd-dmz-k5-1 failed. Upgrade daemon: osd.15: cephadm exited with an error code: 1, stderr: Redeploy daemon osd.15 ... Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df852c3fc\x2d05a0\x2d11e8\x2dbae7\x2d77689751e5e7.slice/ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service Non-zero exit code 1 from systemctl start ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15 systemctl: stderr Job for ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service failed because the control process exited with error code. systemctl: stderr See "systemctl status ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" and "journalctl -xeu ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" for details. Traceback (most recent call last): File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9679, in <module> main() File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9667, in main r = ctx.func(ctx) File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2168, in _default_image return func(ctx) File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 5992, in command_deploy deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid, File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 3301, in deploy_daemon deploy_daemon_units(ctx, fsid, uid, gid, daemon_type, daemon_id, File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 3558, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 1806, in call_throws raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') RuntimeError: Failed command: systemctl start ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15: Job for ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service failed because the control process exited with error code. See "systemctl status ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" and "journalctl -xeu ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" for details. The osd in question seems to be running fine: systemctl status ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service ● ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service - Ceph osd.15 for f852c3fc-05a0-11e8-bae7-77689751e5e7 Loaded: loaded (/etc/systemd/system/ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7@.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2024-11-16 10:02:27 CET; 1 week 2 days ago Main PID: 24583 (conmon) Tasks: 67 (limit: 76281) Memory: 6.0G CPU: 9h 57min 20.017s CGroup: /system.slice/system-ceph\x2df852c3fc\x2d05a0\x2d11e8\x2dbae7\x2d77689751e5e7.slice/ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service ├─libpod-payload-3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e │ ├─24586 /dev/init -- /usr/bin/ceph-osd -n osd.15 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false │ └─24588 /usr/bin/ceph-osd -n osd.15 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false └─supervisor └─24583 /usr/bin/conmon --api-version 1 -c 3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e -u 3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e/userdata -p /run/containers/storage/over> Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.904662) [db/memtable_list.cc:628] [default] Level-0 commit table #794120: memtable #1 done Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.904710) EVENT_LOG_v1 {"time_micros": 1732530194904694, "job": 1660, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 1, 8, 44, 0, 0, 0], "immutable_memtables": 0} Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.904789) [db/db_impl/db_impl_compaction_flush.cc:233] [default] Level summary: files[2 1 8 44 0 0 0] max score 0.78 Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [db/db_impl/db_impl_files.cc:415] [JOB 1660] Try to delete WAL files size 255924988, prev total WAL file size 256244157, number of live WAL files 2. Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [file/delete_scheduler.cc:69] Deleted file db/794117.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.905401) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do Nov 25 11:32:34 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- Nov 25 11:32:34 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 783001.8 total, 600.0 interval Cumulative writes: 24M writes, 97M keys, 24M commit groups, 1.0 writes per commit group, ingest: 119.22 GB, 0.16 MB/s Cumulative WAL: 24M writes, 11M syncs, 2.03 writes per sync, written: 119.22 GB, 0.16 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 17K writes, 61K keys, 17K commit groups, 1.0 writes per commit group, ingest: 95.37 MB, 0.16 MB/s Interval WAL: 17K writes, 8473 syncs, 2.01 writes per sync, written: 0.09 MB, 0.16 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 9.86 MB 0.5 0.0 0.0 0.0 1.9 1.9 0.0 1.0 0.0 29.6 66.03 64.32 505 0.131 0 0 L1 1/0 66.88 MB 0.7 4.6 1.9 2.7 3.3 0.7 0.0 1.8 71.1 52.1 65.56 60.72 126 0.520 100M 4156K L2 8/0 450.76 MB 0.8 7.1 0.7 6.4 6.8 0.4 0.0 10.3 53.7 51.6 135.52 118.93 16 8.470 190M 1298K L3 44/0 2.65 GB 0.1 0.7 0.3 0.4 0.4 -0.0 0.0 1.3 79.5 43.3 8.89 7.81 4 2.223 28M 17M Sum 55/0 3.17 GB 0.0 12.3 2.9 9.5 12.4 3.0 0.0 6.5 45.8 46.2 276.01 251.78 651 0.424 318M 22M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 44.0 0.12 0.12 1 0.124 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.3 2.9 9.5 10.5 1.0 0.0 0.0 60.2 51.4 209.98 187.46 146 1.438 318M 22M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 29.5 66.00 64.32 504 0.131 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 102.0 0.02 0.00 1 0.025 0 0 Uptime(secs): 783001.9 total, 600.0 interval Flush(GB): cumulative 1.906, interval 0.005 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 12.45 GB write, 0.02 MB/s write, 12.35 GB read, 0.02 MB/s read, 276.0 seconds Interval compaction: 0.01 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 9.86 MB 0.5 0.0 0.0 0.0 1.9 1.9 0.0 1.0 0.0 29.6 66.03 64.32 505 0.131 0 0 L1 1/0 66.88 MB 0.7 4.6 1.9 2.7 3.3 0.7 0.0 1.8 71.1 52.1 65.56 60.72 126 0.520 100M 4156K L2 8/0 450.76 MB 0.8 7.1 0.7 6.4 6.8 0.4 0.0 10.3 53.7 51.6 135.52 118.93 16 8.470 190M 1298K L3 44/0 2.65 GB 0.1 0.7 0.3 0.4 0.4 -0.0 0.0 1.3 79.5 43.3 8.89 7.81 4 2.223 28M 17M Sum 55/0 3.17 GB 0.0 12.3 2.9 9.5 12.4 3.0 0.0 6.5 45.8 46.2 276.01 251.78 651 0.424 318M 22M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.3 2.9 9.5 10.5 1.0 0.0 0.0 60.2 51.4 209.98 187.46 146 1.438 318M 22M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 29.5 66.00 64.32 504 0.131 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 102.0 0.02 0.00 1 0.025 0 0 Uptime(secs): 783001.9 total, 0.0 interval Flush(GB): cumulative 1.906, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 12.45 GB write, 0.02 MB/s write, 12.35 GB read, 0.02 MB/s read, 276.0 seconds How do i fix this? We tried redeploying the osd but to no success. Best regards Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 1

Ceph OSD perf metrics missing
by Kilian Ries 25 Nov '24

25 Nov '24

Hi, i'm running a Ceph v18.2.4 cluster. I'm trying to build some latency monitoring with the ceph daemon osd.4 perf dump cli command. On most of the OSDs i get all the metrics i need. On some OSDs i only get zero values: osd.op_latency.avgcount: 0 I already tried restarting the OSD process which didn't help. I also tried to reset the metrics via ceph daemon osd.4 perf reset all but that didn't help either. How can that be that some OSDs don't show values here? How can i fix that? Thanks Regards, Kilian

3 4

Separate gateway for bucket lifecycle
by Szabo, Istvan (Agoda) 25 Nov '24

25 Nov '24

Hi, I remember some discussion that someone is using separated gateways for bucket lifecycles but I couldn't find it. How is that can be possible? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

How to speed up OSD deployment process
by YuFan Chen 23 Nov '24

23 Nov '24

Hi, I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices). I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete. However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next. Is there a way to speed up the OSD deployment process? Thanks in advance for your help! Best regards, Yufan Chen

4 6

multisite sync issue with bucket sync
by Christopher Durham 22 Nov '24

22 Nov '24

I have a 2-site multisite configuration on cdnh 18.2.4 on EL9. After system updates, we discovered that a particular bucket had several thousand objects missing, which the other side had. Newly created objects were being replicated just fine. I decided to 'restart' syncing that bucket. Here is what I did On the side with misisng objects: > radosgw-admin bucket sync init --bucket <bucketname> --src-zone <zone> I restarted the radosgw set up to do the sync thread on the same zone as I ran the radosgw-admin command. Logs on the radosgw src-zone side show GETs with http code 200 for objects that do not exist on the side with missing objects, and GETs with http 304 for objects that already exist on the side with missing objects. So far, so good. As I said, the bucket is active. So on the src-zone side, data is continually being written to /prefixA/../../ There is also data being written to /prefixB/../../ prefixA/ comes lexographically before prefixB/ What happens is that all the 304s happen as it scans the bucket, then starts pulling with GETs and http 200s for the objects the side doing the sync doesnt have. This is on /prefixA. When it 'caches up' with alldata in /prefixA at the moment, the sync seems to START OVER with /prefixA, giving 304s for everything that existed in the bucket up to the moment it caught up, then doing GETs with 200s for the remainingnewer objects. This happens over and over again. It NEVER gets to /prefixB. So it seems to be periodically catching up to /prefixA, but never going on to /prefixB that is also being written to There are 1.2 million objects in this bucket, with about 35 TiB in the bucket. There is a lifecycle expiration happening of 60 days. Any thoughts would be appreciated. -Chris

2 7

please unsubscribe
by Debian 108 22 Nov '24

22 Nov '24

Hi this is the tenth unsubscribe mail I send and after few minutes I receive another email. please, could some admin delete my email from the mailing list ? thanks

2 1

Crush rule examples
by Andre Tann 22 Nov '24

22 Nov '24

Hi all, I'm trying to understand how crush rules need to be set up, and much to my surprise I cannot find examples and/or good explanations (or I'm too stupid to understand them ;) ) My setup looks like this: DC1 node01 node02 node03 node04 node05 DC2 node01 node01 node01 node01 -- Andre Tann

4 10

MDS blocklist/evict clients during network maintenance
by Eugen Block 22 Nov '24

22 Nov '24

Hi, can anyone share some experience with these two configs? ceph config get mds mds_session_blocklist_on_timeout true ceph config get mds mds_session_blocklist_on_evict true If there's some network maintenance going on and the client connection is interrupted, could it help to disable evicting and blocklisting MDS clients? And what risks should we be aware of if we tried that? We're not entirely sure yet if this could be a reasonable approach, but we're trying to figure out how to make network maintenance less painful for clients. I'm also looking at some other possible configs, but let's start with these two first. Any comments would be appreciated! Thanks! Eugen

2 2