- ceph-users - lists.ceph.io

ceph quincy repo update to debian bookworm...?
by Christian Peters 26 Jul '23

26 Jul '23

Hi ceph users/maintainers, I installed ceph quincy on debian bullseye as a ceph client and now want to update to bookworm. I see that there is at the moment only bullseye supported. https://download.ceph.com/debian-quincy/dists/bullseye/ Will there be an update of deb https://download.coeh.com/debian-quincy/ bullseye main to deb https://download.coeh.com/debian-quincy/ boowkworm main in the near future!? Regards, Christian

3 2

cephadm and kernel memory usage
by Luis Domingues 26 Jul '23

26 Jul '23

Hi, So after, looking into OSDs memory usage, which seem to be fine, on a v16.2.13 running with cephadm, on el8, it seems that the kernel is using a lot of memory. # smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 65.0G 18.6G 46.4G userspace memory 50.1G 260.5M 49.9G free memory 9.9G 9.9G 0 ---------------------------------------------------------- 125.0G 28.8G 96.3G Comparing with a similar other cluster, same OS, same ceph version, but running packages instead if containers, and machines have a little bit more memory: # smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 52.8G 50.5G 2.4G userspace memory 123.9G 198.5M 123.7G free memory 10.6G 10.6G 0 ---------------------------------------------------------- 187.3G 61.3G 126.0G Does anyone have an idea why when using containers with podman the kernel needs a lot more memory? Luis Domingues Proton AG

2 6

Signature V4 for Ceph 16.2.4 ( Pacific )
by nguyenvandiep＠baoviet.com.vn 25 Jul '23

25 Jul '23

Can anyone help me ? I need to know do Ceph 16.2.4 support Signature V4 for S3 API ? IF yes, pls guide us Thank u all

2 1

Failing to restart mon and mgr daemons on Pacific
by Renata Callado Borges 25 Jul '23

25 Jul '23

Dear all, How are you? I have a cluster on Pacific with 3 hosts, each one with 1 mon, 1 mgr and 12 OSDs. One of the hosts, darkside1, has been out of quorum according to ceph status. Systemd showed 4 services dead, two mons and two mgrs. I managed to systemctl restart one mon and one mgr, but even after several attempts, the remaining mon and mgr services, when asked to restart, keep returning to a failed state after a few seconds. They try to auto-restart and then go into a failed state where systemd requires me to manually set them to "reset-failed" before trying to start again. But they never stay up. There are no clear messages about the issue in /var/log/ceph/cephadm.log. The host is still out of quorum. I have failed to "turn on debug" as per https://docs.ceph.com/en/pacific/rados/troubleshooting/log-and-debug/. It seems I do not know the proper incantantion for "ceph daemon X config show", no string for X seems to satisfy this command. I have tried adding this: [mon] debug mon = 20 To my ceph.conf, but no additional lines of log are sent to /var/log/cephadm.log so I'm sorry I can´t provide more details. Could someone help me debug this situation? I am sure that if just reboot the machine, it will start up the services properly, as it always has done, but I would prefer to fix this without this action. Cordially, Renata.

2 4

Does ceph permit the definition of new classes?
by wodel youchi 24 Jul '23

24 Jul '23

Hi, Can I define new device classes in ceph, I know that there are hdd, ssd and nvme, but can I define other classes? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 2

Adding datacenter level to CRUSH tree causes rebalancing
by Niklas Hambüchen 24 Jul '23

24 Jul '23

Hi Ceph users, I have a Ceph 16.2.7 cluster that so far has been replicated over the `host` failure domain. All `hosts` have been chosen to be in different `datacenter`s, so that was sufficient. Now I wish to add more hosts, including some in already-used data centers, so I'm planning to use CRUSH's `datacenter` failure domain instead. My problem is that when I add the `datacenter`s into the CRUSH tree, Ceph decides that it should now rebalance the entire cluster. This seems unnecessary, and wrong. Before, `ceph osd tree` (some OSDs omitted for legibility): ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 After assigning of `datacenter` crush buckets: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -18 146.43625 datacenter FSN-DC16 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -17 146.43625 datacenter FSN-DC18 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 -16 146.43625 datacenter FSN-DC4 -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 This shows that the tree is essentially unchanged, it just "gained a level". In `ceph status` I now get: pgs: 1167541260/1595506041 objects misplaced (73.177%) If I remove the `datacenter` level again, then the misplacement disappears. On a minimal testing cluster, this misplacement issue did not appear. Why does Ceph think that these objects are misplaced when I add the datacenter level? Is there a more correct way to do this? Thanks!

4 6

mds terminated
by dxodnd＠naver.com 24 Jul '23

24 Jul '23

hello. I am using ROK CEPH and have 20 MDSs in use. 10 are in rank 0-9 and 10 are in standby. I have one ceph filesystem, and 2 mds are trimming. Under one FILESYSTEM, there are 6 MDSs in RESOLVE, 1 MDS in REPLAY, and 3 in ACTIVE. For some reason, since 36 hours ago, RESOLVE is stuck in TRIMMING, and so are the MDSs in REPLAY. I've also tried FAILing each MDS, but to no avail. I think something should change when the MDS in REPLAY goes to RESOLVE, but I don't know what. Even looking at the logs of the REPLAY MDS, it's hard to see any messages other than it is TERMINATED every 11 minutes. I'm desperate for someone's help.

4 6

cephfs - unable to create new subvolume
by karon karon 24 Jul '23

24 Jul '23

Hello, I recently use cephfs in version 17.2.6 I have a pool named "*data*" and a fs "*kube*" it was working fine until a few days ago, now i can no longer create a new subvolume*, *it gives me the following error: Error EINVAL: invalid value specified for ceph.dir.subvolume > here is the command used: ceph fs subvolume create kube newcsivol --pool_layout data > from what I understand it seems that it creates the subvolume but immediately puts it in the trash !? here is the log : 2023-06-23T08:30:53.307+0000 7f2b929d2700 0 log_channel(audit) log [DBG] : > from='client.86289 -' entity='client.admin' cmd=[{"prefix": "fs subvolume > create", "vol_name": "kube", "sub_name": "newcsivol", "group_name": "csi", > "pool_layout": "data", "target": ["mon-mgr", ""]}]: dispatch > 2023-06-23T08:30:53.307+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Starting _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.327+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_v2] cleaning up subvolume with > path: newcsivol > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_base] subvolume path > 'b'/volumes/csi/newcsivol'' moved to trashcan > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'kube' > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Finishing _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 -1 mgr.server reply reply (22) > Invalid argument invalid value specified for ceph.dir.subvolume > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.507+0000 7f2b3ff33700 0 [prometheus INFO > cherrypy.access.139824530773776] 192.168.240.231 - - [23/Jun/2023:08:30:53] > "GET /metrics HTTP/1.1" 200 194558 "" "Prometheus/2.33.4" > 2023-06-23T08:30:54.219+0000 7f2b3ddaf700 0 [dashboard INFO request] [ > 172.29.2.142:33040] [GET] [200] [0.003s] [admin] [22.0B] > /api/prometheus/notifications > 2023-06-23T08:30:54.223+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.227+0000 7f2b3a5a8700 0 [dashboard INFO request] [ > 172.29.2.142:49348] [GET] [200] [0.019s] [admin] [22.0B] /api/prometheus > 2023-06-23T08:30:54.227+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.231+0000 7f2b3d5ae700 0 [dashboard INFO request] [ > 172.29.2.142:39414] [GET] [200] [0.022s] [admin] [9.3K] > /api/prometheus/rules > 2023-06-23T08:30:54.275+0000 7f2ba39d4700 0 log_channel(cluster) log > [DBG] : pgmap v2116480: 145 pgs: 145 active+clean; 2.8 GiB data, 12 GiB > used, 1.5 TiB / 1.5 TiB avail; 5.5 KiB/s wr, 0 op/s > my fs info : # ceph fs ls > name: kube, metadata pool: metadata, data pools: [data ] > thank for your help best regards Karim

3 3

MDS cache is too large and crashes
by Sake Ceph 24 Jul '23

24 Jul '23

At 01:27 this morning I received the first email about MDS cache is too large (mailing happens every 15 minutes if something happens). Looking into it, it was again a standby-replay host which stops working. At 01:00 a few rsync processes start in parallel on a client machine. This copies data from a NFS share to Cephfs share to sync the latest changes. (we want to switch to Cephfs in the near future). This crashing of the standby-replay mds happend a couple times now, so I think it would be good to get some help. Where should I look next? Some cephfs information ---------------------------------- # ceph fs status atlassian-opl - 8 clients ============= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active atlassian-opl.mds5.zsxfep Reqs: 0 /s 7830 7803 635 3706 0-s standby-replay atlassian-opl.mds6.svvuii Evts: 0 /s 3139 1924 461 0 POOL TYPE USED AVAIL cephfs.atlassian-opl.meta metadata 2186M 1161G cephfs.atlassian-opl.data data 23.0G 1161G atlassian-prod - 12 clients ============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active atlassian-prod.mds1.msydxf Reqs: 0 /s 2703k 2703k 905k 1585 1 active atlassian-prod.mds2.oappgu Reqs: 0 /s 961k 961k 317k 622 2 active atlassian-prod.mds3.yvkjsi Reqs: 0 /s 2083k 2083k 670k 443 0-s standby-replay atlassian-prod.mds4.qlvypn Evts: 0 /s 352k 352k 102k 0 1-s standby-replay atlassian-prod.mds5.egsdfl Evts: 0 /s 873k 873k 277k 0 2-s standby-replay atlassian-prod.mds6.ghonso Evts: 0 /s 2317k 2316k 679k 0 POOL TYPE USED AVAIL cephfs.atlassian-prod.meta metadata 58.8G 1161G cephfs.atlassian-prod.data data 5492G 1161G MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) When looking at the log on the MDS server, I've got the following: 2023-07-21T01:21:01.942+0000 7f668a5e0700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2023-07-21T01:23:13.856+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5671 from mon.1 2023-07-21T01:23:18.369+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5672 from mon.1 2023-07-21T01:23:31.719+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5673 from mon.1 2023-07-21T01:23:35.769+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5674 from mon.1 2023-07-21T01:28:23.764+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5675 from mon.1 2023-07-21T01:29:13.657+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5676 from mon.1 2023-07-21T01:33:43.886+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5677 from mon.1 (and another 20 lines about updating MDS map) Alert mailings: Mail at 01:27 ---------------------------------- HEALTH_WARN --- New --- [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (13GB/9GB); 0 inodes in use by clients, 0 stray files === Full health status === [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (13GB/9GB); 0 inodes in use by clients, 0 stray files Mail at 03:27 ---------------------------------- HEALTH_OK --- Cleared --- [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (14GB/9GB); 0 inodes in use by clients, 0 stray files === Full health status === Mail at 04:12 ---------------------------------- HEALTH_WARN --- New --- [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (15GB/9GB); 0 inodes in use by clients, 0 stray files === Full health status === [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (15GB/9GB); 0 inodes in use by clients, 0 stray files Best regards, Sake

3 3

quincy 17.2.6 - write performance continuously slowing down until OSD restart needed
by Nikola Ciprich 22 Jul '23

22 Jul '23

Hello dear CEPH users and developers, we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD. everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s) When I restart all OSD daemons, numbers immediately return to normal.. volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs. I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash.. I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too Anyone having simimar issue? I'd like to ask for hints on what should I check further.. we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy thanks a lot in advance with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

7 16