- ceph-users - lists.ceph.io

Re: "BUG: soft lockup" with MDS
by Janek Bevendorff 10 Jul '25

10 Jul '25

Yes, they are older Dell PowerEdges. I have to check whether there's newer firmware, but we've been running Ceph for years without these problems. I checked the logs on the host on which I had a lockup just an hour ago, but there's nothing besides the expected hardreset messages. There are two older watchdog messages, but they are from March: -------------------------------------------------------------------------------- SeqNumber = 2089 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:16:03 Message = The watchdog timer expired. RawEventData = 0x03,0x00,0x02,0x33,0xFB,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- SeqNumber = 2088 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:06:41 Message = The watchdog timer expired. RawEventData = 0x02,0x00,0x02,0x01,0xF9,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- I grepped the logs of another host where it happened, but couldn't find any watchtdog messages there. I believe it's also unlikely that suddenly all MDS hosts (we have five active, five hot standbys, and one cold standby) start having hardware issues. I also ran a memtest on one of the hosts last week and couldn't find anything there either. On 16/04/2025 15:14, Anthony D'Atri wrote: > Curious, are your systems Dells? If so you might see some improvement from running DSU to update all the firmware. It might also be illuminating to run `racadm lclog view` > >> On Apr 16, 2025, at 8:32 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> Hi, >> >> Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: >> >> watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] >> >> (it's basically a list of all processes running on the machine). >> >> Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. >> >> I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. >> >> Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. >> >> All hosts run Ubuntu 22.04 with kernel 6.8.0. >> >> Any ideas? Thanks! >> Janek >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 13

RBD quota per namespace
by Eugen Block 10 Jul '25

10 Jul '25

Hi, I asked this question [0] five years ago, I haven't noticed anything wrt rbd quotas in the last releases. Has anyone an update on this topic? I'd appreciate it! Thanks, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZCZ6MTFS645…

1 0

pg repair starts (endless loop ?)
by Dietmar Rieder 10 Jul '25

10 Jul '25

Hi, our ceph cluster reported an inconsistent pg, so we set it to repair: # ceph pg repair 4.b10 # ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 4.b10 is active+clean+scrubbing+deep+inconsistent+repair, acting [109,10,214,148,60,58,199,129,326,165,35] # ceph pg stat 6401 pgs: 1 active+clean+scrubbing+deep+inconsistent+repair, 147 active+clean+scrubbing, 146 active+clean+scrubbing+deep, 6107 active+clean; 1.1 PiB data, 1.5 PiB used, 2.4 PiB / 3.9 PiB avail; 19 MiB/s rd, 11 MiB/s wr, 14 op/s When checking the osd logs on the osd server for osd.109 we just see every couple of seconds repeats of the following messages (which I'm not aware of having seen before when we used pg repair on other occasions): [...] 2025-07-04T09:08:05.076+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:05.076+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts 2025-07-04T09:08:06.123+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:06.123+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts 2025-07-04T09:08:10.173+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:10.196+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts 2025-07-04T09:08:12.162+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:12.162+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts [...] Is this something to worry about? Best Dietmar

2 3

2025-Q3: Stable release recommendation for production clusters
by Özkan Göksu 10 Jul '25

10 Jul '25

Hello. I'm one of the oldest Nautilus 14.2.16 user in the community. It served with honor and great stability for more than 5 years. I want to thank you for its awesome developers. But the day has come and I decided to select a new stable and trusted release/version for the next 3-4 years. I will be so grateful if you share your advice and benchmark or test results to help me select the right Ceph release/version for my 7/24 running production clusters. I have 2 different clusters for specific usages: RBD Cluster: (Win10 VM use cases) - 30 nodes (15node <2x100Gbe 100m> 15node). - I have 2000++ rbd volumes of 50-100GB each. - I use daily snapshot with monthly cycle - I use KVM and libvirt drivers - SSD Pool-RBD: Replication 3 (%95) - SSD Pool-Cephfs: Replication 3 (%5) = For internal cluster usages -------------------------------------- RGW+CephFS Cluster: (S3 Muti-side) - 20 nodes (10node <1Gbit 400km> 10node). - HDD Pool-RGW-Data: 8+2 RGW-S3 use case (%70) - SSD Pool-RGW-MetaData: Replication 3 (%5) - SSD Pool-Cephfs: Replication 3 (%25) -------------------------------------- Let's check our options and what I'm thinking about them. 1. Quincy 17.2.9: is EOL but I never had any issue with it on RBD use in my test clusters. I have a trust feeling for RBD and it is a stable completed release. 2. Reef 18.2.7: Active development for one last month but I never tested it. I have trust issues but also I want to use the latest stable release for better RGW features and stability. 3. Squid 19.2.2: I never use the latest development version for my production but if there are important RGW, S3 or Multiside improvements I should not ignore then I might consider it. . Best regards.

4 5

Managing RGW container logs filling up disk space
by Sinan Polat 10 Jul '25

10 Jul '25

Hi all, I am running a Ceph Octopus cluster deployed with cephadm. I have three RGW nodes, the RGW containers are writing their logs to /var/lib/docker/containers/<container_id>/<container_id>-json.log Over time this causes the mount point to become 100% utilized. What is the recommended way to manage or limit these logs? - Should I configure a log driver in /etc/docker/daemon.json? If so, can this be done through Ceph/cephadm, or should it be configured manually on each node? - Would setting up a logrotate for /var/lib/docker/containers/*/*-json.log be appropriate? - Are there other best practices to control container log growth in this setup? Any advice or examples from others managing this would be greatly appreciated. Thanks! Sinan

2 1

System update on RL9 kills Cephadm host daemons
by Nicola Mori 09 Jul '25

09 Jul '25

Dear Ceph users, I noticed that updating a Rocky Linux 9 host with dnf in my cluster results in some of the Cephadm daemons vanishing. Before dnf update: $ sudo systemctl list-units | grep ceph ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)alertmanager.kili.service loaded active running Ceph alertmanager.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)crash.kili.service loaded active running Ceph crash.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)grafana.kili.service loaded active running Ceph grafana.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)node-exporter.kili.service loaded active running Ceph node-exporter.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.117.service loaded active running Ceph osd.117 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.118.service loaded active running Ceph osd.118 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.119.service loaded active running Ceph osd.119 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.120.service loaded active running Ceph osd.120 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.121.service loaded active running Ceph osd.121 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.122.service loaded active running Ceph osd.122 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.123.service loaded active running Ceph osd.123 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.62.service loaded active running Ceph osd.62 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice loaded active active Slice /system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45.target loaded active active Ceph cluster b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph.target loaded active active All Ceph clusters and services After dnf update: $ sudo systemctl list-units | grep ceph ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)node-exporter.kili.service loaded active running Ceph node-exporter.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice loaded active active Slice /system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45.target loaded active active Ceph cluster b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph.target I guess this is due to dnf updating several Docker packages (docker-ce, docker-ce-cli etc.), but the same does not occur on the Rocky Linux 8 hosts, where after system updates all the Cephadm daemons are still running. Does anybody have an idea about why this might happen? Thanks, Nicola

1 0

reload SSL certificate in radosgw
by Boris 09 Jul '25

09 Jul '25

Hi, is there a way to reload the ceritificate in rgw without downtime? Or if I have multiple rgw daemons to do it one by one and wait for the last one to be active again? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

5 8

Squid and Ganesha and ingress options?
by Nigel Williams 08 Jul '25

08 Jul '25

We deployed a new Cephadm Squid cluster (400 OSDs) and are happy with everything except this time we tried Ganesha again with the ingress option (HAproxy), but soon we ran into a failed daemon event. And there is a github issue for it: https://github.com/nfs-ganesha/nfs-ganesha/issues/1158 Our questions are: Is a hotfix pending for this github issue? Is anyone using Ganesha with or without ingress, is it reliable for you? How is the relative performance between Ganesha and kernel NFS? We seem to try Ganesha every release but return to the usual kernel NFS server setup because Ganesha is unreliable, we would be interested to know what others in the community think about the Ganesha option. thanks.

2 1

Re: Heads up: bad drive for Ceph Western Digital Ultrastar DC HC560
by Konstantin Shalygin 07 Jul '25

07 Jul '25

Hi, > On 4 Jul 2025, at 13:14, Marc <Marc(a)f1-outsourcing.eu> wrote: > > How is it worse than any other hdd of that size? At the moment we have just under 3000pcs of Toshiba MG10ACA and we have not registered any such issues k

2 1

RGW: is it possible to restrict a user access to a realm?
by Michel Jouvin 07 Jul '25

07 Jul '25

Hi, We have several RGW realms hosted in the same Ceph cluster. Looking at how to restrict access to one realm to some users (among all existing ones), I don't find the information. Looking at user/realm/zonegroup/zone parameters, I don't see anything that would allow this. I saw in https://docs.ceph.com/en/latest/radosgw/account/#radosgw-account a few words about tenant isolation but it is not clear for me if it is the same thing and how you achieve it. Thanks in advance for any hint! Best regards, Michel

1 0