September 2023 - ceph-users

Rocksdb compaction and OSD timeout
by J-P Methot 12 Sep '23

12 Sep '23

Hi, We're running latest Pacific on our production cluster and we've been seeing the dreaded 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s' error. We have reasons to believe this happens each time the RocksDB compaction process is launched on an OSD. My question is, does the cluster detecting that an OSD has timed out interrupt the compaction process? This seems to be what's happening, but it's not immediately obvious. We are currently facing an infinite loop of random OSDs timing out and if the compaction process is interrupted without finishing, it may explain that. -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

6 19

ceph orch command hung
by Taku Izumi 12 Sep '23

12 Sep '23

Hi all, I have 4-nodes ceph cluster. After I shutted down my cluster, I tried to start it again, but failed due to ceph orch xxx (such as status) commands hung. How sould I recover from this problem ? root@ceph-manager:/# ceph orch status ==> hung ^CInterrupted root@ceph-manager:/# ceph status cluster: id: 4588ed80-352b-11ee-9eae-157ca4325420 health: HEALTH_ERR 2 failed cephadm daemon(s) 1 filesystem is degraded 1 filesystem is offline pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set 10 slow ops, oldest one blocked for 3736 sec, mon.ceph-osd0 has slow ops services: mon: 4 daemons, quorum ceph-manager,ceph-osd0,ceph-osd1,ceph-osd2 (age 64m) mgr: ceph-manager.kurjlh(active, since 64m), standbys: ceph-osd0.jodevs mds: 0/1 daemons up (1 failed), 2 standby osd: 3 osds: 3 up (since 64m), 3 in (since 2w) flags pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover data: volumes: 0/1 healthy, 1 failed pools: 11 pools, 243 pgs objects: 3.01k objects, 9.4 GiB usage: 28 GiB used, 2.8 TiB / 2.8 TiB avail pgs: 243 active+clean root@ceph-manager:/# ceph health detail HEALTH_ERR 2 failed cephadm daemon(s); 1 filesystem is degraded; 1 filesystem is offline; pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set; 10 slow ops, oldest one blocked for 3741 sec, mon.ceph-osd0 has slow ops [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon rgw.sno_rgw.ceph-manager.umzmku on ceph-manager is in error state daemon rgw.sno_rgw.ceph-osd2.vfpmbs on ceph-osd2 is in error state [WRN] FS_DEGRADED: 1 filesystem is degraded fs sno_cephfs is degraded [ERR] MDS_ALL_DOWN: 1 filesystem is offline fs sno_cephfs is offline because no MDS is active for it. [WRN] OSDMAP_FLAGS: pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set [WRN] SLOW_OPS: 10 slow ops, oldest one blocked for 3741 sec, mon.ceph-osd0 has slow ops

3 3

cannot create new OSDs - ceph version 17.2.6 (810db68029296377607028a6c6da1ec06f5a2b27) quincy (stable)
by Konold, Martin 11 Sep '23

11 Sep '23

Hi, I want to create a new OSD on a 4TB Samsung MZ1L23T8HBLA-00A07 enterprise nvme device. Creating the OSD works but it cannot be initialized and therefore not started. In the log I see an entry about a failed assert. ./src/os/bluestore/fastbmap_allocator_impl.cc: 405: FAILED ceph_assert((aligned_extent.length % l0_granularity) == 0) Is this the culprit? In addition at the end of the logfile there is a failed mount and a failed osd init mentioned. 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluefs _check_allocations OP_FILE_UPDATE_INC invalid extent 1: 0x140000~10000: duplicate reference, ino 30 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluefs mount failed to replay log: (14) Bad address 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 20 bluefs _stop_alloc 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-43) _open_bluefs failed bluefs mount: (14) Bad address 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 10 bluefs maybe_verify_layout no memorized_layout in bluefs superblock 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-43) _open_db failed to prepare db environment: 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 1 bdev(0x5565c261fc00 /var/lib/ceph/osd/ceph-43/block) close 2023-09-11T16:30:04.940+0200 7f99aa28f3c0 -1 osd.43 0 OSD:init: unable to mount object store 2023-09-11T16:30:04.940+0200 7f99aa28f3c0 -1 ** ERROR: osd init failed: (5) Input/output error -- Regards, ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

ceph_leadership_team_meeting_s18e06.mkv
by Ernesto Puerta 11 Sep '23

11 Sep '23

Dear Cephers, Today brought us an eventful CTL meeting: it looks like Jitsi recently started requiring user authentication <https://jitsi.org/blog/authentication-on-meet-jit-si/> (anonymous users will get a "Waiting for a moderator" modal), but authentication didn't work against Google or GitHub accounts, so we had to move to the good old Google Meet. As a result of this, Neha has kindly set up a new private Slack channel (#clt) to allow for quicker communication among CLT members (if you usually attend the CLT meeting and have not been added, please ping any CLT member to request that). Now, let's move on the important stuff: *The latest Pacific Release (v16.2.14)* *The Bad* The 14th drop of the Pacific release has landed with a few hiccups: - Some .deb packages were made available to downloads.ceph.com before the release process completion. Although this is not the first time it happens, we want to ensure this is the last one, so we'd like to gather ideas to improve the release publishing process. Neha encouraged everyone to share ideas here: - https://tracker.ceph.com/issues/62671 - https://tracker.ceph.com/issues/62672 - v16.2.14 also hit issues during the ceph-container stage. Laura wanted to raise awareness of its current setbacks <https://pad.ceph.com/p/16.2.14-struggles> and collect ideas to tackle them: - Enforce reviews and mandatory CI checks - Rework the current approach to use simple Dockerfiles <https://github.com/ceph/ceph/pull/43292> - Call the Ceph community for help: ceph-container is currently maintained part-time by a single contributor (Guillaume Abrioux). This sub-project would benefit from the sound expertise on containers among Ceph users. If you have ever considered contributing to Ceph, but felt a bit intimidated by C++, Paxos and race conditions, ceph-container is a good place to shed your fear. *The Good* Not everything about v16.2.14 was going to be bleak: David Orman brought us really good news. They tested v16.2.14 on a large production cluster (10gbit/s+ RGW and ~13PiB raw) and found that it solved a major issue affecting RGW in Pacific <https://github.com/ceph/ceph/pull/52552>. *The Ugly* During that testing, they noticed that ceph-mgr was occasionally OOM killed (nothing new to 16.2.14, as it was previously reported). They already tried: - Disabling modules (like the restful one, which was a suspect) - Enabling debug 20 - Turning the pg autoscaler off Debugging will continue to characterize this issue: - Enable profiling (Mark Nelson) - Try Bloomberg's Python mem profiler <https://github.com/bloomberg/memray> (Matthew Leonard) *Infrastructure* *Reminder: Infrastructure Meeting Tomorrow. **11:30-12:30 Central Time* Patrick brought up the following topics: - Need to reduce the OVH spending ($72k/year, which is a good cut in the Ceph Foundation budget, that's a lot less avocado sandwiches for the next Cephalocon): - Move services (e.g.: Chacra) to the Sepia lab - Re-use CentOS (and any spared/unused) machines for devel purposes - Current Ceph sys admins are overloaded, so devel/community involvement would be much appreciated. - More to be discussed in tomorrow's meeting. Please join if you think you can help solve/improve the Ceph infrastrucru! *BTW*: today's CDM will be canceled, since no topics were proposed. Kind Regards, Ernesto

5 6

MGR executes config rm all the time
by Frank Schilder 11 Sep '23

11 Sep '23

Hi all, I recently started observing that our MGR seems to execute the same "config rm" commands over and over again, in the audit log: 9/10/23 10:03:24 AM[INF]from='mgr.252911336 ' entity='mgr.ceph-25' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/ceph-25/mirror_snapshot_schedule"}]: dispatch 9/10/23 10:03:24 AM[INF]from='mgr.252911336 192.168.32.68:0/63' entity='mgr.ceph-25' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/ceph-25/mirror_snapshot_schedule"}]: dispatch 9/10/23 10:03:19 AM[INF]from='mgr.252911336 ' entity='mgr.ceph-25' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/ceph-25/trash_purge_schedule"}]: dispatch 9/10/23 10:03:19 AM[INF]from='mgr.252911336 192.168.32.68:0/63' entity='mgr.ceph-25' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/ceph-25/trash_purge_schedule"}]: dispatch 9/10/23 10:02:24 AM[INF]from='mgr.252911336 ' entity='mgr.ceph-25' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/ceph-25/mirror_snapshot_schedule"}]: dispatch We don't have mirroring, ts a single cluster. What is going on here and how can I stop that? I already restarted all MGR daemons to no avail. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

Best practices regarding MDS node restart
by Alexander E. Patrakov 10 Sep '23

10 Sep '23

Hello, I am interested in the best-practice guidance for the following situation. There is a Ceph cluster with CephFS deployed. There are three servers dedicated to running MDS daemons: one active, one standby-replay, and one standby. There is only a single rank. Sometimes, servers need to be rebooted for reasons unrelated to Ceph. What's the proper procedure to follow when restarting a node that currently contains an active MDS server? The goal is to minimize the client downtime. Ideally, they should not notice even if they play MP3s from the CephFS filesystem (note that I haven't tested this exact scenario) - is this achievable? I tried to use the "ceph mds fail mds02" command while mds02 was active and mds03 was standby-replay, to force the fail-over to mds03 so that I could reboot mds02. Result: mds02 became standby, while mds03 went through reconnect (30 seconds), rejoin (another 30 seconds), and replay (5 seconds) phases. During the "reconnect" and "rejoin" phases, the "Activity" column of "ceph fs status" is empty, which concerns me. It looks like I just caused a 65-second downtime. After all of that, mds02 became standby-replay, as expected. Is there a better way? Or, should I have rebooted mds02 without much thinking? -- Alexander E. Patrakov

2 1

CephFS session recovery with different source IP
by caskd 09 Sep '23

09 Sep '23

Hello fellow ceph users, i've been looking for a way to reduce the recovery of a CephFS mount when the source IP changes. It seems like the sessions refuse to be working properly when that happens a reconnect takes ages. Is there a way to reduce that interval or make CephFS work with mobile sessions? Other great examples of this happening is the case of "privacy IPv6 addresses". -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

1 0

Unhappy Cluster
by Dave S 09 Sep '23

09 Sep '23

Hi Everyone, I've been fighting with a ceph cluster that we have recently physically relocated and lost 2 OSDs during the ensuing power down and relocation. After powering everything back on we have 3 incomplete 3 remapped+incomplete And indeed we have 2 OSDs that died along the way. The reason I'm contacting the list is that I'm surprised that these PGs are incomplete. We're running Erasure coding with K=4, M=2 which in my understanding we should be able to lose 2 OSDs without an issue. Am I mis-understanding this or does m=2 mean you can lose m-1 OSDs? Also, these two OSDs happened to be in the same server (#3 of 8 total servers). This is an older cluster running Nautilus 14.2.9. Any thoughts? Thanks -Dave

2 3

MGR Memory Leak in Restful
by Chris Palmer 08 Sep '23

08 Sep '23

I first posted this on 17 April but did not get any response (although IIRC a number of other posts referred to it). Seeing as MGR OOM is being discussed at the moment I am re-posting. These clusters are not containerized. Is this being tracked/fixed or not? Thanks, Chris ------------------------------- We've hit a memory leak in the Manager Restful interface, in versions 17.2.5 & 17.2.6. On our main production cluster the active MGR grew to about 60G until the oom_reaper killed it, causing a successful failover and restart of the failed one. We can then see that the problem is recurring, actually on all 3 of our clusters. We've traced this to when we enabled full Ceph monitoring by Zabbix last week. The leak is about 20GB per day, and seems to be proportional to the number of PGs. For some time we just had the default settings, and no memory leak, but had not got around to finding why many of the Zabbix items were showing as Access Denied. We traced this to the MGR's MON CAPS which were "mon 'profile mgr'". The MON logs showed recurring: log_channel(audit) log [DBG] : from='mgr.284576436 192.168.xxx.xxx:0/2356365' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: access denied Changing the MGR CAPS to "mon 'allow *'" and restarting the MGR immediately allowed that to work, and all the follow-on REST calls worked. log_channel(audit) log [DBG] : from='mgr.283590200 192.168.xxx.xxx:0/1779' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: dispatch However it has also caused the memory leak to start. We've reverted the CAPS and are back to how we were. Two questions: 1) No matter what the REST consumer is doing, the MGR should not accumulate memory, especially as we can see that the REST TCP connections have wrapped up. Is there anything more we can do to diagnose this? 2) Setting "allow *" worked, but is there are better setting just to allow the "pg dump" call (in addition to profile mgr)? Thanks, Chris

2 1

Re: Rocksdb compaction and OSD timeout
by xiaowenhao111 08 Sep '23

08 Sep '23

1 0

failure domain and rack awareness
by Reza Bakhshayeshi 07 Sep '23

07 Sep '23

Hello, What is the best strategy regarding failure domain and rack awareness when there are only 2 physical racks and we need 3 replicas of data? In this scenario what is your point of view if we create 4 artificial racks at least to be able to manage deliberate node maintenance in a more efficient way? Regards, Reza

1 0

Permissions of the .snap directory do not inherit ACLs in 17.2.6
by MARTEL Arnaud 07 Sep '23

07 Sep '23

Hi, I'm facing the same situation as described in bug #57084 (https://tracker.ceph.com/issues/57084) since I upgraded from 16.2.13 to 17.2.6 for example: root@faiserver:~# getfacl /mnt/ceph/default/ # file: mnt/ceph/default/ # owner: 99 # group: nogroup # flags: -s- user::rwx user:s-sac-acquisition:rwx group::rwx group:acquisition:r-x group:SAC_R:r-x mask::rwx other::--- default:user::rwx default:user:s-sac-acquisition:rwx default:group::rwx default:group:acquisition:r-x default:group:SAC_R:r-x default:mask::rwx default:other::--- root@faiserver:~# getfacl /mnt/ceph/default/.snap # file: mnt/ceph/default/.snap # owner: 99 # group: nogroup # flags: -s- user::rwx group::rwx other::r-x </pre> Before creating a new bug report, could you tell me if someone has the same problem with 17.2.6 ?? Kind regards, Arnaud

2 3

Is it possible (or meaningful) to revive old OSDs?
by ceph-mail＠rikdvk.mailer.me 07 Sep '23

07 Sep '23

Hello, I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since. I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start. Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes". I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster? My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard? [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-o… [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-… Many thanks!

5 5

Re: Debian/bullseye build for reef
by Josh Durgin 07 Sep '23

07 Sep '23

We weren't targeting bullseye once we discovered the compiler version problem, the focus shifted to bookworm. If anyone would like to help maintaining debian builds, or looking into these issues, it would be welcome: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 https://tracker.ceph.com/issues/61845 On Mon, Aug 21, 2023 at 7:50 AM Matthew Darwin <bugs(a)mdarwin.ca> wrote: > Thanks for the link to the issue. Any reason it wasn't added to the > release notes (for bullseye). > > I am also waiting for this to be available to start testing. > On 2023-08-21 10:25, Josh Durgin wrote: > > There was difficulty building on bullseye due to the older version of GCC > available: https://tracker.ceph.com/issues/61845 > > On Mon, Aug 21, 2023 at 3:01 AM Chris Palmer <chris.palmer(a)idnet.com> <chris.palmer(a)idnet.com> wrote: > > > I'd like to try reef, but we are on debian 11 (bullseye). > In the ceph repos, there is debian-quincy/bullseye and > debian-quincy/focal, but under reef there is only focal & jammy. > > Is there a reason why there is no reef/bullseye build? I had thought > that the blocker only affected debian-bookworm builds. > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 2

Contionuous spurious repairs without cause?
by Christian Theune 07 Sep '23

07 Sep '23

Hi, this is a bit older cluster (Nautilus, bluestore only). We’ve noticed that the cluster is almost continuously repairing PGs. However, they all finish successfully with “0 fixed”. We do not see the trigger why Ceph decides to repair the PGs and it’s happening for a lot of PGs, not any specific individual one. Deep-scrubs are generally running, but currently a bit late as we had some recoveries in the last week. Logs look regular aside from the number of repairs. Here’s the last weeks from the perspective of a single PG. There’s one repair, but the same thing seems to happen for all PGs. 2023-08-06 16:08:17.870 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-06 16:08:18.270 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-07 21:52:22.299 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-07 21:52:22.711 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-09 00:33:42.587 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-09 00:33:43.049 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-10 09:36:00.590 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-10 09:36:28.811 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-11 12:59:14.219 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-11 12:59:14.567 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-12 13:52:44.073 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-12 13:52:44.483 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-14 01:51:04.774 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-14 01:51:33.113 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-15 05:18:16.093 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-15 05:18:16.520 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-16 09:47:38.520 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-16 09:47:38.930 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-17 19:25:45.352 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-17 19:25:45.775 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-19 05:40:43.663 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-19 05:40:44.073 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-20 12:06:54.343 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-20 12:06:54.809 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-21 19:23:10.801 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-21 19:23:39.936 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-23 03:43:21.391 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-23 03:43:21.844 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-24 04:21:17.004 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-24 04:21:47.972 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-25 06:55:13.588 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-25 06:55:14.087 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-26 09:26:01.174 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-26 09:26:01.561 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-27 11:18:10.828 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-27 11:18:11.264 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-28 19:05:42.104 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-28 19:05:42.693 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-30 07:03:10.327 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-30 07:03:10.805 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-31 14:43:23.849 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-31 14:43:50.723 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-09-01 20:53:42.749 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-01 20:53:43.389 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-02 22:57:49.542 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-02 22:57:50.065 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-04 03:16:14.754 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-04 03:16:15.295 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-05 14:50:36.064 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 repair starts 2023-09-05 14:51:04.407 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 repair ok, 0 fixed Googling didn’t help, unfortunately and the bug tracker doesn’t appear to have any relevant issue either. Any ideas? Liebe Grüße, Christian Theune -- Christian Theune · ct(a)flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

3 4

Join Us for the Relaunch of the Ceph User + Developer Monthly Meeting!
by Laura Flores 06 Sep '23

06 Sep '23

Hi Ceph users and developers, We’re happy to announce a relaunch of the Ceph User + Developer Monthly Meeting – a virtual platform that has long been in our community for encouraging discussion and collaboration between users and developers. With this relaunch, we aim to recenter the platform around user-facing topics that go beyond immediate bug reports, such as long-term improvements and knowledge sharing. Users and developers are encouraged to submit focus topics to this Google form <https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v…> [1] in preparation for each meeting. Read more about what we're looking for in focus topics here: https://ceph.io/en/news/blog/2023/user-dev-meeting-relaunch/ Join us on *September 21st, 10:00am EST *at this link <https://meet.jit.si/ceph-user-dev-monthly> [2] to witness the relaunch! - Laura Flores 1. User + Dev Google form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… 2. Meeting link: https://meet.jit.si/ceph-user-dev-monthly -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Questions about 'public network' and 'cluster nertwork'?
by Louis Koo 06 Sep '23

06 Sep '23

if the public network and cluster network use the same ip, why still need to send heartbeats to hb_front_server and hb _back_server at same time?

1 0

insufficient space ( 10 extents) on vgs lvm detected locked
by absankar89＠gmail.com 06 Sep '23

06 Sep '23

ceph orch device ls - output ( insufficient space ( 10 extents) on vgs lvm detected locked ) Quincy version , Is this just warning or any action should be taken.

2 1

lack of RGW_API_HOST in ceph dashboard, 17.2.6, causes ceph mgr dashboard problems
by Christopher Durham 05 Sep '23

05 Sep '23

Hi, I am using 17.2.6 on Rocky Linux 8 The ceph mgr dashboard, in my situation, (bare metal install, upgraded from 15->16-> 17.2.6), can no longer hit the ObjectStore->(Daemons,Users,Buckets) pages. When I try to hit those pages, it gives an error: RGW REST API failed request with status code 403 {"Code": "AccessDenied", RequestId: "xxxxxxx", HostId: "yyyy-<my zone>"} The log of the rgw server it hit has: "GET /admin/metadata/user?myself HTTP/1.1" 403 125 It appears that the mgr dashboard setting RGW_API_HOST is no longer an option that can be set, nor does that name exist anywhere under /usr/share/ceph/mgr/dashboard, and: # ceph dashboard set-rgw-api-host <host> is no longer in existence in 17.2.6 However, since my situation is an upgrade, the config value still exists in my config, and I can retrieve it with: # ceph dashboard get-rgw-api-host To get the to work in my situation, I have modified /usr/share/ceph/mgr/dashboard/settings.py and re-added RGW_API_HOST to the Options class using RGW_API_HOST = Settings('', [dict,str]) I then modified /usr/share/ceph/mgr/dashboard/services/rgw_request.py such that each rgw daemon retrieved has its 'host' member set to Settings.RGW_API_HOST. Then after restarting the mgr, I was able to access the Objectstore->(Daemons,Users,Buckets) pages in the dashboard. HOWEVER, I know this is NOT the right way to fix this, it is a hack. It seems like the dashboard is trying to contact an rgw server individually. For us, the RGW_API_HOST is a name in DNS: s3.my.dom, that has multiple A records, one for each of our rgw servers, each of which have the *same* SSL cert with CN and SubjectAltNames that allow the cert to present itself as both s3.my.dom as well as the individual host name (SubjectAltName has ALL the rgw servers in it). This works well for us and has done so since 15.x.y, The endpoint for the zone is set to s3.my.dom. Thus my users only have a single endpoint to care about, unless there is a failure situation onan rgw server. (We have other ways of handling that). Any thoughts on the CORRECT way to handle this so I can have the ceph dashboard work with the ObjectStore->(Daemons,Users,Buckets) pages? Thanks. -Chris

2 2

RGW Lua - writable response header/field
by Ondřej Kukla 05 Sep '23

05 Sep '23

Hello, We have a RGW setup that has a bunch of Nginx in front of RGWs to work as a LB. I’m currently working on some metrics and log analysis from the LB logs. At the moment I’m looking at possibilities to recognise the type of s3 request on the LB. I know that matching the format shouldn’t be extremely hard, but I was looking into a possibility to extract the information from RGW as that’s the part that’s aware of that. I was working with the LUA part of RGW before so I know that the Request.RGWOp Field is an great fit. I would like to add this as a some kind of response header, but unfortunately that’s not possible at the moment if I’m not wrong. Has anyone looked into this (wink wink Yuval :))? Or do you have a recommendation how to do it? Thanks a lot. Regards, Ondrej

2 1

rgw replication sync issue
by ankit raikwar 04 Sep '23

04 Sep '23

Hello Users, We have the environment as below. Both environments are the zones of one RGW multisite zonegroup, whereas the DC zone is the primary and the DR zone is the secondary at this point. DC Ceph Version: 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Number of rgw daemons : 25 DR Ceph Version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Number of rgw daemons : 25 Environment description: Both the mentioned zones are in production and the RGW multisite bandwidth is over MPLS of around 3 Gbps. Issue description : We have enabled the multisite between DC-DR almost around a month ago. The total data at the DC zone is around 159 TiB and the sync has been going as expected . But when the sync had gone around 120 TiB we saw the speed drastically fell low, the ideal was around 2 Gbps, and it fell way below 10 Mbps though the link is not saturated. After checking "# radosgw-admin sync status " the output says "metadata is caught up with master" and "data is caught up with source" but with almost 25 TB data behind as compared to DC. It also looks like the sync status of the bucket " radosgw-admin bucket sync status --bucket=<bucket-name>" still bucket is behind shards. Attaching the log and the output below. The possibility of issuing a resync of the data from the beginning is quite low and not feasible in our case. The "# radosgw-admin sync error list" output is also attached with some information redacted and we see errors. radosgw-sync status radosgw-admin sync status realm 6a7fab77-64e3-453e-b54b-066bc8af2f00 (realm0) zonegroup be660604-d853-4f8e-a576-579cae2e07c2 (zg0) zone d06a8dd3-5bcb-486c-945b-2a98969ccd5f (fbd) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: d09d3d16-8601-448b-bf3d-609b8a29647d (ahd) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source radosgw-admin bucket sync status --bucket=<bucket-name> realm 6a7fab77-64e3-453e-b54b-066bc8af2f00 (realm0) zonegroup be660604-d853-4f8e-a576-579cae2e07c2 (zg0) zone d06a8dd3-5bcb-486c-945b-2a98969ccd5f (fbd) bucket :tc******rc-b1[d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1]) source zone d09d3d16-8601-448b-bf3d-609b8a29647d (ahd) source bucket :tc*******arc-b1[d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1]) full sync: 14/9221 shards full sync: 49448693 objects completed incremental sync: 9207/9221 shards bucket is behind on 25 shards behind shards: [9,111,590,826,1774,2968,3132,3382,3386,3409,3685,3820,4174,4544,4708,4811,5733,6285,6558,7288,7417,7443,7876,8151,8878] Error: radosgw-admin sync error list "id": "1_1690799008.725414_3926410.1", "section": "data", "name": "bucket0:d09d3d16-8601-448b-bf3d-609b8a29647d.89871.1:1949", "timestamp": "2023-07-31T10:23:28.725414Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 125, "message": "failed to sync bucket instance: (125) Operation canceled" "id": "1_1690804503.144829_3759212.1", "section": "data", "name": "bucket1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:1232/S01/1/120/2b7ea802-efad-41d3-9d90-9**************523.txt", "timestamp": "2023-07-31T11:54:53.233451Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 5, "message": "failed to sync object(5) Input/output error" Thanks Ankit

2 3

Is it safe to add different OS but same ceph version to the existing cluster?
by Szabo, Istvan (Agoda) 04 Sep '23

04 Sep '23

Hi, I have an octopus cluster on the latest octopus version with mgr/mon/rgw/osds on centos 8. Is it safe to add an ubuntu osd host with the same octopus version? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 3

Running trim / discard on an OSD
by Victor Rodriguez 03 Sep '23

03 Sep '23

TL;DR Is there a way to run trim / discard on an OSD? Long story: I have a Proxmox-Ceph cluster with some OSD as storage for VMs. Discard works perfectly in this cluster. For lab and testing purposes I deploy Proxmox-Ceph clusters as Proxmox VMs in this cluster using nested virtualizacion, each with a few disks acting as OSD. Then a few VMs are configured in the nested Proxmox cluster using the nested Ceph as storage. Hope I've explained myself. The problem I'm facing is that the nested Ceph OSD's are not sending the trim/discard command to the upstream Ceph OSDs, so thin provisioning is not kept. Somehow the discard chain is broken at some point. I've tried using this on the nested ceph OSD: ceph config set global bdev_enable_discard true ceph config set global bdev_async_discard true That somewhat helps, but it does not discard all the used space when data is deleted from the VM or the VM is fully removed, it maybe recovers like 20%. This is why I wonder if there is a way to run trim / discard in an OSD. Thanks in advance --

1 0

Client failing to respond to capability release
by Frank Schilder 01 Sep '23

01 Sep '23

Hi all, I have this warning the whole day already (octopus latest cluster): HEALTH_WARN 4 clients failing to respond to capability release; 1 pgs not deep-scrubbed in time [WRN] MDS_CLIENT_LATE_RELEASE: 4 clients failing to respond to capability release mds.ceph-24(mds.1): Client sn352.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 145698301 mds.ceph-24(mds.1): Client sn463.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 189511877 mds.ceph-24(mds.1): Client sn350.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 189511887 mds.ceph-24(mds.1): Client sn403.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 231250695 If I look at the session info from mds.1 for these clients I see this: # ceph tell mds.1 session ls | jq -c '[.[] | {id: .id, h: .client_metadata.hostname, addr: .inst, fs: .client_metadata.root, caps: .num_caps, req: .request_load_avg}]|sort_by(.caps)|.[]' | grep -e 145698301 -e 189511877 -e 189511887 -e 231250695 {"id":189511887,"h":"sn350.hpc.ait.dtu.dk","addr":"client.189511887 v1:192.168.57.221:0/4262844211","fs":"/hpc/groups","caps":2,"req":0} {"id":231250695,"h":"sn403.hpc.ait.dtu.dk","addr":"client.231250695 v1:192.168.58.18:0/1334540218","fs":"/hpc/groups","caps":3,"req":0} {"id":189511877,"h":"sn463.hpc.ait.dtu.dk","addr":"client.189511877 v1:192.168.58.78:0/3535879569","fs":"/hpc/groups","caps":4,"req":0} {"id":145698301,"h":"sn352.hpc.ait.dtu.dk","addr":"client.145698301 v1:192.168.57.223:0/2146607320","fs":"/hpc/groups","caps":7,"req":0} We have mds_min_caps_per_client=4096, so it looks like the limit is well satisfied. Also, the file system is pretty idle at the moment. Why and what exactly is the MDS complaining about here? Thanks and best regards. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

When to use the auth profiles simple-rados-client and profile simple-rados-client-with-blocklist?
by Christian Rohmann 01 Sep '23

01 Sep '23

Hey ceph-users, 1) When configuring Gnocchi to use Ceph storage (see https://gnocchi.osci.io/install.html#ceph-requirements) I was wondering if one could use any of the auth profiles like * simple-rados-client * simple-rados-client-with-blocklist ? Or are those for different use cases? 2) I was also wondering why the documentation mentions "(Monitor only)" but then it says "Gives a user read-only permissions for monitor, OSD, and PG data."? 3) And are those profiles really for "read-only" users? Why don't they have "read-only" in their name like the rbd and the corresponding "rbd-read-only" profile? Regards Christian

2 1

Critical Information: DELL/Toshiba SSDs dying after 70,000 hours of operation
by Frédéric Nass 01 Sep '23

01 Sep '23

Hello, This message does not concern Ceph itself but a hardware vulnerability which can lead to permanent loss of data on a Ceph cluster equipped with the same hardware in separate fault domains. The DELL / Toshiba PX02SMF020, PX02SMF040, PX02SMF080 and PX02SMB160 SSD drives of the 13G generation of DELL servers are subject to a vulnerability which renders them unusable after 70,000 hours of operation, i.e. approximately 7 years and 11 months of activity. This topic has been discussed here: https://www.dell.com/community/PowerVault/TOSHIBA-PX02SMF080-has-lost-commu… The risk is all the greater since these disks may die at the same time in the same server leading to the loss of all data in the server. To date, DELL has not provided any firmware fixing this vulnerability, the latest firmware version being "A3B3" released on Sept. 12, 2016: https://www.dell.com/support/home/en-us/ drivers/driversdetails?driverid=hhd9k If your have servers running these drives, check their uptime. If they are close to the 70,000 hour limit, replace them immediately. The smartctl tool does not report the uptime for these SSDs, but if you have HDDs in the server, you can query their SMART status and get their uptime, which should be about the same as the SSDs. The smartctl command is: smartctl -a -d megaraid,XX /dev/sdc (where XX is the iSCSI bus number). We have informed DELL about this but have no information yet on the arrival of a fix. We have lost 6 disks, in 3 different servers, in the last few weeks. Our observation shows that the drives don't survive full shutdown and restart of the machine (power off then power on in iDrac), but they may also die during a single reboot (init 6) or even while the machine is running. Fujitsu released a corrective firmware in June 2021 but this firmware is most certainly not applicable to DELL drives: https://www.fujitsu.com/us/imagesgig5/PY-CIB070-00.pdf Regards, Frederic Sous-direction Infrastructure and Services Direction du Numérique Université de Lorraine

2 2

OSDs spam log with scrub starts
by Adrien Georget 01 Sep '23

01 Sep '23

Hello, On our 16.2.14 CephFS cluster, all OSDs are spamming logs with messages like "log_channel(cluster) log [DBG] : xxx scrub starts". All OSDs are concerned, for different PGs. Cluster is healthy without any recovery ops. For a single PG, we can have hundreds of scrub starts msg in less than an hour. With 720 OSDs (8k PG, EC8+2), it can lead to millions of messages by hour... For example with PG 3.1d57 or||3.1988 : |Aug 31 16:02:09 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:09.453+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:11 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:11.446+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:12 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:12.428+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:13 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:13.456+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:14 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:14.431+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:15 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:15.475+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:21 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:21.516+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:23 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:23.555+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:24 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:24.510+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 deep-scrub starts|| Aug 31 16:02:10 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:10.384+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 deep-scrub starts Aug 31 16:02:11 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:11.377+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts Aug 31 16:02:13 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:13.383+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts Aug 31 16:02:15 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:15.383+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 deep-scrub starts Aug 31 16:02:17 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:17.336+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts Aug 31 16:02:19 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:19.328+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts || ||PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN|| ||3.1d57 52757 0 0 0 0 167596026648 0 0 1799 1799 active+clean 2023-08-31T14:27:24.025755+0000 236010'4532653 236011:8745383 [58,421,335,9,59,199,390,481,425,480] 58 [58,421,335,9,59,199,390,481,425,480] 58 231791'4531915 *2023-08-29T22:41:12.266874+0000* 229377'4526369 *2023-08-26T04:30:42.894505+0000* 0| |3.1988 52867 0 0 0 0 168603872808 0 0 1811 1811 active+clean 2023-08-31T14:32:13.361420+0000 236018'4241611 236018:9815753 [276,342,345,299,210,349,85,481,446,46] 276 [276,342,345,299,210,349,85,481,446,46] 276 236012'4241602 *2023-08-31T14:32:13.361343+0000* 228157'4229095 *2023-08-24T05:59:16.573471+0000*| However scrub is working fine, scrub stamp looks OK (2023-08-29 or 2023-08-31) as we have default value for scrub interval (min 24h / max 7days). I tried to play with scrub parameters like osd_scrub_load_threshold (->20), osd_max_scrubs (->3), osd_scrub_*_interval but nothing better. Any idea what's going on and how to fix this? Cheers, Adrien ||

3 3