August 2023 - ceph-users

Ceph Leadership Team Meeting: 2023-08-09 Minutes
by Patrick Donnelly 09 Aug '23

09 Aug '23

Today we discussed: - Delegating more privileges for internal hardware to allow on-call folks to fix issues. - Maybe using CephFS for the teuthology VM /home directory (it became full on Friday night) - Preparation for Open Source Day: we are seeking "low-hanging-fruit" tickets for new developers to try fixing. - Reef is released! Time for blog posts. We are gathering options from PTLs. - Ceph organization Github plan migration from the "bronze legacy plan" to the FOSS "free" plan. There is some uncertainty about surprise drawbacks, Ernesto is continuing his investigation. - Case is updating contributors to generate accurate credits for the new reef release: https://github.com/ceph/ceph/pull/52868 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Backfill Performance for
by Jonathan Suever 08 Aug '23

08 Aug '23

I am in the process of expanding our cluster capacity by ~50% and have noticed some unexpected behavior during the backfill and recovery process that I'd like to understand and see if there is a better configuration that will yield a faster and smoother backfill. Pool Information: OSDs: 243 spinning HDDs PGs: 1024 (yes, this is low for our number of disks) I inherited the cluster and it has the following settings which seem to have been done in an attempt to get the cluster to recover quickly: osd_max_backfills: 6 (default is 1) osd_recovery_sleep_hdd: 0.0 (default is 0.1) osd_recovery_max_active_hdd: 9 When watching the PGs recover I am noticing a few things: - All PGs seem to be backfilling at the same time which seems to be in violation of osd_max_backfills. I understand that there should be 6 readers and 6 writers at a time, but I'm seeing a given OSD participate in more than 6 PG backfills. Is an OSD only considered as backfilling if it is not present in both the UP and ACTING groups (e.g. it will have it's data altered)? - Some PGs are recovering at a much slower rate than others (some as little as kilobytes per second) despite the disks being all of a similar speed. Is there some way to dig into why that may be? - In general, the recovery is happening very slowly (between 1 and 5 objects per second per PG). Is it possible the settings above are too aggressive and causing performance degradation due to disk thrashing? - Currently, all misplaced PGs are backfilling, if I were to change some of the settings above (specifically `osd_max_backfills`) would that essentially pause backfilling PGs or will those backfills have to end and then start over when it is done waiting? - Given that all PGs are backfilling simultaneously there is no way to prioritize one PG over another (we have some disks with very high usage that we're trying to reduce). Would reducing those max backfills allow for proper prioritization of PGs with force-backfill? - We have had some OSDs restart during the process and their misplaced object count is now zero but they are incrementing their recovering objects bytes. Is that expected and is there a way to estimate when that will complete? Thanks for the help! -Jonathan

2 1

Re: cephfs snapshot mirror peer_bootstrap import hung
by Venky Shankar 08 Aug '23

08 Aug '23

Hi Anantha, On Mon, Aug 7, 2023 at 11:52 PM Adiga, Anantha <anantha.adiga(a)intel.com> wrote: > > Hi Venky, > > > > I tried on another secondary Quincy cluster and it is the same problem. The peer_bootstrap mport command hangs. A pacific cluster generated peer token should be importable in a quincy source cluster. Looking at the logs, I suspect that the perceived hang is the mirroring module blocked on connecting to the secondary cluster (to set mirror info xattr). Are you able to connect to the secondary cluster from the host running ceph-mgr on the primary cluster using its monitor address (and a key)? > > > > > > root@fl31ca104ja0201:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ== > > …… > > ……. > > ..command does not complete..waits here > > ^C to exit. > > Thereafter some commands do not complete… > > root@fl31ca104ja0201:/# ceph -s > > cluster: > > id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e > > health: HEALTH_OK > > > > services: > > mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 2d) > > mgr: fl31ca104ja0201.kkoono(active, since 3d), standbys: fl31ca104ja0202, fl31ca104ja0203 > > mds: 1/1 daemons up, 2 standby > > osd: 44 osds: 44 up (since 2d), 44 in (since 5w) > > cephfs-mirror: 1 daemon active (1 hosts) > > rgw: 3 daemons active (3 hosts, 1 zones) > > > > data: > > volumes: 1/1 healthy > > pools: 25 pools, 769 pgs > > objects: 614.40k objects, 1.9 TiB > > usage: 2.9 TiB used, 292 TiB / 295 TiB avail > > pgs: 769 active+clean > > > > io: > > client: 32 KiB/s rd, 0 B/s wr, 33 op/s rd, 1 op/s wr > > > > root@fl31ca104ja0201:/# > > root@fl31ca104ja0201:/# ceph fs status cephfs > > This command also waits. …… > > > > I have attached the mgr log > > root@fl31ca104ja0201:/# ceph service status > > { > > "cephfs-mirror": { > > "5306346": { > > "status_stamp": "2023-08-07T17:35:56.884907+0000", > > "last_beacon": "2023-08-07T17:45:01.903540+0000", > > "status": { > > "status_json": "{\"1\":{\"name\":\"cephfs\",\"directory_count\":0,\"peers\":{}}}" > > } > > } > > > > Quincy secondary cluster > > > > root@a001s008-zz14l47008:/# ceph mgr module enable mirroring > > root@a001s008-zz14l47008:/# ceph fs authorize cephfs client.mirror_remote / rwps > > [client.mirror_remote] > > key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA== > > root@a001s008-zz14l47008:/# ceph auth get client.mirror_remote > > [client.mirror_remote] > > key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA== > > caps mds = "allow rwps fsname=cephfs" > > caps mon = "allow r fsname=cephfs" > > caps osd = "allow rw tag cephfs data=cephfs" > > root@a001s008-zz14l47008:/# > > root@a001s008-zz14l47008:/# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote shgR-site > > {"token": "eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ=="} > > root@a001s008-zz14l47008:/# > > > > Thank you, > > Anantha > > > > From: Adiga, Anantha > Sent: Friday, August 4, 2023 11:55 AM > To: Venky Shankar <vshankar(a)redhat.com>; ceph-users(a)ceph.io > Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung > > > > Hi Venky, > > > > Thank you so much for the guidance. Attached is the mgr log. > > > > Note: the 4th node in the primary cluster has smaller capacity drives, the other 3 nodes have the larger capacity drives. > > 32 ssd 6.98630 1.00000 7.0 TiB 44 GiB 44 GiB 183 KiB 148 MiB 6.9 TiB 0.62 0.64 40 up osd.32 > > -7 76.84927 - 77 TiB 652 GiB 648 GiB 20 MiB 3.0 GiB 76 TiB 0.83 0.86 - host fl31ca104ja0203 > > 1 ssd 6.98630 1.00000 7.0 TiB 73 GiB 73 GiB 8.0 MiB 333 MiB 6.9 TiB 1.02 1.06 54 up osd.1 > > 4 ssd 6.98630 1.00000 7.0 TiB 77 GiB 77 GiB 1.1 MiB 174 MiB 6.9 TiB 1.07 1.11 55 up osd.4 > > 7 ssd 6.98630 1.00000 7.0 TiB 47 GiB 47 GiB 140 KiB 288 MiB 6.9 TiB 0.66 0.68 51 up osd.7 > > 10 ssd 6.98630 1.00000 7.0 TiB 75 GiB 75 GiB 299 KiB 278 MiB 6.9 TiB 1.05 1.09 44 up osd.10 > > 13 ssd 6.98630 1.00000 7.0 TiB 94 GiB 94 GiB 1018 KiB 291 MiB 6.9 TiB 1.31 1.36 72 up osd.13 > > 16 ssd 6.98630 1.00000 7.0 TiB 31 GiB 31 GiB 163 KiB 267 MiB 7.0 TiB 0.43 0.45 49 up osd.16 > > 19 ssd 6.98630 1.00000 7.0 TiB 14 GiB 14 GiB 756 KiB 333 MiB 7.0 TiB 0.20 0.21 50 up osd.19 > > 22 ssd 6.98630 1.00000 7.0 TiB 105 GiB 104 GiB 1.3 MiB 313 MiB 6.9 TiB 1.46 1.51 48 up osd.22 > > 25 ssd 6.98630 1.00000 7.0 TiB 17 GiB 16 GiB 257 KiB 272 MiB 7.0 TiB 0.23 0.24 45 up osd.25 > > 28 ssd 6.98630 1.00000 7.0 TiB 72 GiB 72 GiB 6.1 MiB 180 MiB 6.9 TiB 1.01 1.05 43 up osd.28 > > 31 ssd 6.98630 1.00000 7.0 TiB 47 GiB 46 GiB 592 KiB 358 MiB 6.9 TiB 0.65 0.68 56 up osd.31 > > -9 64.04089 - 64 TiB 728 GiB 726 GiB 17 MiB 1.8 GiB 63 TiB 1.11 1.15 - host fl31ca104ja0302 > > 33 ssd 5.82190 1.00000 5.8 TiB 65 GiB 65 GiB 245 KiB 144 MiB 5.8 TiB 1.09 1.13 47 up osd.33 > > 34 ssd 5.82190 1.00000 5.8 TiB 14 GiB 14 GiB 815 KiB 83 MiB 5.8 TiB 0.24 0.25 55 up osd.34 > > 35 ssd 5.82190 1.00000 5.8 TiB 77 GiB 77 GiB 224 KiB 213 MiB 5.7 TiB 1.30 1.34 44 up osd.35 > > 36 ssd 5.82190 1.00000 5.8 TiB 117 GiB 117 GiB 8.5 MiB 284 MiB 5.7 TiB 1.96 2.03 52 up osd.36 > > 37 ssd 5.82190 1.00000 5.8 TiB 58 GiB 58 GiB 501 KiB 132 MiB 5.8 TiB 0.98 1.01 40 up osd.37 > > 38 ssd 5.82190 1.00000 5.8 TiB 123 GiB 123 GiB 691 KiB 266 MiB 5.7 TiB 2.07 2.14 73 up osd.38 > > 39 ssd 5.82190 1.00000 5.8 TiB 77 GiB 77 GiB 609 KiB 193 MiB 5.7 TiB 1.30 1.34 62 up osd.39 > > 40 ssd 5.82190 1.00000 5.8 TiB 77 GiB 77 GiB 262 KiB 148 MiB 5.7 TiB 1.29 1.34 55 up osd.40 > > 41 ssd 5.82190 1.00000 5.8 TiB 44 GiB 44 GiB 4.4 MiB 140 MiB 5.8 TiB 0.75 0.77 44 up osd.41 > > 42 ssd 5.82190 1.00000 5.8 TiB 45 GiB 45 GiB 886 KiB 135 MiB 5.8 TiB 0.75 0.78 47 up osd.42 > > 43 ssd 5.82190 1.00000 5.8 TiB 28 GiB 28 GiB 187 KiB 104 MiB 5.8 TiB 0.48 0.49 58 up osd.43 > > > > [Also: Yesterday I had two cfs-mirror running one on fl31ca104ja0201 and fl31ca104ja0302. The cfs-mirror on fl31ca104ja0201 was stopped. When the import token was run on fl31ca104ja0302, the cfs-mirror log was active. Just in case it is useful I have attached that log (cfsmirror-container.log) as well. ] > > > > How can I list the token on the target cluster after running the create peer_bootstrap command? > > > > Here is today’s status with your suggestion: > > There is only one cfs-mirror daemon running now. It is on fl31ca104ja0201 node. > > > > root@fl31ca104ja0201:/# ceph -s > > cluster: > > id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e > > health: HEALTH_OK > > > > services: > > mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 7m) > > mgr: fl31ca104ja0201.kkoono(active, since 13m), standbys: fl31ca104ja0202, fl31ca104ja0203 > > mds: 1/1 daemons up, 2 standby > > osd: 44 osds: 44 up (since 7m), 44 in (since 4w) > > cephfs-mirror: 1 daemon active (1 hosts) > > rgw: 3 daemons active (3 hosts, 1 zones) > > > > data: > > volumes: 1/1 healthy > > pools: 25 pools, 769 pgs > > objects: 614.40k objects, 1.9 TiB > > usage: 2.8 TiB used, 292 TiB / 295 TiB avail > > pgs: 769 active+clean > > > > io: > > client: 32 MiB/s rd, 0 B/s wr, 57 op/s rd, 1 op/s wr > > > > root@fl31ca104ja0201:/# > > root@fl31ca104ja0201:/# > > root@fl31ca104ja0201:/# ceph tell mgr.fl31ca104ja0201.kkoono config set debug_mgr 20 > > { > > "success": "" > > } > > root@fl31ca104ja0201:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0= > > ^CInterrupted > > > > Ctrl-C after 15 min. Once the command is run, the health status goes to WARN . > > > > root@fl31ca104ja0201:/# ceph -s > > cluster: > > id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e > > health: HEALTH_WARN > > 6 slow ops, oldest one blocked for 1095 sec, mon.fl31ca104ja0203 has slow ops > > > > services: > > mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 30m) > > mgr: fl31ca104ja0201.kkoono(active, since 35m), standbys: fl31ca104ja0202, fl31ca104ja0203 > > mds: 1/1 daemons up, 2 standby > > osd: 44 osds: 44 up (since 29m), 44 in (since 4w) > > cephfs-mirror: 1 daemon active (1 hosts) > > rgw: 3 daemons active (3 hosts, 1 zones) > > > > data: > > volumes: 1/1 healthy > > pools: 25 pools, 769 pgs > > objects: 614.40k objects, 1.9 TiB > > usage: 2.8 TiB used, 292 TiB / 295 TiB avail > > pgs: 769 active+clean > > > > io: > > client: 67 KiB/s rd, 0 B/s wr, 68 op/s rd, 21 op/s wr > > > > > > -----Original Message----- > From: Venky Shankar <vshankar(a)redhat.com> > Sent: Thursday, August 3, 2023 11:03 PM > To: Adiga, Anantha <anantha.adiga(a)intel.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung > > > > Hi Anantha, > > > > On Fri, Aug 4, 2023 at 2:27 AM Adiga, Anantha <anantha.adiga(a)intel.com> wrote: > > > > > > Hi > > > > > > Could you please provide guidance on how to diagnose this issue: > > > > > > In this case, there are two Ceph clusters: cluster A, 4 nodes and cluster B, 3 node, in different locations. Both are already running RGW multi-site, A is master. > > > > > > Cephfs snapshot mirroring is being configured on the clusters. Cluster A is the primary, cluster B is the peer. Cephfs snapshot mirroring is being configured. The bootstrap import step on the primary node hangs. > > > > > > On the target cluster : > > > --------------------------- > > > "version": "16.2.5", > > > "release": "pacific", > > > "release_type": "stable" > > > > > > root@cr21meg16ba0101:/# ceph fs snapshot mirror peer_bootstrap create > > > cephfs client.mirror_remote flex2-site > > > {"token": > > > "eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJma > > > Wxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiw > > > gInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd > > > 1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTU > > > uNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6M > > > zMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0="} > > > > Seems fine uptil here. > > > > > root@cr21meg16ba0101:/var/run/ceph# > > > > > > On the source cluster: > > > ---------------------------- > > > "version": "17.2.6", > > > "release": "quincy", > > > "release_type": "stable" > > > > > > root@fl31ca104ja0201:/# ceph -s > > > cluster: > > > id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e > > > health: HEALTH_OK > > > > > > services: > > > mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 111m) > > > mgr: fl31ca104ja0201.nwpqlh(active, since 11h), standbys: fl31ca104ja0203, fl31ca104ja0202 > > > mds: 1/1 daemons up, 2 standby > > > osd: 44 osds: 44 up (since 111m), 44 in (since 4w) > > > cephfs-mirror: 1 daemon active (1 hosts) > > > rgw: 3 daemons active (3 hosts, 1 zones) > > > > > > data: > > > volumes: 1/1 healthy > > > pools: 25 pools, 769 pgs > > > objects: 614.40k objects, 1.9 TiB > > > usage: 2.8 TiB used, 292 TiB / 295 TiB avail > > > pgs: 769 active+clean > > > > > > root@fl31ca104ja0302:/# ceph mgr module enable mirroring module > > > 'mirroring' is already enabled root@fl31ca104ja0302:/# ceph fs > > > snapshot mirror peer_bootstrap import cephfs > > > eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaW > > > xlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwg > > > InNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1 > > > h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUu > > > NzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6Mz > > > MwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0= > > > > Going by your description, I'm guessing this is the command that hangs? If that's the case, set `debug_mgr=20`, repeat the token import step and share the ceph-mgr log. Also note that you can check the mirror daemon status as detailed in > > > > https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#mirror-daemon-status > > > > > > > > > > > root@fl31ca104ja0302:/var/run/ceph# ceph --admin-daemon > > > /var/run/ceph/ceph-client.cephfs-mirror.fl31ca104ja0302.sypagt.7.94083135960976.asok status { > > > "metadata": { > > > "ceph_sha1": "d7ff0d10654d2280e08f1ab989c7cdf3064446a5", > > > "ceph_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", > > > "entity_id": "cephfs-mirror.fl31ca104ja0302.sypagt", > > > "hostname": "fl31ca104ja0302", > > > "pid": "7", > > > "root": "/" > > > }, > > > "dentry_count": 0, > > > "dentry_pinned_count": 0, > > > "id": 5194553, > > > "inst": { > > > "name": { > > > "type": "client", > > > "num": 5194553 > > > }, > > > "addr": { > > > "type": "v1", > > > "addr": "10.45.129.5:0", > > > "nonce": 2497002034 > > > } > > > }, > > > "addr": { > > > "type": "v1", > > > "addr": "10.45.129.5:0", > > > "nonce": 2497002034 > > > }, > > > "inst_str": "client.5194553 10.45.129.5:0/2497002034", > > > "addr_str": "10.45.129.5:0/2497002034", > > > "inode_count": 1, > > > "mds_epoch": 118, > > > "osd_epoch": 6266, > > > "osd_epoch_barrier": 0, > > > "blocklisted": false, > > > "fs_name": "cephfs" > > > } > > > > > > root@fl31ca104ja0302:/home/general# docker logs > > > ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-cephfs-mirror-fl31ca104ja030 > > > 2-sypagt --tail 10 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 > > > ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy > > > (stable), process cephfs-mirror, pid 7 debug > > > 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 pidfile_write: ignore > > > empty --pid-file debug 2023-08-03T05:24:27.445+0000 7f8eb6fc0280 1 > > > mgrc service_daemon_register cephfs-mirror.5184622 metadata > > > {arch=x86_64,ceph_release=quincy,ceph_version=ceph version 17.2.6 > > > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy > > > (stable),ceph_version_short=17.2.6,container_hostname=fl31ca104ja0302, > > > container_image=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c > > > 64ca62a0b38ab362e614ad671efa4a0547e,cpu=Intel(R) Xeon(R) Gold 6252 CPU > > > @ 2.10GHz,distro=centos,distro_description=CentOS Stream > > > 8,distro_version=8,hostname=fl31ca104ja0302,id=fl31ca104ja0302.sypagt, > > > instance_id=5184622,kernel_description=#82-Ubuntu SMP Tue Jun 6 > > > 23:10:23 UTC > > > 2023,kernel_version=5.15.0-75-generic,mem_swap_kb=8388604,mem_total_kb > > > =527946928,os=Linux} debug 2023-08-03T05:27:10.419+0000 7f8ea1b2c700 > > > 0 client.5194553 ms_handle_reset on v2:10.45.128.141:3300/0 debug > > > 2023-08-03T05:50:10.917+0000 7f8ea1b2c700 0 client.5194553 > > > ms_handle_reset on v2:10.45.128.139:3300/0 > > > > > > Thank you, > > > Anantha > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > > email to ceph-users-leave(a)ceph.io > > > > > > > > > -- > > Cheers, > > Venky > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

2 6

RBD Disk Usage
by mahnoosh shahidi 07 Aug '23

07 Aug '23

Hi all, I have an rbd image that `rbd disk-usage` shows it has 31GB usage but in the filesystem `du` shows its usage is 40KB. Does anyone know the reason for this difference? Best Regards, Mahnoosh

3 3

help, ceph fs status stuck with no response
by Zhang Bao 07 Aug '23

07 Aug '23

Hi, I have a ceph stucked at `ceph --verbose stats fs fsname`. And in the monitor log, I can found something like `audit [DBG] from='client.431973 -' entity='client.admin' cmd=[{"prefix": "fs status", "fs": "fsname", "target": ["mon-mgr", ""]}]: dispatch`. What happened and what should I do? -- ZhangBao +6585021702

2 1

Problems with UFS / FreeBSD on rbd volumes?
by Roland Giesler 07 Aug '23

07 Aug '23

We have a FreeBSD 12.3 guest machine that works well on an RBD volume until it is live migrated to another node (on Proxmox). After migration, the processes almost all go into D state (waiting for this disk) and they don't exist from it (ie they don't "get" the disk the requested. I'm not sure what it causing this, so I'm asking here if anyone has come across such a problem?

1 0

Multiple CephFS mounts and FSCache
by caskd 07 Aug '23

07 Aug '23

Hello everyone, I've been trying to use CephFS and mix it together with fscache, however i was never able to have multiple mounts with fscache enabled. Is this a known intentional limitation or a bug? It would be possible to work around it by mounting the root of the filesystem and using bind mounts but i have separate volumes that need to be separately mounted. How to replicate: mount -t ceph -o fsc admin@.filesystem1=/path1 /tmp/one # Succeeds mount -t ceph -o fsc admin@.filesystem1=/path2 /tmp/two # Fails complaining about no mds being available The alternative of using no fscache works just fine: mount -t ceph admin@.filesystem1=/path1 /tmp/one # Succeeds mount -t ceph admin@.filesystem1=/path2 /tmp/two # Succeeds Versions: - ceph quincy 17.2.6 - linux 6.4.6 - cachefilesd 0.10.10 -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

2 3

64k buckets for 1 user
by Szabo, Istvan (Agoda) 07 Aug '23

07 Aug '23

Hi, We are in a transition where I'd like to ask my user who stores 2B objects in 1 bucket to split it some way. Thinking for the future we identified to make it future proof and don't store huge amount of objects in 1 bucket, we would need to create 65xxx buckets. Is there anybody aware of any issue with this amount of buckets please? I guess better to split to multiple buckets rather than have gigantic bucket. Thank you the advises ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Natuilus: Taking out OSDs that are 'Failure Pending'
by Dave Hall 04 Aug '23

04 Aug '23

Hello. It's been a while. I have a Nautilus cluster with 72 x 12GB HDD OSDs (BlueStore) and mostly of EC 8+2 pools/PGs. It's been working great - some nodes went nearly 900 days without a reboot. As of yesterday I found that I have 3 OSDs with a Smart status of 'Pending Failure'. New drives are ordered and will be here next week. There is a procedure in the documentation for replacing an OSD, but I can't do that directly until I receive the drives. My inclination is to mark these 3 OSDs 'OUT' before they crash completely, but I want to confirm my understanding of Ceph's response to this. Mainly, given my EC pools (or replicated pools for that matter), if I mark all 3 OSD out all at once will I risk data loss? If I have it right, marking an OSD out will simply cause Ceph to move all of the PG shards from that OSD to other OSDs, so no major risk of data loss. However, if it would be better to do them one per day or something, I'd rather be safe. I also assume that I should wait for the rebalance to complete before I initiate the replacement procedure. Your thoughts? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 4

snapshot timestamp
by Tony Liu 04 Aug '23

04 Aug '23

Hi, We know snapshot is on a point of time. Is this point of time tracked internally by some sort of sequence number, or the timestamp showed by "snap ls", or something else? I noticed that when "deep cp", the timestamps of all snapshot are changed to copy-time. Say I create a snapshot at 1PM and make a copy at 3PM, the timestamp of snapshot in the copy is 3PM. If I rollback the copy to this snapshot, I'd assume it will actually bring me back to the state of 1PM. Is that correct? If the above is true, I won't be able to rely on timestamp to track snapshots. Say I create a snapshot every hour and make a backup by copy at the end of the day. Then the original image is damaged and backup is used to restore the work. On this backup image, how do I know which snapshot was on 1PM, which was on 2PM, etc.? Any advices to track snapshots properly in such case? I can definitely build something else to help on this, but I'd like to know how much Ceph can support it. Thanks! Tony

2 2

What's the max of snap ID?
by Tony Liu 04 Aug '23

04 Aug '23

Hi, There is a snap ID for each snapshot. How is this ID allocated, sequentially? Did some tests, it seems this ID is per pool, starting from 4 and always going up. Is that correct? What's the max of this ID? What's going to happen when ID reaches the max, going back to start from 4 again? Thanks! Tony

3 3

cephfs mount problem - client session lacks required features
by Götz Reinicke 04 Aug '23

04 Aug '23

Hi, During the upgrade from centos7/nautilus to ubuntu 18/nautilus (still updating the MONs) I got a cephfs client who refuses or is refused to mount the ceph fs again. The clients says: mount error 13 = Permission denied The cephmds log: lacks required features 0x0000000000001000 client supports 0x00000000000000ff The mds/mon is still centos7/nautilus. The Clients centos7 as well. Any ideas ? Thx for suggestions and hints . Best Götz

2 2

question about OSD onode hits ratio
by Ben 04 Aug '23

04 Aug '23

Hi, We have a cluster running for a while. From grafana ceph dashboard, I saw OSD onode hits ratio 92% when cluster was just up and running. After couple month, it says now 70%. This is not a good trend I think. Just wondering what should be done to stop this trend. Many thank, Ben

2 1

Ceph Quincy and liburing.so.2 on Rocky Linux 9
by dobrie2＠gmu.edu 04 Aug '23

04 Aug '23

I've been digging and I can't see that this has come up anywhere. I'm trying to update a client from Pacific 17.2.3-2 to 17.2.6-4 and I'm getting the error Error: Problem: cannot install the best update candidate for package ceph-base-2:17.2.3-2.el9s.x86_64 - nothing provides liburing.so.2()(64bit) needed by ceph-base-2:17.2.6-4.el9s.x86_64 - nothing provides liburing.so.2(LIBURING_2.0)(64bit) needed by ceph-base-2:17.2.6-4.el9s.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Did Ceph Pacific switch to requiring liburing 2? Rocky 9 only provides 0.7-7. CentOS stream seems to have 1.0.7-3 (at least back to when I set up that repo on Foreman; I don't remember if I'm keeping it up-to-date). Can I/should I just do --nobest when updating? I could probably build it from a source RPM from another RH-based distro, but I'd rather keep it clean with the same distro.

6 6

MDS nodes blocklisted
by Nathan Harper 04 Aug '23

04 Aug '23

Hi, We're having sporadic problems with a CephFS filesystem where MDSs end up on the OSD blocklist. We're still digging around looking for a cause (Ceph related or other infrastructure cause). The cluster isn't massive (68 OSDs spread over 34 hosts), each host is a VM, with MGR/MON/MDS on non-OSD hosts. Running Ceph 16.2.10 Any suggestions for debugging this further?

3 2

cephfs snapshot mirror peer_bootstrap import hung
by Adiga, Anantha 04 Aug '23

04 Aug '23

Hi Could you please provide guidance on how to diagnose this issue: In this case, there are two Ceph clusters: cluster A, 4 nodes and cluster B, 3 node, in different locations. Both are already running RGW multi-site, A is master. Cephfs snapshot mirroring is being configured on the clusters. Cluster A is the primary, cluster B is the peer. Cephfs snapshot mirroring is being configured. The bootstrap import step on the primary node hangs. On the target cluster : --------------------------- "version": "16.2.5", "release": "pacific", "release_type": "stable" root@cr21meg16ba0101:/# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote flex2-site {"token": "eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0="} root@cr21meg16ba0101:/var/run/ceph# On the source cluster: ---------------------------- "version": "17.2.6", "release": "quincy", "release_type": "stable" root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_OK services: mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 111m) mgr: fl31ca104ja0201.nwpqlh(active, since 11h), standbys: fl31ca104ja0203, fl31ca104ja0202 mds: 1/1 daemons up, 2 standby osd: 44 osds: 44 up (since 111m), 44 in (since 4w) cephfs-mirror: 1 daemon active (1 hosts) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 25 pools, 769 pgs objects: 614.40k objects, 1.9 TiB usage: 2.8 TiB used, 292 TiB / 295 TiB avail pgs: 769 active+clean root@fl31ca104ja0302:/# ceph mgr module enable mirroring module 'mirroring' is already enabled root@fl31ca104ja0302:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0= root@fl31ca104ja0302:/var/run/ceph# ceph --admin-daemon /var/run/ceph/ceph-client.cephfs-mirror.fl31ca104ja0302.sypagt.7.94083135960976.asok status { "metadata": { "ceph_sha1": "d7ff0d10654d2280e08f1ab989c7cdf3064446a5", "ceph_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", "entity_id": "cephfs-mirror.fl31ca104ja0302.sypagt", "hostname": "fl31ca104ja0302", "pid": "7", "root": "/" }, "dentry_count": 0, "dentry_pinned_count": 0, "id": 5194553, "inst": { "name": { "type": "client", "num": 5194553 }, "addr": { "type": "v1", "addr": "10.45.129.5:0", "nonce": 2497002034 } }, "addr": { "type": "v1", "addr": "10.45.129.5:0", "nonce": 2497002034 }, "inst_str": "client.5194553 10.45.129.5:0/2497002034", "addr_str": "10.45.129.5:0/2497002034", "inode_count": 1, "mds_epoch": 118, "osd_epoch": 6266, "osd_epoch_barrier": 0, "blocklisted": false, "fs_name": "cephfs" } root@fl31ca104ja0302:/home/general# docker logs ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-cephfs-mirror-fl31ca104ja0302-sypagt --tail 10 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable), process cephfs-mirror, pid 7 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 pidfile_write: ignore empty --pid-file debug 2023-08-03T05:24:27.445+0000 7f8eb6fc0280 1 mgrc service_daemon_register cephfs-mirror.5184622 metadata {arch=x86_64,ceph_release=quincy,ceph_version=ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable),ceph_version_short=17.2.6,container_hostname=fl31ca104ja0302,container_image=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e,cpu=Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz,distro=centos,distro_description=CentOS Stream 8,distro_version=8,hostname=fl31ca104ja0302,id=fl31ca104ja0302.sypagt,instance_id=5184622,kernel_description=#82-Ubuntu SMP Tue Jun 6 23:10:23 UTC 2023,kernel_version=5.15.0-75-generic,mem_swap_kb=8388604,mem_total_kb=527946928,os=Linux} debug 2023-08-03T05:27:10.419+0000 7f8ea1b2c700 0 client.5194553 ms_handle_reset on v2:10.45.128.141:3300/0 debug 2023-08-03T05:50:10.917+0000 7f8ea1b2c700 0 client.5194553 ms_handle_reset on v2:10.45.128.139:3300/0 Thank you, Anantha

2 5

unbalanced OSDs
by Spiros Papageorgiou 03 Aug '23

03 Aug '23

Hi all, I have a ceph cluster with 3 nodes. ceph version is 16.2.9. There are 7 SSD OSDs on each server and one pool that resides on these OSDs. My OSDs are terribly unbalanced: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -9 28.42200 - 28 TiB 9.3 TiB 9.2 TiB 161 MiB 26 GiB 19 TiB 32.56 1.09 - root ssddisks -2 9.47400 - 9.5 TiB 3.4 TiB 3.4 TiB 66 MiB 9.2 GiB 6.1 TiB 35.52 1.19 - host px1-ssd 0 ssd 1.74599 0.85004 1.7 TiB 810 GiB 807 GiB 3.2 MiB 2.3 GiB 978 GiB 45.28 1.51 26 up osd.0 5 ssd 0.82999 0.85004 850 GiB 581 GiB 580 GiB 22 MiB 912 MiB 269 GiB 68.38 2.29 19 up osd.5 6 ssd 0.82999 1.00000 850 GiB 8.2 GiB 7.8 GiB 9.5 MiB 435 MiB 842 GiB 0.97 0.03 4 up osd.6 7 ssd 0.82999 1.00000 850 GiB 294 GiB 293 GiB 26 MiB 591 MiB 556 GiB 34.60 1.16 11 up osd.7 16 ssd 1.74599 0.85004 1.7 TiB 872 GiB 869 GiB 3.1 MiB 2.3 GiB 916 GiB 48.75 1.63 27 up osd.16 23 ssd 1.74599 1.00000 1.7 TiB 438 GiB 436 GiB 1.5 MiB 1.7 GiB 1.3 TiB 24.48 0.82 14 up osd.23 24 ssd 1.74599 1.00000 1.7 TiB 444 GiB 443 GiB 1.6 MiB 1.0 GiB 1.3 TiB 24.81 0.83 17 up osd.24 -6 9.47400 - 9.5 TiB 2.9 TiB 2.9 TiB 46 MiB 8.1 GiB 6.6 TiB 30.39 1.02 - host px2-ssd 12 ssd 0.82999 1.00000 850 GiB 154 GiB 154 GiB 21 MiB 368 MiB 696 GiB 18.16 0.61 9 up osd.12 13 ssd 0.82999 1.00000 850 GiB 144 GiB 143 GiB 527 KiB 469 MiB 706 GiB 16.92 0.57 4 up osd.13 14 ssd 0.82999 1.00000 850 GiB 149 GiB 149 GiB 16 MiB 299 MiB 700 GiB 17.58 0.59 7 up osd.14 29 ssd 1.74599 1.00000 1.7 TiB 449 GiB 448 GiB 1.6 MiB 1.4 GiB 1.3 TiB 25.11 0.84 20 up osd.29 30 ssd 1.74599 0.85004 1.7 TiB 885 GiB 882 GiB 3.1 MiB 2.3 GiB 903 GiB 49.48 1.65 31 up osd.30 31 ssd 1.74599 1.00000 1.7 TiB 728 GiB 727 GiB 2.6 MiB 1.8 GiB 1.0 TiB 40.74 1.36 22 up osd.31 32 ssd 1.74599 1.00000 1.7 TiB 438 GiB 437 GiB 1.6 MiB 1.4 GiB 1.3 TiB 24.51 0.82 15 up osd.32 -4 9.47400 - 9.5 TiB 3.0 TiB 3.0 TiB 49 MiB 8.7 GiB 6.5 TiB 31.78 1.06 - host px3-ssd 19 ssd 0.82999 1.00000 850 GiB 293 GiB 292 GiB 14 MiB 500 MiB 557 GiB 34.47 1.15 9 up osd.19 20 ssd 0.82999 1.00000 850 GiB 290 GiB 290 GiB 10 MiB 482 MiB 560 GiB 34.15 1.14 10 up osd.20 21 ssd 0.82999 1.00000 850 GiB 148 GiB 147 GiB 16 MiB 428 MiB 702 GiB 17.36 0.58 5 up osd.21 25 ssd 1.74599 1.00000 1.7 TiB 446 GiB 445 GiB 1.8 MiB 1.6 GiB 1.3 TiB 24.96 0.83 19 up osd.25 26 ssd 1.74599 1.00000 1.7 TiB 739 GiB 737 GiB 2.6 MiB 2.0 GiB 1.0 TiB 41.33 1.38 29 up osd.26 27 ssd 1.74599 1.00000 1.7 TiB 725 GiB 723 GiB 2.6 MiB 2.1 GiB 1.0 TiB 40.55 1.36 21 up osd.27 28 ssd 1.74599 1.00000 1.7 TiB 442 GiB 440 GiB 1.6 MiB 1.7 GiB 1.3 TiB 24.72 0.83 17 up osd.28 I have done a "ceph osd reweight-by-utilization" and "ceph osd set-require-min-compat-client luminous". The pool has 32 PGs which were set by autoscale_mode, which is on. Why are my OSDs, so unbalanced? I have osd.5 with 68.3% and osd.6 with 0.97%.... Also when the reweight-by-utilization, osd.5 utilization actually increased... What am i missing here? Sp

3 3

ceph-csi-cephfs - InvalidArgument desc = provided secret is empty
by Shawn Weeks 03 Aug '23

03 Aug '23

I’m attempting to setup the CephFS CSI on K3s managed by Rancher against an external CephFS using the Helm chart. I’m using all default values on the Helm chart accept for cephConf and secret. I’ve verified that the configmap ceph-config get’s created with the values from Helm and I’ve verified that the secret csi-cephfs-secret also get’s created with the same values as seen below. Any attempts to create a PVC result in the following error. The only posts I’ve found are about expansion and I am not trying to expand a CephFS volume, just create one. I0803 19:23:39.715036 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"coder", Name:"test", UID:"9c7e51b6-0321-48e1-9950-444f786c14fb", APIVersion:"v1", ResourceVersion:"4523108", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "cephfs": rpc error: code = InvalidArgument desc = provided secret is empty cephConfConfigMapName: ceph-config cephconf: | [global] fsid = 9b98ccd8-450e-4172-af70-512e4e77bc36 mon_host = [v2:10.0.5.11:3300/0,v1:10.0.5.11:6789/0] [v2:10.0.5.12:3300/0,v1:10.0.5.12:6789/0] [v2:10.0.5.13:3300/0,v1:10.0.5.13:6789/0] commonLabels: {} configMapName: ceph-csi-config csiConfig: null driverName: cephfs.csi.ceph.com externallyManagedConfigmap: false kubeletDir: /var/lib/kubelet logLevel: 5 nodeplugin: affinity: {} fusemountoptions: '' httpMetrics: containerPort: 8081 enabled: true service: annotations: {} clusterIP: '' enabled: true externalIPs: null loadBalancerIP: '' loadBalancerSourceRanges: null servicePort: 8080 type: ClusterIP imagePullSecrets: null kernelmountoptions: '' name: nodeplugin nodeSelector: {} plugin: image: pullPolicy: IfNotPresent repository: quay.io/cephcsi/cephcsi tag: v3.9.0 resources: {} priorityClassName: system-node-critical profiling: enabled: false registrar: image: pullPolicy: IfNotPresent repository: registry.k8s.io/sig-storage/csi-node-driver-registrar tag: v2.8.0 resources: {} tolerations: null updateStrategy: RollingUpdate pluginSocketFile: csi.sock provisioner: affinity: {} enableHostNetwork: false httpMetrics: containerPort: 8081 enabled: true service: annotations: {} clusterIP: '' enabled: true externalIPs: null loadBalancerIP: '' loadBalancerSourceRanges: null servicePort: 8080 type: ClusterIP imagePullSecrets: null name: provisioner nodeSelector: {} priorityClassName: system-cluster-critical profiling: enabled: false provisioner: extraArgs: null image: pullPolicy: IfNotPresent repository: registry.k8s.io/sig-storage/csi-provisioner tag: v3.5.0 resources: {} replicaCount: 3 resizer: enabled: true extraArgs: null image: pullPolicy: IfNotPresent repository: registry.k8s.io/sig-storage/csi-resizer tag: v1.8.0 name: resizer resources: {} setmetadata: true snapshotter: extraArgs: null image: pullPolicy: IfNotPresent repository: registry.k8s.io/sig-storage/csi-snapshotter tag: v6.2.2 resources: {} strategy: rollingUpdate: maxUnavailable: 50% type: RollingUpdate timeout: 60s tolerations: null provisionerSocketFile: csi-provisioner.sock rbac: create: true secret: adminID: <my keyring is for client.home so i put home here> adminKey: <exact value from my keyring here> create: true name: csi-cephfs-secret selinuxMount: true serviceAccounts: nodeplugin: create: true name: null provisioner: create: true name: null sidecarLogLevel: 1 storageClass: allowVolumeExpansion: true annotations: {} clusterID: <cluster-ID> controllerExpandSecret: csi-cephfs-secret controllerExpandSecretNamespace: '' create: false fsName: myfs fuseMountOptions: '' kernelMountOptions: '' mountOptions: null mounter: '' name: csi-cephfs-sc nodeStageSecret: csi-cephfs-secret nodeStageSecretNamespace: '' pool: '' provisionerSecret: csi-cephfs-secret provisionerSecretNamespace: '' reclaimPolicy: Delete volumeNamePrefix: '' global: cattle: clusterId: c-m-xschvkd5 clusterName: dev-cluster rkePathPrefix: '' rkeWindowsPathPrefix: '' systemProjectId: p-g6rqs url: https://rancher.example.com

1 0

Luminous Bluestore issues and RGW Multi-site Recovery
by Gregory O'Neill 03 Aug '23

03 Aug '23

Hello, I have two main questions here. 1. What can I do when `ceph-bluestore-tool` outputs a stack trace for `fsck`? 2. How does one recover from lost PGs / data corruption in an RGW Multi-site setup? --- I have a Luminous 12.2.12 cluster built on ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 for all daemons, no ceph packages are installed on the systems. The OSD nodes have 128GB RAM, 6 SATA SSDs (Micron 5200, 2TB) and 1 NVMe SSD split into 4 OSDs. osd_memory_target is set to 10GB and the OSD nodes have 128GB of RAM. That should put me at 100/128GB used. There are 3 PGs down, 3 of the OSDs that had those PGs won't stay online, and they crash fairly quickly after starting. These are running on SATA SSDs which are being replaced with NVMe SSDs. Crush re-weighting the SATA drives down causes some SATA OSDs to crash and some NVMe drives have slow or blocked ops (related to the down PGs). I installed the ceph-osd package on one OSD host. When I ran `ceph-bluestore-tool`, I got a bunch of tcmalloc and unexpected aio errors. Exact output below. I also tried `ceph-objectstore-tool` but received similar results. I cloned the other OSD that has the affected PGs to have a copy I can work on, but I got the exact same results as before. --- From what I can see, this is likely due to bad drives and automation trying to restart down OSDs several times. With 3 down PGs, I am assuming my next step would be to mark those PGs lost. From there, I am unsure what the recovery procedure is to sync "clean" data from other zones into the cluster that was impacted. Is RGW able to handle this? Do I need to use `rclone`? --- $ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-11 fsck tcmalloc: large alloc 1283989504 bytes == 0x557fdbe46000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 2567970816 bytes == 0x5580286c8000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 5135933440 bytes == 0x5580c17ca000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 3025510400 bytes == 0x557f8f6e6000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc87335582b 0x557f89d75d19 0x557f89d2edda 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 2269913088 bytes == 0x55832469e000 @ 0x7fc87e3f2e50 0x7fc87e4121b9 0x7fc8756ca4f7 0x7fc8756cd304 0x557f89cc4661 0x557f89ad0858 0x557f89ad2224 0x557f89cb7b1d 0x557f89de584c 0x557f89de6a7e 0x557f89e05e7b 0x557f89d2cf48 0x557f89d2efd2 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240 /var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but returned: 2147479552/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2023-07-30 08:27:27.544215 7fc86f689700 -1 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1> 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240 /var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but returned: 2147479552 0> 2023-07-30 08:27:27.544215 7fc86f689700 -1 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 0> 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted $ ceph-objectstore-tool --data-path=/var/lib/ceph/osd/ceph-11 --op list-pgs tcmalloc: large alloc 1283989504 bytes == 0x5649b1bdc000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 2567970816 bytes == 0x5649fe45e000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 5135933440 bytes == 0x564a97560000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 3025510400 bytes == 0x56496547c000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbc82b 0x56495df34079 0x56495deed22a 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 2269913088 bytes == 0x564cfa402000 @ 0x7f3af5e55e50 0x7f3af5e751b9 0x7f3aed12d4f7 0x7f3aed130304 0x56495de9fbc1 0x56495de7a5f8 0x56495de7bfc4 0x56495de9307d 0x56495dfa32dc 0x56495dfa450e 0x56495dfc34db 0x56495deeb398 0x56495deed422 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f3ae72f0700 time 2023-07-30 08:37:16.531432 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3aed1872c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd] 4: (()+0x74a4) [0x7f3aeba734a4] 5: (clone()+0x3f) [0x7f3aea6ccd0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (Aborted) ** in thread 7f3ae72f0700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x94a0f4) [0x56495debe0f4] 2: (()+0x110e0) [0x7f3aeba7d0e0] 3: (gsignal()+0xcf) [0x7f3aea616fff] 4: (abort()+0x16a) [0x7f3aea61842a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7f3aed18744e] 6: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd] 8: (()+0x74a4) [0x7f3aeba734a4] 9: (clone()+0x3f) [0x7f3aea6ccd0f] Aborted -- Gregory O’Neill

3 3

Re: ceph-volume lvm migrate error
by Roland Giesler 03 Aug '23

03 Aug '23

On 2023/08/02 13:29, Roland Giesler wrote: > > On 2023/08/02 12:53, Igor Fedotov wrote: >> Roland, >> >> First of all there are no block.db/block.wal symlinks in OSD folder. >> Which means there are no standalone DB/WAL any more. > > That is surprising. So ceph-volume is not able to extract the DB/WAL > from an OSD to migrate it it seems? I figured out the if one doesn't specify and separate LV for the DB/WAL, it is integrated into the data drive. However, one can create a new DB/WAL for and OSD as follows: # systemctl stop ceph-osd@14 # ceph-bluestore-tool bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-14 --dev-target /dev/NodeC-nvme1/NodeC-nvme-LV-RocksDB1 --bluestore-block-db-size 45G inferring bluefs devices from bluestore path DB device added /dev/dm-20 # systemctl start ceph-osd@14 And, viola!, it did it. # ls -la /var/lib/ceph/osd/ceph-14/block* lrwxrwxrwx 1 ceph ceph 50 Dec 25 2022 /var/lib/ceph/osd/ceph-14/block -> /dev/mapper/0GVWr9-dQ65-LHcx-y6fD-z7fI-10A9-gVWZkY lrwxrwxrwx 1 root root 10 Aug 2 21:17 /var/lib/ceph/osd/ceph-14/block.db -> /dev/dm-20 I'm just check it out now, so see if there are no errors and that it actually does what I think it does.

2 2

ref v18.2.0 QE Validation status
by Yuri Weinstein 03 Aug '23

03 Aug '23

Details of this release are summarized here: https://tracker.ceph.com/issues/62231#note-1 Seeking approvals/reviews for: smoke - Laura, Radek rados - Neha, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade-clients:client-upgrade* - in progress powercycle - Brad Please reply to this email with approval and/or trackers of known issues/PRs to address them. bookworm distro support is an outstanding issue. TIA YuriW

11 19

mgr services frequently crash on nodes 2,3,4
by Adiga, Anantha 03 Aug '23

03 Aug '23

Hi, Mgr service crash frequently on nodes 2 3 and 4 with the same condition after the 4th node was added. root@zp3110b001a0104:/# ceph crash stat 19 crashes recorded 16 older than 1 days old: 2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189 2023-07-29T03:35:32.055174Z_a2ee1e23-5f41-4dbe-86ff-643fbf870dc9 2023-07-29T14:34:13.752432Z_39b6a0d9-1bc3-4481-9a14-c92fea6c2710 2023-07-30T03:02:57.510867Z_df595e04-0ac2-4e3d-93be-a7225348ea19 2023-07-30T06:20:09.322530Z_0c2485f8-281c-4440-8b08-89b08a669de4 2023-07-30T10:16:46.798405Z_79082f37-ee08-4a2b-84d1-d96c4026f321 2023-07-30T10:16:46.843441Z_788391d6-3278-48c4-a95b-1934ee3265c1 2023-07-31T02:26:55.903966Z_416a1e94-a8e1-4057-a683-a907faf400a1 2023-07-31T04:40:10.216044Z_bef9d811-4e92-45cd-bcd7-3282962c8dfe 2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6 2023-07-31T09:22:12.527968Z_f136c93b-7156-4176-a734-66a5a62513a4 2023-07-31T15:22:08.417988Z_b80c6255-5eb3-41dd-b0b1-8bc5b070094f 2023-07-31T23:05:16.589501Z_20ed8ef9-a478-49de-a371-08ea7a9937e5 2023-08-01T01:26:01.911387Z_670f9e3c-7fbe-497f-9f0b-abeaefd8f2b3 2023-08-01T01:51:39.759874Z_ff8206e4-34aa-44fe-82ac-7339e6714bb7 2023-08-01T01:56:21.955706Z_98c86cdd-45ec-47dc-8f0c-2e5e09731db8 7 older than 3 days old: 2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189 2023-07-29T03:35:32.055174Z_a2ee1e23-5f41-4dbe-86ff-643fbf870dc9 2023-07-29T14:34:13.752432Z_39b6a0d9-1bc3-4481-9a14-c92fea6c2710 2023-07-30T03:02:57.510867Z_df595e04-0ac2-4e3d-93be-a7225348ea19 2023-07-30T06:20:09.322530Z_0c2485f8-281c-4440-8b08-89b08a669de4 2023-07-30T10:16:46.798405Z_79082f37-ee08-4a2b-84d1-d96c4026f321 2023-07-30T10:16:46.843441Z_788391d6-3278-48c4-a95b-1934ee3265c1 root@zp3110b001a0104:/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6#<mailto:root@zp3110b001a0104:/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6#> cat meta { "crash_id": "2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6", "timestamp": "2023-07-31T08:44:20.893344Z", "process_name": "ceph-mgr", "entity_name": "mgr.zp3110b001a0104.tmbkzq", "ceph_version": "16.2.5", "utsname_hostname": "zp3110b001a0104", "utsname_sysname": "Linux", "utsname_release": "5.4.0-153-generic", "utsname_version": "#170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023", "utsname_machine": "x86_64", "os_name": "CentOS Linux", "os_id": "centos", "os_version_id": "8", "os_version": "8", "assert_condition": "pending_service_map.epoch > service_map.epoch", "assert_func": "DaemonServer::got_service_map()::<lambda(const ServiceMap&)>", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc", "assert_line": 2932, "assert_thread_name": "ms_dispatch", "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc: In function 'DaemonServer::got_service_map()::<lambda(const ServiceMap&)>' thread 7f127440a700 time 2023-07-31T08:44:20.887150+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mgr/DaemonServer.cc: 2932: FAILED ceph_assert(pending_service_map.epoch > service_map.epoch)\n", "backtrace": [ "/lib64/libpthread.so.0(+0x12b20) [0x7f127c611b20]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f127da26b75]", "/usr/lib64/ceph/libceph-common.so.2(+0x276d3e) [0x7f127da26d3e]", "(DaemonServer::got_service_map()+0xb2d) [0x5625aee23a4d]", "(Mgr::handle_service_map(boost::intrusive_ptr<MServiceMap>)+0x1b6) [0x5625aee527c6]", "(Mgr::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x894) [0x5625aee55424]", "(MgrStandby::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0xb0) [0x5625aee5ec10]", "(DispatchQueue::entry()+0x126a) [0x7f127dc610ca]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7f127dd11591]", "/lib64/libpthread.so.0(+0x814a) [0x7f127c60714a]", "clone()" ] } root@zp3110b001a0104:/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6#<mailto:root@zp3110b001a0104:/var/lib/ceph/8dbfcd81-fee3-49d2-ac0c-e988c8be7178/crash/posted/2023-07-31T08:44:20.893344Z_037688ae-266f-4879-932c-2239f4679fd6#> more log --- begin dump of recent events --- -9999> 2023-07-31T08:27:14.084+0000 7f126fc01700 10 monclient: _send_mon_message to mon.zp3110b001a0104 at v2:XX.XXX.26.4:3300/0 -9998> 2023-07-31T08:27:14.216+0000 7f1272406700 10 monclient: tick -9997> 2023-07-31T08:27:14.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:44.220044+0000) -9996> 2023-07-31T08:27:15.216+0000 7f1272406700 10 monclient: tick -9995> 2023-07-31T08:27:15.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:45.220236+0000) -9994> 2023-07-31T08:27:16.108+0000 7f126fc01700 10 monclient: _send_mon_message to mon.zp3110b001a0104 at v2:XX.XXX.26.4:3300/0 -9993> 2023-07-31T08:27:16.216+0000 7f1272406700 10 monclient: tick -9992> 2023-07-31T08:27:16.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:46.220455+0000) -9991> 2023-07-31T08:27:17.216+0000 7f1272406700 10 monclient: tick -9990> 2023-07-31T08:27:17.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:47.220605+0000) -9989> 2023-07-31T08:27:18.132+0000 7f126fc01700 10 monclient: _send_mon_message to mon.zp3110b001a0104 at v2:XX.XXX.26.4:3300/0 -9988> 2023-07-31T08:27:18.216+0000 7f1272406700 10 monclient: tick -9987> 2023-07-31T08:27:18.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:48.220802+0000) -9986> 2023-07-31T08:27:19.216+0000 7f1272406700 10 monclient: tick -9985> 2023-07-31T08:27:19.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:49.220991+0000) -9982> 2023-07-31T08:27:20.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:50.221181+0000) -9981> 2023-07-31T08:27:21.216+0000 7f1272406700 10 monclient: tick -9980> 2023-07-31T08:27:21.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:51.221376+0000) -9979> 2023-07-31T08:27:22.180+0000 7f126fc01700 10 monclient: _send_mon_message to mon.zp3110b001a0104 at v2:XX.XXX.26.4:3300/0 -9978> 2023-07-31T08:27:22.216+0000 7f1272406700 10 monclient: tick -9977> 2023-07-31T08:27:22.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:52.221566+0000) -9976> 2023-07-31T08:27:23.216+0000 7f1272406700 10 monclient: tick -9975> 2023-07-31T08:27:23.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:53.221770+0000) -9974> 2023-07-31T08:27:23.512+0000 7f11664d5700 4 mgr get_config get_config key: mgr/dashboard/AUDIT_API_ENABLED -9973> 2023-07-31T08:27:23.512+0000 7f11664d5700 4 ceph_get_module_option AUDIT_API_ENABLED not found -9972> 2023-07-31T08:27:23.512+0000 7f11664d5700 4 mgr get_config get_config key: mgr/dashboard/standby_behaviour -9971> 2023-07-31T08:27:23.512+0000 7f11664d5700 4 ceph_get_module_option standby_behaviour not found -9970> 2023-07-31T08:27:24.184+0000 7f126fc01700 10 monclient: _send_mon_message to mon.zp3110b001a0104 at v2:XX.XXX.26.4:3300/0 -9969> 2023-07-31T08:27:24.216+0000 7f1272406700 10 monclient: tick -9968> 2023-07-31T08:27:24.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 1T08:26:54.221952+0000) -9967> 2023-07-31T08:27:25.216+0000 7f1272406700 10 monclient: tick -9966> 2023-07-31T08:27:25.216+0000 7f1272406700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-07-3 root@zp3110b0

3 2

Disk device path changed - cephadm faild to apply osd service
by Kilian Ries 02 Aug '23

02 Aug '23

Hi, it seems that after reboot / OS update my disk labels / device paths may have changed. Since then i get an error like this: CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd-12-22_hdd-2 ### RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9025b6db2e8be9173186e068a79a0da5a54ada -e NODE_NAME=ceph-osd07.intern -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd-12-22_hdd-2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/run/ceph:z -v /var/log/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/log/ceph:z -v /var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2cvmr5lf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpb38cuw7q:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9025b6db2e8be9173186e068a79a0da5a54ada lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq --db-devices /dev/sdg --yes --no-systemd /bin/docker: stderr Traceback (most recent call last): /bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /bin/docker: stderr self.main(self.argv) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /bin/docker: stderr return f(*a, **kw) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /bin/docker: stderr instance.main() /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /bin/docker: stderr terminal.dispatch(self.mapper, self.argv) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 192, in dispatch /bin/docker: stderr instance = mapper.get(arg)(argv[count:]) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 348, in __init__ /bin/docker: stderr self.args = parser.parse_args(argv) /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1734, in parse_args /bin/docker: stderr args, argv = self.parse_known_args(args, namespace) /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1766, in parse_known_args /bin/docker: stderr namespace, args = self._parse_known_args(args, namespace) /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1954, in _parse_known_args /bin/docker: stderr positionals_end_index = consume_positionals(start_index) /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1931, in consume_positionals /bin/docker: stderr take_action(action, args) /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1824, in take_action /bin/docker: stderr argument_values = self._get_values(action, argument_strings) /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2279, in _get_values /bin/docker: stderr value = [self._get_value(action, v) for v in arg_strings] /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2279, in <listcomp> /bin/docker: stderr value = [self._get_value(action, v) for v in arg_strings] /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2294, in _get_value /bin/docker: stderr result = type_func(arg_string) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 116, in __call__ /bin/docker: stderr return self._format_device(self._is_valid_device()) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 127, in _is_valid_device /bin/docker: stderr super()._is_valid_device(raise_sys_exit=False) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 104, in _is_valid_device /bin/docker: stderr super()._is_valid_device() /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 69, in _is_valid_device /bin/docker: stderr super()._is_valid_device() /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 47, in _is_valid_device /bin/docker: stderr raise RuntimeError("Device {} has partitions.".format(self.dev_path)) /bin/docker: stderr RuntimeError: Device /dev/sdq has partitions. Traceback (most recent call last): File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9309, in <module> main() File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9297, in main r = ctx.func(ctx) File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1941, in _infer_config return func(ctx) File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1872, in _infer_fsid return func(ctx) File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1969, in _infer_image return func(ctx) File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1859, in _validate_fsid return func(ctx) File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 5366, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd()) File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1661, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) ### /dev/sdg is my boot device at the moment wich was formerly /dev/sda. Is it safe to edit the ceph orch yaml file and change the device pathes to the new format? Like this: ceph orch ls --service_name=<service-name> --export > myservice.yaml vi (change device pathes in spec -> data_devices -> path | db_devices -> path) ceph orch apply -i myservice.yaml [--dry-run] Is that ok / expected behaviour? Or is there a better way? However can see that mgr detects new devices in ceph orch log: mgr.ceph-mon03.lrfomu [INF] Detected new or changed devices on ceph-osd07 Regards, Kilian

3 6

RHEL / CephFS / Pacific / SELinux unavoidable "relabel inode" error?
by Harry G Coin 02 Aug '23

02 Aug '23

Hi! No matter what I try, using the latest cephfs on an all ceph-pacific setup, I've not been able to avoid this error message, always similar to this on RHEL family clients: SELinux: inode=1099954719159 on dev=ceph was found to have an invalid context=system_u:object_r:unlabeled_t:s0. This indicates you may need to relabel the inode or the filesystem in question. What's the answer? Thanks Harry Coin

2 1

ceph-volume lvm migrate error
by Roland Giesler 02 Aug '23

02 Aug '23

I need some help with this please. The command below gives and error which is not helpful to me. ceph-volume lvm migrate --osd-id 14 --osd-fsid 4de2a617-4452-420d-a99b-9e0cd6b2a99b --from db wal --target NodeC-nvme1/NodeC-nvme-LV-RocksDB1 --> Source device list is empty Unable to migrate to : NodeC-nvme1/NodeC-nvme-LV-RocksDB1 Alternatively I have tried to only specify --from db instead of including wal, but it makes no difference. Here is the OSD in question. # ls -la /dev/ceph-025b887e-4f06-468f-845c-0ddf9ad04990/ lrwxrwxrwx 1 root root 7 Dec 25 2022 osd-block-4de2a617-4452-420d-a99b-9e0cd6b2a99b -> ../dm-4 What is happening here? I want to move the DB/WAL to NVMe storage without trashing the data OSD and having to go through rebalancing for each drive I do this for. thanks Roland

2 1

1 Large omap object found
by Mark Johnson 02 Aug '23

02 Aug '23

I've been going round and round in circles trying to work this one out but I'm getting nowhere. We're running a 4 node quincy cluster (17.2.6) which recently reported the following: ceph.log-20230729.gz:2023-07-28T08:31:42.390003+0000 osd.26 (osd.26) 13834 : cluster [WRN] Large omap object found. Object: 5:6c65dd84:users.uid::callrecordings$callrecordings_rw.buckets:head PG: 5.21bba636 (5.16) Key count: 378454 Size (bytes): 75565579 This happened a week or so ago (only the key count was only just over the 200000 threshold on that occasion) and after much searching around, I found an article that suggested a deep scrub on the pg would likely resolve the issue, so I forced a deep scrub and shortly after, the warning cleared. Came into the office today to discover the above. It's on the same PG as before which is in the default.rgw.meta pool. This time, after forcing a deep-scrub on that PG, nothing changed. I did it a second time just to be sure but got the same result. I keep finding a suse article that simply suggests increasing the threshold to the previous default of 2,000,000, but other articles I read say it was lowered for a reason and that by the time it hits that figure, it's too late so I don't want to just mask it. Problem is that I don't really understand it. I found a thread here from a bit over two years ago but their issue was in the default.rgw.buckets.index pool. A step in the solution was to list out the problematic object id and check the objects per shard however, if I issue the command "rados -p default.rgw.meta ls" it returns nothing. I get a big list from "rados -p default.rgw.buckets.index ls" just nothing from the first pool. I think it may be because the meta pool isn't indexed based on something I read, but I really don't know what I'm talking about tbh. I don't know if this is helpful, but if I list out all the PGs for that pool, there are 32 PGs and 5.16 shows 80186950 bytes and 401505 keys. PG 5.c has 75298 and 384 keys. The remaining 30 PGs show zero bytes and zero keys. I'm really not sure how to troubleshoot and resolve from here. For the record, dynamic resharding is enabled in that no options have been set in the config and that is the default setting. Based on the suse article I mentioned which also references the default.rgw.meta pool, I'm gathering our issue is because we have so many buckets that are all owned by the one user and the solution is either: * delete unused buckets * create multiple users and spread buckets evenly across all users (not something we can do) * increase the threshold to stop the warning Problem is that I'm having trouble verifying this is the issue. I've tried dumping out bucket stats to a file (radosgw-admin bucket stats > bucket_stats.txt) but after three hours this is still running with no output. Thanks for your time, Mark

3 12

veeam backup on rgw - error - op->ERRORHANDLER: err_no=-2 new_err_no=-2
by xadhoom76＠gmail.com 01 Aug '23

01 Aug '23

Hi , we have random error with rgw during a backup from veeam . Daemons go in error state. Where we can find the appropriate logs about it ? I just find out something related to this -1788> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s getting op 0 -1787> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj verifying requester -1786> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj normalizing buckets and tenants -1785> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj init permissions -1784> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj recalculating target -1783> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj reading permissions -1782> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 req 10656715914266436796 0.004000044s op->ERRORHANDLER: err_no=-2 new_err_no=-2 -1781> 2023-07-31T19:51:21.173+0000 7f04567d3700 2 req 10656715914266436796 0.004000044s s3:get_obj op status=0 -1780> 2023-07-31T19:51:21.173+0000 7f04567d3700 2 req 10656715914266436796 0.004000044s s3:get_obj http status=404 -1779> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 ====== req done req=0x7f03aa67a710 op status=0 http_status=404 latency=0.004000044s ====== -1778> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 beast: 0x7f03aa67a710: 172.16.199.11 - veeam90 [31/Jul/2023:19:51:21.169 +0000] "GET /veeam90/Veeam/Backup/veeam90/Clients/%7Bd14cd688-57b4-4809-a1d9-14cafd191b11%7D/15fb2d73-11f5-4ef2-8d17-c9848eb40c93/Metadata/Lock/create.checkpoint/lock HTTP/1.1" 404 207 - "APN/1.0 Veeam/1.0 Backup/12.0" - latency=0.004000044s Best regards

1 0