- ceph-users - lists.ceph.io

Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.
by Bruno Gomes Pessanha 06 Jan '25

06 Jan '25

+ceph-users On Mon, 6 Jan 2025 at 15:50, Bruno Gomes Pessanha <bruno.pessanha(a)gmail.com> wrote: > `ceph osd df`, look at the PGS column. Could be you’re hitting the limit >> on some OSDs. It’s odd to stop at a non power of 2. >> > > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS > 8 ssd 13.97069 1.00000 14 TiB 9.4 TiB 9.4 TiB 9.8 MiB 21 > GiB 4.6 TiB 67.09 1.07 63 up > 18 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 8.5 MiB 19 > GiB 5.7 TiB 59.02 0.94 54 up > 28 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 1.5 GiB 19 > GiB 5.9 TiB 58.02 0.93 54 up > 37 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 84 MiB 31 > GiB 1.2 TiB 91.14 1.45 135 up > 45 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 542 MiB 22 > GiB 4.8 TiB 65.79 1.05 66 up > 56 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 5.0 MiB 19 > GiB 5.7 TiB 59.19 0.94 56 up > 66 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.8 TiB 358 MiB 20 > GiB 5.1 TiB 63.39 1.01 54 up > 77 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 3.7 MiB 19 > GiB 5.3 TiB 62.18 0.99 53 up > 6 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 355 MiB 18 > GiB 5.6 TiB 59.67 0.95 52 up > 16 ssd 13.97069 1.00000 14 TiB 7.3 TiB 7.3 TiB 572 MiB 18 > GiB 6.7 TiB 52.05 0.83 47 up > 26 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 921 MiB 20 > GiB 5.4 TiB 61.55 0.98 57 up > 36 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.2 GiB 31 > GiB 1.2 TiB 91.45 1.46 141 up > 46 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 419 MiB 19 > GiB 5.9 TiB 57.77 0.92 57 up > 54 ssd 13.97069 1.00000 14 TiB 9.0 TiB 8.9 TiB 1011 MiB 20 > GiB 5.0 TiB 64.10 1.02 65 up > 64 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 962 MiB 19 > GiB 6.0 TiB 56.87 0.91 59 up > 74 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 879 MiB 17 > GiB 6.8 TiB 51.23 0.82 49 up > 3 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.1 TiB 2.0 GiB 20 > GiB 4.8 TiB 65.45 1.04 71 up > 12 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 433 MiB 19 > GiB 5.6 TiB 59.92 0.96 61 up > 22 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.8 TiB 501 MiB 20 > GiB 5.2 TiB 63.01 1.01 66 up > 31 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 932 MiB 18 > GiB 5.8 TiB 58.20 0.93 53 up > 41 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.6 TiB 627 MiB 20 > GiB 5.3 TiB 61.96 0.99 54 up > 51 ssd 13.97069 1.00000 14 TiB 6.9 TiB 6.9 TiB 1.4 GiB 16 > GiB 7.1 TiB 49.18 0.78 43 up > 61 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1000 MiB 31 > GiB 1.4 TiB 89.99 1.44 134 up > 71 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 553 MiB 19 > GiB 5.9 TiB 57.63 0.92 51 up > 4 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 525 MiB 15 > GiB 6.9 TiB 50.87 0.81 47 up > 15 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 606 MiB 19 > GiB 5.2 TiB 62.55 1.00 55 up > 23 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.8 TiB 423 MiB 18 > GiB 6.1 TiB 56.32 0.90 49 up > 34 ssd 13.97069 1.00000 14 TiB 10 TiB 10 TiB 551 MiB 24 > GiB 3.5 TiB 74.79 1.19 72 up > 44 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 77 MiB 28 > GiB 1.4 TiB 89.90 1.43 132 up > 59 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.0 TiB 510 MiB 20 > GiB 4.9 TiB 64.91 1.04 61 up > 69 ssd 13.97069 1.00000 14 TiB 9.7 TiB 9.7 TiB 1.0 GiB 22 > GiB 4.2 TiB 69.62 1.11 71 up > 79 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 1.0 GiB 17 > GiB 6.8 TiB 51.26 0.82 47 up > 9 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 1.1 MiB 18 > GiB 5.8 TiB 58.81 0.94 49 up > 19 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 421 MiB 21 > GiB 4.8 TiB 65.68 1.05 63 up > 29 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.7 TiB 511 MiB 21 > GiB 5.2 TiB 62.68 1.00 61 up > 39 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 556 MiB 20 > GiB 5.6 TiB 60.15 0.96 58 up > 49 ssd 13.97069 1.00000 14 TiB 7.0 TiB 7.0 TiB 2.9 MiB 15 > GiB 7.0 TiB 49.92 0.80 45 up > 58 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 432 MiB 16 > GiB 6.8 TiB 51.22 0.82 49 up > 67 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.1 GiB 17 > GiB 6.6 TiB 52.80 0.84 47 up > 78 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 574 MiB 19 > GiB 6.3 TiB 54.94 0.88 55 up > 0 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 455 MiB 19 > GiB 5.8 TiB 58.66 0.94 58 up > 10 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 439 MiB 19 > GiB 5.4 TiB 61.07 0.97 48 up > 21 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 646 MiB 30 > GiB 1.3 TiB 90.87 1.45 135 up > 32 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 3.2 MiB 19 > GiB 5.4 TiB 61.60 0.98 56 up > 42 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 582 MiB 33 > GiB 1.3 TiB 90.78 1.45 133 up > 52 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.4 GiB 17 > GiB 6.6 TiB 52.74 0.84 47 up > 62 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 1.5 GiB 16 > GiB 6.6 TiB 52.68 0.84 49 up > 72 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 1.2 GiB 17 > GiB 6.3 TiB 55.20 0.88 53 up > 1 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 1.5 GiB 19 > GiB 5.9 TiB 57.73 0.92 64 up > 11 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 422 MiB 19 > GiB 5.5 TiB 60.37 0.96 62 up > 20 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 405 MiB 17 > GiB 6.3 TiB 55.02 0.88 52 up > 30 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 494 MiB 19 > GiB 5.9 TiB 57.84 0.92 49 up > 40 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 376 MiB 22 > GiB 4.7 TiB 66.17 1.06 74 up > 50 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.0 GiB 20 > GiB 5.7 TiB 59.43 0.95 54 up > 60 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 2.1 GiB 32 > GiB 1.2 TiB 91.27 1.46 136 up > 70 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 498 MiB 19 > GiB 5.6 TiB 59.64 0.95 56 up > 2 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.1 GiB 31 > GiB 1.2 TiB 91.62 1.46 133 up > 14 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 9.8 MiB 22 > GiB 4.7 TiB 66.05 1.05 60 up > 24 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 426 MiB 17 > GiB 6.5 TiB 53.13 0.85 50 up > 33 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.9 TiB 413 MiB 20 > GiB 5.1 TiB 63.54 1.01 56 up > 43 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 529 MiB 18 > GiB 5.8 TiB 58.58 0.93 58 up > 53 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.6 TiB 533 MiB 17 > GiB 6.4 TiB 54.29 0.87 52 up > 63 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 569 MiB 19 > GiB 5.4 TiB 61.20 0.98 58 up > 73 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 536 MiB 20 > GiB 5.3 TiB 62.14 0.99 63 up > 5 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.3 GiB 19 > GiB 5.7 TiB 59.46 0.95 53 up > 13 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 923 MiB 21 > GiB 5.4 TiB 61.60 0.98 49 up > 25 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 4.3 MiB 16 > GiB 6.6 TiB 53.03 0.85 46 up > 35 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 397 MiB 17 > GiB 6.6 TiB 52.70 0.84 45 up > 47 ssd 13.97069 1.00000 14 TiB 9.0 TiB 9.0 TiB 10 MiB 21 > GiB 4.9 TiB 64.74 1.03 60 up > 57 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 418 MiB 17 > GiB 6.8 TiB 51.13 0.82 50 up > 68 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 590 MiB 30 > GiB 1.2 TiB 91.28 1.46 141 up > 76 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 976 MiB 16 > GiB 6.7 TiB 51.70 0.82 46 up > 7 ssd 13.97069 1.00000 14 TiB 7.8 TiB 7.8 TiB 2.6 MiB 18 > GiB 6.1 TiB 56.03 0.89 49 up > 17 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 443 MiB 16 > GiB 6.8 TiB 51.34 0.82 43 up > 27 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.4 TiB 8.6 MiB 21 > GiB 5.5 TiB 60.51 0.97 54 up > 38 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 1005 MiB 18 > GiB 6.0 TiB 56.88 0.91 52 up > 48 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.5 TiB 2.0 MiB 17 > GiB 6.4 TiB 54.08 0.86 45 up > 55 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 509 MiB 30 > GiB 1.2 TiB 91.50 1.46 137 up > 65 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 901 MiB 17 > GiB 6.3 TiB 55.07 0.88 51 up > 75 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 6.0 MiB 19 > GiB 5.5 TiB 60.66 0.97 58 up > TOTAL 1.1 PiB 701 TiB 699 TiB 49 GiB 1.6 > TiB 417 TiB 62.70 > MIN/MAX VAR: 0.78/1.46 STDDEV: 11.76 > > We’re a community! That curve is way gentler than it used to be. >> > 🙌👍 > > > On Sun, 5 Jan 2025 at 14:59, Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > >> >> >> Do you use the autoscalar or did you trigger a manual PG increment of >> the >> >> pool? >> > >> > The pool had autoscale enabled until 2 days ago when I thought it was >> > better to change things manually in order to have a more deterministic >> > result. Yes, I wanted to increase from "1" to something like "1024" but >> it >> > looks like it was capped to the 144 no matter what I do: >> >> `ceph osd df`, look at the PGS column. Could be you’re hitting the limit >> on some OSDs. It’s odd to stop at a non power of 2. >> >> > Is it correct to say that every PG/OSD change can potentially cause data >> > misplacements, unbalanced osd's and long backfills? I'll be way more >> > careful before tuning it if that's the case. >> >> The autoscaler will usually only bump pg_num for a pool when the value is >> <half what it thinks it should be. >> >> I suggest setting the ‘bulk’ flag one pool at a time to effectively >> pre-split PGs as if the pool were full of data already. Ignore .mgr. >> Start with the pools with the fewest , let the cluster settle between each >> adjustment. That way the autoscaler will only make changes if cluster >> topology changes. >> >> >> > Thank you both so much! It definitely helped me to understand Ceph >> better. >> > It is kind of a steep curve :). >> >> We’re a community! That curve is way gentler than it used to be. > > > > -- > Bruno Gomes Pessanha > -- Bruno Gomes Pessanha

1 0

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

recovery a downed/inaccessible pg
by Nick Anderson 02 Jan '25

02 Jan '25

Hello ceph user-community, We have a reef (18.2.2) cluster with an erasure coded pool of k=8, m=4 that has a pg that is currently down with the following: ``` WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete pg 2.699 is remapped+incomplete, acting [NONE,NONE,NONE,NONE,NONE,353,488,333,408,282,167,145] (reducing pool i1-sea-rbd-01-data min_size from 9 may help; search ceph.com/docs for 'incomplete') ``` Timeline leading up to our current state noted above. We had an issue with a host that had a flappy network and during our efforts to restore the network some prolonged OSD heartbeats not reachable seems to have caused two OSDs to go down/out. The OSD containers were stuck in a restart loop and eventually were auto-out'd from the cluster. Restarting the affected OSD containers had no effect and the pg was stuck in a "down+remapped state." ``` [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg down pg 2.699 is down+remapped, acting [NONE,NONE,NONE,NONE,613,353,488,333,408,282,167,145] ``` Looking over the `pg query 2.699` we could the following: ``` "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 38, 120 ], "peering_blocked_by": [ { "osd": 38, "current_lost_at": 90605, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 120, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, ``` With that output and reading ceph documentation[1], we decided to mark osd.38 as lost (`ceph osd lost 38`). Within a couple minutes, a 3rd OSD went down in osd.613 and exhibited the same restart loop behavior. Leaving us with the following state from a ` ceph pg query 2.699` ``` "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 38, 120, 613 ], "peering_blocked_by": [ { "osd": 38, "current_lost_at": 90605, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 120, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 613, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] }, { "name": "Started", "enter_time": "2024-12-21T17:13:18.435769+0000" } ], "agent_state": {} ``` After sleeping on it and letting the cluster recover from another failed OSD, we decided to mark all three as lost (osd.38, osd.120, and osd.613). Unfortunately, this caused two additional OSDs (osd.226 and osd.339) to go down. This changed the pg state from `down+remapped` to`remapped+incomplete`. We now have 5 down/out OSDs and are wondering if there is anyone to recover this PG? This one PG has caused our 1.6 PiB to be inaccessible.. We've gleaned some documentation[2] involving exporting the affected pg data with ceph-objectstore-tool and importing into a new OSD. Is this our only route to possibly recover this pg and its data? The strange thing is the OSDs underlying disk seems fine in that the ceph volume-group/LVM are intact. There are 0 issues with the disks from what I can glean or any indication of disk issues (the last 3 disks that were outed were part of a successful recovery/backfill with no issues). I tested the two original outed OSDs and the underlying disks all pass smartctl short tests and have seen no indication of hardware issues with these spinning disks. Looking over the ceph documentation, it looks like there is a newer tool that is cephadm/bluestore friendly in ceph-bluestore-tool[3] that does the same job as ceph-objectstore-tool? We are hoping there is a way to recover this PG and appreciate any advice. Currently our production cluster is down and are looking for recovery avenues. - Nick Links: [1] https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#pla… [2] https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor… [3] https://docs.ceph.com/en/reef/man/8/ceph-bluestore-tool/

3 2

disregard Re: Missing Release file? (cephadm add-repo —release squid fails on Ubuntu 24.04.1 LTS)
by Christian Kuhtz 31 Dec '24

31 Dec '24

Ooops, I can’t read. 24.04.1 LTS isn’t on the OS list, only 22.04. :-( On Dec 30, 2024 at 13:29:39, Christian Kuhtz <christian(a)kuhtz.com> wrote: > Hello, > > Host is an Ubuntu 24.04.1 LTS server image. Following instructions at > https://docs.ceph.com/en/squid/cephadm/install/ for cephadm install with > curl. The command > > ./cephadm add-repo --release squid > > fails because it can’t find the Release file and is getting a 404 from the > repo. Full output is below. It does install /etc/apt/sources.d/ceph.list > with the line below > > deb https://download.ceph.com/debian-squid/ noble main > > before failing. If I modify ceph.list with bookworm instead of noble, apt > update succeeds. If this is tested on Ubuntu 24.01.4 LTS as > https://docs.ceph.com/en/squid/start/os-recommendations/ suggests, > shouldn’t this work out of the box? > > Thanks, > Christian > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 24.04.1 LTS > Release: 24.04 > Codename: noble > # /cephadm add-repo --release squid > Installing repo GPG key from https://download.ceph.com/keys/release.gpg... > Installing repo file at /etc/apt/sources.list.d/ceph.list... > Updating package list... > Non-zero exit code 100 from apt-get update > apt-get: stdout Hit:1 http://security.ubuntu.com/ubuntu noble-security > InRelease > apt-get: stdout Ign:2 https://download.ceph.com/debian-squid noble > InRelease > apt-get: stdout Err:3 https://download.ceph.com/debian-squid noble Release > apt-get: stdout 404 Not Found [IP: 158.69.68.124 443] > apt-get: stdout Hit:4 http://us.archive.ubuntu.com/ubuntu noble InRelease > apt-get: stdout Hit:5 http://us.archive.ubuntu.com/ubuntu noble-updates > InRelease > apt-get: stdout Hit:6 http://us.archive.ubuntu.com/ubuntu noble-backports > InRelease > apt-get: stdout Reading package lists... > apt-get: stderr E: The repository 'https://download.ceph.com/debian-squid > noble Release' does not have a Release file. > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File "/root/./cephadm/__main__.py", line 5579, in <module> > File "/root/./cephadm/__main__.py", line 5567, in main > File "/root/./cephadm/__main__.py", line 4575, in command_add_repo > File "/root/./cephadm/cephadmlib/packagers.py", line 192, in add_repo > File "/root/./cephadm/cephadmlib/packagers.py", line 213, in update > File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in > call_throws > RuntimeError: Failed command: apt-get update: E: The repository ' > https://download.ceph.com/debian-squid noble Release' does not have a > Release file. >

1 0

Missing Release file? (cephadm add-repo —release squid fails on Ubuntu 24.04.1 LTS)
by Christian Kuhtz 31 Dec '24

31 Dec '24

Hello, Host is an Ubuntu 24.04.1 LTS server image. Following instructions at https://docs.ceph.com/en/squid/cephadm/install/ for cephadm install with curl. The command ./cephadm add-repo --release squid fails because it can’t find the Release file and is getting a 404 from the repo. Full output is below. It does install /etc/apt/sources.d/ceph.list with the line below deb https://download.ceph.com/debian-squid/ noble main before failing. If I modify ceph.list with bookworm instead of noble, apt update succeeds. If this is tested on Ubuntu 24.01.4 LTS as https://docs.ceph.com/en/squid/start/os-recommendations/ suggests, shouldn’t this work out of the box? Thanks, Christian # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 24.04.1 LTS Release: 24.04 Codename: noble # /cephadm add-repo --release squid Installing repo GPG key from https://download.ceph.com/keys/release.gpg... Installing repo file at /etc/apt/sources.list.d/ceph.list... Updating package list... Non-zero exit code 100 from apt-get update apt-get: stdout Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease apt-get: stdout Ign:2 https://download.ceph.com/debian-squid noble InRelease apt-get: stdout Err:3 https://download.ceph.com/debian-squid noble Release apt-get: stdout 404 Not Found [IP: 158.69.68.124 443] apt-get: stdout Hit:4 http://us.archive.ubuntu.com/ubuntu noble InRelease apt-get: stdout Hit:5 http://us.archive.ubuntu.com/ubuntu noble-updates InRelease apt-get: stdout Hit:6 http://us.archive.ubuntu.com/ubuntu noble-backports InRelease apt-get: stdout Reading package lists... apt-get: stderr E: The repository 'https://download.ceph.com/debian-squid noble Release' does not have a Release file. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 5579, in <module> File "/root/./cephadm/__main__.py", line 5567, in main File "/root/./cephadm/__main__.py", line 4575, in command_add_repo File "/root/./cephadm/cephadmlib/packagers.py", line 192, in add_repo File "/root/./cephadm/cephadmlib/packagers.py", line 213, in update File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: apt-get update: E: The repository ' https://download.ceph.com/debian-squid noble Release' does not have a Release file.

1 0

download.ceph.com TLS cert expired 29/12/2024
by Zakhar Kirpichenko 30 Dec '24

30 Dec '24

Hi! Looks like the TLS certificate at download.ceph.com expired on 29/12/2024: Certificate chain 0 s:CN = download.ceph.com i:C = US, O = Let's Encrypt, CN = R11 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Sep 30 23:02:29 2024 GMT; NotAfter: Dec 29 23:02:28 2024 GMT This affects multiple things, such as for example updating package repositories on multiple systems. Perhaps someone could please look into this. Best regards, Zakhar

3 4

cephadm rollout behavior and post adoption issues
by Nima AbolhassanBeigi 29 Dec '24

29 Dec '24

Hi everybody, I'm fairly new to cephadm. I'm trying to get some hands-on experience. I have a test cluster consisting of: 3 Monitor/Manager nodes. 6 OSD nodes 3 RGW nodes Pacific version containerized deployment using: quay.io/ceph/daemon:v6.0.11-stable-6.0-pacific-centos-stream8 I've adopted this cluster using the cephadm adopt playbook in the ceph-ansible project. infrastructure-playbooks/cephadm-adopt.yml I set the default container_image to a certain image and I checked the configuration database and it was set to the correct image. The adoption process for monitor, manager, and RGW daemons (redeployment in the case of RGWs) used the value for container_image, but the OSDs used the old image they were using before they were adopted. container_image value: quay.io/ceph/ceph:v16.2.13 I was wondering what could be the reason for the image not changing for OSDs. After the adoption, I wanted to mount the client.admin.keyring to monitor daemons. I used the extra_container_args service specification and added the volume. After I applied the mon service spec file to the cluster using ceph orch apply -i <filename>, I witnessed something odd. I read that cephadm has a rollout-like behavior for redeploying daemons. But the delay between the redeploys was 4 seconds and there was a period when all the monitor daemons were in the starting state. Is there any mechanism to increase the delay between the redeployments? Is there a way to enforce the redeployments to only 1 daemon? I've provided some logs, at the end of the message, that I think can be helpful. Thanks in advance. Regards Nima AbolhassanBeigi NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mon.stg-pacific-mon1 stg-pacific-mon1 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon2 stg-pacific-mon2 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon3 stg-pacific-mon3 starting - - - 6144M <unknown> <unknon w> Every 1.0s: ceph orch ps --service_name=mon --refresh stg-pacific-mon1: Sun Dec 29 15:52:15 2024 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mon.stg-pacific-mon1 stg-pacific-mon1 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon2 stg-pacific-mon2 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon3 stg-pacific-mon3 starting - - - 6144M <unknown> <unknow n> Every 1.0s: ceph orch ps --service_name=mon --refresh stg-pacific-mon1: Sun Dec 29 15:52:41 2024 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.stg-pacific-mon1 stg-pacific-mon1 running (37s) 2s ago 6d 30.2M 6144M 16.2.13 9e29efbb67d5 367a7f3bac31 mon.stg-pacific-mon2 stg-pacific-mon2 running (33s) 2s ago 11d 30.5M 6144M 16.2.13 9e29efbb67d5 43828e2453da mon.stg-pacific-mon3 stg-pacific-mon3 running (29s) 2s ago 11d 24.8M 6144M 16.2.13 9e29efbb67d5 d698a809370f 2024-12-29T12:18:16.974710+0000 mgr.stg-pacific-mon1 (mgr.173193) 322363 : cephadm [INF] Saving service mon spec with placement count:7;label:mons 2024-12-29T12:19:44.140240+0000 mgr.stg-pacific-mon1 (mgr.173193) 322545 : cephadm [INF] Saving service mon spec with placement count:3;label:mons 2024-12-29T12:19:47.497916+0000 mgr.stg-pacific-mon1 (mgr.173193) 322555 : cephadm [INF] Redeploying mon.stg-pacific-mon1, (container cli args changed) . . . 2024-12-29T12:19:47.507683+0000 mgr.stg-pacific-mon1 (mgr.173193) 322556 : cephadm [INF] Deploying daemon mon.stg-pacific-mon1 on stg-pacific-mon1 2024-12-29T12:19:50.970499+0000 mgr.stg-pacific-mon1 (mgr.173193) 322562 : cephadm [INF] Redeploying mon.stg-pacific-mon2, (container cli args changed) . . . 2024-12-29T12:19:50.982244+0000 mgr.stg-pacific-mon1 (mgr.173193) 322563 : cephadm [INF] Deploying daemon mon.stg-pacific-mon2 on stg-pacific-mon2 2024-12-29T12:19:55.774118+0000 mgr.stg-pacific-mon1 (mgr.173193) 322573 : cephadm [INF] Redeploying mon.stg-pacific-mon3, (container cli args changed) . . . 2024-12-29T12:19:55.782550+0000 mgr.stg-pacific-mon1 (mgr.173193) 322574 : cephadm [INF] Deploying daemon mon.stg-pacific-mon3 on stg-pacific-mon3 2024-12-29T12:21:56.620507+0000 mgr.stg-pacific-mon1 (mgr.173193) 322830 : cephadm [INF] Saving service mon spec with placement count:3;label:mons 2024-12-29T12:22:02.198770+0000 mgr.stg-pacific-mon1 (mgr.173193) 322844 : cephadm [INF] Redeploying mon.stg-pacific-mon1, (container cli args changed) . . . 2024-12-29T12:22:02.210484+0000 mgr.stg-pacific-mon1 (mgr.173193) 322845 : cephadm [INF] Deploying daemon mon.stg-pacific-mon1 on stg-pacific-mon1 2024-12-29T12:22:05.654926+0000 mgr.stg-pacific-mon1 (mgr.173193) 322851 : cephadm [INF] Redeploying mon.stg-pacific-mon2, (container cli args changed) . . . 2024-12-29T12:22:05.667782+0000 mgr.stg-pacific-mon1 (mgr.173193) 322852 : cephadm [INF] Deploying daemon mon.stg-pacific-mon2 on stg-pacific-mon2 2024-12-29T12:22:10.332756+0000 mgr.stg-pacific-mon1 (mgr.173193) 322862 : cephadm [INF] Redeploying mon.stg-pacific-mon3, (container cli args changed) . . . 2024-12-29T12:22:10.344298+0000 mgr.stg-pacific-mon1 (mgr.173193) 322863 : cephadm [INF] Deploying daemon mon.stg-pacific-mon3 on stg-pacific-mon3

1 0

Tpm2 in squid
by Ehsan Golpayegani 29 Dec '24

29 Dec '24

Hi ceph users, Has anyone successfully installed Ceph Squid with TPM2 enabled on an OSD? I'm encountering an error during installation: "stderr_lines": ["Error EINVAL: Failed to validate OSD spec "<unnamed>": Feature `tpm2` is not supported"]. I've also attempted to prepare the disks with TPM2 before adding them to the cluster, but I receive the following error: "/usr/bin/podman: stderr ceph-volume lvm prepare: error: unrecognized arguments: --with-tpm". For reference, my cluster is deployed via Cephadm and I'm using Squid 19.2.0. Any guidance would be greatly appreciated. Thanks, Ehsan -------------- Kind Regards <http://about.me/ehsan310?promo=email_sig> Ehsan Golpayegani [image: https://]www.ehsan.pw <https://ehsan.pw?promo=email_sig&utm_source=product&utm_medium=email_sig&ut…> SAVE PAPER - THINK BEFORE YOU PRINT!

2 3

Pls stuck in snaptrim
by bellow.oar_0t＠icloud.com 28 Dec '24

28 Dec '24

Greetings, I am having trouble with (the same) PGs stuck in snaptrrim and snaptrrim_wait for over half a year now. I don’t know how to debug/fix that, I tried bringing down the cluster, rebooted everything, failed the mds, mgr, upgraded to a new mentor version, upgraded to a new major version, all to no avail. Ceph version is 19.2.0 # ceph pg dump|grep snaptrim 9.7d 123817 0 0 0 0 479247022395 0 0 4375 346 4375 active+clean+snaptrim_wait 2024-10-13T00:00:12.050817+0000 134416'449058 134416:4144441 [9,14,0,6] 9 [9,14,0,6] 9 134054'434172 2024-10-11T01:26:12.751712+0000 134054'434172 2024-10-11T01:26:12.751712+0000 1256 138366 user requested deep scrub scheduled @ 2024-12-27T08:47:58.847491+0000 114488 0 9.74 123271 0 0 0 0 478810964688 0 0 4440 1140 4440 active+clean+snaptrim 2024-10-13T00:02:12.299691+0000 134416'451014 134416:4037366 [9,8,6,14] 9 [9,8,6,14] 9 134049'436191 2024-10-10T09:25:49.744261+0000 134049'436191 2024-10-10T09:25:49.744261+0000 1256 80746 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094636+0000 114101 0 9.71 124359 0 0 0 0 482220121204 0 0 4374 878 4374 active+clean+snaptrim_wait 2024-10-13T00:02:05.517504+0000 134416'454057 134416:4334146 [9,8,7,3] 9 [9,8,7,3] 9 134049'438914 2024-10-10T05:16:04.290745+0000 134049'438914 2024-10-10T05:16:04.290745+0000 1256 65761 user requested deep scrub scheduled @ 2024-12-27T08:48:03.954808+0000 115086 0 9.6b 123942 0 0 0 0 480451675897 0 0 4358 345 4358 active+clean+snaptrim_wait 2024-10-13T00:00:12.540600+0000 134416'450439 134416:4705616 [9,4,0,11] 9 [9,4,0,11] 9 134049'435438 2024-10-10T18:35:30.564215+0000 134049'435438 2024-10-10T18:35:30.564215+0000 1256 113715 user requested deep scrub scheduled @ 2024-12-27T08:47:37.794121+0000 114709 0 9.6a 123870 0 0 0 0 480164531666 0 0 4364 879 4364 active+clean+snaptrim 2024-10-13T00:02:05.652753+0000 134416'456701 134416:4003484 [9,4,0,13] 9 [9,4,0,13] 9 134049'441302 2024-10-10T12:42:55.289279+0000 134049'441302 2024-10-10T12:42:55.289279+0000 1256 92560 user requested deep scrub scheduled @ 2024-12-27T08:48:09.914747+0000 114496 0 9.5f 123879 0 0 0 0 479941485759 0 0 4431 1141 4431 active+clean+snaptrim_wait 2024-10-13T00:00:13.340335+0000 134416'450791 134416:4123574 [9,3,2,6] 9 [9,3,2,6] 9 134054'435775 2024-10-11T01:27:38.427214+0000 134054'435775 2024-10-11T01:27:38.427214+0000 1256 138452 user requested deep scrub scheduled @ 2024-12-27T08:49:04.452674+0000 114542 0 27.19 7017 0 0 0 0 29241743329 0 0 4358 0 4358 active+clean+snaptrim_wait 2024-10-13T00:02:05.652729+0000 134416'5924654 134416:11859363 [9,7,3,4] 9 [9,7,3,4] 9 133910'5800741 2024-10-09T13:55:08.732933+0000 133910'5800741 2024-10-09T13:55:08.732933+0000 1256 10492 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053054+0000 6293 0 27.4 7072 0 0 0 0 29496160608 0 0 4401 0 4401 active+clean+snaptrim_wait 2024-10-13T00:00:13.740349+0000 134416'5920493 134416:12128517 [9,0,14,2] 9 [9,0,14,2] 9 133972'5796609 2024-10-09T12:20:39.099453+0000 133972'5796609 2024-10-09T12:20:39.099453+0000 1256 4833 user requested deep scrub scheduled @ 2024-12-27T08:47:47.863907+0000 6249 0 9.1c 124329 0 0 0 0 482320315530 0 0 4368 878 4368 active+clean+snaptrim_wait 2024-10-13T00:02:05.518217+0000 134416'453725 134416:3954611 [9,7,11,0] 9 [9,7,11,0] 9 134049'438455 2024-10-10T06:40:15.938302+0000 134049'438455 2024-10-10T06:40:15.938302+0000 1256 70803 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053068+0000 114948 0 9.19 123790 0 0 0 0 480077901246 0 0 4411 218 4411 active+clean+snaptrim_wait 2024-10-13T00:00:12.539642+0000 134416'451736 134416:3826333 [9,6,7,1] 9 [9,6,7,1] 9 134049'436829 2024-10-10T21:35:24.519252+0000 134049'436829 2024-10-10T21:35:24.519252+0000 1256 138616 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053063+0000 114595 0 dumped all 9.13 124027 0 0 0 0 480656352004 0 0 4370 0 4370 active+clean+snaptrim_wait 2024-10-13T00:00:06.753470+0000 134416'447134 134416:3903589 [9,14,13,0] 9 [9,14,13,0] 9 134049'431887 2024-10-10T03:55:32.120382+0000 134049'431887 2024-10-10T03:55:32.120382+0000 1256 60916 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053062+0000 114776 0 9.28 123470 0 0 0 0 478607727659 0 0 4355 0 4355 active+clean+snaptrim_wait 2024-10-13T00:02:12.298971+0000 134416'450238 134416:3500480 [9,0,1,3] 9 [9,0,1,3] 9 134054'435412 2024-10-11T01:23:22.978541+0000 134054'435412 2024-10-11T01:23:22.978541+0000 1256 138187 user requested deep scrub scheduled @ 2024-12-27T08:48:20.260350+0000 114182 0 9.37 123338 0 0 0 0 478217011616 0 0 4391 248 4391 active+clean+snaptrim_wait 2024-10-13T00:02:06.016700+0000 134416'448180 134416:3783995 [9,14,7,13] 9 [9,14,7,13] 9 134054'433338 2024-10-11T01:26:24.029875+0000 134054'433338 2024-10-11T01:26:24.029875+0000 1256 138368 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053069+0000 114085 0 9.42 123550 0 0 0 0 479437295594 0 0 4385 415 4385 active+clean+snaptrim_wait 2024-10-13T00:00:13.264210+0000 134416'446881 134416:3886466 [9,7,1,6] 9 [9,7,1,6] 9 134054'431973 2024-10-11T04:24:50.076511+0000 134054'431973 2024-10-11T04:24:50.076511+0000 1256 163177 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053050+0000 114369 0 9.48 123504 0 0 0 0 478672624191 0 0 4370 0 4370 active+clean+snaptrim_wait 2024-10-13T00:00:15.366907+0000 134416'447888 134416:3476913 [9,2,13,7] 9 [9,2,13,7] 9 134049'432764 2024-10-10T18:43:14.290802+0000 134049'432764 2024-10-10T18:43:14.290802+0000 1256 114183 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094642+0000 114094 0 9.4c 123799 0 0 0 0 480058080747 0 0 4367 298 4367 active+clean+snaptrim_wait 2024-10-13T00:00:15.365742+0000 134416'450187 134416:4460709 [9,7,4,0] 9 [9,7,4,0] 9 134049'435272 2024-10-10T12:45:20.588451+0000 134049'435272 2024-10-10T12:45:20.588451+0000 1256 92705 user requested deep scrub scheduled @ 2024-12-27T08:48:09.914749+0000 114623 0 9.4d 124344 0 0 0 0 480982998237 0 0 4349 878 4349 active+clean+snaptrim_wait 2024-10-13T00:02:12.299642+0000 134416'445413 134416:4170713 [9,14,0,7] 9 [9,14,0,7] 9 134049'430041 2024-10-10T18:35:55.877861+0000 134049'430041 2024-10-10T18:35:55.877861+0000 1255 113735 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094640+0000 114850 0 9.5c 123379 0 0 0 0 478151706144 0 0 4377 0 4377 active+clean+snaptrim_wait 2024-10-13T00:02:06.017489+0000 134416'452726 134416:4446619 [9,11,6,13] 9 [9,11,6,13] 9 134049'437743 2024-10-10T18:43:28.133384+0000 134049'437743 2024-10-10T18:43:28.133384+0000 1255 114188 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094632+0000 114239 0 pgs: 206 active+clean 16 active+clean+snaptrim_wait 2 active+clean+snaptrim 1 active+clean+scrubbing+deep Please advice. Daniel

2 1

Re: Ceph Reef 18.2.2 - stucked pgs in not scrubbed and deep-scrubbed in time
by Frédéric Nass 28 Dec '24

28 Dec '24

Hi, Can you try setting osd_scrub_disable_reservation_queuing to 'true' and report back? There's an ongoing bug [1] in Squid that closely matches what you're describing. This issue has been reported to affect multiple Ceph versions as well. Regards, Frédéric. [1] https://tracker.ceph.com/issues/69078 ________________________________ De : Saint Kid <saint8kid(a)gmail.com> Envoyé : samedi 28 décembre 2024 00:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph Reef 18.2.2 - stucked pgs in not scrubbed and deep-scrubbed in time Hello, I have adjusted the scrubbing schedule and manually kick started the process but it appears the warning remains on these 15 PGs. I am thinking of restarting the OSDs which those PGs are and hoping to clear this condition. Any suggestions? Best Regards, Saint Kid _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0