- ceph-users - lists.ceph.io

tentacle 20.2.0 GA QE validation status
by Yuri Weinstein 27 Oct '25

27 Oct '25

This release is RC1 and cherry picks from the list https://pad.ceph.com/p/tentacle_RC1_must-have Some suites need a rerun, but we are waiting for build fixing. Details of this release are summarized here: https://tracker.ceph.com/issues/73556#note-1 Release Notes - https://github.com/ceph/ceph/pull/65971 LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade/squid-x: NEED RERUN WAITING FOR BUILD upgrade/reef-x: NEED RERUN WAITING FOR BUILD upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: PASSED TIA

10 12

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

Hi Kirby, Not at all. It still sends the ticks and you can see the last lines of the ongoing logs below: Oct 24 19:58:35 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:35.798+0000 7e2cb3476640 20 osd.11 70297 check_full_status cur ratio 0.77673, physical ratio 0.77673, new state none Oct 24 19:58:36 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:36 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:36.106+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:36 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:36 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:36 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:36.661+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:36 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:36.661+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:37 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:37 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:37.094+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:37 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:37 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:37.681+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:37 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:37 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:37.681+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:38 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:38 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:38.074+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:38 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:38 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:38.644+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:38 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:38.644+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:38 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:39 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:39 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:39.064+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:39 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:39 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:39 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:39.658+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:39 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:39.658+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.071+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 reports for 0 queries Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 collect_pg_stats Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.415+0000 7e2cc18ba640 20 osd.11 70297 reports for 0 queries Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.415+0000 7e2cc18ba640 15 osd.11 70297 collect_pg_stats Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.624+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.624+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:41 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:41.048+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Selamlar, Huseyin Cotuk hcotuk(a)gmail.com > On 24 Oct 2025, at 19:57, Kirby Haze <kirbyhaze01(a)gmail.com> wrote: > > Are these the end of the logs? This actually looks normal and fine but eventually it will crash? or does the systemd unit stay running and they are stuck on these logs? > > On Fri, Oct 24, 2025 at 9:26 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >> journalctl logs of a failing OSD starts like in the link: https://pastes.io/osd11-restart-log >> >> Oct 24 18:50:46 ank-backup01 systemd[1]: Started ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f(a)osd.11.service - Ceph osd.11 for 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f. >> Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: set uid:gid to 167:167 (ceph:ceph) >> Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 2 >> Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: pidfile_write: ignore empty --pid-file >> >> I can see that it runs with the Ceph version 19.2.3. >> >> After the activation process (it takes some time), it does not crash actually. AFAIR from the logs, it mounts bluefs, starts the rocksdb version 7.9.2 and loads the tables. It evens compacts the rocksdb. >> >> Afterwards, it triggers the boot process as follows. >> >> Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 journal looks like hdd >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 journal looks like hdd >> Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 boot >> Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 configured osd_max_object_name[space]_len looks ok >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 boot >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 20 osd.11 0 configured osd_max_object_name[space]_len looks ok >> >> I noted that at this point, it loads the osdmap 70297 >> >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.706+0000 7e2ccf90a740 20 osd.11 0 get_map 70297 - loading and decoding 0x565a07bb0000 >> >> There are lots of missing logs: >> >> Oct 24 18:53:48 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:48.603+0000 7e2ccf90a740 20 read_log_and_missing 68276'8386071 (0'0) modify 23:bf0d9243:::48f964cb-12c5-4689-8aa4-0b8e573f41d5.37815204.2_001%2f184%2f288%2f052.zip:head by client.43525885.0:1306616953 2025-10-09T11:06:44.324360+0000 0 ObjectCleanRegions clean_offsets: [(48911, 18446744073709502704)], clean_omap: true, new_object: false >> >> After all these missing logs, it starts and loads the PGs. >> >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 load_pgs opened 66 pgs >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 superblock: I am osd.11 >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 heartbeat osd_stat(store_statfs(0x19ff4006000/0x0/0x74702400000, data 0x4b4a481b898/0x4b4a4e68000, compress 0x0/0x0/0x0, omap 0x1a50, meta 0xf26958e5b0), peers [] op hist []) >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 done with init, starting boot process >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 start_boot >> >> It periodically dumps stats that shows that OSD is running (up) for xxx secs. >> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Uptime(secs): 600.6 total, 600.0 interval >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Flush(GB): cumulative 0.000, interval 0.000 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(GB): cumulative 0.000, interval 0.000 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Total Files): cumulative 0, interval 0 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(L0 Files): cumulative 0, interval 0 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Keys): cumulative 0, interval 0 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_b> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache BinnedLRUCache@0x5659dca05350#2 capacity: 1.06 GB usage: 100.56 MB table_size: 0 occupancy: 18446744073709551615 collections: 2 last_copies: 8> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache entry stats(count,size,portion): DataBlock(24571,97.85 MB,8.99328%) FilterBlock(14,1.17 MB,0.107375%) IndexBlock(20,1.54 MB,0.141681%) Misc(1,> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: >> >> >> The problem is, it can not register itself to the mon as UP and running. There are lots of ticks and tick_without_osd_lock messages in the log file. >> >> Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 10 osd.11 70297 tick >> Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 >> Oct 24 18:57:14 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock >> Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.729+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock >> Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 reports for 0 queries >> Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 collect_pg_stats >> >> It looks like it stays in a loop that send ticks with the older epoch (70297). >> >> Thanks for your help >> >> BR, >> Huseyin Cotuk >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >> >> >> >> >>> On 24 Oct 2025, at 18:36, Kirby Haze <kirbyhaze01(a)gmail.com <mailto:kirbyhaze01@gmail.com>> wrote: >>> >>> if you look at the osd’s logs and follow it while trigger a restart through systemctl >>> >>> systemctl list-units | grep osd >>> >>> journalctl -u <osd unit> >>> >>> When the osd boots up it spits out the version, I would be curious to see how it crashes and the stack trace >>> >>> On Fri, Oct 24, 2025 at 8:31 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >>>> Hi Kirby, >>>> >>>> Is there any way to check whether these OSDs have completed the upgrade successfully or not? >>>> >>>> Any help will be appreciated >>>> >>>> BR, >>>> Huseyin >>>> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >>>> >>>> >>>> >>>> >>>>> On 24 Oct 2025, at 18:14, Kirby Haze <kirbyhaze01(a)gmail.com <mailto:kirbyhaze01@gmail.com>> wrote: >>>>> >>>>> Osds usually try to load the last clean map on boot, but if these osds finished the upgrade it must have hit an assert I think >>>>> >>>>> On Fri, Oct 24, 2025 at 8:06 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >>>>>> Hi Can, >>>>>> >>>>>> No, I did not add or recreate these OSDs. I tried cephadm to run failing OSDs with the help of following commands without success: >>>>>> >>>>>> cephadm --image quay.io/ceph/ceph:v19 <http://quay.io/ceph/ceph:v19> unit --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f restart >>>>>> >>>>>> cephadm --image aade1b12b8e6 run --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f >>>>>> >>>>>> BR, >>>>>> Huseyin >>>>>> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> > On 24 Oct 2025, at 17:55, Can Özyurt <acozyurt(a)gmail.com <mailto:acozyurt@gmail.com>> wrote: >>>>>> > >>>>>> > Hi Huseyin, >>>>>> > >>>>>> > Did you add or recreate these OSDs after the upgrade? >>>>>> > >>>>>> > On Fri, 24 Oct 2025 at 16:42, Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>>> wrote: >>>>>> >> More debug logs from a failing OSD: >>>>>> >> >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 70297 reports for 0 queries >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 70297 collect_pg_stats >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.5fs2( v 69081'8395094 (68276'8390628,69081'8395094] local-lis/les=69078/69079 n=8316950 ec=21807/21807 lis/c=69078/68833 les/c/f=69079/68834/39143 sis=70280) [NONE,NONE,11,8,14,16,0,20,17,18,NONE]p11(2) r=2 lpr=70297 pi=[68833,70280)/7 crt=69081'8395094 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>>>> >> >>>>>> >> >>>>>> >> Selamlar, >>>>>> >> Huseyin Cotuk >>>>>> >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> _______________________________________________ >>>>>> >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> >>>>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >>

2 1

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

journalctl logs of a failing OSD starts like in the link: https://pastes.io/osd11-restart-log Oct 24 18:50:46 ank-backup01 systemd[1]: Started ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f(a)osd.11.service - Ceph osd.11 for 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f. Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: set uid:gid to 167:167 (ceph:ceph) Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 2 Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: pidfile_write: ignore empty --pid-file I can see that it runs with the Ceph version 19.2.3. After the activation process (it takes some time), it does not crash actually. AFAIR from the logs, it mounts bluefs, starts the rocksdb version 7.9.2 and loads the tables. It evens compacts the rocksdb. Afterwards, it triggers the boot process as follows. Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 journal looks like hdd Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 journal looks like hdd Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 boot Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 configured osd_max_object_name[space]_len looks ok Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 boot Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 20 osd.11 0 configured osd_max_object_name[space]_len looks ok I noted that at this point, it loads the osdmap 70297 Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.706+0000 7e2ccf90a740 20 osd.11 0 get_map 70297 - loading and decoding 0x565a07bb0000 There are lots of missing logs: Oct 24 18:53:48 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:48.603+0000 7e2ccf90a740 20 read_log_and_missing 68276'8386071 (0'0) modify 23:bf0d9243:::48f964cb-12c5-4689-8aa4-0b8e573f41d5.37815204.2_001%2f184%2f288%2f052.zip:head by client.43525885.0:1306616953 2025-10-09T11:06:44.324360+0000 0 ObjectCleanRegions clean_offsets: [(48911, 18446744073709502704)], clean_omap: true, new_object: false After all these missing logs, it starts and loads the PGs. Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 load_pgs opened 66 pgs Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 superblock: I am osd.11 Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 heartbeat osd_stat(store_statfs(0x19ff4006000/0x0/0x74702400000, data 0x4b4a481b898/0x4b4a4e68000, compress 0x0/0x0/0x0, omap 0x1a50, meta 0xf26958e5b0), peers [] op hist []) Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 done with init, starting boot process Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 start_boot It periodically dumps stats that shows that OSD is running (up) for xxx secs. Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Uptime(secs): 600.6 total, 600.0 interval Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Flush(GB): cumulative 0.000, interval 0.000 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(GB): cumulative 0.000, interval 0.000 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Total Files): cumulative 0, interval 0 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(L0 Files): cumulative 0, interval 0 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Keys): cumulative 0, interval 0 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_b> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache BinnedLRUCache@0x5659dca05350#2 capacity: 1.06 GB usage: 100.56 MB table_size: 0 occupancy: 18446744073709551615 collections: 2 last_copies: 8> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache entry stats(count,size,portion): DataBlock(24571,97.85 MB,8.99328%) FilterBlock(14,1.17 MB,0.107375%) IndexBlock(20,1.54 MB,0.141681%) Misc(1,> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: The problem is, it can not register itself to the mon as UP and running. There are lots of ticks and tick_without_osd_lock messages in the log file. Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 18:57:14 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.729+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 reports for 0 queries Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 collect_pg_stats It looks like it stays in a loop that send ticks with the older epoch (70297). Thanks for your help BR, Huseyin Cotuk hcotuk(a)gmail.com > On 24 Oct 2025, at 18:36, Kirby Haze <kirbyhaze01(a)gmail.com> wrote: > > if you look at the osd’s logs and follow it while trigger a restart through systemctl > > systemctl list-units | grep osd > > journalctl -u <osd unit> > > When the osd boots up it spits out the version, I would be curious to see how it crashes and the stack trace > > On Fri, Oct 24, 2025 at 8:31 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >> Hi Kirby, >> >> Is there any way to check whether these OSDs have completed the upgrade successfully or not? >> >> Any help will be appreciated >> >> BR, >> Huseyin >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >> >> >> >> >>> On 24 Oct 2025, at 18:14, Kirby Haze <kirbyhaze01(a)gmail.com <mailto:kirbyhaze01@gmail.com>> wrote: >>> >>> Osds usually try to load the last clean map on boot, but if these osds finished the upgrade it must have hit an assert I think >>> >>> On Fri, Oct 24, 2025 at 8:06 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >>>> Hi Can, >>>> >>>> No, I did not add or recreate these OSDs. I tried cephadm to run failing OSDs with the help of following commands without success: >>>> >>>> cephadm --image quay.io/ceph/ceph:v19 <http://quay.io/ceph/ceph:v19> unit --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f restart >>>> >>>> cephadm --image aade1b12b8e6 run --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f >>>> >>>> BR, >>>> Huseyin >>>> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >>>> >>>> >>>> >>>> >>>> > On 24 Oct 2025, at 17:55, Can Özyurt <acozyurt(a)gmail.com <mailto:acozyurt@gmail.com>> wrote: >>>> > >>>> > Hi Huseyin, >>>> > >>>> > Did you add or recreate these OSDs after the upgrade? >>>> > >>>> > On Fri, 24 Oct 2025 at 16:42, Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>>> wrote: >>>> >> More debug logs from a failing OSD: >>>> >> >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 70297 reports for 0 queries >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 70297 collect_pg_stats >>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.5fs2( v 69081'8395094 (68276'8390628,69081'8395094] local-lis/les=69078/69079 n=8316950 ec=21807/21807 lis/c=69078/68833 les/c/f=69079/68834/39143 sis=70280) [NONE,NONE,11,8,14,16,0,20,17,18,NONE]p11(2) r=2 lpr=70297 pi=[68833,70280)/7 crt=69081'8395094 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>> >> >>>> >> >>>> >> Selamlar, >>>> >> Huseyin Cotuk >>>> >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> >>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

More debug logs from a failing OSD: Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 70297 reports for 0 queries Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 70297 collect_pg_stats Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.5fs2( v 69081'8395094 (68276'8390628,69081'8395094] local-lis/les=69078/69079 n=8316950 ec=21807/21807 lis/c=69078/68833 les/c/f=69079/68834/39143 sis=70280) [NONE,NONE,11,8,14,16,0,20,17,18,NONE]p11(2) r=2 lpr=70297 pi=[68833,70280)/7 crt=69081'8395094 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] Selamlar, Huseyin Cotuk hcotuk(a)gmail.com

3 5

Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

Hi Cephers, I have recently completed Reef to Squid upgrade in our test environment, After the upgrade, some of the OSDs did not come up. podman containers are up and running but some OSDs did not join the cluster and seem down. After digging into logs, I saw that down OSDs try to boot with osdmap epoch 70297 but the current osdmap epoch is 72555. All the running OSDs use the latest osdmap on the logs, but the failing ones use the same and older osdmap (epcoh 70297). This is a single-node test cluster and has been running for years. All the OSDs are on the same host and all the containers related to OSDs are running. After enabling debug on a failing OSD, I can see that there are several ticks with the given epoch like below. 2025-10-23T21:00:23.945+0000 7f3e05b79640 20 osd.41 70297 tick last_purged_snaps_scrub 2025-10-22T22:20:42.570629+0000 next 2025-10-24T03:13:12.570629+000 I tried to get the latest osdmap via “ceph osd getmap 72555 > /tmp/osdmap.72555” and set with the command: CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-41/ --op set-osdmap --file /tmp/osd_map_72555" but it fails with the error: osdmap (#-1:9c8e9ef2:::osdmap.72555:0#) does not exist. I can verify all the PGs and the underlying objects are staying safe on the disk by using the ceph-bluestore-tool. Does anyone have a clue why some of the OSDs do not get the latest osdmap from the mon? Thanks in advance, Huseyin hcotuk(a)gmail.com

1 0

Next Cephalocon (2026) - when/where?
by Sebastian 24 Oct '25

24 Oct '25

Hey, I know it is too early, but maybe you already know where it will be? Unfortunately, I have not received permission from the company to visit Cephalocon ‘25 in Vancouver. The rule is simple where I work. Conferences are allowed in the European continent. So when and where do you plan the next Cephalocon to be? I also consider the option to pay for it myself if it will be outside Europe. BTW, too much knowledge to lose IMO :) BR, Sebastian

3 2

Attention: Documentation
by abobakr osman 23 Oct '25

23 Oct '25

Small typo in the sentence[0]: " Baseboard Management Controller (BMC)...Additionally BMCs as of 2023 rarely sport network connections faster than 1 Gb/s, so dedicated and inexpensive 1 Gb/s switches for BMC administrative traffic may reduce costs by wasting fewer expenive ports on faster host switches. " Not sure if sport is intended here as well. [0] https://docs.ceph.com/en/latest/start/hardware-recommendations/#baseboard-m… Best regards Abobakr Mohammed

2 1

Very slow mirroring operation
by GLE, Vivien 23 Oct '25

23 Oct '25

Hi, I'm testing mirroring in ceph and dealt with very slow mirroring operation, Cluster A (primary) is on a Ceph hyperconverged on proxmox and cluster B (secondary) is a Ceph cluster managed by cephadm. These 2 have been peered via bootstrap. Moving 60 GiB VM to the mirrored pool in cluster A it took 5-6 hours to fully sync in the cluster B (SSD device only) Moving 60 GiB VM from cluster A to cluster B in replica 3 SSD pool took 2 minutes (SSD device only) It looks like for this purpose cluster wont be higher than 6-7 MiB/s, is this a normal behaviour ? Ceph -s during mirroring output and rbd info below ceph -s cluster: id: ID health: HEALTH_OK services: mon: 3 daemons, quorum r620-13-1,r620-13-9,r620-13-4 (age 12d) mgr: r620-13-7.zxzajo(active, since 24h), standbys: r620-13-10.wmzodp mds: 1/1 daemons up osd: 24 osds: 24 up (since 6d), 24 in (since 6d) rbd-mirror: 1 daemon active (1 hosts) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 1287 pgs objects: 151.21k objects, 551 GiB usage: 1.6 TiB used, 6.3 TiB / 7.9 TiB avail pgs: 1287 active+clean io: client: 15 KiB/s rd, 6.5 MiB/s wr, 11 op/s rd, 419 op/s wr -------------------------------------------- rbd info ceph_proxmox/vm-118-disk-0 rbd image 'vm-118-disk-0': size 60 GiB in 15360 objects order 22 (4 MiB objects) snapshot_count: 0 id: abe8c560f69498 block_name_prefix: rbd_data.abe8c560f69498 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Wed Oct 22 07:52:31 2025 access_timestamp: Wed Oct 22 07:52:31 2025 modify_timestamp: Wed Oct 22 07:52:31 2025 journal: abe8c560f69498 mirroring state: enabled mirroring mode: journal mirroring global id: id mirroring primary: false ----------------------------------------- In cluster A ceph.conf rbd_default_features = 125 Thanks ! Vivien

3 3

newbie CRUSH rules / rbd pool
by Alexander Leutz 23 Oct '25

23 Oct '25

Hi everybody, I'm new to ceph. I'm testing 4 proxmox ceph nodes. I researched a lot on CRUSH rules and I build a rbd pool for use with my test-vms. I'm reading a lot in the official docs under docs.ceph.com. But I still have a lot of white space on my personal knowledge map. Are there any other recommendations for getting a full / deeper understanding of ceph? books? Best regards, Alexander Concat AG IT Solutions | Berliner Ring 127 - 129 | 64625 Bensheim | www.concat.de alexander.leutz(a)concat.de | Tel.: +49 6251 7026-143 | Mobil: +49 151 18856009 | Fax: +49 6251 7026-356 Hauptsitz Bensheim a. d. Bergstraße | Vorstand Olaf von Heyer Aufsichtsratsvorsitzender Tim Patronik | Amtsgericht Darmstadt (HRB 25326)

4 6

Ceph RGW Barbican Integration — Tenant isolation issue (Barbican keys from one project usable by another)
by Vignesh Varma 22 Oct '25

22 Oct '25

Hi Team, Lab Environment * Ceph: 19.2.x (Squid) * OpenStack: Caracal * OS: Ubuntu 22.04 LTS * Deployment: `cephadm` --- Setup Summary We have a working Ceph RGW + Barbican integration with: * Keystone auth: `rgw_s3_auth_use_keystone = true` * SSE-KMS backend: `rgw_crypt_s3_kms_backend = barbican` * Single RGW Barbican service user under a single project: ``` rgw_keystone_barbican_user = rgw-barbican rgw_keystone_barbican_project = service rgw_keystone_barbican_domain = default rgw_barbican_url = http://controller:9311 rgw_crypt_s3_kms_backend = barbican rgw_crypt_s3_kms_encryption_keys = {"coke":"<COKE_KEY_ID>","pepsi":"<PEPSI_KEY_ID>"} rgw_crypt_require_ssl = false ``` Each tenant (e.g., Coke, Pepsi) has: * Its own OpenStack project * Separate EC2 credentials * Its own Barbican AES-256 key (with `rgw-barbican` added to the secret ACL for read) S3 operations work for both tenants using their own EC2 credentials. --- Observed Issue Cross-tenant Barbican key usage is allowed: | Tenant | Key Used | Result | | ----------------- | ---------- | -------------- | | Coke → Coke key | ✅ Success | expected | | Pepsi → Pepsi key | ✅ Success | expected | | Coke → Pepsi key | ⚠️ Success | unexpected | | Pepsi → Coke key | ⚠️ Success | unexpected | If a client supplies any valid Barbican key UUID (one that `rgw-barbican` can read via ACL), RGW accepts it—even if the key belongs to a different project. This suggests RGW does not validate that the Barbican key is owned by (or scoped to) the same project as the authenticated S3 user. --- Minimal Repro 1. Create projects `coke` and `pepsi`; create per-project EC2 creds. 2. Store one Barbican key per project; grant `rgw-barbican` read ACL on both secrets. 3. Configure RGW as above with the `rgw_crypt_s3_kms_encryption_keys` mapping. 4. Using Coke EC2 creds, run `put-object` with Pepsi key UUID via `--ssekms-key-id <PEPSI_KEY_ID>`. → Upload succeeds; encryption uses Pepsi key. --- What We Need * A way for RGW to validate that the SSE-KMS key used in the request belongs to the same OpenStack project (tenant) as the authenticated S3 user. * Cross-project use of KMS keys should fail with `AccessDenied` (or similar). --- Questions for Guidance 1. Is there a supported (or planned) way to restrict RGW so the Barbican key must belong to the same project as the authenticated S3 user? 2. Can RGW be configured to derive the project scope from the S3 auth context when requesting the key from Barbican (instead of always using the fixed `rgw_keystone_barbican_project` for the token)? --- Best Regards, Vignesh

3 3