- ceph-users - lists.ceph.io

Re: resize/extend pool/fs storage size?
by listy 02 Mar '26

02 Mar '26

-> $ ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.001667", "last_optimize_started": "Mon Mar 2 04:08:35 2026", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.07428 root default -3 0.34180 host podster1 9 ssd 0.04880 osd.9 up 1.00000 1.00000 10 ssd 0.29300 osd.10 up 1.00000 1.00000 -7 0.39069 host podster2 0 ssd 0.09769 osd.0 up 1.00000 1.00000 4 ssd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 ssd 0.04880 osd.1 up 1.00000 1.00000 5 ssd 0.29300 osd.5 up 1.00000 1.00000 -> $ ceph osd dump | grep ratio full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 I've changed upmap_max_deviation = 1 I also did: -> $ ceph osd reweight osd.0 1.0 and ceased all clients' activity to FSes I've extend disk-devices ceph uses twice, for 'health' complained about 'backfillull' like here, only _pg_ was there, is is now below: -> $ ceph health detail HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5 pool(s) backfillfull [WRN] OSD_BACKFILLFULL: 1 backfillfull osd(s) osd.0 is backfill full [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull pg 3.1b is active+remapped+backfill_toofull, acting [4,5,10] pg 3.21 is active+remapped+backfill_toofull, acting [4,5,10] pg 3.23 is active+remapped+backfill_toofull, acting [10,5,4] [WRN] POOL_BACKFILLFULL: 5 pool(s) backfillfull pool '.mgr' is backfillfull pool 'cephfs.APKI.meta' is backfillfull pool 'cephfs.APKI.data' is backfillfull pool 'cephfs.MONERO.meta' is backfillfull pool 'cephfs.MONERO.data' is backfillfull Each time I "extended" devices-disks, cluster went to use osd.0 and fill it up. Yes everything is small in this cluster, it's a lab. -> $ ceph -w cluster: id: 9f4f9dba-72c7-11f0-8052-525400519d29 health: HEALTH_WARN 1 backfillfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 5 pool(s) backfillfull services: mon: 3 daemons, quorum podster3,podster2,podster1 (age 2d) [leader: podster3] mgr: podster1.qzojrl(active, since 2d), standbys: podster3.kyyolr mds: 2/2 daemons up, 2 standby osd: 6 osds: 6 up (since 12h), 6 in (since 2d); 3 remapped pgs data: volumes: 2/2 healthy pools: 5 pools, 289 pgs objects: 102.74k objects, 355 GiB usage: 1.1 TiB used, 523 GiB / 1.6 TiB avail pgs: 737/308229 objects misplaced (0.239%) 286 active+clean 3 active+remapped+backfill_toofull 2026-03-02T04:10:00.000125+0000 mon.podster3 [WRN] overall HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5 pool(s) backfillfull 2026-03-02T04:20:00.000091+0000 mon.podster3 [WRN] overall HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5 pool(s) backfillfull 2026-03-02T04:26:38.174003+0000 mon.podster3 [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2026-03-02T04:26:44.210790+0000 mon.podster3 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2026-03-02T04:30:00.000140+0000 mon.podster3 [WRN] overall HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5 pool(s) backfillfull 2026-03-02T04:40:00.000118+0000 mon.podster3 [WRN] overall HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5 pool(s) backfillfull 2026-03-02T04:50:00.000114+0000 mon.podster3 [WRN] overall HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5 pool(s) backfillfull If the cluster is doing something good, something it should be doing to heal, it's barely happening so slow it is - given no clients work now + as you said, storage capacities are minute in comparison to anything production. 7 hours later, still: -> $ ceph osd df tree | egrep '(osd.0|ID)' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 0 ssd 0.09769 1.00000 150 GiB 140 GiB 88 GiB 1.0 MiB 2.0 GiB 10 GiB 93.29 1.37 72 up osd.0 Now 5 hours later - since i started this draft - I added: -> $ ceph osd reweight-by-utilization and before that cmd above I also noticed: -> $ ceph config get mgr mgr/balancer/begin_weekday 0 -> $ ceph config get mgr mgr/balancer/end_weekday 0 which was done by 'deployment' process - cephadm bootstrap - and made me wonder: does that mean that auto-rebalance runs only on Sunday? I changed: end_weekday = 6 _reweight-by-utilization_ I notice, changed REWEIGHT for osd.0 and that did something, I think. So now _active+remapped+backfill_toofull_ are gone from 'pcs' part of health report. RAW USE & DATA are down, but stil: -> $ ceph osd df tree | egrep '(osd.0|ID)' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB 1.1 MiB 2.0 GiB 16 GiB 89.51 1.31 68 up osd.0 and when compared to other host-ods which use "identical" disk-drives: -> $ ceph osd df tree | egrep '(osd\.[0,1,9] |ID)' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB 624 KiB 1.4 GiB 98 GiB 34.83 0.51 43 up osd.9 0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB 1.1 MiB 2.0 GiB 16 GiB 89.51 1.31 68 up osd.0 1 ssd 0.04880 1.00000 150 GiB 53 GiB 53 GiB 526 KiB 254 MiB 97 GiB 35.40 0.52 44 up osd.1 Perhaps cluster goes only as far as to satisfy _backfill_toofull_ be gone and then "gives up"? The "other" disk-drives: -> $ ceph osd df tree | egrep '(osd\.(5|4|10)\ |ID)' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 2.0 MiB 2.1 GiB 93 GiB 76.68 1.12 246 up osd.10 4 ssd 0.29300 1.00000 400 GiB 275 GiB 273 GiB 3.2 MiB 2.1 GiB 125 GiB 68.83 1.01 221 up osd.4 5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB 3.0 MiB 2.5 GiB 95 GiB 76.27 1.12 245 up osd.5 Seems that _host podster2_ balances its osds 4 & 0 "differently" to what other two hosts do - if so then why?

1 0

Re: resize/extend pool/fs storage size?
by Anthony D'Atri 01 Mar '26

01 Mar '26

> On Mar 1, 2026, at 7:58 AM, listy via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi guys. > > I've extended capacity sizes of my drives How? Are they virtual drives of some sort? VMDKs? > - then used 'bluefs-bdev-expand' to tell it to _ceph_, but it seems pools/fs did not pick those up - should they not? Please share exactly what you did. Compare your `ceph osd df` output from before and after, did the SIZE and/or WEIGHT columns change? > -> $ ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS > 9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB 506 KiB 1.4 GiB 98 GiB 34.86 0.51 45 up > 10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 1.5 MiB 2.0 GiB 93 GiB 76.67 1.12 244 up > 0 ssd 0.09769 1.00000 150 GiB 143 GiB 91 GiB 835 KiB 2.0 GiB 7.5 GiB 95.02 1.39 78 up > 4 ssd 0.29300 1.00000 400 GiB 267 GiB 265 GiB 1.1 MiB 2.1 GiB 133 GiB 66.77 0.98 211 up > 1 ssd 0.04880 1.00000 150 GiB 54 GiB 53 GiB 392 KiB 1.1 GiB 96 GiB 36.00 0.53 47 up > 5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB 1.6 MiB 2.5 GiB 95 GiB 76.26 1.12 242 up > TOTAL 1.6 TiB 1.1 TiB 1.0 TiB 5.9 MiB 11 GiB 522 GiB 68.34 > MIN/MAX VAR: 0.51/1.39 STDDEV: 22.41 These are extremely small OSDs, and I’ve seen reports of such filling up before one would think they would, perhaps due to rounding or various math not taking metadata into account. Also, these are very imbalanced, which doesn’t help. Please share `ceph balancer status` and `ceph osd tree`. Ensure that you have this set, which may help: mgr advanced mgr/balancer/upmap_max_deviation 1 > -> $ rados df > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR > .mgr 1.3 MiB 2 0 6 0 0 0 13951 35 MiB 5271 54 MiB 0 B 0 B > cephfs.APKI.data 329 GiB 31352 0 94056 0 0 0 153508700 219 TiB 136896977 2.4 TiB 0 B 0 B > cephfs.APKI.meta 723 MiB 8513 0 25539 0 0 0 267014 11 GiB 17205610 69 GiB 0 B 0 B > cephfs.MONERO.data 736 GiB 62784 0 188352 0 0 0 6553814758 29 TiB 1776390595 8.7 TiB 0 B 0 B > cephfs.MONERO.meta 776 MiB 92 0 276 0 0 0 21866 22 GiB 4219726 17 GiB 0 B 0 B > > total_objects 102743 > total_used 1.1 TiB > total_avail 522 GiB > total_space 1.6 TiB > And what do you mean exactly by "it seems pools/fs did not pick those up “? I suspect you mean in terms of MAX AVAIL. “Ceph” picks up capacity if it shows in the RAW STORAGE numbers from `ceph df`. We don’t have the before output, so we can’t tell. Assuming this is what you mean, this is a commonly misundertood nuance. MAX AVAIL is a function of the delta between how full the MOST full OSD of the pool’s device class is vs the fullness ratios. # ceph osd dump | grep ratio full_ratio 0.95 backfillfull_ratio 0.90 nearfull_ratio 0.85 Your numbers may look different from these, which I think are the defaults. Notice your osd.0 is 95% full. I should think that `ceph status`, which you didn’t include in your message, has clues. So for your immediate question, I think if you fix your balancing issue you’ll see a bit of space show under MAX AVAIL. > -> $ ceph df > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > ssd 1.6 TiB 522 GiB 1.1 TiB 1.1 TiB 68.34 > TOTAL 1.6 TiB 522 GiB 1.1 TiB 1.1 TiB 68.34 > > --- POOLS --- > POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL > .mgr 1 1 449 KiB 2 1.3 MiB 100.00 0 B > cephfs.APKI.meta 2 16 241 MiB 8.51k 723 MiB 100.00 0 B > cephfs.APKI.data 3 128 110 GiB 31.35k 329 GiB 100.00 0 B > cephfs.MONERO.meta 4 16 259 MiB 92 776 MiB 100.00 0 B > cephfs.MONERO.data 5 128 245 GiB 62.78k 736 GiB 100.00 0 B > > I do not use quotas, no subvols. > I must be missing something obvious - right? - how to tell ceph to use that newly added capacity? > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

resize/extend pool/fs storage size?
by listy 01 Mar '26

01 Mar '26

Hi guys. I've extended capacity sizes of my drives - then used 'bluefs-bdev-expand' to tell it to _ceph_, but it seems pools/fs did not pick those up - should they not? -> $ ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB 506 KiB 1.4 GiB 98 GiB 34.86 0.51 45 up 10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 1.5 MiB 2.0 GiB 93 GiB 76.67 1.12 244 up 0 ssd 0.09769 1.00000 150 GiB 143 GiB 91 GiB 835 KiB 2.0 GiB 7.5 GiB 95.02 1.39 78 up 4 ssd 0.29300 1.00000 400 GiB 267 GiB 265 GiB 1.1 MiB 2.1 GiB 133 GiB 66.77 0.98 211 up 1 ssd 0.04880 1.00000 150 GiB 54 GiB 53 GiB 392 KiB 1.1 GiB 96 GiB 36.00 0.53 47 up 5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB 1.6 MiB 2.5 GiB 95 GiB 76.26 1.12 242 up TOTAL 1.6 TiB 1.1 TiB 1.0 TiB 5.9 MiB 11 GiB 522 GiB 68.34 MIN/MAX VAR: 0.51/1.39 STDDEV: 22.41 -> $ rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR .mgr 1.3 MiB 2 0 6 0 0 0 13951 35 MiB 5271 54 MiB 0 B 0 B cephfs.APKI.data 329 GiB 31352 0 94056 0 0 0 153508700 219 TiB 136896977 2.4 TiB 0 B 0 B cephfs.APKI.meta 723 MiB 8513 0 25539 0 0 0 267014 11 GiB 17205610 69 GiB 0 B 0 B cephfs.MONERO.data 736 GiB 62784 0 188352 0 0 0 6553814758 29 TiB 1776390595 8.7 TiB 0 B 0 B cephfs.MONERO.meta 776 MiB 92 0 276 0 0 0 21866 22 GiB 4219726 17 GiB 0 B 0 B total_objects 102743 total_used 1.1 TiB total_avail 522 GiB total_space 1.6 TiB -> $ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.6 TiB 522 GiB 1.1 TiB 1.1 TiB 68.34 TOTAL 1.6 TiB 522 GiB 1.1 TiB 1.1 TiB 68.34 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 1.3 MiB 100.00 0 B cephfs.APKI.meta 2 16 241 MiB 8.51k 723 MiB 100.00 0 B cephfs.APKI.data 3 128 110 GiB 31.35k 329 GiB 100.00 0 B cephfs.MONERO.meta 4 16 259 MiB 92 776 MiB 100.00 0 B cephfs.MONERO.data 5 128 245 GiB 62.78k 736 GiB 100.00 0 B I do not use quotas, no subvols. I must be missing something obvious - right? - how to tell ceph to use that newly added capacity? many thanks, L.

1 0

Re: Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch
by Igor Fedotov 28 Feb '26

28 Feb '26

Hi Theo, Sorry I can't tell for sure what would marking OSD lost do with its encryption keys. Likely - yes, they'll be lost. But instead of going this way I'd rather suggest you to add another two OSDs and let Ceph recover more PGs replicas into them. Just in case - didn't you overwrite existing PG replicas at target OSDs when exporting PGs back to OSDs 1&3? The same PG can't have two replicas/shards at a single OSD while your OSD count is pretty limited.. Just curious for now - that still shouldn't be an issue given you have at least 2 replicas/shards for all the pools anyway. Thanks, Igor On 2/21/2026 11:00 PM, Theo Cabrerizo Diem via ceph-users wrote: > Hello Igor, > > First of all, sorry about the late reply. It took me a while to export all > shards that weren't available from the osd.2 (1 and 3 were fine, 2 didn't > start but i could use `ceph-objectstore-tool ... --op list-pgs` while osd.0 > I couldn't even list the pgs, it threw an error right away - more about it > later in the email) > > Two of the unavailable shards, when exporting, ceph-objectstore-tool core > dumped with the same issue in the rocksdb, but I should have enough chunks > to not need them - just mentioning in case is useful: > > sh-5.1# ceph-objectstore-tool --data-path /var/lib/ceph/osd --pgid 11.19s2 > --op export --file pg.11.19s2.dat > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: In function > 'virtual int RocksDBStore::get(const std::string&, const std::string&, > ceph::bufferlist*)' thread 7ff3be4ca800 time 2026-02-04T09:42:00.743877+0000 > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: 1961: > ceph_abort_msg("block checksum mismatch: stored = 246217859, computed = > 2155741315, type = 4 in db/170027.sst offset 28264757 size 1417") > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle > (stable - RelWithDebInfo) > 1: (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0xc9) [0x7ff3bf5391fd] > 2: (RocksDBStore::get(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc) > [0x555667b340bc] > 3: > (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, > ghobject_t const&, > std::set<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > > > const&, > std::map<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, ceph::buffer::v15_2_0::list, > std::less<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, > std::allocator<std::pair<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const, > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1] > 4: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361) > [0x5556675e0101] > 5: main() > 6: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610] > 7: __libc_start_main() > 8: _start() > *** Caught signal (Aborted) ** > in thread 7ff3be4ca800 thread_name:ceph-objectstor > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle > (stable - RelWithDebInfo) > 1: /lib64/libc.so.6(+0x3fc30) [0x7ff3be945c30] > 2: /lib64/libc.so.6(+0x8d03c) [0x7ff3be99303c] > 3: raise() > 4: abort() > 5: (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0x186) [0x7ff3bf5392ba] > 6: (RocksDBStore::get(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc) > [0x555667b340bc] > 7: > (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, > ghobject_t const&, > std::set<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > > > const&, > std::map<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, ceph::buffer::v15_2_0::list, > std::less<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, > std::allocator<std::pair<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const, > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1] > 8: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361) > [0x5556675e0101] > 9: main() > 10: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610] > 11: __libc_start_main() > 12: _start() > Aborted (core dumped) > > > > After importing all shards that I could recover that weren't available, I > don't have any "unknown" pgs anymore. I still have lots of PGs in "down" > state, which I assume I need to flag both "dead" OSDs as lost to unstuck > them. Since it is an operation I cannot go back, I would like to confirm > that is indeed the correct next step to take. > > I have a few questions to understand "what happens" in the next step > (marking osd as lost?): > > Shall I assume that once I flag an OSD as lost, I won't be able to > "activate" it since I use encryption when initializing the bluestore OSD, > or flagging them as lost won't destroy their unlocking keys? (which means > any hope of further extracting data to be gone, mostly on the osd.0 which I > couldn't use ceph-objectstore-tool at all since the power loss). > > I think I should have all the shards from the PGs but just in case, I've > managed to make a clone of the osd.0 on a different physical disk (the > other reason I took long to answer). But still ceph-objectstore-tool > refuses to run: > > # ceph-objectstore-tool --data-path /var/lib/ceph/osd --op list-pgs > Mount failed with '(5) Input/output error' > > # ls -l /var/lib/ceph/osd > total 28 > lrwxrwxrwx 1 ceph ceph 50 Feb 4 08:26 block -> > /dev/mapper/zNPZJR-i0TZ-6NtK-URto-tjfs-iJRb-GCAYEm > -rw------- 1 ceph ceph 37 Feb 4 08:26 ceph_fsid > -rw------- 1 ceph ceph 37 Feb 4 08:26 fsid > -rw------- 1 ceph ceph 55 Feb 4 08:26 keyring > -rw------- 1 ceph ceph 106 Jan 24 00:44 lockbox.keyring > -rw------- 1 ceph ceph 6 Feb 4 08:26 ready > -rw------- 1 ceph ceph 10 Feb 4 08:26 type > -rw------- 1 ceph ceph 2 Feb 4 08:26 whoami > > Just as information, all except 2 pools in my cluster are "replicated". > Pools id 11 and 16 are erasure coded (2+1). If I understood correctly, as > long as I have two acting shards (and at most one "NONE"), data should be > available (at least in read-only) once I mark the down OSDs as lost. Is > that understanding correct? > > Another information, pools 10 and 15 are the "replicated root pools" before > the erasure coded pools were created. > > Ignoring osd.0 for now, here are the current state of my cluster (mds is > intentionally not started while I try to fix the PGs): > ### ceph osd lspools > 3 .rgw.root > 4 default.rgw.log > 5 default.rgw.control > 6 default.rgw.meta > 10 ark.data > 11 ark.data_ec > 12 ark.metadata > 14 .mgr > 15 limbo > 16 limbo.data_ec > 18 default.rgw.buckets.index > 19 default.rgw.buckets.data > ### > > ### ceph health > # ceph -s > cluster: > id: 021f058f-dbf3-4a23-adb5-21d83f3f1bb6 > health: HEALTH_ERR > 1 filesystem is degraded > 1 filesystem has a failed mds daemon > 1 filesystem is offline > insufficient standby MDS daemons available > Reduced data availability: 143 pgs inactive, 143 pgs down > Degraded data redundancy: 1303896/7149898 objects degraded > (18.237%), 218 pgs degraded, 316 pgs undersized > 144 pgs not deep-scrubbed in time > 459 pgs not scrubbed in time > 256 slow ops, oldest one blocked for 1507794 sec, osd.1 has > slow ops > too many PGs per OSD (657 > max 500) > > services: > mon: 2 daemons, quorum ceph-ymir-mon2,ceph-ymir-mon1 (age 2w) > mgr: ceph-ymir-mgr1(active, since 2w) > mds: 0/1 daemons up (1 failed) > osd: 4 osds: 2 up (since 29m), 2 in (since 4w); 24 remapped pgs > > data: > volumes: 0/1 healthy, 1 failed > pools: 12 pools, 529 pgs > objects: 2.46M objects, 7.4 TiB > usage: 8.3 TiB used, 13 TiB / 22 TiB avail > pgs: 27.032% pgs not active > 1303896/7149898 objects degraded (18.237%) > 306628/7149898 objects misplaced (4.289%) > 218 active+undersized+degraded > 143 down > 98 active+undersized > 45 active+clean > 19 active+clean+remapped > 4 active+clean+remapped+scrubbing+deep > 1 active+clean+remapped+scrubbing > 1 active+clean+scrubbing+deep > ### ceph health > > ### ceph health detail > # ceph health detail > HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds daemon; > 1 filesystem is offline; insufficient standby > MDS daemons available; Reduced data availability: 143 pgs inactive, 143 > pgs down; Degraded data redundancy: 1303896/714 > 9898 objects degraded (18.237%), 218 pgs degraded, 316 pgs undersized; 144 > pgs not deep-scrubbed in time; 459 pgs not sc > rubbed in time; 256 slow ops, oldest one blocked for 1508207 sec, osd.1 has > slow ops; too many PGs per OSD (657 > max 50 > 0) > [WRN] FS_DEGRADED: 1 filesystem is degraded > fs ark is degraded > [WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon > fs ark has 1 failed mds > [ERR] MDS_ALL_DOWN: 1 filesystem is offline > fs ark is offline because no MDS is active for it. > [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available > have 0; want 1 more > [WRN] PG_AVAILABILITY: Reduced data availability: 143 pgs inactive, 143 pgs > down > pg 10.11 is down, acting [1,3] > pg 10.18 is down, acting [3,1] > pg 10.1d is down, acting [1,3] > pg 10.1f is down, acting [1,3] > pg 11.10 is down, acting [3,1,NONE] > pg 11.12 is down, acting [1,NONE,3] > pg 11.18 is stuck inactive for 4w, current state down, last acting > [1,3,NONE] > pg 11.19 is down, acting [3,1,NONE] > pg 11.1b is down, acting [1,NONE,3] > pg 11.62 is down, acting [NONE,3,1] > pg 11.63 is down, acting [3,NONE,1] > pg 11.64 is down, acting [NONE,1,3] > pg 11.66 is down, acting [NONE,3,1] > pg 11.67 is down, acting [1,NONE,3] > pg 11.68 is down, acting [3,NONE,1] > pg 11.69 is down, acting [NONE,1,3] > pg 11.6a is down, acting [1,NONE,3] > pg 11.6b is down, acting [NONE,1,3] > pg 11.6f is down, acting [NONE,3,1] > pg 11.71 is down, acting [1,3,NONE] > pg 11.72 is down, acting [1,3,NONE] > pg 11.74 is down, acting [NONE,3,1] > pg 11.76 is down, acting [1,NONE,3] > pg 11.78 is down, acting [3,1,NONE] > pg 11.7d is down, acting [NONE,3,1] > pg 11.7e is down, acting [NONE,1,3] > pg 15.15 is down, acting [1,3] > pg 15.16 is down, acting [3,1] > pg 15.17 is down, acting [1,3] > pg 15.1a is down, acting [3,1] > pg 16.1 is down, acting [1,3,NONE] > pg 16.4 is down, acting [1,3,NONE] > pg 16.b is down, acting [3,NONE,1] > pg 16.60 is down, acting [3,1,NONE] > pg 16.61 is down, acting [3,1,NONE] > pg 16.62 is down, acting [3,NONE,1] > pg 16.63 is down, acting [3,NONE,1] > pg 16.65 is down, acting [NONE,3,1] > pg 16.67 is down, acting [1,NONE,3] > pg 16.68 is down, acting [1,NONE,3] > pg 16.69 is down, acting [3,1,NONE] > pg 16.6a is down, acting [1,3,NONE] > pg 16.6c is down, acting [1,3,NONE] > pg 16.70 is down, acting [3,NONE,1] > pg 16.73 is down, acting [3,NONE,1] > pg 16.74 is down, acting [1,3,NONE] > pg 16.75 is down, acting [3,1,NONE] > pg 16.79 is down, acting [3,NONE,1] > pg 16.7a is down, acting [1,3,NONE] > pg 16.7e is down, acting [1,3,NONE] > pg 16.7f is down, acting [3,NONE,1] > [WRN] PG_DEGRADED: Degraded data redundancy: 1303896/7149898 objects > degraded (18.237%), 218 pgs degraded, 316 pgs under > sized > pg 3.18 is stuck undersized for 36m, current state active+undersized, > last acting [1,3] > ...<snipped for brevity> > ### > > Once again, I cannot thank you enough for looking into my issue. > I have the impression that being able to recover the data I need is just > around the corner. Although the croit.io blog did mention flagging the osd > as lost, I would like to double check it to avoid losing any possibility to > recover the data. > > If there's anything further I could check or if you need full output of the > commands, let me know. > > Thanks in advance. > > On Tue, 3 Feb 2026 at 10:26, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > >> Hi Theo, >> >> you might want to try to use PG export/import using ceph-objectstore-tool. >> >> Please find more details here >> https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor… >> >> >> Thanks, >> >> Igor >> On 03/02/2026 02:38, Theo Cabrerizo Diem via ceph-users wrote: >> >> :12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd) >> fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks >> fsck status: remaining 1 error(s) and warning(s) >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: Prometheus cannot connect to Ceph-Exporter
by Eugen Block 28 Feb '26

28 Feb '26

Hi, to me it seems like it's not available yet. Looking at [0], there is a section to update the exporter: security_enabled, _, _ = self.mgr._get_security_config() if security_enabled: exporter_config.update({'https_enabled': True}) tls_creds = self.get_certificates(daemon_spec) exporter_config['files'] = { 'ceph-exporter.crt': tls_creds.cert, 'ceph-exporter.key': tls_creds.key } But I don't see this section neither in current Tentacle [1] nor in current Squid [2]. Tentacle does already support setting a cert and key file, but apparently not enabling https: tentacle:~ # ceph config ls | grep exporter exporter_sock_dir exporter_addr exporter_http_port exporter_cert_file exporter_key_file exporter_prio_limit exporter_stats_period exporter_sort_metrics But I'm not a dev, so take that with a grain of salt. Regards, Eugen [0] https://github.com/ceph/ceph/blob/1ef2745047383fbb712b9494e125b2a312fa2f39/… [1] https://github.com/ceph/ceph/blob/v20.2.0/src/pybind/mgr/cephadm/services/c… [2] https://github.com/ceph/ceph/blob/v19.2.3/src/pybind/mgr/cephadm/services/c… Zitat von "Maintz, Thomas via ceph-users" <ceph-users(a)ceph.io>: > Hello everyone, > > We deployed the monitoring stack with Grafana and Prometheus via > cephadm and set “mgr/cephadm/secure_monitoring_stack true.” > To get the OSD latencies into the dashboard, we deployed the > ceph-exporter on the hosts. > However, due to the secure_monitoring_stack, Prometheus tries to > access the Ceph Exporter via https, as the schema has been set to > https by cephadm. > The Ceph exporter can be accessed locally via http with curl, but > not via https. > If I manually configure the schema for the Ceph exporter to http in > prometheus.yml and restart the service, I get the data correctly. > However, a redeploy overwrites my change. > Can someone tell me how to either configure the Ceph-exporter to > respond to https or tell Prometheus to only access the Ceph exporter > via http? Unfortunately, there is no documentation on this. > > Best regards and many thanks in advance. > Thomas Maintz > > > > > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellvertretende Vorsitzende), > Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > --------------------------------------------------------------------------------------------- > ---------------------------------------------------------------------------------------------

1 0

should ceph mount propagate inotify?
by lejeczek 27 Feb '26

27 Feb '26

HI guys. Should ceph propagate inotify/logs "backwards" across all client-mounts? Say... clientA locks -> mount -> ceph fs <- mount <- clientB gets/sees the file has a lock/inotify does that make sense and if so & ceph should do that - what have I missed, possibly, for I do not get this behaviour. many thanks, L.

1 0

Re: 8443: connect: connection refused
by lejeczek 27 Feb '26

27 Feb '26

I did not think that ceph should be aware of any "external" proxy being used - it runs on one of ceph nodes - I assumed such a proxy would only be relevant to users/clients looking at ceph via web-browser. As far a ceph goes I did: -> $ ceph config set mgr mgr/dashboard/standby_behaviour "redirect" # set it back to default -> $ ceph config set mgr mgr/dashboard/standby_behaviour "error" -> $ ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr true # resolve to hostnames -> $ ceph config set mgr mgr/dashboard/ssl false -> $ ceph mgr services { "dashboard": "http://10.1.1.63:8080/", "prometheus": "http://10.1.1.63:9283/" } Rest is Nginx's, what I'd call a "standard" proxy configuration. If needed - I should spam it here? thanks, L.

1 0

Prometheus cannot connect to Ceph-Exporter
by Maintz, Thomas 27 Feb '26

27 Feb '26

Hello everyone, We deployed the monitoring stack with Grafana and Prometheus via cephadm and set “mgr/cephadm/secure_monitoring_stack true.” To get the OSD latencies into the dashboard, we deployed the ceph-exporter on the hosts. However, due to the secure_monitoring_stack, Prometheus tries to access the Ceph Exporter via https, as the schema has been set to https by cephadm. The Ceph exporter can be accessed locally via http with curl, but not via https. If I manually configure the schema for the Ceph exporter to http in prometheus.yml and restart the service, I get the data correctly. However, a redeploy overwrites my change. Can someone tell me how to either configure the Ceph-exporter to respond to https or tell Prometheus to only access the Ceph exporter via http? Unfortunately, there is no documentation on this. Best regards and many thanks in advance. Thomas Maintz --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

1 0

RGW Admin Ops: question about /admin/account permissions
by Jeffrey van Doesburg 25 Feb '26

25 Feb '26

Hello, I am experiencing some issues with the RADOSGW Admin Ops API in Ceph Tentacle (20.2.0). According to the latest documentation[1], it should be possible (since Squid) to perform calls to /admin/account. However, in my setup this endpoint consistently returns 403 AccessDenied. What I find confusing is that these /admin/account options are only present in the latest documentation. They do not appear in the Squid or Tentacle version specific documentation. My user is created with the following caps: radosgw-admin caps add \ --uid="<user>" \ -- caps="accounts=*;buckets=*;info=read;metadata=read;usage=read;users=*" For example, a call to the metadata endpoint works without any issues: curl -sS \ --user "${ACCESS_KEY}:${SECRET_KEY}" \ --aws-sigv4 "aws:amz:us-east-1:s3" \ "${RGW_URL}/admin/metadata/account?format=json" Response: ["RGW67272693614667731","RGW60101944561690517","RGW31205003107553515"] However, a call to /admin/account returns AccessDenied: curl -sS \ --user "${ACCESS_KEY}:${SECRET_KEY}" \ --aws-sigv4 "aws:amz:us-east-1:s3" \ "${RGW_URL}/admin/account?id=RGW67272693614667731&format=json" Response: {"Code":"AccessDenied","Message":"","RequestId":"tx00000f330b10a36af581 f-00699ef8b2-24707-default","HostId":"24707-default-default"} From my testing, the /admin/account endpoint works when the user is marked as a system user. But returns AccessDenied when using a "regular" user that has accounts=* caps. Should accounts=* be sufficient to access /admin/account, as the documentation suggests? Or is a system user explicitly required for these endpoints? Additionally, can someone clarify whether /admin/account is fully supported in Tentacle? Given that it only appears in the latest documentation and not in the version specific Tentacle docs? Thanks in advance! Kind regards, Jeffrey [1] https://docs.ceph.com/en/latest/radosgw/adminops/#get-account-info

1 0

Re: metrics duplicate in prometheus after migrate and upgrade to cephadm
by Francesco Usseglio Gaudi 25 Feb '26

25 Feb '26

Hi Roberto, thank you very much, this is the right parameter to toggle! I have change it from "false" to "true" and now we have graphs working again. Have a good day Francesco Il 25/02/26 14:20, Roberto Valverde Cameselle ha scritto: > Hi Francesco, > Can you check if you have this feature set? > mgr advanced mgr/prometheus/exclude_perf_counters > Make sure that is set to true so then performance metrics are gathered > from the ceph-exporters and not from the manager. > If your cluster is small, you can also not add ceph-exporters and set > the above setting to false so the manager will be also exposing the > performance metrics. > Cheers > > > *-----------------------------------------------------------------* > *Roberto VALVERDE CAMESELLE* > IT Storage And Data Management > CERN, European Organization for Nuclear Research > Esplanade des Particules 1, Geneve (Switzerland) > ------------------------------------------------------------------------ > *From:* Francesco Usseglio Gaudi via ceph-users <ceph-users(a)ceph.io> > *Sent:* Tuesday, February 24, 2026 6:17 PM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] metrics duplicate in prometheus after migrate > and upgrade to cephadm > Hi all, > > we have a 10 nodes cluster used for S3 with radosgw. > > Initially the cluster has been installed with ceph-ansible and was at > version 18. > > Some months ago an external consultant converted to cephadm and upgrade > to 19.2.3 > > Almost everything is ok but we have some issues with dashboard and > grafana graphs. > > In Dashboard → Cluster → OSDs → Overall Performance we can't see "OSD > Read Latencies" and "OSD Write Latencies" graphs, they have red triangle > icons with this error popup: > > "execution: found duplicate series for the match group > {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: > [{ceph_daemon=\"osd.0\", > cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", > instance=\"ceph_cluster\", job=\"ceph\"}, {ceph_daemon=\"osd.0\", > cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"node1\", > job=\"ceph-exporter\"}];many-to-many matching not allowed: matching > labels must be unique on one side" > > After some my investigations I found out there are some duplicate > metrics in prometheus db, one coming from ceph (I think mgr module?) and > one from ceph-exporter, for example this metric: > > ceph_osd_op_r_latency_sum > > be there with two distinct values with different job labels (values are > exactly the same) > > Every node has a running ceph-exporter container and there are 3 nodes > with mgr services. > > I have tried to modify grafana dashboard queries to use only one of two > metrics specifying the job label but they are readonly (I can make a > copy and modifying it that works, but i don't know how to use this new > grafana dashboard in ceph dashboard). > > I have read from ceph documentation and I understand ceph-exporter is no > longer required but on another ceph cluster recently set up with cephadm > version 19.2.3 from the very start, ceph-exporter container are presents > on every nodes, so I'm not sure if I should keep ceph-exporter > containers running > > Which is the right way of extract these metrics from ceph? > > Thank you all > > Francesco Usseglio > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0