ceph-users
Threads by month
- ----- 2026 -----
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- 40 participants
- 9156 discussions
Re: Recover after multiple OSD failures (was Re: Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch)
by Eugen Block 03 Mar '26
by Eugen Block 03 Mar '26
03 Mar '26
Hi,
I didn't read all the details of this thread, but if you want to
prevent freshly created OSDs from receiving recovery traffic, you
might want to set:
ceph config set osd osd_crush_initial_weight 0
This allows you to create and start OSDs without a crush weight, so
there won't be any traffic to this OSD at all until you reweight its
crush weight. But the OSD will be up and in, which should allow you to
import PGs anyway.
Zitat von Theo Cabrerizo Diem via ceph-users <ceph-users(a)ceph.io>:
> Hello Igor, Hello all,
>
> First, I've already accepted the fact that my data most likely is
> unrecoverable by now due to my own fault.. I'm using it to learn and
> hopefully document the information gained as I couldn't find much recent
> information regarding the process of recovery and improvement of my
> understanding about ceph, if someone is willing to further chime in.
>
> The current situation is that I've had multiple OSD failures and some of
> the ceph-osd processes would refuse to start (corruption on rocksdb). I've
> decided to follow
> https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor…
> as a suggested mechanism to export the PGs and re-import on a fresh OSD.
>
> I had an attempt (sort of documented in this thread) to do it with
> "unexpected" results. I have written down a lot of information, states,
> ceph pg queries, etc (so I can provide their outputs if relevant. I have
> not marked any of the OSDs as lost at any time and the monitors have been
> running without issue since the beginning.
>
> I have some questions regarding my observations following the information
> on that blog (sorry for my lack of experience):
>
> - Is the process for setting up a new "temporary" OSD to import PGs
> correct? (short story: "ceph-volume lvm prepare", start the osd and as soon
> as possible run "ceph osd crush reweight osd.XX 0")
>
> - Creating the OSD as described above, running "ceph-objectstore-tool --op
> list-pgs" on this OSD showed lots of PGs (which I assume they were
> "pre-allocation" from crush) but "ceph osd df" confirmed no data was on the
> osd (only very very little, like less than 2Gb). Is there a way to have an
> OSD "flushed out" so I can import further PGs?
>
> - Running "ceph pg XX.XX query" into some of the imported PGs after
> starting ceph-osd again doesn't seem to reliably reflect my progress. Is
> there a different way? Is it because the PG is still in down state because
> of the dead OSDs?
> For example, pg 11.17 of which had only shard-0 available because only one
> OSD was up. I've imported shard-1 to osd.10 , but "ceph pg 11.17 query"
> shows under "recovery_state":
>
> "intervals": [
> {
> "first": "2882",
> "last": "2883",
> "acting": "1(1),3(0)"
> },
> {
> "first": "3021",
> "last": "3023",
> "acting": "3(0),8(1)"
> },
> {
> "first": "3024",
> "last": "3026",
> "acting": "3(0),8(1),10(2)"
> }
> ]
>
> Running ceph-objectstore-tool --op list-pgs on osd.10 (stopped) confirms
> that 11.17s1 is listed, and running ceph-objectstore-tool --op list-pgs on
> osd.8 (stopped) doesn't show 11.17 at all (none of its shards)
>
> Should I instead keep track of my progress using "ceph-objectstore-tool
> --op list" looking for a "oid" present?
>
> This might reflect on my lack of knowledge regarding how ceph osd "works
> internally", so feel free to correct me or suggest a better approach. I
> still have the 3x original OSDs (out of 4, one, as mentioned on the thread,
> have a bigger corruption and ceph-objectstore-tool fails) and 8x 2Tb disks
> that I can load as new OSDs to import the old data (I had less than 6Tb
> used before the crash).
>
> Should I continue exporting all PGs and keep importing them this way?
>
> Thanks
>
>
> On Sat, 28 Feb 2026 at 16:50, Theo Cabrerizo Diem via ceph-users <
> ceph-users(a)ceph.io> wrote:
>
>> Hello all,
>>
>> I've managed to get a bunch of 2Tb disks for setting up a few OSDs but
>> before I even started adding them to my monitors, I decided to check my
>> cluster state and noticed another OSD died. Trying to start it, revealed a
>> rocksdb corruption:
>>
>> # /usr/bin/ceph-osd -f --id "1" --osd-data "/var/lib/ceph/osd" --cluster
>> "ceph" --setuser "ceph" --setgroup "ceph"
>> 2026-02-28T15:34:16.596+0000 7f793ce718c0 -1 Falling back to public
>> interface
>>
>> 2026-02-28T15:35:04.304+0000 7f792c1c5640 -1 rocksdb: submit_common error:
>> Corruption: block checksum mismatch: stored = 0, computed = 1265684702,
>> type = 4 in db/180433.sst offset 1048892 size 1429 code = ☻ Rocksdb
>> transaction:
>> PutCF( prefix = O key =
>> 0x7F8000000000000006D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value
>> size = 33)
>> PutCF( prefix = S key = 'nid_max' value size = 8)
>> PutCF( prefix = S key = 'blobid_max' value size = 8)
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
>> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
>> time 2026-02-28T15:35:04.305926+0000
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
>> FAILED ceph_assert(r == 0)
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x11f) [0x557b7cbf6236]
>> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>> 2026-02-28T15:35:04.309+0000 7f792c1c5640 -1
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
>> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
>> time 2026-02-28T15:35:04.305926+0000
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
>> FAILED ceph_assert(r == 0)
>>
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x11f) [0x557b7cbf6236]
>> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>>
>> *** Caught signal (Aborted) **
>> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
>> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
>> 3: raise()
>> 4: abort()
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x17a) [0x557b7cbf6291]
>> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>> 2026-02-28T15:35:04.320+0000 7f792c1c5640 -1 *** Caught signal (Aborted) **
>> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>>
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
>> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
>> 3: raise()
>> 4: abort()
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x17a) [0x557b7cbf6291]
>> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> -2421> 2026-02-28T15:34:16.596+0000 7f793ce718c0 -1 Falling back to public
>> interface
>> -8> 2026-02-28T15:35:04.304+0000 7f792c1c5640 -1 rocksdb: submit_common
>> error: Corruption: block checksum mismatch: stored = 0, computed =
>> 1265684702, type = 4 in db/180433.sst offset 1048892 size 1429 code = ☻
>> Rocksdb transaction:
>> PutCF( prefix = O key =
>> 0x7F8000000000000006D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value
>> size = 33)
>> PutCF( prefix = S key = 'nid_max' value size = 8)
>> PutCF( prefix = S key = 'blobid_max' value size = 8)
>> -7> 2026-02-28T15:35:04.309+0000 7f792c1c5640 -1
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
>> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
>> time 2026-02-28T15:35:04.305926+0000
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
>> FAILED ceph_assert(r == 0)
>>
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x11f) [0x557b7cbf6236]
>> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>>
>> 0> 2026-02-28T15:35:04.320+0000 7f792c1c5640 -1 *** Caught signal
>> (Aborted) **
>> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>>
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
>> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
>> 3: raise()
>> 4: abort()
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x17a) [0x557b7cbf6291]
>> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> -2445> 2026-02-28T15:34:16.596+0000 7f793ce718c0 -1 Falling back to public
>> interface
>> -32> 2026-02-28T15:35:04.304+0000 7f792c1c5640 -1 rocksdb: submit_common
>> error: Corruption: block checksum mismatch: stored = 0, computed =
>> 1265684702, type = 4 in db/180433.sst offset 1048892 size 1429 code = ☻
>> Rocksdb transaction:
>> PutCF( prefix = O key =
>> 0x7F8000000000000006D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value
>> size = 33)
>> PutCF( prefix = S key = 'nid_max' value size = 8)
>> PutCF( prefix = S key = 'blobid_max' value size = 8)
>> -31> 2026-02-28T15:35:04.309+0000 7f792c1c5640 -1
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
>> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
>> time 2026-02-28T15:35:04.305926+0000
>> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
>> FAILED ceph_assert(r == 0)
>>
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x11f) [0x557b7cbf6236]
>> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>>
>> -24> 2026-02-28T15:35:04.320+0000 7f792c1c5640 -1 *** Caught signal
>> (Aborted) **
>> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>>
>> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
>> (stable - RelWithDebInfo)
>> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
>> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
>> 3: raise()
>> 4: abort()
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x17a) [0x557b7cbf6291]
>> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
>> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
>> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
>> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
>> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> Aborted (core dumped)
>>
>> There was no sign of hardware failure on kernel logs. I think, for now, I
>> should move away from alpine binaries and use ceph's official containers.
>> Is there any tool that can try to fix this rocksdb issue? Or is the
>> recommended way to export all PGs from this osd and re-import on a new one?
>>
>> I agree that at this point I should consider rechecking all the hardware
>> involved. I plan to decommission this system once I (if possible) I get the
>> data out.
>>
>> Thanks,
>> Theo
>>
>> On Tue, 24 Feb 2026 at 22:22, Theo Cabrerizo Diem <diem(a)carpediem.sh>
>> wrote:
>>
>> > Hello Igor,
>> >
>> > > Just in case - didn't you overwrite existing PG replicas at target
>> OSDs
>> > > when exporting PGs back to OSDs 1&3?
>> >
>> > Now that you mention that an OSD cannot have two shards, I think I did
>> put
>> > myself now in a tricky place.
>> > I had 4 OSDs, two "died" .. leaving two online, so when I did export
>> > shards from osd.2 I did end up importing them in osd.3 (which had other
>> > shards for the same PG).
>> >
>> > Doesn't look "that bad" ... (*looking up to the sky*) ...
>> > ## on osd.3
>> > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ --op list-pgs |
>> > grep 16.b
>> > 16.bs1
>> > 16.bs0
>> > (and the s2 should be in osd.1, s1 was imported)
>> >
>> > ## ceph health reports
>> > pg 16.b is down, acting [3,NONE,1]
>> >
>> > I might have some spares HDDs to set up a new OSD (one though) with
>> enough
>> > space for it to balance out, but as I've mentioned, there are quite a
>> bunch
>> > of PGs in down state. Should I set-up a new OSD and wait until it
>> "settles
>> > up"? The exported shards from the dead osd.2 that ceph health was
>> > complaining about sums to about 400Gb.
>> >
>> > Is it possible to set-up an OSD and have ceph not push too many shards
>> > into it? (the replica pools are "small"). I ask this because it should be
>> > easy to get a 1Tb/2Tb disk, but if it should balance completely for the
>> PGs
>> > to "unstuck", I need a bigger disk, which I don't have at the moment (and
>> > currently not the best market prices to get them).
>> >
>> > Another option is if osd.0 is gone for good (the one that doesn't even
>> > allows me to run 'ceph-objectstore-tool --list-pgs'), I can wipe its
>> > disk/clone and use it instead, it is the same capacity as other OSDs (I
>> > used a spare I had at home). If you believe it is still workable, I can
>> try
>> > to find some more disks and not touch those for the moment.
>> >
>> > Thanks for the patience so far.
>> >
>> > On Tue, 24 Feb 2026 at 16:39, Igor Fedotov <igor.fedotov(a)croit.io>
>> wrote:
>> >
>> >> Hi Theo,
>> >>
>> >> Sorry I can't tell for sure what would marking OSD lost do with its
>> >> encryption keys. Likely - yes, they'll be lost.
>> >>
>> >> But instead of going this way I'd rather suggest you to add another two
>> >> OSDs and let Ceph recover more PGs replicas into them.
>> >>
>> >> Just in case - didn't you overwrite existing PG replicas at target OSDs
>> >> when exporting PGs back to OSDs 1&3? The same PG can't have two
>> >> replicas/shards at a single OSD while your OSD count is pretty limited..
>> >> Just curious for now - that still shouldn't be an issue given you have
>> >> at least 2 replicas/shards for all the pools anyway.
>> >>
>> >> Thanks,
>> >>
>> >> Igor
>> >>
>> >> On 2/21/2026 11:00 PM, Theo Cabrerizo Diem via ceph-users wrote:
>> >> > Hello Igor,
>> >> >
>> >> > First of all, sorry about the late reply. It took me a while to export
>> >> all
>> >> > shards that weren't available from the osd.2 (1 and 3 were fine, 2
>> >> didn't
>> >> > start but i could use `ceph-objectstore-tool ... --op list-pgs` while
>> >> osd.0
>> >> > I couldn't even list the pgs, it threw an error right away - more
>> about
>> >> it
>> >> > later in the email)
>> >> >
>> >> > Two of the unavailable shards, when exporting, ceph-objectstore-tool
>> >> core
>> >> > dumped with the same issue in the rocksdb, but I should have enough
>> >> chunks
>> >> > to not need them - just mentioning in case is useful:
>> >> >
>> >> > sh-5.1# ceph-objectstore-tool --data-path /var/lib/ceph/osd --pgid
>> >> 11.19s2
>> >> > --op export --file pg.11.19s2.dat
>> >> > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: In function
>> >> > 'virtual int RocksDBStore::get(const std::string&, const std::string&,
>> >> > ceph::bufferlist*)' thread 7ff3be4ca800 time
>> >> 2026-02-04T09:42:00.743877+0000
>> >> > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: 1961:
>> >> > ceph_abort_msg("block checksum mismatch: stored = 246217859, computed
>> =
>> >> > 2155741315, type = 4 in db/170027.sst offset 28264757 size 1417")
>> >> > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489)
>> >> tentacle
>> >> > (stable - RelWithDebInfo)
>> >> > 1: (ceph::__ceph_abort(char const*, int, char const*,
>> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
>> >> > std::allocator<char> > const&)+0xc9) [0x7ff3bf5391fd]
>> >> > 2: (RocksDBStore::get(std::__cxx11::basic_string<char,
>> >> > std::char_traits<char>, std::allocator<char> > const&,
>> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
>> >> > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc)
>> >> > [0x555667b340bc]
>> >> > 3:
>> >> >
>> >>
>> (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
>> >> > ghobject_t const&,
>> >> > std::set<std::__cxx11::basic_string<char,std::char_traits<char>,
>> >> > std::allocator<char> >, std::less<std::__cxx11::basic_string<char,
>> >> > std::char_traits<char>, std::allocator<char> > >,
>> >> > std::allocator<std::__cxx11::basic_string<char,
>> std::char_traits<char>,
>> >> > std::allocator<char> > > > const&,
>> >> > std::map<std::__cxx11::basic_string<char,std::char_traits<char>,
>> >> > std::allocator<char> >, ceph::buffer::v15_2_0::list,
>> >> > std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
>> >> > std::allocator<char> > >,
>> >> > std::allocator<std::pair<std::__cxx11::basic_string<char,
>> >> > std::char_traits<char>, std::allocator<char> > const,
>> >> > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1]
>> >> > 4: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361)
>> >> > [0x5556675e0101]
>> >> > 5: main()
>> >> > 6: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610]
>> >> > 7: __libc_start_main()
>> >> > 8: _start()
>> >> > *** Caught signal (Aborted) **
>> >> > in thread 7ff3be4ca800 thread_name:ceph-objectstor
>> >> > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489)
>> >> tentacle
>> >> > (stable - RelWithDebInfo)
>> >> > 1: /lib64/libc.so.6(+0x3fc30) [0x7ff3be945c30]
>> >> > 2: /lib64/libc.so.6(+0x8d03c) [0x7ff3be99303c]
>> >> > 3: raise()
>> >> > 4: abort()
>> >> > 5: (ceph::__ceph_abort(char const*, int, char const*,
>> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
>> >> > std::allocator<char> > const&)+0x186) [0x7ff3bf5392ba]
>> >> > 6: (RocksDBStore::get(std::__cxx11::basic_string<char,
>> >> > std::char_traits<char>, std::allocator<char> > const&,
>> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
>> >> > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc)
>> >> > [0x555667b340bc]
>> >> > 7:
>> >> >
>> >>
>> (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
>> >> > ghobject_t const&,
>> >> > std::set<std::__cxx11::basic_string<char,std::char_traits<char>,
>> >> > std::allocator<char> >, std::less<std::__cxx11::basic_string<char,
>> >> > std::char_traits<char>, std::allocator<char> > >,
>> >> > std::allocator<std::__cxx11::basic_string<char,
>> std::char_traits<char>,
>> >> > std::allocator<char> > > > const&,
>> >> > std::map<std::__cxx11::basic_string<char,std::char_traits<char>,
>> >> > std::allocator<char> >, ceph::buffer::v15_2_0::list,
>> >> > std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
>> >> > std::allocator<char> > >,
>> >> > std::allocator<std::pair<std::__cxx11::basic_string<char,
>> >> > std::char_traits<char>, std::allocator<char> > const,
>> >> > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1]
>> >> > 8: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361)
>> >> > [0x5556675e0101]
>> >> > 9: main()
>> >> > 10: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610]
>> >> > 11: __libc_start_main()
>> >> > 12: _start()
>> >> > Aborted (core dumped)
>> >> >
>> >> >
>> >> >
>> >> > After importing all shards that I could recover that weren't
>> available,
>> >> I
>> >> > don't have any "unknown" pgs anymore. I still have lots of PGs in
>> "down"
>> >> > state, which I assume I need to flag both "dead" OSDs as lost to
>> unstuck
>> >> > them. Since it is an operation I cannot go back, I would like to
>> confirm
>> >> > that is indeed the correct next step to take.
>> >> >
>> >> > I have a few questions to understand "what happens" in the next step
>> >> > (marking osd as lost?):
>> >> >
>> >> > Shall I assume that once I flag an OSD as lost, I won't be able to
>> >> > "activate" it since I use encryption when initializing the bluestore
>> >> OSD,
>> >> > or flagging them as lost won't destroy their unlocking keys? (which
>> >> means
>> >> > any hope of further extracting data to be gone, mostly on the osd.0
>> >> which I
>> >> > couldn't use ceph-objectstore-tool at all since the power loss).
>> >> >
>> >> > I think I should have all the shards from the PGs but just in case,
>> I've
>> >> > managed to make a clone of the osd.0 on a different physical disk (the
>> >> > other reason I took long to answer). But still ceph-objectstore-tool
>> >> > refuses to run:
>> >> >
>> >> > # ceph-objectstore-tool --data-path /var/lib/ceph/osd --op list-pgs
>> >> > Mount failed with '(5) Input/output error'
>> >> >
>> >> > # ls -l /var/lib/ceph/osd
>> >> > total 28
>> >> > lrwxrwxrwx 1 ceph ceph 50 Feb 4 08:26 block ->
>> >> > /dev/mapper/zNPZJR-i0TZ-6NtK-URto-tjfs-iJRb-GCAYEm
>> >> > -rw------- 1 ceph ceph 37 Feb 4 08:26 ceph_fsid
>> >> > -rw------- 1 ceph ceph 37 Feb 4 08:26 fsid
>> >> > -rw------- 1 ceph ceph 55 Feb 4 08:26 keyring
>> >> > -rw------- 1 ceph ceph 106 Jan 24 00:44 lockbox.keyring
>> >> > -rw------- 1 ceph ceph 6 Feb 4 08:26 ready
>> >> > -rw------- 1 ceph ceph 10 Feb 4 08:26 type
>> >> > -rw------- 1 ceph ceph 2 Feb 4 08:26 whoami
>> >> >
>> >> > Just as information, all except 2 pools in my cluster are
>> "replicated".
>> >> > Pools id 11 and 16 are erasure coded (2+1). If I understood correctly,
>> >> as
>> >> > long as I have two acting shards (and at most one "NONE"), data should
>> >> be
>> >> > available (at least in read-only) once I mark the down OSDs as lost.
>> Is
>> >> > that understanding correct?
>> >> >
>> >> > Another information, pools 10 and 15 are the "replicated root pools"
>> >> before
>> >> > the erasure coded pools were created.
>> >> >
>> >> > Ignoring osd.0 for now, here are the current state of my cluster (mds
>> is
>> >> > intentionally not started while I try to fix the PGs):
>> >> > ### ceph osd lspools
>> >> > 3 .rgw.root
>> >> > 4 default.rgw.log
>> >> > 5 default.rgw.control
>> >> > 6 default.rgw.meta
>> >> > 10 ark.data
>> >> > 11 ark.data_ec
>> >> > 12 ark.metadata
>> >> > 14 .mgr
>> >> > 15 limbo
>> >> > 16 limbo.data_ec
>> >> > 18 default.rgw.buckets.index
>> >> > 19 default.rgw.buckets.data
>> >> > ###
>> >> >
>> >> > ### ceph health
>> >> > # ceph -s
>> >> > cluster:
>> >> > id: 021f058f-dbf3-4a23-adb5-21d83f3f1bb6
>> >> > health: HEALTH_ERR
>> >> > 1 filesystem is degraded
>> >> > 1 filesystem has a failed mds daemon
>> >> > 1 filesystem is offline
>> >> > insufficient standby MDS daemons available
>> >> > Reduced data availability: 143 pgs inactive, 143 pgs down
>> >> > Degraded data redundancy: 1303896/7149898 objects
>> degraded
>> >> > (18.237%), 218 pgs degraded, 316 pgs undersized
>> >> > 144 pgs not deep-scrubbed in time
>> >> > 459 pgs not scrubbed in time
>> >> > 256 slow ops, oldest one blocked for 1507794 sec, osd.1
>> has
>> >> > slow ops
>> >> > too many PGs per OSD (657 > max 500)
>> >> >
>> >> > services:
>> >> > mon: 2 daemons, quorum ceph-ymir-mon2,ceph-ymir-mon1 (age 2w)
>> >> > mgr: ceph-ymir-mgr1(active, since 2w)
>> >> > mds: 0/1 daemons up (1 failed)
>> >> > osd: 4 osds: 2 up (since 29m), 2 in (since 4w); 24 remapped pgs
>> >> >
>> >> > data:
>> >> > volumes: 0/1 healthy, 1 failed
>> >> > pools: 12 pools, 529 pgs
>> >> > objects: 2.46M objects, 7.4 TiB
>> >> > usage: 8.3 TiB used, 13 TiB / 22 TiB avail
>> >> > pgs: 27.032% pgs not active
>> >> > 1303896/7149898 objects degraded (18.237%)
>> >> > 306628/7149898 objects misplaced (4.289%)
>> >> > 218 active+undersized+degraded
>> >> > 143 down
>> >> > 98 active+undersized
>> >> > 45 active+clean
>> >> > 19 active+clean+remapped
>> >> > 4 active+clean+remapped+scrubbing+deep
>> >> > 1 active+clean+remapped+scrubbing
>> >> > 1 active+clean+scrubbing+deep
>> >> > ### ceph health
>> >> >
>> >> > ### ceph health detail
>> >> > # ceph health detail
>> >> > HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds
>> >> daemon;
>> >> > 1 filesystem is offline; insufficient standby
>> >> > MDS daemons available; Reduced data availability: 143 pgs inactive,
>> >> 143
>> >> > pgs down; Degraded data redundancy: 1303896/714
>> >> > 9898 objects degraded (18.237%), 218 pgs degraded, 316 pgs undersized;
>> >> 144
>> >> > pgs not deep-scrubbed in time; 459 pgs not sc
>> >> > rubbed in time; 256 slow ops, oldest one blocked for 1508207 sec,
>> osd.1
>> >> has
>> >> > slow ops; too many PGs per OSD (657 > max 50
>> >> > 0)
>> >> > [WRN] FS_DEGRADED: 1 filesystem is degraded
>> >> > fs ark is degraded
>> >> > [WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon
>> >> > fs ark has 1 failed mds
>> >> > [ERR] MDS_ALL_DOWN: 1 filesystem is offline
>> >> > fs ark is offline because no MDS is active for it.
>> >> > [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons
>> >> available
>> >> > have 0; want 1 more
>> >> > [WRN] PG_AVAILABILITY: Reduced data availability: 143 pgs inactive,
>> 143
>> >> pgs
>> >> > down
>> >> > pg 10.11 is down, acting [1,3]
>> >> > pg 10.18 is down, acting [3,1]
>> >> > pg 10.1d is down, acting [1,3]
>> >> > pg 10.1f is down, acting [1,3]
>> >> > pg 11.10 is down, acting [3,1,NONE]
>> >> > pg 11.12 is down, acting [1,NONE,3]
>> >> > pg 11.18 is stuck inactive for 4w, current state down, last
>> acting
>> >> > [1,3,NONE]
>> >> > pg 11.19 is down, acting [3,1,NONE]
>> >> > pg 11.1b is down, acting [1,NONE,3]
>> >> > pg 11.62 is down, acting [NONE,3,1]
>> >> > pg 11.63 is down, acting [3,NONE,1]
>> >> > pg 11.64 is down, acting [NONE,1,3]
>> >> > pg 11.66 is down, acting [NONE,3,1]
>> >> > pg 11.67 is down, acting [1,NONE,3]
>> >> > pg 11.68 is down, acting [3,NONE,1]
>> >> > pg 11.69 is down, acting [NONE,1,3]
>> >> > pg 11.6a is down, acting [1,NONE,3]
>> >> > pg 11.6b is down, acting [NONE,1,3]
>> >> > pg 11.6f is down, acting [NONE,3,1]
>> >> > pg 11.71 is down, acting [1,3,NONE]
>> >> > pg 11.72 is down, acting [1,3,NONE]
>> >> > pg 11.74 is down, acting [NONE,3,1]
>> >> > pg 11.76 is down, acting [1,NONE,3]
>> >> > pg 11.78 is down, acting [3,1,NONE]
>> >> > pg 11.7d is down, acting [NONE,3,1]
>> >> > pg 11.7e is down, acting [NONE,1,3]
>> >> > pg 15.15 is down, acting [1,3]
>> >> > pg 15.16 is down, acting [3,1]
>> >> > pg 15.17 is down, acting [1,3]
>> >> > pg 15.1a is down, acting [3,1]
>> >> > pg 16.1 is down, acting [1,3,NONE]
>> >> > pg 16.4 is down, acting [1,3,NONE]
>> >> > pg 16.b is down, acting [3,NONE,1]
>> >> > pg 16.60 is down, acting [3,1,NONE]
>> >> > pg 16.61 is down, acting [3,1,NONE]
>> >> > pg 16.62 is down, acting [3,NONE,1]
>> >> > pg 16.63 is down, acting [3,NONE,1]
>> >> > pg 16.65 is down, acting [NONE,3,1]
>> >> > pg 16.67 is down, acting [1,NONE,3]
>> >> > pg 16.68 is down, acting [1,NONE,3]
>> >> > pg 16.69 is down, acting [3,1,NONE]
>> >> > pg 16.6a is down, acting [1,3,NONE]
>> >> > pg 16.6c is down, acting [1,3,NONE]
>> >> > pg 16.70 is down, acting [3,NONE,1]
>> >> > pg 16.73 is down, acting [3,NONE,1]
>> >> > pg 16.74 is down, acting [1,3,NONE]
>> >> > pg 16.75 is down, acting [3,1,NONE]
>> >> > pg 16.79 is down, acting [3,NONE,1]
>> >> > pg 16.7a is down, acting [1,3,NONE]
>> >> > pg 16.7e is down, acting [1,3,NONE]
>> >> > pg 16.7f is down, acting [3,NONE,1]
>> >> > [WRN] PG_DEGRADED: Degraded data redundancy: 1303896/7149898 objects
>> >> > degraded (18.237%), 218 pgs degraded, 316 pgs under
>> >> > sized
>> >> > pg 3.18 is stuck undersized for 36m, current state
>> >> active+undersized,
>> >> > last acting [1,3]
>> >> > ...<snipped for brevity>
>> >> > ###
>> >> >
>> >> > Once again, I cannot thank you enough for looking into my issue.
>> >> > I have the impression that being able to recover the data I need is
>> just
>> >> > around the corner. Although the croit.io blog did mention flagging
>> the
>> >> osd
>> >> > as lost, I would like to double check it to avoid losing any
>> >> possibility to
>> >> > recover the data.
>> >> >
>> >> > If there's anything further I could check or if you need full output
>> of
>> >> the
>> >> > commands, let me know.
>> >> >
>> >> > Thanks in advance.
>> >> >
>> >> > On Tue, 3 Feb 2026 at 10:26, Igor Fedotov <igor.fedotov(a)croit.io>
>> >> wrote:
>> >> >
>> >> >> Hi Theo,
>> >> >>
>> >> >> you might want to try to use PG export/import using
>> >> ceph-objectstore-tool.
>> >> >>
>> >> >> Please find more details here
>> >> >>
>> >>
>> https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor…
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Igor
>> >> >> On 03/02/2026 02:38, Theo Cabrerizo Diem via ceph-users wrote:
>> >> >>
>> >> >> :12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd)
>> >> >> fsck error: free extent 0x1714c521000~978b26df000 intersects
>> >> allocatedblocks
>> >> >> fsck status: remaining 1 error(s) and warning(s)
>> >> >>
>> >> >>
>> >> > _______________________________________________
>> >> > ceph-users mailing list -- ceph-users(a)ceph.io
>> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> >>
>> >
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0
I recommend to crush reweight your OSDs to reflect their actual sizes
in TB. According to your crush tree output it would be these commands:
ceph osd crush reweight osd.0 0.15
ceph osd crush reweight osd.1 0.15
ceph osd crush reweight osd.9 0.15
ceph osd crush reweight osd.4 0.4
ceph osd crush reweight osd.5 0.4
ceph osd crush reweight osd.10 0.4
I expect this to have a positive effect on PG distribution, but it
will cause some data movement, of course.
Zitat von listy via ceph-users <ceph-users(a)ceph.io>:
> I don't understand - have what I showed already does reveal
> underlying disk-devices sizes, does not do it clearly?
> Perhaps ceph tools/cmd might do it incorrectly - which is why you
> are asking bout 10GB? - or doing that jokingly?
>
> Does that matter why I use something, why anybody runs whatever - I
> already answered these questions - mate(s) if you ever were to work
> a proverbial "front-desk" (facing customers) then I share a friendly
> advice (if you take those).. teach yourself to resit the urge....
>
> It was, still is a small lab - all what I've showed.
> No, I do _not_ use 10GB drives - perhaps long time ago when the lab
> was first spun up, I don't remember.
> I've not touched _crush_ at all - what to do next?
> Poke the cluster somehow to "fix" this? Do some changes/tweaks manually?
>
> That would be a great thing to have - a more seamless path to resize
> (mostly extend) OSD's disks - in ubiquitous cloud with IaaS but not
> only there, in-house many virtualize too(yes, with ceph above, not
> only underneath it), probably many _add_ drives but extending a
> drive is not uncommon - great, I can only upvote it, if @devel read
> here.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0
Hi everyone,
I’m reaching out for some guidance on troubleshooting periodic latency
spikes we’re seeing in our Ceph cluster. At times simple file operations
are occasionally taking several seconds to complete and this is impacting
our users.
We suspect the issue lies with our MDS load balancing, specifically an
imbalance where our rank 0 MDS consistently has ~3-4 billion open caps
(which seems to be about where its RAM maxes out), while the rank 1 and
rank 2 sit around 3-7 million.
Cluster Details
-
Ceph Version: 18.2.4
-
OS: QuantaStor 6.6.9
-
MDS Configuration: 3 active, 5 standby.
-
Storage Pools:
-
Data: 16 node HDD pool (8k+3m erasure coding).
-
Metadata: NVMe pool (triple replication).
-
We have ~150 nodes in an HPC cluster that use this storage in som
We’ve tried tuning some ceph config parameters to try and make the MDS
balance better, but this didn’t seem to help much:
-
mds mds_recall_max_caps 50000
-
mds mds_max_caps_per_client 100000
-
mds mds_bal_split_size 5000
-
mds mds_bal_interval 5
Additionally, we’ve also attempted pinning our highest-load user to the
rank 2 MDS. While this did successfully increase utilization on that rank,
we are still seeing significant latency spikes for small file read/write
operations across the board. If it helps, this user has a volume of 505TB,
we have ten other volumes in the 10-100TB range, and then 175 other volumes
under 10TB in usage.
Is our diagnosis correct that the MDS is what’s causing the poor latency
with file operations? If so, is it possible to have our MDS services
distribute load more evenly? And as another question, does the size of a
volume in ceph matter, meaning should we limit volume sizes to under 100TB
for example?
Thanks in advance for any guidance in diagnosing and resolving this issue.
Best regards,
Greg
--
Gregory Ballantine
System Administrator for Research and Enterprise Computing
UMBC - DoIT
1
0
The V release will follow Umbrella. It is targeting a release date of
Spring 2027.
Please share name ideas and vote on your favorite with a "+1" at
https://pad.ceph.com/p/v
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
1
1
>
> Now 5 hours later - since i started this draft - I added:
> -> $ ceph osd reweight-by-utilization
This is a legacy that is mostly obviated by the upmap balancer. It is best to not use this and leave all REWEIGHT values 1.00000 as it and the upmap balancer don’t play well together.
> and before that cmd above I also noticed:
> -> $ ceph config get mgr mgr/balancer/begin_weekday
> 0
> -> $ ceph config get mgr mgr/balancer/end_weekday
> 0
> which was done by 'deployment' process - cephadm bootstrap - and made me wonder:
> does that mean that auto-rebalance runs only on Sunday?
By default it runs all the time.
> I changed: end_weekday = 6
>
> _reweight-by-utilization_ I notice, changed REWEIGHT for osd.0 and that did something, I think.
> So now _active+remapped+backfill_toofull_ are gone from 'pcs' part of health report.
> RAW USE & DATA are down, but stil:
> -> $ ceph osd df tree | egrep '(osd.0|ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
> 0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB 1.1 MiB 2.0 GiB 16 GiB 89.51 1.31 68 up osd.0
> and when compared to other host-ods which use "identical" disk-drives:
Do you really have a 10 GiB OSD drive? As noted, your CRUSH weights do not match the ostensible device sizes, which is a significant part of your problem. What is underlying these OSDs? Why do you have OSDs as ostensibly small as 150 GiB?
While there is a bit of work under way to better handle OSD devices whose capacity can change, today one has to ensure that CRUSH weights are adjusted to correlate with device size.
> -> $ ceph osd df tree | egrep '(osd\.[0,1,9] |ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
> 9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB 624 KiB 1.4 GiB 98 GiB 34.83 0.51 43 up osd.9
> 0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB 1.1 MiB 2.0 GiB 16 GiB 89.51 1.31 68 up osd.0
> 1 ssd 0.04880 1.00000 150 GiB 53 GiB 53 GiB 526 KiB 254 MiB 97 GiB 35.40 0.52 44 up osd.1
>
> Perhaps cluster goes only as far as to satisfy _backfill_toofull_ be gone and then "gives up"?
> The "other" disk-drives:
> -> $ ceph osd df tree | egrep '(osd\.(5|4|10)\ |ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
> 10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 2.0 MiB 2.1 GiB 93 GiB 76.68 1.12 246 up osd.10
> 4 ssd 0.29300 1.00000 400 GiB 275 GiB 273 GiB 3.2 MiB 2.1 GiB 125 GiB 68.83 1.01 221 up osd.4
> 5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB 3.0 MiB 2.5 GiB 95 GiB 76.27 1.12 245 up osd.5
>
> Seems that _host podster2_ balances its osds 4 & 0 "differently" to what other two hosts do - if so then why?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2
1
Re: Recover after multiple OSD failures (was Re: Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch)
by Theo Cabrerizo Diem 02 Mar '26
by Theo Cabrerizo Diem 02 Mar '26
02 Mar '26
Hello Igor, Hello all,
First, I've already accepted the fact that my data most likely is
unrecoverable by now due to my own fault.. I'm using it to learn and
hopefully document the information gained as I couldn't find much recent
information regarding the process of recovery and improvement of my
understanding about ceph, if someone is willing to further chime in.
The current situation is that I've had multiple OSD failures and some of
the ceph-osd processes would refuse to start (corruption on rocksdb). I've
decided to follow
https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor…
as a suggested mechanism to export the PGs and re-import on a fresh OSD.
I had an attempt (sort of documented in this thread) to do it with
"unexpected" results. I have written down a lot of information, states,
ceph pg queries, etc (so I can provide their outputs if relevant. I have
not marked any of the OSDs as lost at any time and the monitors have been
running without issue since the beginning.
I have some questions regarding my observations following the information
on that blog (sorry for my lack of experience):
- Is the process for setting up a new "temporary" OSD to import PGs
correct? (short story: "ceph-volume lvm prepare", start the osd and as soon
as possible run "ceph osd crush reweight osd.XX 0")
- Creating the OSD as described above, running "ceph-objectstore-tool --op
list-pgs" on this OSD showed lots of PGs (which I assume they were
"pre-allocation" from crush) but "ceph osd df" confirmed no data was on the
osd (only very very little, like less than 2Gb). Is there a way to have an
OSD "flushed out" so I can import further PGs?
- Running "ceph pg XX.XX query" into some of the imported PGs after
starting ceph-osd again doesn't seem to reliably reflect my progress. Is
there a different way? Is it because the PG is still in down state because
of the dead OSDs?
For example, pg 11.17 of which had only shard-0 available because only one
OSD was up. I've imported shard-1 to osd.10 , but "ceph pg 11.17 query"
shows under "recovery_state":
"intervals": [
{
"first": "2882",
"last": "2883",
"acting": "1(1),3(0)"
},
{
"first": "3021",
"last": "3023",
"acting": "3(0),8(1)"
},
{
"first": "3024",
"last": "3026",
"acting": "3(0),8(1),10(2)"
}
]
Running ceph-objectstore-tool --op list-pgs on osd.10 (stopped) confirms
that 11.17s1 is listed, and running ceph-objectstore-tool --op list-pgs on
osd.8 (stopped) doesn't show 11.17 at all (none of its shards)
Should I instead keep track of my progress using "ceph-objectstore-tool
--op list" looking for a "oid" present?
This might reflect on my lack of knowledge regarding how ceph osd "works
internally", so feel free to correct me or suggest a better approach. I
still have the 3x original OSDs (out of 4, one, as mentioned on the thread,
have a bigger corruption and ceph-objectstore-tool fails) and 8x 2Tb disks
that I can load as new OSDs to import the old data (I had less than 6Tb
used before the crash).
Should I continue exporting all PGs and keep importing them this way?
Thanks
On Sat, 28 Feb 2026 at 16:50, Theo Cabrerizo Diem via ceph-users <
ceph-users(a)ceph.io> wrote:
> Hello all,
>
> I've managed to get a bunch of 2Tb disks for setting up a few OSDs but
> before I even started adding them to my monitors, I decided to check my
> cluster state and noticed another OSD died. Trying to start it, revealed a
> rocksdb corruption:
>
> # /usr/bin/ceph-osd -f --id "1" --osd-data "/var/lib/ceph/osd" --cluster
> "ceph" --setuser "ceph" --setgroup "ceph"
> 2026-02-28T15:34:16.596+0000 7f793ce718c0 -1 Falling back to public
> interface
>
> 2026-02-28T15:35:04.304+0000 7f792c1c5640 -1 rocksdb: submit_common error:
> Corruption: block checksum mismatch: stored = 0, computed = 1265684702,
> type = 4 in db/180433.sst offset 1048892 size 1429 code = ☻ Rocksdb
> transaction:
> PutCF( prefix = O key =
> 0x7F8000000000000006D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value
> size = 33)
> PutCF( prefix = S key = 'nid_max' value size = 8)
> PutCF( prefix = S key = 'blobid_max' value size = 8)
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
> time 2026-02-28T15:35:04.305926+0000
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
> FAILED ceph_assert(r == 0)
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x11f) [0x557b7cbf6236]
> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
> 2026-02-28T15:35:04.309+0000 7f792c1c5640 -1
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
> time 2026-02-28T15:35:04.305926+0000
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
> FAILED ceph_assert(r == 0)
>
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x11f) [0x557b7cbf6236]
> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>
> *** Caught signal (Aborted) **
> in thread 7f792c1c5640 thread_name:bstore_kv_sync
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
> 3: raise()
> 4: abort()
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x17a) [0x557b7cbf6291]
> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
> 2026-02-28T15:35:04.320+0000 7f792c1c5640 -1 *** Caught signal (Aborted) **
> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
> 3: raise()
> 4: abort()
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x17a) [0x557b7cbf6291]
> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> to interpret this.
>
> -2421> 2026-02-28T15:34:16.596+0000 7f793ce718c0 -1 Falling back to public
> interface
> -8> 2026-02-28T15:35:04.304+0000 7f792c1c5640 -1 rocksdb: submit_common
> error: Corruption: block checksum mismatch: stored = 0, computed =
> 1265684702, type = 4 in db/180433.sst offset 1048892 size 1429 code = ☻
> Rocksdb transaction:
> PutCF( prefix = O key =
> 0x7F8000000000000006D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value
> size = 33)
> PutCF( prefix = S key = 'nid_max' value size = 8)
> PutCF( prefix = S key = 'blobid_max' value size = 8)
> -7> 2026-02-28T15:35:04.309+0000 7f792c1c5640 -1
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
> time 2026-02-28T15:35:04.305926+0000
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
> FAILED ceph_assert(r == 0)
>
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x11f) [0x557b7cbf6236]
> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>
> 0> 2026-02-28T15:35:04.320+0000 7f792c1c5640 -1 *** Caught signal
> (Aborted) **
> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
> 3: raise()
> 4: abort()
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x17a) [0x557b7cbf6291]
> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> to interpret this.
>
> -2445> 2026-02-28T15:34:16.596+0000 7f793ce718c0 -1 Falling back to public
> interface
> -32> 2026-02-28T15:35:04.304+0000 7f792c1c5640 -1 rocksdb: submit_common
> error: Corruption: block checksum mismatch: stored = 0, computed =
> 1265684702, type = 4 in db/180433.sst offset 1048892 size 1429 code = ☻
> Rocksdb transaction:
> PutCF( prefix = O key =
> 0x7F8000000000000006D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value
> size = 33)
> PutCF( prefix = S key = 'nid_max' value size = 8)
> PutCF( prefix = S key = 'blobid_max' value size = 8)
> -31> 2026-02-28T15:35:04.309+0000 7f792c1c5640 -1
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function
> 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7f792c1c5640
> time 2026-02-28T15:35:04.305926+0000
> /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 14539:
> FAILED ceph_assert(r == 0)
>
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x11f) [0x557b7cbf6236]
> 2: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 3: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 4: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 5: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 6: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
>
> -24> 2026-02-28T15:35:04.320+0000 7f792c1c5640 -1 *** Caught signal
> (Aborted) **
> in thread 7f792c1c5640 thread_name:bstore_kv_sync
>
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable - RelWithDebInfo)
> 1: /lib64/libc.so.6(+0x3fc30) [0x7f793d2ecc30]
> 2: /lib64/libc.so.6(+0x8d03c) [0x7f793d33a03c]
> 3: raise()
> 4: abort()
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x17a) [0x557b7cbf6291]
> 6: /usr/bin/ceph-osd(+0x44a3ef) [0x557b7cbd03ef]
> 7: (BlueStore::_kv_sync_thread()+0xaf1) [0x557b7d276191]
> 8: /usr/bin/ceph-osd(+0xa790d1) [0x557b7d1ff0d1]
> 9: /lib64/libc.so.6(+0x8b2fa) [0x7f793d3382fa]
> 10: /lib64/libc.so.6(+0x110400) [0x7f793d3bd400]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> to interpret this.
>
> Aborted (core dumped)
>
> There was no sign of hardware failure on kernel logs. I think, for now, I
> should move away from alpine binaries and use ceph's official containers.
> Is there any tool that can try to fix this rocksdb issue? Or is the
> recommended way to export all PGs from this osd and re-import on a new one?
>
> I agree that at this point I should consider rechecking all the hardware
> involved. I plan to decommission this system once I (if possible) I get the
> data out.
>
> Thanks,
> Theo
>
> On Tue, 24 Feb 2026 at 22:22, Theo Cabrerizo Diem <diem(a)carpediem.sh>
> wrote:
>
> > Hello Igor,
> >
> > > Just in case - didn't you overwrite existing PG replicas at target
> OSDs
> > > when exporting PGs back to OSDs 1&3?
> >
> > Now that you mention that an OSD cannot have two shards, I think I did
> put
> > myself now in a tricky place.
> > I had 4 OSDs, two "died" .. leaving two online, so when I did export
> > shards from osd.2 I did end up importing them in osd.3 (which had other
> > shards for the same PG).
> >
> > Doesn't look "that bad" ... (*looking up to the sky*) ...
> > ## on osd.3
> > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ --op list-pgs |
> > grep 16.b
> > 16.bs1
> > 16.bs0
> > (and the s2 should be in osd.1, s1 was imported)
> >
> > ## ceph health reports
> > pg 16.b is down, acting [3,NONE,1]
> >
> > I might have some spares HDDs to set up a new OSD (one though) with
> enough
> > space for it to balance out, but as I've mentioned, there are quite a
> bunch
> > of PGs in down state. Should I set-up a new OSD and wait until it
> "settles
> > up"? The exported shards from the dead osd.2 that ceph health was
> > complaining about sums to about 400Gb.
> >
> > Is it possible to set-up an OSD and have ceph not push too many shards
> > into it? (the replica pools are "small"). I ask this because it should be
> > easy to get a 1Tb/2Tb disk, but if it should balance completely for the
> PGs
> > to "unstuck", I need a bigger disk, which I don't have at the moment (and
> > currently not the best market prices to get them).
> >
> > Another option is if osd.0 is gone for good (the one that doesn't even
> > allows me to run 'ceph-objectstore-tool --list-pgs'), I can wipe its
> > disk/clone and use it instead, it is the same capacity as other OSDs (I
> > used a spare I had at home). If you believe it is still workable, I can
> try
> > to find some more disks and not touch those for the moment.
> >
> > Thanks for the patience so far.
> >
> > On Tue, 24 Feb 2026 at 16:39, Igor Fedotov <igor.fedotov(a)croit.io>
> wrote:
> >
> >> Hi Theo,
> >>
> >> Sorry I can't tell for sure what would marking OSD lost do with its
> >> encryption keys. Likely - yes, they'll be lost.
> >>
> >> But instead of going this way I'd rather suggest you to add another two
> >> OSDs and let Ceph recover more PGs replicas into them.
> >>
> >> Just in case - didn't you overwrite existing PG replicas at target OSDs
> >> when exporting PGs back to OSDs 1&3? The same PG can't have two
> >> replicas/shards at a single OSD while your OSD count is pretty limited..
> >> Just curious for now - that still shouldn't be an issue given you have
> >> at least 2 replicas/shards for all the pools anyway.
> >>
> >> Thanks,
> >>
> >> Igor
> >>
> >> On 2/21/2026 11:00 PM, Theo Cabrerizo Diem via ceph-users wrote:
> >> > Hello Igor,
> >> >
> >> > First of all, sorry about the late reply. It took me a while to export
> >> all
> >> > shards that weren't available from the osd.2 (1 and 3 were fine, 2
> >> didn't
> >> > start but i could use `ceph-objectstore-tool ... --op list-pgs` while
> >> osd.0
> >> > I couldn't even list the pgs, it threw an error right away - more
> about
> >> it
> >> > later in the email)
> >> >
> >> > Two of the unavailable shards, when exporting, ceph-objectstore-tool
> >> core
> >> > dumped with the same issue in the rocksdb, but I should have enough
> >> chunks
> >> > to not need them - just mentioning in case is useful:
> >> >
> >> > sh-5.1# ceph-objectstore-tool --data-path /var/lib/ceph/osd --pgid
> >> 11.19s2
> >> > --op export --file pg.11.19s2.dat
> >> > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: In function
> >> > 'virtual int RocksDBStore::get(const std::string&, const std::string&,
> >> > ceph::bufferlist*)' thread 7ff3be4ca800 time
> >> 2026-02-04T09:42:00.743877+0000
> >> > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: 1961:
> >> > ceph_abort_msg("block checksum mismatch: stored = 246217859, computed
> =
> >> > 2155741315, type = 4 in db/170027.sst offset 28264757 size 1417")
> >> > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489)
> >> tentacle
> >> > (stable - RelWithDebInfo)
> >> > 1: (ceph::__ceph_abort(char const*, int, char const*,
> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > std::allocator<char> > const&)+0xc9) [0x7ff3bf5391fd]
> >> > 2: (RocksDBStore::get(std::__cxx11::basic_string<char,
> >> > std::char_traits<char>, std::allocator<char> > const&,
> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc)
> >> > [0x555667b340bc]
> >> > 3:
> >> >
> >>
> (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
> >> > ghobject_t const&,
> >> > std::set<std::__cxx11::basic_string<char,std::char_traits<char>,
> >> > std::allocator<char> >, std::less<std::__cxx11::basic_string<char,
> >> > std::char_traits<char>, std::allocator<char> > >,
> >> > std::allocator<std::__cxx11::basic_string<char,
> std::char_traits<char>,
> >> > std::allocator<char> > > > const&,
> >> > std::map<std::__cxx11::basic_string<char,std::char_traits<char>,
> >> > std::allocator<char> >, ceph::buffer::v15_2_0::list,
> >> > std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > std::allocator<char> > >,
> >> > std::allocator<std::pair<std::__cxx11::basic_string<char,
> >> > std::char_traits<char>, std::allocator<char> > const,
> >> > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1]
> >> > 4: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361)
> >> > [0x5556675e0101]
> >> > 5: main()
> >> > 6: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610]
> >> > 7: __libc_start_main()
> >> > 8: _start()
> >> > *** Caught signal (Aborted) **
> >> > in thread 7ff3be4ca800 thread_name:ceph-objectstor
> >> > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489)
> >> tentacle
> >> > (stable - RelWithDebInfo)
> >> > 1: /lib64/libc.so.6(+0x3fc30) [0x7ff3be945c30]
> >> > 2: /lib64/libc.so.6(+0x8d03c) [0x7ff3be99303c]
> >> > 3: raise()
> >> > 4: abort()
> >> > 5: (ceph::__ceph_abort(char const*, int, char const*,
> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > std::allocator<char> > const&)+0x186) [0x7ff3bf5392ba]
> >> > 6: (RocksDBStore::get(std::__cxx11::basic_string<char,
> >> > std::char_traits<char>, std::allocator<char> > const&,
> >> > std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc)
> >> > [0x555667b340bc]
> >> > 7:
> >> >
> >>
> (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
> >> > ghobject_t const&,
> >> > std::set<std::__cxx11::basic_string<char,std::char_traits<char>,
> >> > std::allocator<char> >, std::less<std::__cxx11::basic_string<char,
> >> > std::char_traits<char>, std::allocator<char> > >,
> >> > std::allocator<std::__cxx11::basic_string<char,
> std::char_traits<char>,
> >> > std::allocator<char> > > > const&,
> >> > std::map<std::__cxx11::basic_string<char,std::char_traits<char>,
> >> > std::allocator<char> >, ceph::buffer::v15_2_0::list,
> >> > std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > std::allocator<char> > >,
> >> > std::allocator<std::pair<std::__cxx11::basic_string<char,
> >> > std::char_traits<char>, std::allocator<char> > const,
> >> > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1]
> >> > 8: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361)
> >> > [0x5556675e0101]
> >> > 9: main()
> >> > 10: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610]
> >> > 11: __libc_start_main()
> >> > 12: _start()
> >> > Aborted (core dumped)
> >> >
> >> >
> >> >
> >> > After importing all shards that I could recover that weren't
> available,
> >> I
> >> > don't have any "unknown" pgs anymore. I still have lots of PGs in
> "down"
> >> > state, which I assume I need to flag both "dead" OSDs as lost to
> unstuck
> >> > them. Since it is an operation I cannot go back, I would like to
> confirm
> >> > that is indeed the correct next step to take.
> >> >
> >> > I have a few questions to understand "what happens" in the next step
> >> > (marking osd as lost?):
> >> >
> >> > Shall I assume that once I flag an OSD as lost, I won't be able to
> >> > "activate" it since I use encryption when initializing the bluestore
> >> OSD,
> >> > or flagging them as lost won't destroy their unlocking keys? (which
> >> means
> >> > any hope of further extracting data to be gone, mostly on the osd.0
> >> which I
> >> > couldn't use ceph-objectstore-tool at all since the power loss).
> >> >
> >> > I think I should have all the shards from the PGs but just in case,
> I've
> >> > managed to make a clone of the osd.0 on a different physical disk (the
> >> > other reason I took long to answer). But still ceph-objectstore-tool
> >> > refuses to run:
> >> >
> >> > # ceph-objectstore-tool --data-path /var/lib/ceph/osd --op list-pgs
> >> > Mount failed with '(5) Input/output error'
> >> >
> >> > # ls -l /var/lib/ceph/osd
> >> > total 28
> >> > lrwxrwxrwx 1 ceph ceph 50 Feb 4 08:26 block ->
> >> > /dev/mapper/zNPZJR-i0TZ-6NtK-URto-tjfs-iJRb-GCAYEm
> >> > -rw------- 1 ceph ceph 37 Feb 4 08:26 ceph_fsid
> >> > -rw------- 1 ceph ceph 37 Feb 4 08:26 fsid
> >> > -rw------- 1 ceph ceph 55 Feb 4 08:26 keyring
> >> > -rw------- 1 ceph ceph 106 Jan 24 00:44 lockbox.keyring
> >> > -rw------- 1 ceph ceph 6 Feb 4 08:26 ready
> >> > -rw------- 1 ceph ceph 10 Feb 4 08:26 type
> >> > -rw------- 1 ceph ceph 2 Feb 4 08:26 whoami
> >> >
> >> > Just as information, all except 2 pools in my cluster are
> "replicated".
> >> > Pools id 11 and 16 are erasure coded (2+1). If I understood correctly,
> >> as
> >> > long as I have two acting shards (and at most one "NONE"), data should
> >> be
> >> > available (at least in read-only) once I mark the down OSDs as lost.
> Is
> >> > that understanding correct?
> >> >
> >> > Another information, pools 10 and 15 are the "replicated root pools"
> >> before
> >> > the erasure coded pools were created.
> >> >
> >> > Ignoring osd.0 for now, here are the current state of my cluster (mds
> is
> >> > intentionally not started while I try to fix the PGs):
> >> > ### ceph osd lspools
> >> > 3 .rgw.root
> >> > 4 default.rgw.log
> >> > 5 default.rgw.control
> >> > 6 default.rgw.meta
> >> > 10 ark.data
> >> > 11 ark.data_ec
> >> > 12 ark.metadata
> >> > 14 .mgr
> >> > 15 limbo
> >> > 16 limbo.data_ec
> >> > 18 default.rgw.buckets.index
> >> > 19 default.rgw.buckets.data
> >> > ###
> >> >
> >> > ### ceph health
> >> > # ceph -s
> >> > cluster:
> >> > id: 021f058f-dbf3-4a23-adb5-21d83f3f1bb6
> >> > health: HEALTH_ERR
> >> > 1 filesystem is degraded
> >> > 1 filesystem has a failed mds daemon
> >> > 1 filesystem is offline
> >> > insufficient standby MDS daemons available
> >> > Reduced data availability: 143 pgs inactive, 143 pgs down
> >> > Degraded data redundancy: 1303896/7149898 objects
> degraded
> >> > (18.237%), 218 pgs degraded, 316 pgs undersized
> >> > 144 pgs not deep-scrubbed in time
> >> > 459 pgs not scrubbed in time
> >> > 256 slow ops, oldest one blocked for 1507794 sec, osd.1
> has
> >> > slow ops
> >> > too many PGs per OSD (657 > max 500)
> >> >
> >> > services:
> >> > mon: 2 daemons, quorum ceph-ymir-mon2,ceph-ymir-mon1 (age 2w)
> >> > mgr: ceph-ymir-mgr1(active, since 2w)
> >> > mds: 0/1 daemons up (1 failed)
> >> > osd: 4 osds: 2 up (since 29m), 2 in (since 4w); 24 remapped pgs
> >> >
> >> > data:
> >> > volumes: 0/1 healthy, 1 failed
> >> > pools: 12 pools, 529 pgs
> >> > objects: 2.46M objects, 7.4 TiB
> >> > usage: 8.3 TiB used, 13 TiB / 22 TiB avail
> >> > pgs: 27.032% pgs not active
> >> > 1303896/7149898 objects degraded (18.237%)
> >> > 306628/7149898 objects misplaced (4.289%)
> >> > 218 active+undersized+degraded
> >> > 143 down
> >> > 98 active+undersized
> >> > 45 active+clean
> >> > 19 active+clean+remapped
> >> > 4 active+clean+remapped+scrubbing+deep
> >> > 1 active+clean+remapped+scrubbing
> >> > 1 active+clean+scrubbing+deep
> >> > ### ceph health
> >> >
> >> > ### ceph health detail
> >> > # ceph health detail
> >> > HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds
> >> daemon;
> >> > 1 filesystem is offline; insufficient standby
> >> > MDS daemons available; Reduced data availability: 143 pgs inactive,
> >> 143
> >> > pgs down; Degraded data redundancy: 1303896/714
> >> > 9898 objects degraded (18.237%), 218 pgs degraded, 316 pgs undersized;
> >> 144
> >> > pgs not deep-scrubbed in time; 459 pgs not sc
> >> > rubbed in time; 256 slow ops, oldest one blocked for 1508207 sec,
> osd.1
> >> has
> >> > slow ops; too many PGs per OSD (657 > max 50
> >> > 0)
> >> > [WRN] FS_DEGRADED: 1 filesystem is degraded
> >> > fs ark is degraded
> >> > [WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon
> >> > fs ark has 1 failed mds
> >> > [ERR] MDS_ALL_DOWN: 1 filesystem is offline
> >> > fs ark is offline because no MDS is active for it.
> >> > [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons
> >> available
> >> > have 0; want 1 more
> >> > [WRN] PG_AVAILABILITY: Reduced data availability: 143 pgs inactive,
> 143
> >> pgs
> >> > down
> >> > pg 10.11 is down, acting [1,3]
> >> > pg 10.18 is down, acting [3,1]
> >> > pg 10.1d is down, acting [1,3]
> >> > pg 10.1f is down, acting [1,3]
> >> > pg 11.10 is down, acting [3,1,NONE]
> >> > pg 11.12 is down, acting [1,NONE,3]
> >> > pg 11.18 is stuck inactive for 4w, current state down, last
> acting
> >> > [1,3,NONE]
> >> > pg 11.19 is down, acting [3,1,NONE]
> >> > pg 11.1b is down, acting [1,NONE,3]
> >> > pg 11.62 is down, acting [NONE,3,1]
> >> > pg 11.63 is down, acting [3,NONE,1]
> >> > pg 11.64 is down, acting [NONE,1,3]
> >> > pg 11.66 is down, acting [NONE,3,1]
> >> > pg 11.67 is down, acting [1,NONE,3]
> >> > pg 11.68 is down, acting [3,NONE,1]
> >> > pg 11.69 is down, acting [NONE,1,3]
> >> > pg 11.6a is down, acting [1,NONE,3]
> >> > pg 11.6b is down, acting [NONE,1,3]
> >> > pg 11.6f is down, acting [NONE,3,1]
> >> > pg 11.71 is down, acting [1,3,NONE]
> >> > pg 11.72 is down, acting [1,3,NONE]
> >> > pg 11.74 is down, acting [NONE,3,1]
> >> > pg 11.76 is down, acting [1,NONE,3]
> >> > pg 11.78 is down, acting [3,1,NONE]
> >> > pg 11.7d is down, acting [NONE,3,1]
> >> > pg 11.7e is down, acting [NONE,1,3]
> >> > pg 15.15 is down, acting [1,3]
> >> > pg 15.16 is down, acting [3,1]
> >> > pg 15.17 is down, acting [1,3]
> >> > pg 15.1a is down, acting [3,1]
> >> > pg 16.1 is down, acting [1,3,NONE]
> >> > pg 16.4 is down, acting [1,3,NONE]
> >> > pg 16.b is down, acting [3,NONE,1]
> >> > pg 16.60 is down, acting [3,1,NONE]
> >> > pg 16.61 is down, acting [3,1,NONE]
> >> > pg 16.62 is down, acting [3,NONE,1]
> >> > pg 16.63 is down, acting [3,NONE,1]
> >> > pg 16.65 is down, acting [NONE,3,1]
> >> > pg 16.67 is down, acting [1,NONE,3]
> >> > pg 16.68 is down, acting [1,NONE,3]
> >> > pg 16.69 is down, acting [3,1,NONE]
> >> > pg 16.6a is down, acting [1,3,NONE]
> >> > pg 16.6c is down, acting [1,3,NONE]
> >> > pg 16.70 is down, acting [3,NONE,1]
> >> > pg 16.73 is down, acting [3,NONE,1]
> >> > pg 16.74 is down, acting [1,3,NONE]
> >> > pg 16.75 is down, acting [3,1,NONE]
> >> > pg 16.79 is down, acting [3,NONE,1]
> >> > pg 16.7a is down, acting [1,3,NONE]
> >> > pg 16.7e is down, acting [1,3,NONE]
> >> > pg 16.7f is down, acting [3,NONE,1]
> >> > [WRN] PG_DEGRADED: Degraded data redundancy: 1303896/7149898 objects
> >> > degraded (18.237%), 218 pgs degraded, 316 pgs under
> >> > sized
> >> > pg 3.18 is stuck undersized for 36m, current state
> >> active+undersized,
> >> > last acting [1,3]
> >> > ...<snipped for brevity>
> >> > ###
> >> >
> >> > Once again, I cannot thank you enough for looking into my issue.
> >> > I have the impression that being able to recover the data I need is
> just
> >> > around the corner. Although the croit.io blog did mention flagging
> the
> >> osd
> >> > as lost, I would like to double check it to avoid losing any
> >> possibility to
> >> > recover the data.
> >> >
> >> > If there's anything further I could check or if you need full output
> of
> >> the
> >> > commands, let me know.
> >> >
> >> > Thanks in advance.
> >> >
> >> > On Tue, 3 Feb 2026 at 10:26, Igor Fedotov <igor.fedotov(a)croit.io>
> >> wrote:
> >> >
> >> >> Hi Theo,
> >> >>
> >> >> you might want to try to use PG export/import using
> >> ceph-objectstore-tool.
> >> >>
> >> >> Please find more details here
> >> >>
> >>
> https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor…
> >> >>
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Igor
> >> >> On 03/02/2026 02:38, Theo Cabrerizo Diem via ceph-users wrote:
> >> >>
> >> >> :12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd)
> >> >> fsck error: free extent 0x1714c521000~978b26df000 intersects
> >> allocatedblocks
> >> >> fsck status: remaining 1 error(s) and warning(s)
> >> >>
> >> >>
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users(a)ceph.io
> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >>
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1
0
I did do, myself, anything to the crush.
systemctl stop
ceph-9f4f9dba-72c7-11f0-8052-525400519d29(a)osd.0.service &&
lvresize -l 100%vg
/dev/ceph-cae223cb-c326-4560-b411-120bf5396fbb/osd-block-6efed8b4-6f4f-4f3f-8a1d-432f5ce38a52
&& ceph-bluestore-tool bluefs-bdev-expand --path
/var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.0/ ;
ceph-bluestore-tool bluefs-bdev-sizes --path
/var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.0/ &&
systemctl start
ceph-9f4f9dba-72c7-11f0-8052-525400519d29(a)osd.0.service
Without stopping - at least on my _cephadm_ deployed cluster
- the container, resizing with _ceph-bluestore-tool_ is not
possible, also, I did that with all disk-devices, all OSDs.
1
0
OSD.0 is 150 GB in size but its crush weight is only 0.09769 (10 GB).
And you didn't provide the history of commands how you extended
exactly the OSDs.
Zitat von listy via ceph-users <ceph-users(a)ceph.io>:
> -> $ ceph balancer status
> {
> "active": true,
> "last_optimize_duration": "0:00:00.001667",
> "last_optimize_started": "Mon Mar 2 04:08:35 2026",
> "mode": "upmap",
> "no_optimization_needed": true,
> "optimize_result": "Unable to find further optimization, or
> pool(s) pg_num is decreasing, or distribution is already perfect",
> "plans": []
> }
>
> -> $ ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 1.07428 root default
> -3 0.34180 host podster1
> 9 ssd 0.04880 osd.9 up 1.00000 1.00000
> 10 ssd 0.29300 osd.10 up 1.00000 1.00000
> -7 0.39069 host podster2
> 0 ssd 0.09769 osd.0 up 1.00000 1.00000
> 4 ssd 0.29300 osd.4 up 1.00000 1.00000
> -5 0.34180 host podster3
> 1 ssd 0.04880 osd.1 up 1.00000 1.00000
> 5 ssd 0.29300 osd.5 up 1.00000 1.00000
>
> -> $ ceph osd dump | grep ratio
> full_ratio 0.95
> backfillfull_ratio 0.9
> nearfull_ratio 0.85
>
> I've changed upmap_max_deviation = 1
> I also did:
> -> $ ceph osd reweight osd.0 1.0
> and ceased all clients' activity to FSes
>
> I've extend disk-devices ceph uses twice, for 'health' complained
> about 'backfillull' like here, only _pg_ was there, is is now below:
>
> -> $ ceph health detail
> HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add
> storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5
> pool(s) backfillfull
> [WRN] OSD_BACKFILLFULL: 1 backfillfull osd(s)
> osd.0 is backfill full
> [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if
> this doesn't resolve itself): 3 pgs backfill_toofull
> pg 3.1b is active+remapped+backfill_toofull, acting [4,5,10]
> pg 3.21 is active+remapped+backfill_toofull, acting [4,5,10]
> pg 3.23 is active+remapped+backfill_toofull, acting [10,5,4]
> [WRN] POOL_BACKFILLFULL: 5 pool(s) backfillfull
> pool '.mgr' is backfillfull
> pool 'cephfs.APKI.meta' is backfillfull
> pool 'cephfs.APKI.data' is backfillfull
> pool 'cephfs.MONERO.meta' is backfillfull
> pool 'cephfs.MONERO.data' is backfillfull
>
> Each time I "extended" devices-disks, cluster went to use osd.0 and
> fill it up.
> Yes everything is small in this cluster, it's a lab.
>
> -> $ ceph -w
> cluster:
> id: 9f4f9dba-72c7-11f0-8052-525400519d29
> health: HEALTH_WARN
> 1 backfillfull osd(s)
> Low space hindering backfill (add storage if this
> doesn't resolve itself): 3 pgs backfill_toofull
> 5 pool(s) backfillfull
>
> services:
> mon: 3 daemons, quorum podster3,podster2,podster1 (age 2d)
> [leader: podster3]
> mgr: podster1.qzojrl(active, since 2d), standbys: podster3.kyyolr
> mds: 2/2 daemons up, 2 standby
> osd: 6 osds: 6 up (since 12h), 6 in (since 2d); 3 remapped pgs
>
> data:
> volumes: 2/2 healthy
> pools: 5 pools, 289 pgs
> objects: 102.74k objects, 355 GiB
> usage: 1.1 TiB used, 523 GiB / 1.6 TiB avail
> pgs: 737/308229 objects misplaced (0.239%)
> 286 active+clean
> 3 active+remapped+backfill_toofull
>
>
> 2026-03-02T04:10:00.000125+0000 mon.podster3 [WRN] overall
> HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add
> storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5
> pool(s) backfillfull
> 2026-03-02T04:20:00.000091+0000 mon.podster3 [WRN] overall
> HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add
> storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5
> pool(s) backfillfull
> 2026-03-02T04:26:38.174003+0000 mon.podster3 [WRN] Health check
> failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)
> 2026-03-02T04:26:44.210790+0000 mon.podster3 [INF] Health check
> cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg
> peering)
> 2026-03-02T04:30:00.000140+0000 mon.podster3 [WRN] overall
> HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add
> storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5
> pool(s) backfillfull
> 2026-03-02T04:40:00.000118+0000 mon.podster3 [WRN] overall
> HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add
> storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5
> pool(s) backfillfull
> 2026-03-02T04:50:00.000114+0000 mon.podster3 [WRN] overall
> HEALTH_WARN 1 backfillfull osd(s); Low space hindering backfill (add
> storage if this doesn't resolve itself): 3 pgs backfill_toofull; 5
> pool(s) backfillfull
>
> If the cluster is doing something good, something it should be doing
> to heal, it's barely happening so slow it is - given no clients work
> now + as you said, storage capacities are minute in comparison to
> anything production.
> 7 hours later, still:
> -> $ ceph osd df tree | egrep '(osd.0|ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS TYPE NAME
> 0 ssd 0.09769 1.00000 150 GiB 140 GiB 88 GiB 1.0 MiB 2.0
> GiB 10 GiB 93.29 1.37 72 up osd.0
>
> Now 5 hours later - since i started this draft - I added:
> -> $ ceph osd reweight-by-utilization
> and before that cmd above I also noticed:
> -> $ ceph config get mgr mgr/balancer/begin_weekday
>
> -> $ ceph config get mgr mgr/balancer/end_weekday
>
> which was done by 'deployment' process - cephadm bootstrap - and
> made me wonder:
> does that mean that auto-rebalance runs only on Sunday?
> I changed: end_weekday = 6
>
> _reweight-by-utilization_ I notice, changed REWEIGHT for osd.0 and
> that did something, I think.
> So now _active+remapped+backfill_toofull_ are gone from 'pcs' part
> of health report.
> RAW USE & DATA are down, but stil:
> -> $ ceph osd df tree | egrep '(osd.0|ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS TYPE NAME
> 0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB 1.1 MiB 2.0
> GiB 16 GiB 89.51 1.31 68 up osd.0
> and when compared to other host-ods which use "identical" disk-drives:
> -> $ ceph osd df tree | egrep '(osd\.[0,1,9] |ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS TYPE NAME
> 9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB 624 KiB 1.4
> GiB 98 GiB 34.83 0.51 43 up osd.9
> 0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB 1.1 MiB 2.0
> GiB 16 GiB 89.51 1.31 68 up osd.0
> 1 ssd 0.04880 1.00000 150 GiB 53 GiB 53 GiB 526 KiB 254
> MiB 97 GiB 35.40 0.52 44 up osd.1
>
> Perhaps cluster goes only as far as to satisfy _backfill_toofull_ be
> gone and then "gives up"?
> The "other" disk-drives:
> -> $ ceph osd df tree | egrep '(osd\.(5|4|10)\ |ID)'
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS TYPE NAME
> 10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 2.0 MiB 2.1
> GiB 93 GiB 76.68 1.12 246 up osd.10
> 4 ssd 0.29300 1.00000 400 GiB 275 GiB 273 GiB 3.2 MiB 2.1
> GiB 125 GiB 68.83 1.01 221 up osd.4
> 5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB 3.0 MiB 2.5
> GiB 95 GiB 76.27 1.12 245 up osd.5
>
> Seems that _host podster2_ balances its osds 4 & 0 "differently" to
> what other two hosts do - if so then why?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0
-> $ ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:00.001667",
"last_optimize_started": "Mon Mar 2 04:08:35 2026",
"mode": "upmap",
"no_optimization_needed": true,
"optimize_result": "Unable to find further
optimization, or pool(s) pg_num is decreasing, or
distribution is already perfect",
"plans": []
}
-> $ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.07428 root default
-3 0.34180 host podster1
9 ssd 0.04880 osd.9 up 1.00000
1.00000
10 ssd 0.29300 osd.10 up 1.00000 1.00000
-7 0.39069 host podster2
0 ssd 0.09769 osd.0 up 1.00000
1.00000
4 ssd 0.29300 osd.4 up 1.00000
1.00000
-5 0.34180 host podster3
1 ssd 0.04880 osd.1 up 1.00000
1.00000
5 ssd 0.29300 osd.5 up 1.00000
1.00000
-> $ ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
I've changed upmap_max_deviation = 1
I also did:
-> $ ceph osd reweight osd.0 1.0
and ceased all clients' activity to FSes
I've extend disk-devices ceph uses twice, for 'health'
complained about 'backfillull' like here, only _pg_ was
there, is is now below:
-> $ ceph health detail
HEALTH_WARN 1 backfillfull osd(s); Low space hindering
backfill (add storage if this doesn't resolve itself): 3 pgs
backfill_toofull; 5 pool(s) backfillfull
[WRN] OSD_BACKFILLFULL: 1 backfillfull osd(s)
osd.0 is backfill full
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add
storage if this doesn't resolve itself): 3 pgs backfill_toofull
pg 3.1b is active+remapped+backfill_toofull, acting
[4,5,10]
pg 3.21 is active+remapped+backfill_toofull, acting
[4,5,10]
pg 3.23 is active+remapped+backfill_toofull, acting
[10,5,4]
[WRN] POOL_BACKFILLFULL: 5 pool(s) backfillfull
pool '.mgr' is backfillfull
pool 'cephfs.APKI.meta' is backfillfull
pool 'cephfs.APKI.data' is backfillfull
pool 'cephfs.MONERO.meta' is backfillfull
pool 'cephfs.MONERO.data' is backfillfull
Each time I "extended" devices-disks, cluster went to use
osd.0 and fill it up.
Yes everything is small in this cluster, it's a lab.
-> $ ceph -w
cluster:
id: 9f4f9dba-72c7-11f0-8052-525400519d29
health: HEALTH_WARN
1 backfillfull osd(s)
Low space hindering backfill (add storage if
this doesn't resolve itself): 3 pgs backfill_toofull
5 pool(s) backfillfull
services:
mon: 3 daemons, quorum podster3,podster2,podster1 (age
2d) [leader: podster3]
mgr: podster1.qzojrl(active, since 2d), standbys:
podster3.kyyolr
mds: 2/2 daemons up, 2 standby
osd: 6 osds: 6 up (since 12h), 6 in (since 2d); 3
remapped pgs
data:
volumes: 2/2 healthy
pools: 5 pools, 289 pgs
objects: 102.74k objects, 355 GiB
usage: 1.1 TiB used, 523 GiB / 1.6 TiB avail
pgs: 737/308229 objects misplaced (0.239%)
286 active+clean
3 active+remapped+backfill_toofull
2026-03-02T04:10:00.000125+0000 mon.podster3 [WRN] overall
HEALTH_WARN 1 backfillfull osd(s); Low space hindering
backfill (add storage if this doesn't resolve itself): 3 pgs
backfill_toofull; 5 pool(s) backfillfull
2026-03-02T04:20:00.000091+0000 mon.podster3 [WRN] overall
HEALTH_WARN 1 backfillfull osd(s); Low space hindering
backfill (add storage if this doesn't resolve itself): 3 pgs
backfill_toofull; 5 pool(s) backfillfull
2026-03-02T04:26:38.174003+0000 mon.podster3 [WRN] Health
check failed: Reduced data availability: 1 pg peering
(PG_AVAILABILITY)
2026-03-02T04:26:44.210790+0000 mon.podster3 [INF] Health
check cleared: PG_AVAILABILITY (was: Reduced data
availability: 1 pg peering)
2026-03-02T04:30:00.000140+0000 mon.podster3 [WRN] overall
HEALTH_WARN 1 backfillfull osd(s); Low space hindering
backfill (add storage if this doesn't resolve itself): 3 pgs
backfill_toofull; 5 pool(s) backfillfull
2026-03-02T04:40:00.000118+0000 mon.podster3 [WRN] overall
HEALTH_WARN 1 backfillfull osd(s); Low space hindering
backfill (add storage if this doesn't resolve itself): 3 pgs
backfill_toofull; 5 pool(s) backfillfull
2026-03-02T04:50:00.000114+0000 mon.podster3 [WRN] overall
HEALTH_WARN 1 backfillfull osd(s); Low space hindering
backfill (add storage if this doesn't resolve itself): 3 pgs
backfill_toofull; 5 pool(s) backfillfull
If the cluster is doing something good, something it should
be doing to heal, it's barely happening so slow it is -
given no clients work now + as you said, storage capacities
are minute in comparison to anything production.
7 hours later, still:
-> $ ceph osd df tree | egrep '(osd.0|ID)'
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
0 ssd 0.09769 1.00000 150 GiB 140 GiB 88 GiB
1.0 MiB 2.0 GiB 10 GiB 93.29 1.37 72 up
osd.0
Now 5 hours later - since i started this draft - I added:
-> $ ceph osd reweight-by-utilization
and before that cmd above I also noticed:
-> $ ceph config get mgr mgr/balancer/begin_weekday
0
-> $ ceph config get mgr mgr/balancer/end_weekday
0
which was done by 'deployment' process - cephadm bootstrap -
and made me wonder:
does that mean that auto-rebalance runs only on Sunday?
I changed: end_weekday = 6
_reweight-by-utilization_ I notice, changed REWEIGHT for
osd.0 and that did something, I think.
So now _active+remapped+backfill_toofull_ are gone from
'pcs' part of health report.
RAW USE & DATA are down, but stil:
-> $ ceph osd df tree | egrep '(osd.0|ID)'
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB
1.1 MiB 2.0 GiB 16 GiB 89.51 1.31 68 up
osd.0
and when compared to other host-ods which use "identical"
disk-drives:
-> $ ceph osd df tree | egrep '(osd\.[0,1,9] |ID)'
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB
624 KiB 1.4 GiB 98 GiB 34.83 0.51 43 up
osd.9
0 ssd 0.09769 0.90002 150 GiB 134 GiB 82 GiB
1.1 MiB 2.0 GiB 16 GiB 89.51 1.31 68 up
osd.0
1 ssd 0.04880 1.00000 150 GiB 53 GiB 53 GiB
526 KiB 254 MiB 97 GiB 35.40 0.52 44 up
osd.1
Perhaps cluster goes only as far as to satisfy
_backfill_toofull_ be gone and then "gives up"?
The "other" disk-drives:
-> $ ceph osd df tree | egrep '(osd\.(5|4|10)\ |ID)'
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 2.0
MiB 2.1 GiB 93 GiB 76.68 1.12 246 up osd.10
4 ssd 0.29300 1.00000 400 GiB 275 GiB 273 GiB
3.2 MiB 2.1 GiB 125 GiB 68.83 1.01 221 up
osd.4
5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB
3.0 MiB 2.5 GiB 95 GiB 76.27 1.12 245 up
osd.5
Seems that _host podster2_ balances its osds 4 & 0
"differently" to what other two hosts do - if so then why?
1
0
> On Mar 1, 2026, at 7:58 AM, listy via ceph-users <ceph-users(a)ceph.io> wrote:
>
> Hi guys.
>
> I've extended capacity sizes of my drives
How? Are they virtual drives of some sort? VMDKs?
> - then used 'bluefs-bdev-expand' to tell it to _ceph_, but it seems pools/fs did not pick those up - should they not?
Please share exactly what you did. Compare your `ceph osd df` output from before and after, did the SIZE and/or WEIGHT columns change?
> -> $ ceph osd df
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
> 9 ssd 0.04880 1.00000 150 GiB 52 GiB 51 GiB 506 KiB 1.4 GiB 98 GiB 34.86 0.51 45 up
> 10 ssd 0.29300 1.00000 400 GiB 307 GiB 305 GiB 1.5 MiB 2.0 GiB 93 GiB 76.67 1.12 244 up
> 0 ssd 0.09769 1.00000 150 GiB 143 GiB 91 GiB 835 KiB 2.0 GiB 7.5 GiB 95.02 1.39 78 up
> 4 ssd 0.29300 1.00000 400 GiB 267 GiB 265 GiB 1.1 MiB 2.1 GiB 133 GiB 66.77 0.98 211 up
> 1 ssd 0.04880 1.00000 150 GiB 54 GiB 53 GiB 392 KiB 1.1 GiB 96 GiB 36.00 0.53 47 up
> 5 ssd 0.29300 1.00000 400 GiB 305 GiB 303 GiB 1.6 MiB 2.5 GiB 95 GiB 76.26 1.12 242 up
> TOTAL 1.6 TiB 1.1 TiB 1.0 TiB 5.9 MiB 11 GiB 522 GiB 68.34
> MIN/MAX VAR: 0.51/1.39 STDDEV: 22.41
These are extremely small OSDs, and I’ve seen reports of such filling up before one would think they would, perhaps due to rounding or various math not taking metadata into account.
Also, these are very imbalanced, which doesn’t help.
Please share `ceph balancer status` and `ceph osd tree`.
Ensure that you have this set, which may help:
mgr advanced mgr/balancer/upmap_max_deviation 1
> -> $ rados df
> POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
> .mgr 1.3 MiB 2 0 6 0 0 0 13951 35 MiB 5271 54 MiB 0 B 0 B
> cephfs.APKI.data 329 GiB 31352 0 94056 0 0 0 153508700 219 TiB 136896977 2.4 TiB 0 B 0 B
> cephfs.APKI.meta 723 MiB 8513 0 25539 0 0 0 267014 11 GiB 17205610 69 GiB 0 B 0 B
> cephfs.MONERO.data 736 GiB 62784 0 188352 0 0 0 6553814758 29 TiB 1776390595 8.7 TiB 0 B 0 B
> cephfs.MONERO.meta 776 MiB 92 0 276 0 0 0 21866 22 GiB 4219726 17 GiB 0 B 0 B
>
> total_objects 102743
> total_used 1.1 TiB
> total_avail 522 GiB
> total_space 1.6 TiB
>
And what do you mean exactly by "it seems pools/fs did not pick those up “? I suspect you mean in terms of MAX AVAIL. “Ceph” picks up capacity if it shows in the RAW STORAGE numbers from `ceph df`. We don’t have the before output, so we can’t tell.
Assuming this is what you mean, this is a commonly misundertood nuance. MAX AVAIL is a function of the delta between how full the MOST full OSD of the pool’s device class is vs the fullness ratios.
# ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.90
nearfull_ratio 0.85
Your numbers may look different from these, which I think are the defaults.
Notice your osd.0 is 95% full. I should think that `ceph status`, which you didn’t include in your message, has clues.
So for your immediate question, I think if you fix your balancing issue you’ll see a bit of space show under MAX AVAIL.
> -> $ ceph df
> --- RAW STORAGE ---
> CLASS SIZE AVAIL USED RAW USED %RAW USED
> ssd 1.6 TiB 522 GiB 1.1 TiB 1.1 TiB 68.34
> TOTAL 1.6 TiB 522 GiB 1.1 TiB 1.1 TiB 68.34
>
> --- POOLS ---
> POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
> .mgr 1 1 449 KiB 2 1.3 MiB 100.00 0 B
> cephfs.APKI.meta 2 16 241 MiB 8.51k 723 MiB 100.00 0 B
> cephfs.APKI.data 3 128 110 GiB 31.35k 329 GiB 100.00 0 B
> cephfs.MONERO.meta 4 16 259 MiB 92 776 MiB 100.00 0 B
> cephfs.MONERO.data 5 128 245 GiB 62.78k 736 GiB 100.00 0 B
>
> I do not use quotas, no subvols.
> I must be missing something obvious - right? - how to tell ceph to use that newly added capacity?
> many thanks, L.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0