22 Sep
2020
22 Sep
'20
10:02 p.m.
Hi,
this morning I woke up to a degraded test ceph cluster (managed by rook,
but it does not really change anything for the question I'm about to ask).
After checking logs I have found that bluestore on one of the OSDs run out
of space.
Some cluster details:
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus
(stable)
it runs on 3 little OSDs 10Gb each
`ceph osd df` returned RAW USE of about 4.5GB on every node, happily
reporting about 5.5GB of AVAIL.
Yet:
debug -9> 2020-09-22T20:23:15.421+0000 7f29e9798f40 4 rocksdb:
EVENT_LOG_v1 {"time_micros": 1600806195424423, "job": 1,
"event":
"recovery_started", "log_files": [347, 350]}
debug -8> 2020-09-22T20:23:15.421+0000 7f29e9798f40 4 rocksdb:
[db/db_impl_open.cc:583] Recovering log #347 mode 0
debug -7> 2020-09-22T20:23:16.465+0000 7f29e9798f40 4 rocksdb:
[db/db_impl_open.cc:583] Recovering log #350 mode 0
debug -6> 2020-09-22T20:23:18.689+0000 7f29e9798f40 1 bluefs _allocate
failed to allocate 0x17a2360 on bdev 1, free 0x390000; fallback to bdev 2
debug -5> 2020-09-22T20:23:18.689+0000 7f29e9798f40 1 bluefs _allocate
unable to allocate 0x17a2360 on bdev 2, free 0xffffffffffffffff; fallback
to slow device expander
debug -4> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1
bluestore(/var/lib/ceph/osd/ceph-0) allocate_bluefs_freespace failed to
allocate on 0x39a20000 min_size 0x17b0000 > allocated total 0x6250000
bluefs_shared_alloc_size 0x10000 allocated 0x0 available 0x 12ee32000
debug -3> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluefs _allocate
failed to expand slow device to fit +0x17a2360
debug -2> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluefs
_flush_range allocated: 0x0 offset: 0x0 length: 0x17a2360
debug -1> 2020-09-22T20:23:18.693+0000 7f29e9798f40 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4/rpm/el8/BUILD/ceph-15.2.4/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f29e9798f40 time 2020-09-22T20:23:18.690014+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4/rpm/el8/BUILD/ceph-15.2.4/src/os/bluestore/BlueFS.cc:
2696: ceph_abort_msg("bluefs enospc")
So, my question would be: how could I have prevented that? From monitoring
I have (prometheus) - OSDs are healthy, have plenty of space, yet they are
not.
What command (and prometheus metric) would help me understand the actual
real bluestore use? Or am I missing something?
Oh, and I "fixed" the cluster by expanding the broken osd.0 with a larger
15GB volume. And 2 other OSDs still run on 10GB volumes.
Thanks in advance for any thoughts.
--
With best regards, Ivan Kurnosov
22 Sep
22 Sep
11:31 p.m.
As far as I know, bluestore doesn't like super small sizes. Normally odd
should stop doing funny things as full mark, but if device is too small it
may be too late and bluefs run out of space.
Two things:
1. Don't use too small osd
2. Have a spare area on the drive. I usually reserve 1% for emergency
extension (and to give ssd firmware a bit if space to breath).
On Wed, Sep 23, 2020, 01:03 Ivan Kurnosov <zerkms(a)zerkms.com> wrote:
Hi,
this morning I woke up to a degraded test ceph cluster (managed by rook,
but it does not really change anything for the question I'm about to ask).
After checking logs I have found that bluestore on one of the OSDs run out
of space.
Some cluster details:
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus
(stable)
it runs on 3 little OSDs 10Gb each
`ceph osd df` returned RAW USE of about 4.5GB on every node, happily
reporting about 5.5GB of AVAIL.
Yet:
...
So, my question would be: how could I have prevented that? From monitoring
I have (prometheus) - OSDs are healthy, have plenty of space, yet they are
not.
What command (and prometheus metric) would help me understand the actual
real bluestore use? Or am I missing something?
Oh, and I "fixed" the cluster by expanding the broken osd.0 with a larger
15GB volume. And 2 other OSDs still run on 10GB volumes.
Thanks in advance for any thoughts.
--
With best regards, Ivan Kurnosov
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Sep
23 Sep
12:27 a.m.
You can also expand the OSD. ceph-bluestore-tool has an option for expansion of the OSD.
I'm not 100% sure if that would solve the rockdb out of space issue. I think it will,
though. If not, you can move rockdb to a separate block device.
September 22, 2020 7:31 PM, "George Shuklin" <george.shuklin(a)gmail.com>
wrote:
> As far as I know, bluestore doesn't like super small sizes. Normally odd
> should stop doing funny things as full mark, but if device is too small it
> may be too late and bluefs run out of space.
>
> Two things:
> 1. Don't use too small osd
> 2. Have a spare area on the drive. I usually reserve 1% for emergency
> extension (and to give ssd firmware a bit if space to breath).
>
> On Wed, Sep 23, 2020, 01:03 Ivan Kurnosov <zerkms(a)zerkms.com> wrote:
>
>> Hi,
>>
>> this morning I woke up to a degraded test ceph cluster (managed by rook,
>> but it does not really change anything for the question I'm about to ask).
>>
>> After checking logs I have found that bluestore on one of the OSDs run out
>> of space.
>>
>> Some cluster details:
>>
>> ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus
>> (stable)
>> it runs on 3 little OSDs 10Gb each
>>
>> `ceph osd df` returned RAW USE of about 4.5GB on every node, happily
>> reporting about 5.5GB of AVAIL.
>>
>> Yet:
>>
>> ...
>> So, my question would be: how could I have prevented that? From monitoring
>> I have (prometheus) - OSDs are healthy, have plenty of space, yet they are
>> not.
>>
>> What command (and prometheus metric) would help me understand the actual
>> real bluestore use? Or am I missing something?
>>
>> Oh, and I "fixed" the cluster by expanding the broken osd.0 with a
larger
>> 15GB volume. And 2 other OSDs still run on 10GB volumes.
>>
>> Thanks in advance for any thoughts.
>>
>> --
>> With best regards, Ivan Kurnosov
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:09 a.m.
On Wed, Sep 23, 2020 at 3:03 AM Ivan Kurnosov <zerkms(a)zerkms.com> wrote:
Hi,
this morning I woke up to a degraded test ceph cluster (managed by rook,
but it does not really change anything for the question I'm about to ask).
After checking logs I have found that bluestore on one of the OSDs run out
of space.
I think this is a consequence, and the real error is something else
that happened before.
The problem is that, if the cluster is unhealthy, the MON storage
accumulates a lot of osdmaps and pgmaps, and is not cleaned up
automatically, because the MONs think that these old versions might be
needed. And OSDs also get a copy of these osdmaps and pgmaps, if I
understand correctly, that's why small OSDs get quickly filled up if
the cluster stays unhealthy for a few hours.
So, my question would be: how could I have prevented
that? From monitoring
I have (prometheus) - OSDs are healthy, have plenty of space, yet they are
not.
What command (and prometheus metric) would help me understand the actual
real bluestore use? Or am I missing something?
You can fix monitoring by setting the "mon data size warn" to
something like 1 GB or even less.
Oh, and I "fixed" the cluster by expanding
the broken osd.0 with a larger
15GB volume. And 2 other OSDs still run on 10GB volumes.
Sometimes this doesn't help. For data recovery purposes, the most
helpful step if you get the "bluefs enospc" error is to add a separate
db device, like this:
systemctl disable --now ceph-osd@${OSDID}
truncate -s 32G /junk/osd.${OSDID}-recover/block.db
sgdisk -n 0:0:0 /junk/osd.${OSDID}-recover/block.db
ceph-bluestore-tool \
bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-${OSDID} \
--dev-target /junk/osd.${OSDID}-recover/block.db \
--bluestore-block-db-size=31G --bluefs-log-compact-min-size=31G
Of course you can use a real block device instead of just a file.
After that, export all PGs using ceph-objecttstore-tool and re-import
into a fresh OSD, then destroy or purge the full one.
Here is why the options:
--bluestore-block-db-size=31G: ceph-bluestore-tool refuses to do
anything if this option is not set to any value
--bluefs-log-compact-min-size=31G: make absolutely sure that log
compaction doesn't happen, because it would hit "bluefs enospc" again.
--
Alexander E. Patrakov
CV: http://pc.cd/PLz7
10:52 a.m.
On 23/09/2020 04:09, Alexander E. Patrakov wrote:
Sometimes this doesn't help. For data recovery purposes, the most
helpful step if you get the "bluefs enospc" error is to add a separate
db device, like this:
systemctl disable --now ceph-osd@${OSDID}
truncate -s 32G /junk/osd.${OSDID}-recover/block.db
sgdisk -n 0:0:0 /junk/osd.${OSDID}-recover/block.db
ceph-bluestore-tool \
bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-${OSDID} \
--dev-target /junk/osd.${OSDID}-recover/block.db \
--bluestore-block-db-size=31G --bluefs-log-compact-min-size=31G
Of course you can use a real block device instead of just a file.
After that, export all PGs using ceph-objecttstore-tool and re-import
into a fresh OSD, then destroy or purge the full one.
Here is why the options:
--bluestore-block-db-size=31G: ceph-bluestore-tool refuses to do
anything if this option is not set to any value
--bluefs-log-compact-min-size=31G: make absolutely sure that log
compaction doesn't happen, because it would hit "bluefs enospc" again.
Oh, you went this way... I solved my 'pocket ceph' needs by exporting
disk images (from files) via iscsi and mounting them back to localhost.
That gives me a perfect 'scsi' devices which work exactly as in
production. I have a little playbook (iscsi_loopback) to setup it on
random scrap (including VMs) for development purposes. After iscsi is
loopback-mounted, all other code works exactly the same as it would in
production.
I've got this issue few times on small 10GB osds, so I moved to 15Gb and
it become a less often. I never have had this in real-hardware tests
with real disk sizes (>>100G per OSD).
1322
days inactive
1323
days old
4 comments
4 participants
participants (4)
-
Alexander E. Patrakov -
George Shuklin -
Ivan Kurnosov -
tri＠postix.net