Here's a different cluster we upgraded luminous -> nautilus in October: 2020-10-14 13:22:51.860 7f78e3d20a80 0 ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable), process ceph-osd, pid 119714 ... 2020-10-14 13:27:50.368 7f78e3d20a80 1 bluestore(/var/lib/ceph/osd/ceph-136) _fsck_on_open checking shared_blobs 2020-10-14 13:27:50.368 7f78e3d20a80 1 bluestore(/var/lib/ceph/osd/ceph-136) _fsck_on_open checking pool_statfs 2020-10-14 13:27:50.368 7f78e3d20a80 -1 bluestore(/var/lib/ceph/osd/ceph-136) fsck error: legacy statfs record found, removing 2020-10-14 13:27:50.368 7f78e3d20a80 -1 bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS record for pool 1 2020-10-14 13:27:50.368 7f78e3d20a80 -1 bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS record for pool 2 2020-10-14 13:27:50.368 7f78e3d20a80 -1 bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS record for pool 5 2020-10-14 13:27:50.368 7f78e3d20a80 -1 bluestore(/var/lib/ceph/osd/ceph-136) fsck error: missing Pool StatFS record for pool ffffffffffffffff and ceph df right now showing stored == used in the pool stats: # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.2 PiB 2.3 PiB 842 TiB 843 TiB 26.01 TOTAL 3.2 PiB 2.3 PiB 842 TiB 843 TiB 26.00 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 252 TiB 270.90M 252 TiB 20.48 326 TiB cephfs_metadata 2 72 GiB 22.03M 72 GiB 0 326 TiB test 5 9.3 GiB 10.96k 9.3 GiB 0 326 TiB Cheers, Dan On Thu, Nov 26, 2020 at 6:00 PM Dan van der Ster <dan@vanderster.com> wrote:

Hi Igor,

No BLUESTORE_LEGACY_STATFS warning, and bluestore_warn_on_legacy_statfs is the default true on this (and all) clusters. I'm quite sure we did the statfs conversion during one of the recent upgrades (I forget which one exactly).

# ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true #

Is there a command to see the statfs reported by an individual OSD ? We have a mix of ~year old and recently recreated OSDs, so I could try to see if they differ.

Thanks!

Dan

On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Hi Dan

don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced by bluestore_warn_on_legacy_statfs param) for the older cluster?

Thanks,

Igor

On 11/26/2020 7:29 PM, Dan van der Ster wrote:

...

Hi,

Depending on which cluster I look at (all running v14.2.11), the bytes_used is reporting raw space or stored bytes variably.

Here's a 7 year old cluster:

# ceph df -f json | jq .pools[0] { "name": "volumes", "id": 4, "stats": { "stored": 1229308190855881, "objects": 294401604, "kb_used": 1200496280133, "bytes_used": 1229308190855881, "percent_used": 0.4401889145374298, "max_avail": 521125025021952 } }

Note that stored == bytes_used for that pool. (this is a 3x replica pool).

But here's a newer cluster (installed recently with nautilus)

# ceph df -f json | jq .pools[0] { "name": "volumes", "id": 1, "stats": { "stored": 680977600893041, "objects": 163155803, "kb_used": 1995736271829, "bytes_used": 2043633942351985, "percent_used": 0.23379847407341003, "max_avail": 2232457428467712 } }

In the second cluster, bytes_used is 3x stored.

Does anyone know why these are not reported consistently? Noticing this just now, I'll update our monitoring to plot stored rather than bytes_used from now on.

Thanks!

Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

0. OK, not sure this is informative: 2020-11-26 19:40:01.726 7fd10d09b700 20 bluestore(/var/lib/ceph/osd/ceph-252) statfs store_statfs(0x18391370000/0x227b00000/0x74702800000, data 0x5c0ef8b8393/0x5c149980000, compress 0x0/0x0/0x0, omap 0x9163, meta 0x227af6e9d) 2020-11-26 19:40:02.078 7fd0fe87e700 20 bluestore(/var/lib/ceph/osd/ceph-252) pool_statfs pool 68 2020-11-26 19:40:02.078 7fd0fe87e700 10 bluestore(/var/lib/ceph/osd/ceph-252) pool_statfsstore_statfs(0x0/0x0/0x0, data 0x5c090850000/0x5c0e7fa0000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) 2020-11-26 19:40:02.078 7fd0fe87e700 20 bluestore(/var/lib/ceph/osd/ceph-252) pool_statfs pool 72 2020-11-26 19:40:02.078 7fd0fe87e700 10 bluestore(/var/lib/ceph/osd/ceph-252) pool_statfsstore_statfs(0x0/0x0/0x0, data 0x4cc00018/0x4cc10000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) 2020-11-26 19:40:02.768 7fd10d09b700 20 bluestore(/var/lib/ceph/osd/ceph-252) statfs store_statfs(0x18391370000/0x227b00000/0x74702800000, data 0x5c0ef8b8393/0x5c149980000, compress 0x0/0x0/0x0, omap 0x9163, meta 0x227af6e9d) 2020-11-26 19:40:03.728 7fd10d09b700 20 bluestore(/var/lib/ceph/osd/ceph-252) statfs store_statfs(0x18391370000/0x227b00000/0x74702800000, data 0x5c0ef8b8393/0x5c149980000, compress 0x0/0x0/0x0, omap 0x9163, meta 0x227af6e9d) 1. Anyway, I found something weird... I created a new 1-PG pool "foo" on a different cluster and wrote some data to it. The stored and used are equal. Thu 26 Nov 19:26:58 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB But I tried restarting the relevant three OSDs, and the bytes_used are temporarily reported correctly: Thu 26 Nov 19:27:00 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB But then a few seconds later it's back to used == stored: Thu 26 Nov 19:27:03 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB It seems to report the correct stats only when the PG is peering (so some other transition state). I've restarted all three relevant OSDs now -- the stats are reported as stored == used. 2. Another data point -- I found another old cluster that reports stored/used correctly. I have no idea what might be different about that cluster -- we updated it just like the others. Cheers, Dan On Thu, Nov 26, 2020 at 6:22 PM Igor Fedotov <ifedotov@suse.de> wrote:

For specific BlueStore instance you can learn relevant statfs output by

setting debug_bluestore to 20 and leaving OSD for 5-10 seconds (or may be a couple of minutes - don't remember exact statsfs poll period ).

Then grep osd log for "statfs" and/or "pool_statfs" and get the output formatted as per the following operator (taken from src/osd/osd_types.cc):

ostream& operator<<(ostream& out, const store_statfs_t &s) { out << std::hex << "store_statfs(0x" << s.available << "/0x" << s.internally_reserved << "/0x" << s.total << ", data 0x" << s.data_stored << "/0x" << s.allocated << ", compress 0x" << s.data_compressed << "/0x" << s.data_compressed_allocated << "/0x" << s.data_compressed_original << ", omap 0x" << s.omap_allocated << ", meta 0x" << s.internal_metadata << std::dec << ")"; return out; }

But honestly I doubt this is BlueStore which reports incorrectly since it doesn't care about replication.

It rather looks like lack of stats from some replicas or improper pg replica factor processing...

Perhaps legacy vs. new pool what matters... Can you try to create a new pool at old cluster and fill it with some data (e.g. just a single 64K object) and check the stats?

Thanks,

Igor

On 11/26/2020 8:00 PM, Dan van der Ster wrote:

...

Hi Igor,

No BLUESTORE_LEGACY_STATFS warning, and bluestore_warn_on_legacy_statfs is the default true on this (and all) clusters. I'm quite sure we did the statfs conversion during one of the recent upgrades (I forget which one exactly).

# ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true #

Is there a command to see the statfs reported by an individual OSD ? We have a mix of ~year old and recently recreated OSDs, so I could try to see if they differ.

Thanks!

Dan

On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Hi Dan

don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced by bluestore_warn_on_legacy_statfs param) for the older cluster?

Thanks,

Igor

On 11/26/2020 7:29 PM, Dan van der Ster wrote:

...

Hi,

Depending on which cluster I look at (all running v14.2.11), the bytes_used is reporting raw space or stored bytes variably.

Here's a 7 year old cluster:

# ceph df -f json | jq .pools[0] { "name": "volumes", "id": 4, "stats": { "stored": 1229308190855881, "objects": 294401604, "kb_used": 1200496280133, "bytes_used": 1229308190855881, "percent_used": 0.4401889145374298, "max_avail": 521125025021952 } }

Note that stored == bytes_used for that pool. (this is a 3x replica pool).

But here's a newer cluster (installed recently with nautilus)

# ceph df -f json | jq .pools[0] { "name": "volumes", "id": 1, "stats": { "stored": 680977600893041, "objects": 163155803, "kb_used": 1995736271829, "bytes_used": 2043633942351985, "percent_used": 0.23379847407341003, "max_avail": 2232457428467712 } }

In the second cluster, bytes_used is 3x stored.

Does anyone know why these are not reported consistently? Noticing this just now, I'll update our monitoring to plot stored rather than bytes_used from now on.

Thanks!

Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

There are a couple gaps, yes: https://termbin.com/9mx1 What should I do? -- dan On Thu, Nov 26, 2020 at 7:52 PM Igor Fedotov <ifedotov@suse.de> wrote:

Does "ceph osd df tree" show stats properly (I mean there are no evident gaps like unexpected zero values) for all the daemons?

...

1. Anyway, I found something weird...

I created a new 1-PG pool "foo" on a different cluster and wrote some data to it.

The stored and used are equal.

Thu 26 Nov 19:26:58 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

But I tried restarting the relevant three OSDs, and the bytes_used are temporarily reported correctly:

Thu 26 Nov 19:27:00 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB

But then a few seconds later it's back to used == stored:

Thu 26 Nov 19:27:03 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

It seems to report the correct stats only when the PG is peering (so some other transition state). I've restarted all three relevant OSDs now -- the stats are reported as stored == used.

2. Another data point -- I found another old cluster that reports stored/used correctly. I have no idea what might be different about that cluster -- we updated it just like the others.

Cheers, Dan

On Thu, Nov 26, 2020 at 6:22 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

For specific BlueStore instance you can learn relevant statfs output by

setting debug_bluestore to 20 and leaving OSD for 5-10 seconds (or may be a couple of minutes - don't remember exact statsfs poll period ).

Then grep osd log for "statfs" and/or "pool_statfs" and get the output formatted as per the following operator (taken from src/osd/osd_types.cc):

ostream& operator<<(ostream& out, const store_statfs_t &s) { out << std::hex << "store_statfs(0x" << s.available << "/0x" << s.internally_reserved << "/0x" << s.total << ", data 0x" << s.data_stored << "/0x" << s.allocated << ", compress 0x" << s.data_compressed << "/0x" << s.data_compressed_allocated << "/0x" << s.data_compressed_original << ", omap 0x" << s.omap_allocated << ", meta 0x" << s.internal_metadata << std::dec << ")"; return out; }

But honestly I doubt this is BlueStore which reports incorrectly since it doesn't care about replication.

It rather looks like lack of stats from some replicas or improper pg replica factor processing...

Perhaps legacy vs. new pool what matters... Can you try to create a new pool at old cluster and fill it with some data (e.g. just a single 64K object) and check the stats?

Thanks,

Igor

On 11/26/2020 8:00 PM, Dan van der Ster wrote:

...

Hi Igor,

No BLUESTORE_LEGACY_STATFS warning, and bluestore_warn_on_legacy_statfs is the default true on this (and all) clusters. I'm quite sure we did the statfs conversion during one of the recent upgrades (I forget which one exactly).

# ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true #

Is there a command to see the statfs reported by an individual OSD ? We have a mix of ~year old and recently recreated OSDs, so I could try to see if they differ.

Thanks!

Dan

On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Hi Dan

don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced by bluestore_warn_on_legacy_statfs param) for the older cluster?

Thanks,

Igor

On 11/26/2020 7:29 PM, Dan van der Ster wrote:

...

Hi,

Depending on which cluster I look at (all running v14.2.11), the bytes_used is reporting raw space or stored bytes variably.

Here's a 7 year old cluster:

# ceph df -f json | jq .pools[0] { "name": "volumes", "id": 4, "stats": { "stored": 1229308190855881, "objects": 294401604, "kb_used": 1200496280133, "bytes_used": 1229308190855881, "percent_used": 0.4401889145374298, "max_avail": 521125025021952 } }

Note that stored == bytes_used for that pool. (this is a 3x replica pool).

But here's a newer cluster (installed recently with nautilus)

# ceph df -f json | jq .pools[0] { "name": "volumes", "id": 1, "stats": { "stored": 680977600893041, "objects": 163155803, "kb_used": 1995736271829, "bytes_used": 2043633942351985, "percent_used": 0.23379847407341003, "max_avail": 2232457428467712 } }

In the second cluster, bytes_used is 3x stored.

Does anyone know why these are not reported consistently? Noticing this just now, I'll update our monitoring to plot stored rather than bytes_used from now on.

Thanks!

Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hey that's it! I stopped the up but out OSDs (100 and 177), and now the stats are correct! # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.56M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB On Thu, Nov 26, 2020 at 8:02 PM Dan van der Ster <dan@vanderster.com> wrote:

There are a couple gaps, yes: https://termbin.com/9mx1

What should I do?

-- dan

On Thu, Nov 26, 2020 at 7:52 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Does "ceph osd df tree" show stats properly (I mean there are no evident gaps like unexpected zero values) for all the daemons?

...

1. Anyway, I found something weird...

I created a new 1-PG pool "foo" on a different cluster and wrote some data to it.

The stored and used are equal.

Thu 26 Nov 19:26:58 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

But I tried restarting the relevant three OSDs, and the bytes_used are temporarily reported correctly:

Thu 26 Nov 19:27:00 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB

But then a few seconds later it's back to used == stored:

Thu 26 Nov 19:27:03 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

It seems to report the correct stats only when the PG is peering (so some other transition state). I've restarted all three relevant OSDs now -- the stats are reported as stored == used.

2. Another data point -- I found another old cluster that reports stored/used correctly. I have no idea what might be different about that cluster -- we updated it just like the others.

Cheers, Dan

On Thu, Nov 26, 2020 at 6:22 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

For specific BlueStore instance you can learn relevant statfs output by

setting debug_bluestore to 20 and leaving OSD for 5-10 seconds (or may be a couple of minutes - don't remember exact statsfs poll period ).

Then grep osd log for "statfs" and/or "pool_statfs" and get the output formatted as per the following operator (taken from src/osd/osd_types.cc):

ostream& operator<<(ostream& out, const store_statfs_t &s) { out << std::hex << "store_statfs(0x" << s.available << "/0x" << s.internally_reserved << "/0x" << s.total << ", data 0x" << s.data_stored << "/0x" << s.allocated << ", compress 0x" << s.data_compressed << "/0x" << s.data_compressed_allocated << "/0x" << s.data_compressed_original << ", omap 0x" << s.omap_allocated << ", meta 0x" << s.internal_metadata << std::dec << ")"; return out; }

But honestly I doubt this is BlueStore which reports incorrectly since it doesn't care about replication.

It rather looks like lack of stats from some replicas or improper pg replica factor processing...

Perhaps legacy vs. new pool what matters... Can you try to create a new pool at old cluster and fill it with some data (e.g. just a single 64K object) and check the stats?

Thanks,

Igor

On 11/26/2020 8:00 PM, Dan van der Ster wrote:

...

Hi Igor,

No BLUESTORE_LEGACY_STATFS warning, and bluestore_warn_on_legacy_statfs is the default true on this (and all) clusters. I'm quite sure we did the statfs conversion during one of the recent upgrades (I forget which one exactly).

# ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true #

Is there a command to see the statfs reported by an individual OSD ? We have a mix of ~year old and recently recreated OSDs, so I could try to see if they differ.

Thanks!

Dan

On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Hi Dan

don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced by bluestore_warn_on_legacy_statfs param) for the older cluster?

Thanks,

Igor

On 11/26/2020 7:29 PM, Dan van der Ster wrote: > Hi, > > Depending on which cluster I look at (all running v14.2.11), the > bytes_used is reporting raw space or stored bytes variably. > > Here's a 7 year old cluster: > > # ceph df -f json | jq .pools[0] > { > "name": "volumes", > "id": 4, > "stats": { > "stored": 1229308190855881, > "objects": 294401604, > "kb_used": 1200496280133, > "bytes_used": 1229308190855881, > "percent_used": 0.4401889145374298, > "max_avail": 521125025021952 > } > } > > Note that stored == bytes_used for that pool. (this is a 3x replica pool). > > But here's a newer cluster (installed recently with nautilus) > > # ceph df -f json | jq .pools[0] > { > "name": "volumes", > "id": 1, > "stats": { > "stored": 680977600893041, > "objects": 163155803, > "kb_used": 1995736271829, > "bytes_used": 2043633942351985, > "percent_used": 0.23379847407341003, > "max_avail": 2232457428467712 > } > } > > In the second cluster, bytes_used is 3x stored. > > Does anyone know why these are not reported consistently? > Noticing this just now, I'll update our monitoring to plot stored > rather than bytes_used from now on. > > Thanks! > > Dan > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

Also wondering if you have the same "gap" OSDs at different cluster(s) which show stats improperly? On 11/26/2020 10:08 PM, Dan van der Ster wrote:

Hey that's it!

I stopped the up but out OSDs (100 and 177), and now the stats are correct!

# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.56M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB

On Thu, Nov 26, 2020 at 8:02 PM Dan van der Ster <dan@vanderster.com> wrote:

...

There are a couple gaps, yes: https://termbin.com/9mx1

What should I do?

-- dan

On Thu, Nov 26, 2020 at 7:52 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Does "ceph osd df tree" show stats properly (I mean there are no evident gaps like unexpected zero values) for all the daemons?

...

1. Anyway, I found something weird...

I created a new 1-PG pool "foo" on a different cluster and wrote some data to it.

The stored and used are equal.

Thu 26 Nov 19:26:58 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

But I tried restarting the relevant three OSDs, and the bytes_used are temporarily reported correctly:

Thu 26 Nov 19:27:00 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB

But then a few seconds later it's back to used == stored:

Thu 26 Nov 19:27:03 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

It seems to report the correct stats only when the PG is peering (so some other transition state). I've restarted all three relevant OSDs now -- the stats are reported as stored == used.

2. Another data point -- I found another old cluster that reports stored/used correctly. I have no idea what might be different about that cluster -- we updated it just like the others.

Cheers, Dan

On Thu, Nov 26, 2020 at 6:22 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

For specific BlueStore instance you can learn relevant statfs output by

setting debug_bluestore to 20 and leaving OSD for 5-10 seconds (or may be a couple of minutes - don't remember exact statsfs poll period ).

Then grep osd log for "statfs" and/or "pool_statfs" and get the output formatted as per the following operator (taken from src/osd/osd_types.cc):

ostream& operator<<(ostream& out, const store_statfs_t &s) { out << std::hex << "store_statfs(0x" << s.available << "/0x" << s.internally_reserved << "/0x" << s.total << ", data 0x" << s.data_stored << "/0x" << s.allocated << ", compress 0x" << s.data_compressed << "/0x" << s.data_compressed_allocated << "/0x" << s.data_compressed_original << ", omap 0x" << s.omap_allocated << ", meta 0x" << s.internal_metadata << std::dec << ")"; return out; }

But honestly I doubt this is BlueStore which reports incorrectly since it doesn't care about replication.

It rather looks like lack of stats from some replicas or improper pg replica factor processing...

Perhaps legacy vs. new pool what matters... Can you try to create a new pool at old cluster and fill it with some data (e.g. just a single 64K object) and check the stats?

Thanks,

Igor

On 11/26/2020 8:00 PM, Dan van der Ster wrote:

...

Hi Igor,

No BLUESTORE_LEGACY_STATFS warning, and bluestore_warn_on_legacy_statfs is the default true on this (and all) clusters. I'm quite sure we did the statfs conversion during one of the recent upgrades (I forget which one exactly).

# ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true #

Is there a command to see the statfs reported by an individual OSD ? We have a mix of ~year old and recently recreated OSDs, so I could try to see if they differ.

Thanks!

Dan

On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@suse.de> wrote: > Hi Dan > > don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced > by bluestore_warn_on_legacy_statfs param) for the older cluster? > > > Thanks, > > Igor > > > On 11/26/2020 7:29 PM, Dan van der Ster wrote: >> Hi, >> >> Depending on which cluster I look at (all running v14.2.11), the >> bytes_used is reporting raw space or stored bytes variably. >> >> Here's a 7 year old cluster: >> >> # ceph df -f json | jq .pools[0] >> { >> "name": "volumes", >> "id": 4, >> "stats": { >> "stored": 1229308190855881, >> "objects": 294401604, >> "kb_used": 1200496280133, >> "bytes_used": 1229308190855881, >> "percent_used": 0.4401889145374298, >> "max_avail": 521125025021952 >> } >> } >> >> Note that stored == bytes_used for that pool. (this is a 3x replica pool). >> >> But here's a newer cluster (installed recently with nautilus) >> >> # ceph df -f json | jq .pools[0] >> { >> "name": "volumes", >> "id": 1, >> "stats": { >> "stored": 680977600893041, >> "objects": 163155803, >> "kb_used": 1995736271829, >> "bytes_used": 2043633942351985, >> "percent_used": 0.23379847407341003, >> "max_avail": 2232457428467712 >> } >> } >> >> In the second cluster, bytes_used is 3x stored. >> >> Does anyone know why these are not reported consistently? >> Noticing this just now, I'll update our monitoring to plot stored >> rather than bytes_used from now on. >> >> Thanks! >> >> Dan >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io

OK, cool! Will try to reproduce this locally tomorrow... Thanks, Igor On 11/26/2020 10:19 PM, Dan van der Ster wrote:

Those osds are intentionally out, yes. (They were drained to be replaced).

I have fixed 2 clusters' stats already with this method ... both had up but out osds, and stopping the up/out osd fixed the stats.

I opened a tracker for this: https://tracker.ceph.com/issues/48385

-- dan

On Thu, Nov 26, 2020 at 8:14 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Also wondering if you have the same "gap" OSDs at different cluster(s) which show stats improperly?

On 11/26/2020 10:08 PM, Dan van der Ster wrote:

...

Hey that's it!

I stopped the up but out OSDs (100 and 177), and now the stats are correct!

# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.56M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB

On Thu, Nov 26, 2020 at 8:02 PM Dan van der Ster <dan@vanderster.com> wrote:

...

There are a couple gaps, yes: https://termbin.com/9mx1

What should I do?

-- dan

On Thu, Nov 26, 2020 at 7:52 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Does "ceph osd df tree" show stats properly (I mean there are no evident gaps like unexpected zero values) for all the daemons?

...

1. Anyway, I found something weird...

I created a new 1-PG pool "foo" on a different cluster and wrote some data to it.

The stored and used are equal.

Thu 26 Nov 19:26:58 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

But I tried restarting the relevant three OSDs, and the bytes_used are temporarily reported correctly:

Thu 26 Nov 19:27:00 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 4.3 PiB 84.55 538 TiB test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB

But then a few seconds later it's back to used == stored:

Thu 26 Nov 19:27:03 CET 2020 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47

POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB test 71 29 MiB 6.56k 29 MiB 0 269 TiB foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB

It seems to report the correct stats only when the PG is peering (so some other transition state). I've restarted all three relevant OSDs now -- the stats are reported as stored == used.

2. Another data point -- I found another old cluster that reports stored/used correctly. I have no idea what might be different about that cluster -- we updated it just like the others.

Cheers, Dan

On Thu, Nov 26, 2020 at 6:22 PM Igor Fedotov <ifedotov@suse.de> wrote: > For specific BlueStore instance you can learn relevant statfs output by > > setting debug_bluestore to 20 and leaving OSD for 5-10 seconds (or may > be a couple of minutes - don't remember exact statsfs poll period ). > > Then grep osd log for "statfs" and/or "pool_statfs" and get the output > formatted as per the following operator (taken from src/osd/osd_types.cc): > > ostream& operator<<(ostream& out, const store_statfs_t &s) > { > out << std::hex > << "store_statfs(0x" << s.available > << "/0x" << s.internally_reserved > << "/0x" << s.total > << ", data 0x" << s.data_stored > << "/0x" << s.allocated > << ", compress 0x" << s.data_compressed > << "/0x" << s.data_compressed_allocated > << "/0x" << s.data_compressed_original > << ", omap 0x" << s.omap_allocated > << ", meta 0x" << s.internal_metadata > << std::dec > << ")"; > return out; > } > > But honestly I doubt this is BlueStore which reports incorrectly since > it doesn't care about replication. > > It rather looks like lack of stats from some replicas or improper pg > replica factor processing... > > Perhaps legacy vs. new pool what matters... Can you try to create a new > pool at old cluster and fill it with some data (e.g. just a single 64K > object) and check the stats? > > > Thanks, > > Igor > > On 11/26/2020 8:00 PM, Dan van der Ster wrote: >> Hi Igor, >> >> No BLUESTORE_LEGACY_STATFS warning, and >> bluestore_warn_on_legacy_statfs is the default true on this (and all) >> clusters. >> I'm quite sure we did the statfs conversion during one of the recent >> upgrades (I forget which one exactly). >> >> # ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true >> # >> >> Is there a command to see the statfs reported by an individual OSD ? >> We have a mix of ~year old and recently recreated OSDs, so I could try >> to see if they differ. >> >> Thanks! >> >> Dan >> >> >> On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@suse.de> wrote: >>> Hi Dan >>> >>> don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced >>> by bluestore_warn_on_legacy_statfs param) for the older cluster? >>> >>> >>> Thanks, >>> >>> Igor >>> >>> >>> On 11/26/2020 7:29 PM, Dan van der Ster wrote: >>>> Hi, >>>> >>>> Depending on which cluster I look at (all running v14.2.11), the >>>> bytes_used is reporting raw space or stored bytes variably. >>>> >>>> Here's a 7 year old cluster: >>>> >>>> # ceph df -f json | jq .pools[0] >>>> { >>>> "name": "volumes", >>>> "id": 4, >>>> "stats": { >>>> "stored": 1229308190855881, >>>> "objects": 294401604, >>>> "kb_used": 1200496280133, >>>> "bytes_used": 1229308190855881, >>>> "percent_used": 0.4401889145374298, >>>> "max_avail": 521125025021952 >>>> } >>>> } >>>> >>>> Note that stored == bytes_used for that pool. (this is a 3x replica pool). >>>> >>>> But here's a newer cluster (installed recently with nautilus) >>>> >>>> # ceph df -f json | jq .pools[0] >>>> { >>>> "name": "volumes", >>>> "id": 1, >>>> "stats": { >>>> "stored": 680977600893041, >>>> "objects": 163155803, >>>> "kb_used": 1995736271829, >>>> "bytes_used": 2043633942351985, >>>> "percent_used": 0.23379847407341003, >>>> "max_avail": 2232457428467712 >>>> } >>>> } >>>> >>>> In the second cluster, bytes_used is 3x stored. >>>> >>>> Does anyone know why these are not reported consistently? >>>> Noticing this just now, I'll update our monitoring to plot stored >>>> rather than bytes_used from now on. >>>> >>>> Thanks! >>>> >>>> Dan >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io