August 2019 - ceph-users

removing/flattening a bucket without data movement?

by Zoltan Arnold Nagy

Hi folks, Originally our osd tree looked like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2073.15186 root default -14 176.63100 rack s01-rack -19 176.63100 host s01 <snip osds> -15 171.29900 rack s02-rack -20 171.29900 host s02 <snip osds> etc. You get the idea. It was a legacy thing as we've been upgrading this cluster since probably firefly, and started with way less hardware. The crush rule was set up like this originally: step take default step chooseleaf firstn 0 type rack which we have modified to step take default step chooseleaf firstn 0 type host taking advantage of chooseleaf's behavior (eg searching in depth instead of just a single level). Now we thought we could get rid of the rack buckets simply by moving the host buckets to the root using "ceph osd crush move s01 root=default", however this resulted in a bunch of data movement. Swapping the IDs manually in the crushmap seems to work (verified via crushtool's --compare), eg. changing the ID of s01 to s01-rack's and vice versa, including all shadow trees. Looking around I saw that there is a swap-bucket command but that does not swap the IDs just bucket contents, so would result in data movement. Other than manually editing the crushmap is there a better way to achieve this? Is this way the most optimal? Cheers, Zoltan

3 years, 6 months

3
7
0 0

Safe to reboot host?

by Brett Chancellor

Before I write something that's already been done, are there any built in utilities or tools that can tell me if it's safe to reboot a host? I'm looking for something better than just checking the health status, but rather checking pg status and ensuring that a reboot wouldn't take any undersized PGs offline. Thanks. -Brett

3 years, 6 months

2
1
0 0

Howto define OSD weight in Crush map

by 74cmonty＠gmail.com

Hi, after adding an OSD to Ceph it is adviseable to create a relevant entry in Crush map using a weight size depending on disk size. Example: ceph osd crush set osd.<id> <weight> root=default host=<hostname> Question: How is the weight defined depending on disk size? Which algorithm can be used to calculate the weight? From my first Ceph installation (luminous) the Crush map entry for - HDD device with 1.80TB size (output of lsscsi -s): weight was 1.627229 - NVMI device with 3.20TB size (output of lsscsi -s): weight was 2.910889 THX

3 years, 6 months

2
1
0 0

Error: err=rados: File too large caller=cephstorage_linux.go:231

by tapas＠wolk.com

Hi, I am trying to upload image size=>190 MB got error "err=rados: File too large caller=cephstorage_linux.go:231" Please help Thanks!!!! Tapas

3 years, 6 months

1
0
0 0

Re: backfill_toofull after adding new OSDs

by Frank Schilder

I observe the same issue after adding two new OSD hosts to an almost empty mimic cluster. > Let's try to restrict discussion to the original thread > "backfill_toofull while OSDs are not full" and get a tracker opened up > for this issue. Is this the issue you are referring to: https://tracker.ceph.com/issues/41255 ? I have a number of larger rebalance operations ahead and will probably see this for a couple of days. If there is any information (logs etc.) I can provide, please let me know. Status right now is: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_ERR 15227159/90990337 objects misplaced (16.735%) Degraded data redundancy (low space): 64 pgs backfill_toofull too few PGs per OSD (29 < min 30) services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs-1/1/1 up {0=ceph-12=up:active}, 1 up:standby-replay osd: 208 osds: 208 up, 208 in; 273 remapped pgs data: pools: 7 pools, 790 pgs objects: 9.45 M objects, 17 TiB usage: 21 TiB used, 1.4 PiB / 1.4 PiB avail pgs: 15227159/90990337 objects misplaced (16.735%) 517 active+clean 190 active+remapped+backfill_wait 64 active+remapped+backfill_wait+backfill_toofull 19 active+remapped+backfilling io: client: 893 KiB/s rd, 6.3 MiB/s wr, 208 op/s rd, 306 op/s wr recovery: 298 MiB/s, 156 objects/s Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 years, 6 months

1
0
0 0

Howto add DB (aka RockDB) device to existing OSD on HDD

by 74cmonty＠gmail.com

Hi, I have created OSD on HDD w/o putting DB on faster drive. In order to improve performance I have now a single SSD drive with 3.8TB. I modified /etc/ceph/ceph.conf by adding this in [global]: bluestore_block_db_size = 53687091200 This should create RockDB with size 50GB. Then I tried to move DB to a new device (SSD) that is not formatted: root@ld5505:~# ceph-bluestore-tool bluefs-bdev-new-db –-path /var/lib/ceph/osd/ceph-76 --dev-target /dev/sdbk too many positional options have been specified on the command line Checking the content of /var/lib/ceph/osd/ceph-76 it appears that there's no link to block.db: root@ld5505:~# ls -l /var/lib/ceph/osd/ceph-76/ insgesamt 52 -rw-r--r-- 1 ceph ceph 418 Aug 27 11:08 activate.monmap lrwxrwxrwx 1 ceph ceph 93 Aug 27 11:08 block -> /dev/ceph-8cd045dc-9eb2-47ad-9668-116cf425a66a/osd-block-9c51bde1-3c75-4767-8808-f7e7b58b8f97 -rw-r--r-- 1 ceph ceph 2 Aug 27 11:08 bluefs -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 ceph_fsid -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 fsid -rw------- 1 ceph ceph 56 Aug 27 11:08 keyring -rw-r--r-- 1 ceph ceph 8 Aug 27 11:08 kv_backend -rw-r--r-- 1 ceph ceph 21 Aug 27 11:08 magic -rw-r--r-- 1 ceph ceph 4 Aug 27 11:08 mkfs_done -rw-r--r-- 1 ceph ceph 41 Aug 27 11:08 osd_key -rw-r--r-- 1 ceph ceph 6 Aug 27 11:08 ready -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 require_osd_release -rw-r--r-- 1 ceph ceph 10 Aug 27 11:08 type -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 whoami root@ld5505:~# more /var/lib/ceph/osd/ceph-76/bluefs 1 Questions: How can I add DB device for every single existing OSD to this new SSD drive? How can I increase the DB size later in case it's insufficient? THX

3 years, 6 months

2
1
1 0

help

by Amudhan P

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning . "ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%) "ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+ pg 2.3 is stuck undersized for 1076.456113, current state active+undersized+ pg 2.7 is stuck undersized for 1076.456342, current state active+undersized+ pg 2.8 is stuck undersized for 1076.455920, current state active+undersized+ pg 2.a is stuck undersized for 1076.486412, current state active+undersized+ pg 2.b is stuck undersized for 1076.485975, current state active+undersized+ pg 2.f is stuck undersized for 1076.486953, current state active+undersized+ pg 2.10 is stuck undersized for 1076.486763, current state active+undersized pg 2.12 is stuck undersized for 1076.486539, current state active+undersized pg 2.13 is stuck undersized for 1075.419199, current state active+undersized pg 2.17 is stuck undersized for 1076.455424, current state active+undersized pg 2.18 is stuck undersized for 1075.419639, current state active+undersized pg 2.1a is stuck undersized for 1076.455966, current state active+undersized pg 2.1b is stuck undersized for 1076.486677, current state active+undersized pg 2.1f is stuck undersized for 1076.455572, current state active+undersized how to bring it health status OK regards Amudhan

3 years, 6 months

5
10
0 0

Which CephFS clients send a compressible hint?

by Erwin Bogaard

We're mainly using CephFS using the Centos/Rhel 7 kernel client and I'm pondering if I should go for bluestore compression mode" = passive or aggressive with this client to get compression on (preferably) only compressible objects. Is there any list of CephFS clients that send compressible hints? If not, is there any other way to detect this, other than checking a client by just writing some compressible and some uncompressible data? The documentation is rather sparse on this matter, as far as I can see.

3 years, 6 months

2
1
0 0

Deleted snapshot still having error, how to fix (pg repair is not working)

by Marc Roos

I was a little bit afraid I would be deleting this snapshot without result. How do I fix this error (pg repair is not working) pg 17.36 is active+clean+inconsistent, acting [7,29,12] 2019-08-30 10:40:04.580470 7f9b3f061700 -1 log_channel(cluster) log [ERR] : repair 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing

3 years, 6 months

1
0
0 0

645% Clean PG's in Dashboard

by c.lilja＠falseprivacy.org

Hi, I've upgraded to Nautilus from Mimic a while ago and enabled the pg_autoscaler. When pg_autoscaler was activated I got a HEALTH_WARN regarding: POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_ratio 0.000 on pools [] Both target_size_bytes and target_size_ratio on all the pools are set to 0, so I started to wonder why this error message appear. My autoscale-status looks like this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 16708M 4.0 34465G 0.0019 1.0 8 warn cephfs_data_reduced 15506G 2.0 34465G 0.8998 1.0 375 warn cephfs_data 6451G 3.0 34465G 0.5616 1.0 250 warn So the ratio in total is 1.4633.. Isn't 1.0 of the combined ratio of all pools equal of full? I also enabled the Dashboard and saw that the PG Status showed "645% clean" PG's. This cluster was originally installed with version Jewel, so may it be any legacy setting or such that causing this?

3 years, 6 months

1
0
0 0

2023

2022

2021

2020

2019

ceph-users August 2019