- ceph-users - lists.ceph.io

Fwd: Kinetic support
by Johan Thomsen 02 Sep '19

02 Sep '19

Hey, What is the current status of Kinetic KV support in Ceph? I'm asking because: https://www.crn.com.au/news/seagate-quietly-bins-open-storage-project-519345 .. and the fact that kinetic-cpp-client hasn't been updated in four years and only compiles against OpenSSL 1.0.2, which will become EOL by the end of 2019. Or am I totally wrong? ^ Thank you for reply in advance, /Johan

1 0

removing/flattening a bucket without data movement?
by Zoltan Arnold Nagy 02 Sep '19

02 Sep '19

Hi folks, Originally our osd tree looked like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2073.15186 root default -14 176.63100 rack s01-rack -19 176.63100 host s01 <snip osds> -15 171.29900 rack s02-rack -20 171.29900 host s02 <snip osds> etc. You get the idea. It was a legacy thing as we've been upgrading this cluster since probably firefly, and started with way less hardware. The crush rule was set up like this originally: step take default step chooseleaf firstn 0 type rack which we have modified to step take default step chooseleaf firstn 0 type host taking advantage of chooseleaf's behavior (eg searching in depth instead of just a single level). Now we thought we could get rid of the rack buckets simply by moving the host buckets to the root using "ceph osd crush move s01 root=default", however this resulted in a bunch of data movement. Swapping the IDs manually in the crushmap seems to work (verified via crushtool's --compare), eg. changing the ID of s01 to s01-rack's and vice versa, including all shadow trees. Looking around I saw that there is a swap-bucket command but that does not swap the IDs just bucket contents, so would result in data movement. Other than manually editing the crushmap is there a better way to achieve this? Is this way the most optimal? Cheers, Zoltan

3 7

trouble with grafana dashboards in nautilus
by Rory Schramm 01 Sep '19

01 Sep '19

HI, I recently upgraded my cluster from 12.2 to 14.2 and I'm having some trouble getting the mgr dashboards for grafana working. I setup Prometheus and Grafana per https://docs.ceph.com/docs/nautilus/mgr/prometheus/#mgr-prometheus However, for the osd disk performance statistics graphs on the host details dashboard I'm getting the following error: "found duplicate series for the match group {device="dm-5", instance=":9100"} on the right hand-side of the operation: [{name="ceph_disk_occupation", ceph_daemon="osd.13", db_device="/dev/dm-8", device="dm-5", instance=":9100", job="ceph"}, {name="ceph_disk_occupation", ceph_daemon="osd.15", db_device="/dev/dm-10", device="dm-5", instance=":9100", job="ceph"}];many-to-many matching not allowed: matching labels must be unique on one side" This also happens on the following graphs: Host Overview/AVG Disk Utilization Host Details/OSD Disk Performance Statistics/* Also the following graphs show no data points: OSD Details/Physical Device Performance/* prometheus version: 2.12.0 node exporter: 0.15.2 grafana version: 6.3.3 note that my osds all have separate data and rocks db devices. I have also upgraded all the osds to nautilus via ceph-bluestore-tool repair. Any idea what's needed to fix this? Thanks below are the Prometheus config files prometheus.yml global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: 'node' file_sd_configs: - files: - node_targets.yml - job_name: 'ceph' honor_labels: true file_sd_configs: - files: - ceph_targets.yml ---- node_targets.yml: [ { "targets": [ "nas-osd-01:9100" ], "labels": { "instance": "nas-osd-01" } }, { "targets": [ "nas-osd-02:9100" ], "labels": { "instance": "nas-osd-02" } }, { "targets": [ "nas-osd-02:9100" ], "labels": { "instance": "nas-osd-03" } } ] --- ceph_targets.yml: [ { "targets": [ "nas-osd-01:9283" ], "labels": {} }, { "targets": [ "nas-osd-02:9283" ], "labels": {} }, { "targets": [ "nas-osd-03:9283" ], "labels": {} } ]

1 0

Safe to reboot host?
by Brett Chancellor 31 Aug '19

31 Aug '19

Before I write something that's already been done, are there any built in utilities or tools that can tell me if it's safe to reboot a host? I'm looking for something better than just checking the health status, but rather checking pg status and ensuring that a reboot wouldn't take any undersized PGs offline. Thanks. -Brett

2 1

Howto define OSD weight in Crush map
by 74cmonty＠gmail.com 31 Aug '19

31 Aug '19

Hi, after adding an OSD to Ceph it is adviseable to create a relevant entry in Crush map using a weight size depending on disk size. Example: ceph osd crush set osd.<id> <weight> root=default host=<hostname> Question: How is the weight defined depending on disk size? Which algorithm can be used to calculate the weight? From my first Ceph installation (luminous) the Crush map entry for - HDD device with 1.80TB size (output of lsscsi -s): weight was 1.627229 - NVMI device with 3.20TB size (output of lsscsi -s): weight was 2.910889 THX

2 1

Error: err=rados: File too large caller=cephstorage_linux.go:231
by tapas＠wolk.com 30 Aug '19

30 Aug '19

Hi, I am trying to upload image size=>190 MB got error "err=rados: File too large caller=cephstorage_linux.go:231" Please help Thanks!!!! Tapas

1 0

Re: backfill_toofull after adding new OSDs
by Frank Schilder 30 Aug '19

30 Aug '19

I observe the same issue after adding two new OSD hosts to an almost empty mimic cluster. > Let's try to restrict discussion to the original thread > "backfill_toofull while OSDs are not full" and get a tracker opened up > for this issue. Is this the issue you are referring to: https://tracker.ceph.com/issues/41255 ? I have a number of larger rebalance operations ahead and will probably see this for a couple of days. If there is any information (logs etc.) I can provide, please let me know. Status right now is: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_ERR 15227159/90990337 objects misplaced (16.735%) Degraded data redundancy (low space): 64 pgs backfill_toofull too few PGs per OSD (29 < min 30) services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs-1/1/1 up {0=ceph-12=up:active}, 1 up:standby-replay osd: 208 osds: 208 up, 208 in; 273 remapped pgs data: pools: 7 pools, 790 pgs objects: 9.45 M objects, 17 TiB usage: 21 TiB used, 1.4 PiB / 1.4 PiB avail pgs: 15227159/90990337 objects misplaced (16.735%) 517 active+clean 190 active+remapped+backfill_wait 64 active+remapped+backfill_wait+backfill_toofull 19 active+remapped+backfilling io: client: 893 KiB/s rd, 6.3 MiB/s wr, 208 op/s rd, 306 op/s wr recovery: 298 MiB/s, 156 objects/s Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Howto add DB (aka RockDB) device to existing OSD on HDD
by 74cmonty＠gmail.com 30 Aug '19

30 Aug '19

Hi, I have created OSD on HDD w/o putting DB on faster drive. In order to improve performance I have now a single SSD drive with 3.8TB. I modified /etc/ceph/ceph.conf by adding this in [global]: bluestore_block_db_size = 53687091200 This should create RockDB with size 50GB. Then I tried to move DB to a new device (SSD) that is not formatted: root@ld5505:~# ceph-bluestore-tool bluefs-bdev-new-db –-path /var/lib/ceph/osd/ceph-76 --dev-target /dev/sdbk too many positional options have been specified on the command line Checking the content of /var/lib/ceph/osd/ceph-76 it appears that there's no link to block.db: root@ld5505:~# ls -l /var/lib/ceph/osd/ceph-76/ insgesamt 52 -rw-r--r-- 1 ceph ceph 418 Aug 27 11:08 activate.monmap lrwxrwxrwx 1 ceph ceph 93 Aug 27 11:08 block -> /dev/ceph-8cd045dc-9eb2-47ad-9668-116cf425a66a/osd-block-9c51bde1-3c75-4767-8808-f7e7b58b8f97 -rw-r--r-- 1 ceph ceph 2 Aug 27 11:08 bluefs -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 ceph_fsid -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 fsid -rw------- 1 ceph ceph 56 Aug 27 11:08 keyring -rw-r--r-- 1 ceph ceph 8 Aug 27 11:08 kv_backend -rw-r--r-- 1 ceph ceph 21 Aug 27 11:08 magic -rw-r--r-- 1 ceph ceph 4 Aug 27 11:08 mkfs_done -rw-r--r-- 1 ceph ceph 41 Aug 27 11:08 osd_key -rw-r--r-- 1 ceph ceph 6 Aug 27 11:08 ready -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 require_osd_release -rw-r--r-- 1 ceph ceph 10 Aug 27 11:08 type -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 whoami root@ld5505:~# more /var/lib/ceph/osd/ceph-76/bluefs 1 Questions: How can I add DB device for every single existing OSD to this new SSD drive? How can I increase the DB size later in case it's insufficient? THX

2 1

help
by Amudhan P 30 Aug '19

30 Aug '19

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning . "ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%) "ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+ pg 2.3 is stuck undersized for 1076.456113, current state active+undersized+ pg 2.7 is stuck undersized for 1076.456342, current state active+undersized+ pg 2.8 is stuck undersized for 1076.455920, current state active+undersized+ pg 2.a is stuck undersized for 1076.486412, current state active+undersized+ pg 2.b is stuck undersized for 1076.485975, current state active+undersized+ pg 2.f is stuck undersized for 1076.486953, current state active+undersized+ pg 2.10 is stuck undersized for 1076.486763, current state active+undersized pg 2.12 is stuck undersized for 1076.486539, current state active+undersized pg 2.13 is stuck undersized for 1075.419199, current state active+undersized pg 2.17 is stuck undersized for 1076.455424, current state active+undersized pg 2.18 is stuck undersized for 1075.419639, current state active+undersized pg 2.1a is stuck undersized for 1076.455966, current state active+undersized pg 2.1b is stuck undersized for 1076.486677, current state active+undersized pg 2.1f is stuck undersized for 1076.455572, current state active+undersized how to bring it health status OK regards Amudhan

5 10

Which CephFS clients send a compressible hint?
by Erwin Bogaard 30 Aug '19

30 Aug '19

We're mainly using CephFS using the Centos/Rhel 7 kernel client and I'm pondering if I should go for bluestore compression mode" = passive or aggressive with this client to get compression on (preferably) only compressible objects. Is there any list of CephFS clients that send compressible hints? If not, is there any other way to detect this, other than checking a client by just writing some compressible and some uncompressible data? The documentation is rather sparse on this matter, as far as I can see.

2 1