- ceph-users - lists.ceph.io

trouble with grafana dashboards in nautilus
by Rory Schramm 01 Sep '19

01 Sep '19

HI, I recently upgraded my cluster from 12.2 to 14.2 and I'm having some trouble getting the mgr dashboards for grafana working. I setup Prometheus and Grafana per https://docs.ceph.com/docs/nautilus/mgr/prometheus/#mgr-prometheus However, for the osd disk performance statistics graphs on the host details dashboard I'm getting the following error: "found duplicate series for the match group {device="dm-5", instance=":9100"} on the right hand-side of the operation: [{name="ceph_disk_occupation", ceph_daemon="osd.13", db_device="/dev/dm-8", device="dm-5", instance=":9100", job="ceph"}, {name="ceph_disk_occupation", ceph_daemon="osd.15", db_device="/dev/dm-10", device="dm-5", instance=":9100", job="ceph"}];many-to-many matching not allowed: matching labels must be unique on one side" This also happens on the following graphs: Host Overview/AVG Disk Utilization Host Details/OSD Disk Performance Statistics/* Also the following graphs show no data points: OSD Details/Physical Device Performance/* prometheus version: 2.12.0 node exporter: 0.15.2 grafana version: 6.3.3 note that my osds all have separate data and rocks db devices. I have also upgraded all the osds to nautilus via ceph-bluestore-tool repair. Any idea what's needed to fix this? Thanks below are the Prometheus config files prometheus.yml global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: 'node' file_sd_configs: - files: - node_targets.yml - job_name: 'ceph' honor_labels: true file_sd_configs: - files: - ceph_targets.yml ---- node_targets.yml: [ { "targets": [ "nas-osd-01:9100" ], "labels": { "instance": "nas-osd-01" } }, { "targets": [ "nas-osd-02:9100" ], "labels": { "instance": "nas-osd-02" } }, { "targets": [ "nas-osd-02:9100" ], "labels": { "instance": "nas-osd-03" } } ] --- ceph_targets.yml: [ { "targets": [ "nas-osd-01:9283" ], "labels": {} }, { "targets": [ "nas-osd-02:9283" ], "labels": {} }, { "targets": [ "nas-osd-03:9283" ], "labels": {} } ]

1 0

Safe to reboot host?
by Brett Chancellor 31 Aug '19

31 Aug '19

Before I write something that's already been done, are there any built in utilities or tools that can tell me if it's safe to reboot a host? I'm looking for something better than just checking the health status, but rather checking pg status and ensuring that a reboot wouldn't take any undersized PGs offline. Thanks. -Brett

2 1

Howto define OSD weight in Crush map
by 74cmonty＠gmail.com 31 Aug '19

31 Aug '19

Hi, after adding an OSD to Ceph it is adviseable to create a relevant entry in Crush map using a weight size depending on disk size. Example: ceph osd crush set osd.<id> <weight> root=default host=<hostname> Question: How is the weight defined depending on disk size? Which algorithm can be used to calculate the weight? From my first Ceph installation (luminous) the Crush map entry for - HDD device with 1.80TB size (output of lsscsi -s): weight was 1.627229 - NVMI device with 3.20TB size (output of lsscsi -s): weight was 2.910889 THX

2 1

Error: err=rados: File too large caller=cephstorage_linux.go:231
by tapas＠wolk.com 30 Aug '19

30 Aug '19

Hi, I am trying to upload image size=>190 MB got error "err=rados: File too large caller=cephstorage_linux.go:231" Please help Thanks!!!! Tapas

1 0

Re: backfill_toofull after adding new OSDs
by Frank Schilder 30 Aug '19

30 Aug '19

I observe the same issue after adding two new OSD hosts to an almost empty mimic cluster. > Let's try to restrict discussion to the original thread > "backfill_toofull while OSDs are not full" and get a tracker opened up > for this issue. Is this the issue you are referring to: https://tracker.ceph.com/issues/41255 ? I have a number of larger rebalance operations ahead and will probably see this for a couple of days. If there is any information (logs etc.) I can provide, please let me know. Status right now is: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_ERR 15227159/90990337 objects misplaced (16.735%) Degraded data redundancy (low space): 64 pgs backfill_toofull too few PGs per OSD (29 < min 30) services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs-1/1/1 up {0=ceph-12=up:active}, 1 up:standby-replay osd: 208 osds: 208 up, 208 in; 273 remapped pgs data: pools: 7 pools, 790 pgs objects: 9.45 M objects, 17 TiB usage: 21 TiB used, 1.4 PiB / 1.4 PiB avail pgs: 15227159/90990337 objects misplaced (16.735%) 517 active+clean 190 active+remapped+backfill_wait 64 active+remapped+backfill_wait+backfill_toofull 19 active+remapped+backfilling io: client: 893 KiB/s rd, 6.3 MiB/s wr, 208 op/s rd, 306 op/s wr recovery: 298 MiB/s, 156 objects/s Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Howto add DB (aka RockDB) device to existing OSD on HDD
by 74cmonty＠gmail.com 30 Aug '19

30 Aug '19

Hi, I have created OSD on HDD w/o putting DB on faster drive. In order to improve performance I have now a single SSD drive with 3.8TB. I modified /etc/ceph/ceph.conf by adding this in [global]: bluestore_block_db_size = 53687091200 This should create RockDB with size 50GB. Then I tried to move DB to a new device (SSD) that is not formatted: root@ld5505:~# ceph-bluestore-tool bluefs-bdev-new-db –-path /var/lib/ceph/osd/ceph-76 --dev-target /dev/sdbk too many positional options have been specified on the command line Checking the content of /var/lib/ceph/osd/ceph-76 it appears that there's no link to block.db: root@ld5505:~# ls -l /var/lib/ceph/osd/ceph-76/ insgesamt 52 -rw-r--r-- 1 ceph ceph 418 Aug 27 11:08 activate.monmap lrwxrwxrwx 1 ceph ceph 93 Aug 27 11:08 block -> /dev/ceph-8cd045dc-9eb2-47ad-9668-116cf425a66a/osd-block-9c51bde1-3c75-4767-8808-f7e7b58b8f97 -rw-r--r-- 1 ceph ceph 2 Aug 27 11:08 bluefs -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 ceph_fsid -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 fsid -rw------- 1 ceph ceph 56 Aug 27 11:08 keyring -rw-r--r-- 1 ceph ceph 8 Aug 27 11:08 kv_backend -rw-r--r-- 1 ceph ceph 21 Aug 27 11:08 magic -rw-r--r-- 1 ceph ceph 4 Aug 27 11:08 mkfs_done -rw-r--r-- 1 ceph ceph 41 Aug 27 11:08 osd_key -rw-r--r-- 1 ceph ceph 6 Aug 27 11:08 ready -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 require_osd_release -rw-r--r-- 1 ceph ceph 10 Aug 27 11:08 type -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 whoami root@ld5505:~# more /var/lib/ceph/osd/ceph-76/bluefs 1 Questions: How can I add DB device for every single existing OSD to this new SSD drive? How can I increase the DB size later in case it's insufficient? THX

2 1

help
by Amudhan P 30 Aug '19

30 Aug '19

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning . "ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%) "ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+ pg 2.3 is stuck undersized for 1076.456113, current state active+undersized+ pg 2.7 is stuck undersized for 1076.456342, current state active+undersized+ pg 2.8 is stuck undersized for 1076.455920, current state active+undersized+ pg 2.a is stuck undersized for 1076.486412, current state active+undersized+ pg 2.b is stuck undersized for 1076.485975, current state active+undersized+ pg 2.f is stuck undersized for 1076.486953, current state active+undersized+ pg 2.10 is stuck undersized for 1076.486763, current state active+undersized pg 2.12 is stuck undersized for 1076.486539, current state active+undersized pg 2.13 is stuck undersized for 1075.419199, current state active+undersized pg 2.17 is stuck undersized for 1076.455424, current state active+undersized pg 2.18 is stuck undersized for 1075.419639, current state active+undersized pg 2.1a is stuck undersized for 1076.455966, current state active+undersized pg 2.1b is stuck undersized for 1076.486677, current state active+undersized pg 2.1f is stuck undersized for 1076.455572, current state active+undersized how to bring it health status OK regards Amudhan

5 10

Which CephFS clients send a compressible hint?
by Erwin Bogaard 30 Aug '19

30 Aug '19

We're mainly using CephFS using the Centos/Rhel 7 kernel client and I'm pondering if I should go for bluestore compression mode" = passive or aggressive with this client to get compression on (preferably) only compressible objects. Is there any list of CephFS clients that send compressible hints? If not, is there any other way to detect this, other than checking a client by just writing some compressible and some uncompressible data? The documentation is rather sparse on this matter, as far as I can see.

2 1

Deleted snapshot still having error, how to fix (pg repair is not working)
by Marc Roos 30 Aug '19

30 Aug '19

I was a little bit afraid I would be deleting this snapshot without result. How do I fix this error (pg repair is not working) pg 17.36 is active+clean+inconsistent, acting [7,29,12] 2019-08-30 10:40:04.580470 7f9b3f061700 -1 log_channel(cluster) log [ERR] : repair 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing

1 0

645% Clean PG's in Dashboard
by c.lilja＠falseprivacy.org 30 Aug '19

30 Aug '19

Hi, I've upgraded to Nautilus from Mimic a while ago and enabled the pg_autoscaler. When pg_autoscaler was activated I got a HEALTH_WARN regarding: POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_ratio 0.000 on pools [] Both target_size_bytes and target_size_ratio on all the pools are set to 0, so I started to wonder why this error message appear. My autoscale-status looks like this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 16708M 4.0 34465G 0.0019 1.0 8 warn cephfs_data_reduced 15506G 2.0 34465G 0.8998 1.0 375 warn cephfs_data 6451G 3.0 34465G 0.5616 1.0 250 warn So the ratio in total is 1.4633.. Isn't 1.0 of the combined ratio of all pools equal of full? I also enabled the Dashboard and saw that the PG Status showed "645% clean" PG's. This cluster was originally installed with version Jewel, so may it be any legacy setting or such that causing this?

1 0