- ceph-users - lists.ceph.io

Copying out bluestore's rocksdb, compact, then put back in - Mimic 13.2.6/13.2.8
by Paul Choi 20 Dec '19

20 Dec '19

Hi, I have a weird situation where an OSD's rocksdb fails to compact, because the OSD became full and the osd-full-ratio was 1.0 (not a good idea, I know). Hitting "bluefs enospc" while compacting: -376> 2019-12-18 15:48:16.492 7f2e0a5ac700 1 bluefs _allocate failed to allocate 0x40da486 on bdev 1, free 0x38b0000; fallback to bdev 2 -376> 2019-12-18 15:48:16.492 7f2e0a5ac700 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne -376> 2019-12-18 15:48:16.492 7f2e0a5ac700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40da486 -376> 2019-12-18 15:48:16.500 7f2e0a5ac700 -1 /build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uin t64_t, uint64_t)' thread 7f2e0a5ac700 time 2019-12-18 15:48:16.499599 /build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") So my idea is to copy out rocksdb somewhere else (bluefs-export), compact, then copy back in. Is there a way to do this? Mounting bluefs seems to be part of the OSD code, so there's no easy way to do this it seems. Because the OSD died at 100% full, I can't do bluefs-bdev-expand, and repair/fsck fail too. Thanks in advance.

1 1

Prometheus endpoint hanging with 13.2.7 release?
by Paul Choi 20 Dec '19

20 Dec '19

Hello, Anybody seeing the Prometheus endpoint hanging with the new 13.2.7 release? With 13.2.6 the endpoint would respond with a payload of 15MB in less than 10 seconds. Now, if you restart ceph-mgr, the Prometheus endpoint responds quickly for the first run, then successive runs get slower and slower, until it takes several minutes. I have no customization for the mgr module. Except for the Prometheus module, the "status" module and "Zabbix" module are working fine. This is on Ubuntu 16 LTS: ii ceph-mgr 13.2.7-1xenial amd64 manager for the ceph distributed storage system I'd love to know if there's a way to diagnose this issue - I tried upping the debug ms level but that doesn't seem to yield useful information. I don't know if this useful, but "prometheus self-test" is fine too. $ sudo ceph tell mgr.0 prometheus self-test Self-test OK pchoi@pchoi-desktop:~$ ceph mgr module ls { "enabled_modules": [ "prometheus", "status", "zabbix" ], "disabled_modules": [ { "name": "balancer", "can_run": true, "error_string": "" }, { "name": "crash", "can_run": true, "error_string": "" }, { "name": "dashboard", "can_run": true, "error_string": "" }, { "name": "hello", "can_run": true, "error_string": "" }, { "name": "influx", "can_run": false, "error_string": "influxdb python module not found" }, { "name": "iostat", "can_run": true, "error_string": "" }, { "name": "localpool", "can_run": true, "error_string": "" }, { "name": "restful", "can_run": true, "error_string": "" }, { "name": "selftest", "can_run": true, "error_string": "" }, { "name": "smart", "can_run": true, "error_string": "" }, { "name": "telegraf", "can_run": true, "error_string": "" }, { "name": "telemetry", "can_run": true, "error_string": "" } ] } pchoi@pchoi-desktop:~$ ceph mgr services { "prometheus": "http://woodenbox2:9283/" }

2 2

can run more than one rgw multisite realm on one ceph cluster
by 黄明友 20 Dec '19

20 Dec '19

I want run two realm on one ceph cluster, but I found rgw will use only one .rgw.root pool save rgw meta data, so , if run two realm on one ceph cluster, will be error ? 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

3 5

PG deep-scrubs ... triggered by backfill?
by Stefan Kooman 20 Dec '19

20 Dec '19

Hi, We have been rebalacing our cluster (CRUSH policy change for cephfs metadata pool). The rebalancing is done ... but now there are PG deep-scrubs going on (mostly on the pool that got almost completely backfilled). We have had the no-(deep)-scrub flags active ... but that does not prevent new (deep-)scrubs from happening. Also disabling them completely on the specific pool did not help ... The oldest deep-scrub was from the 14th of december ... Is there logic coded somewhere that a PG that got backfilled needs to be deep-scrubbed after that? I remember this was a thing on filestore (to check integrity) ... should not be needed anymore with bluestore, right (CRC checking)? All this on Mimic 13.2.8 Thanks, Stefan P.s. I updated https://tracker.ceph.com/issues/11202 -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 0

v14.2.5 Nautilus released
by Abhishek Lekshmanan 20 Dec '19

20 Dec '19

This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release. For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/ Notable Changes --------------- Critical fix: * This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used. New health warnings: * Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded):: ceph crash ls-new To acknowledge a particular crash (or all crashes) and silence the health warning:: ceph crash archive <crash-id> ceph crash archive-all * Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false * Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false * A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD. Changes in the telemetry module: * The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show ceph telemetry show device # specifically show the device channel If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in: ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on * The telemetry module now reports more information about CephFS file systems, including: - how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached We have also added: - which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use If you had telemetry enabled, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of the new info) OSD: * A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues. * Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote (replica) PGs. RGW: * RGW now supports S3 Object Lock set of APIs allowing for a WORM model for storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold. * RGW now supports List Objects V2 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

9 11

PG-upmap offline optimization is not working as expected
by Thomas Schneider 20 Dec '19

20 Dec '19

Hi, I have tested the PG-upmap offline optimization with 1 of my pools: ssd This pool is unbalanced; here's the ouput of ceph osd df tree before the optimization: root@ld3955:~# ceph osd df tree class ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1532.88501 - 27 TiB 27 GiB 4.4 GiB 816 KiB 23 GiB 27 TiB 0.10 1.00 - root default -46 353.82300 - 1.1 TiB 3.7 GiB 702 MiB 144 KiB 3.0 GiB 1.1 TiB 0.33 3.35 - host ld4257 20 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.35 25 up osd.20 21 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 27 up osd.21 22 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 80 KiB 1024 MiB 370 GiB 0.33 3.35 26 up osd.22 -34 356.22299 - 4.6 TiB 4.9 GiB 936 MiB 228 KiB 4.0 GiB 4.6 TiB 0.10 1.06 - host ld4464 23 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.35 29 up osd.23 24 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 40 KiB 1024 MiB 370 GiB 0.33 3.35 31 up osd.24 25 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 52 KiB 1024 MiB 370 GiB 0.33 3.35 31 up osd.25 26 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 234 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.35 265 up osd.26 -37 356.22299 - 4.6 TiB 4.9 GiB 936 MiB 152 KiB 4.0 GiB 4.6 TiB 0.10 1.06 - host ld4465 27 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 24 up osd.27 28 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 28 up osd.28 29 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 22 up osd.29 30 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 234 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.35 258 up osd.30 -3 116.65399 - 4.2 TiB 3.5 GiB 491 MiB 76 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5505 8 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 20 KiB 1024 MiB 3.5 TiB 0.03 0.33 288 up osd.8 9 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 371 GiB 0.31 3.16 28 up osd.9 10 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 32 KiB 1024 MiB 371 GiB 0.31 3.16 31 up osd.10 -7 116.65399 - 4.2 TiB 3.5 GiB 491 MiB 72 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5506 11 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 371 GiB 0.31 3.16 36 up osd.11 12 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 32 KiB 1024 MiB 3.5 TiB 0.03 0.33 260 up osd.12 13 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 16 KiB 1024 MiB 371 GiB 0.31 3.16 28 up osd.13 -10 116.65399 - 4.2 TiB 3.5 GiB 491 MiB 80 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5507 14 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 371 GiB 0.31 3.16 24 up osd.14 15 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 32 KiB 1024 MiB 371 GiB 0.31 3.16 26 up osd.15 16 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.16 -13 116.65399 - 4.2 TiB 3.5 GiB 490 MiB 64 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5508 17 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 28 KiB 1024 MiB 371 GiB 0.31 3.16 19 up osd.17 18 ssd 0.37199 1.00000 372 GiB 1.2 GiB 163 MiB 8 KiB 1024 MiB 371 GiB 0.31 3.16 24 up osd.18 19 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 28 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.19 TOTAL 27 TiB 27 GiB 4.4 GiB 823 KiB 23 GiB 27 TiB 0.10 MIN/MAX VAR: 0.33/3.35 STDDEV: 0.20 The output of osdmaptool implies many modifications affecting osd.11 and osd.12, means the optimizer wants to shift PGs from 12 to 11. root@ld3955:~# source out_ssd.txt set 66.41 pg_upmap_items mapping to [12->13] set 66.4e pg_upmap_items mapping to [22->20] set 66.7c pg_upmap_items mapping to [28->29] set 66.9f pg_upmap_items mapping to [12->11] set 66.147 pg_upmap_items mapping to [12->11] set 66.1b1 pg_upmap_items mapping to [12->11] set 66.203 pg_upmap_items mapping to [12->11] set 66.257 pg_upmap_items mapping to [28->30] set 66.27d pg_upmap_items mapping to [28->30] set 66.300 pg_upmap_items mapping to [12->11] set 66.354 pg_upmap_items mapping to [28->29] set 66.35b pg_upmap_items mapping to [28->30] set 66.38a pg_upmap_items mapping to [12->11] set 66.3d0 pg_upmap_items mapping to [28->30] However this makes no sense as osd.11 has already more PGs than other OSDs. In fact there's one OSD with least PGs: osd.17 Why is the optimizer not shifting PGs to osd.17? Here's the output of ceph osd df tree after the optimization: root@ld3955:~# ceph osd df tree class ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1532.88501 - 27 TiB 27 GiB 3.2 GiB 816 KiB 23 GiB 27 TiB 0.10 1.00 - root default -46 353.82300 - 1.1 TiB 3.6 GiB 502 MiB 144 KiB 3.0 GiB 1.1 TiB 0.33 3.40 - host ld4257 20 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.49 25 up osd.20 21 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.44 27 up osd.21 22 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 80 KiB 1024 MiB 370 GiB 0.31 3.28 26 up osd.22 -34 356.22299 - 4.6 TiB 4.8 GiB 668 MiB 228 KiB 4.0 GiB 4.6 TiB 0.10 1.07 - host ld4464 23 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.44 29 up osd.23 24 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 40 KiB 1024 MiB 370 GiB 0.31 3.28 31 up osd.24 25 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 52 KiB 1024 MiB 370 GiB 0.33 3.49 31 up osd.25 26 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 167 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.34 265 up osd.26 -37 356.22299 - 4.6 TiB 4.8 GiB 669 MiB 152 KiB 4.0 GiB 4.6 TiB 0.10 1.07 - host ld4465 27 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.31 3.28 24 up osd.27 28 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.49 28 up osd.28 29 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.31 3.28 23 up osd.29 30 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 167 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.36 257 up osd.30 -3 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 76 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5505 8 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 20 KiB 1024 MiB 3.5 TiB 0.03 0.33 288 up osd.8 9 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 371 GiB 0.30 3.13 28 up osd.9 10 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 32 KiB 1024 MiB 371 GiB 0.30 3.13 31 up osd.10 -7 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 72 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5506 11 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 371 GiB 0.30 3.13 41 up osd.11 12 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 32 KiB 1024 MiB 3.5 TiB 0.03 0.33 254 up osd.12 13 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 16 KiB 1024 MiB 371 GiB 0.30 3.13 29 up osd.13 -10 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 80 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5507 14 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 371 GiB 0.30 3.13 24 up osd.14 15 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 32 KiB 1024 MiB 371 GiB 0.30 3.13 26 up osd.15 16 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.16 -13 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 64 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5508 17 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 28 KiB 1024 MiB 371 GiB 0.30 3.13 19 up osd.17 18 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 8 KiB 1024 MiB 371 GiB 0.30 3.13 24 up osd.18 19 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 28 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.19 TOTAL 27 TiB 27 GiB 3.2 GiB 823 KiB 23 GiB 27 TiB 0.10 MIN/MAX VAR: 0.33/3.49 STDDEV: 0.19 Regards Thomas

1 0

ceph radosgw failed to initialize
by dayong tian 20 Dec '19

20 Dec '19

The radosgw can't start normally, the error in log file: ------------------------------- 2019-12-20 14:37:04.058 7fd5b088f700 -1 Initialization timeout, failed to initialize 2019-12-20 14:37:04.304 7fe7148c0780 0 deferred set uid:gid to 167:167 (ceph:ceph) 2019-12-20 14:37:04.304 7fe7148c0780 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process radosgw, pid 3474 2019-12-20 14:37:04.333 7fe6fe47e700 20 reqs_thread_entry: start 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: starting msgr at 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: starting objecter 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: setting wanted keys 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: calling monclient init 2019-12-20 14:37:04.340 7fe7148c0780 1 librados: init done 2019-12-20 14:37:04.340 7fe7148c0780 20 rados->read ofs=0 len=0 2019-12-20 14:37:04.340 7fe7148c0780 10 librados: wait_for_osdmap waiting 2019-12-20 14:37:04.341 7fe7148c0780 10 librados: wait_for_osdmap done waiting 2019-12-20 14:37:04.341 7fe7148c0780 10 librados: read oid=default.realm nspace= 2019-12-20 14:42:04.304 7fe700f0d700 -1 Initialization timeout, failed to initialize ------------------------------- The config I used: ------------------------------- [client.rgw.ceph-test-f01] rgw_host = ceph-test-f01 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-test-f01/keyring rgw_frontends = civetweb port=8099 ------------------------------- The env I set as follows: ------------------------------- # cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) # ceph -v ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) # ceph -s cluster: id: ac018df0-2e71-4c0f-a8a1-0ea29b8a7eb1 health: HEALTH_WARN Reduced data availability: 8 pgs inactive Degraded data redundancy: 8 pgs undersized services: mon: 1 daemons, quorum ceph-test-f01 (age 2h) mgr: ceph-test-f01(active, since 61m) osd: 1 osds: 1 up (since 2h), 1 in (since 2h) data: pools: 2 pools, 40 pgs objects: 1 objects, 1.6 KiB usage: 1.0 GiB used, 98 GiB / 99 GiB avail pgs: 20.000% pgs not active 32 active+clean 8 undersized+peered # ceph osd pool ls .rgw.root panama ------------------------------- I also tried to follow the ceph document to create the default pools such as .default.rgw.control .default.rgw.gc .default.rgw.buckets .default.rgw.buckets.index .default.rgw.buckets.extra .default.log .default.intent-log .default.usage .default.users .default.users.email .default.users.swift .default.users.default.uid The radosgw service still can't start normally, any advices will be appreciated. And all the radosgw-admin utilities hang like: # radosgw-admin zone get 2019-12-20 15:29:03.322 7f18c65876c0 1 librados: starting msgr at 2019-12-20 15:29:03.322 7f18c65876c0 1 librados: starting objecter 2019-12-20 15:29:03.323 7f18c65876c0 1 librados: setting wanted keys 2019-12-20 15:29:03.323 7f18c65876c0 1 librados: calling monclient init 2019-12-20 15:29:03.325 7f18c65876c0 1 librados: init done 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: starting msgr at 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: starting objecter 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: setting wanted keys 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: calling monclient init 2019-12-20 15:29:03.333 7f18c65876c0 1 librados: init done 2019-12-20 15:29:03.334 7f188ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-20 15:29:03.334 7f18c65876c0 20 rados->read ofs=0 len=0 2019-12-20 15:29:03.334 7f188f7fe700 20 reqs_thread_entry: start 2019-12-20 15:29:03.335 7f18c65876c0 10 librados: read oid=default.realm nspace= 2019-12-20 15:29:25.335 7f188ffff700 2 RGWDataChangesLog::ChangesRenewThread: start

1 0

RGW bucket stats extremely slow to respond
by David Monschein 19 Dec '19

19 Dec '19

Hi all! Reaching out again about this issue since I haven't had much luck. We've been seeing some strange behavior with our object storage cluster. While bucket stats (radosgw-admin bucket stats) normally return in a matter of seconds, we frequently observe it taking almost ten minutes, which is not convenient since we use those bucket stats for billing/accounting. Restarting the radosgw process on the RGWs fixes this issue until it crops up again in maybe a few days. Someone mentioned that they think this might have to do with bucket deletions, or more specifically, lifecycle policies to abort incomplete multipart uploads. He mentioned there was an item in the bug tracker for this, but I have not been able to find said bug in the tracker. I have no clue if this is the case or not, but I figured I'd throw it out there to see if anyone else has run into this problem. I have seen many of these messages in my RGW logs: 2019-12-02 13:12:52.882 7faa7018f700 0 abort_bucket_multiparts WARNING : aborted 8553000 incomplete multipart uploads So maybe there is some truth to the aborted multipart uploads causing problems? My cluster has over 200 OSDs, 10 RGWs, about 2200 buckets. Running Nautilus 14.2.5. If anyone has run into this or has any information I'd appreciate it. Merry Christmas & Happy Holidays, - Dave

1 0

Re: Pool Max Avail and Ceph Dashboard Pool Useage on Nautilus giving different percentages
by Stephan Mueller 19 Dec '19

19 Dec '19

Hi, if "MAX AVAIL" displays the wrong data, the bug is just made more visible through the dashboard, as the calculation is correct. To get the right percentage you have to divide the used space through the total, and the total can only consist of two states used and not used space, so both states will be added together to get the total. Or in short: used / (avail + used) Just looked into the C++ code - Max avail will be calculated the following way: avail_res = avail / raw_used_rate ( https://github.com/ceph/ceph/blob/nautilus/src/mon/PGMap.cc#L905) raw_used_rate *= (sum.num_object_copies - sum.num_objects_degraded) / sum.num_object_copies (https://github.com/ceph/ceph/blob/nautilus/src/mon/PGMap.cc#L892) Am Dienstag, den 17.12.2019, 07:07 +0100 schrieb ceph(a)elchaka.de: > I have observed this in the ceph nautilus dashboard too - and Think > it is a Display Bug... but sometimes it Shows tue right values > > > Which nautilus u use? > > > Am 10. Dezember 2019 14:31:05 MEZ schrieb "David Majchrzak, ODERLAND > Webbhotell AB" <david(a)oderland.se>: > > Hi! > > > > While browsing /#/pool in nautilus ceph dashboard I noticed it said > > 93% > > used on the single pool we have (3x replica). > > > > ceph df detal however shows 81% used on the pool and 67% raw > > useage. > > > > # ceph df detail > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW > > USED > > ssd 478 TiB 153 TiB 324 TiB 325 > > TiB 67.96 > > TOTAL 478 TiB 153 TiB 324 TiB 325 > > TiB 67.96 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED > > > > MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED > > COMPR UNDER COMPR > > echo 3 108 TiB 29.49M 324 > > TiB 81.61 24 > > TiB N/A N/A 29.49M 0 > > B 0 B I manually calculated the used percentage to get "avail" in your case it seems to be 73 TiB. That means the the total space available for your pool would be 397 TiB. I'm not sure why that is, but it's what the math behind those calculations say. (Found a thread regarding that on the new mailing list (ceph- users(a)ceph.io) -> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NH2LMMX5KVR… ) 0.8161 = used (324) / total => total = 397 Than I looked at the remaining calculations: raw_used_rate *= (sum.num_object_copies - sum.num_objects_degraded) / sum.num_object_copies and avail_res = avail / raw_used_rate First I looked up the init value for "raw_used_rate" for replicated pools. It's their size so we can put in 3 here and for "avail_res" is 24. So I first calculated the final "raw_used_rate" which is 3.042. That means that you have around 4.2% degraded pg's in your pool. > > > > > > I know we're looking at the most full OSD (210PGs, 79% used, 1.17 > > VAR) > > and count max avail from that. But where's the 93% full from in > > dashboard? As said above the calculation is right but the data is wrong... As it uses the real data that can be put in the selected pool, but it uses everywhere else sizes that consider all pool replicas. I created an issue to fix this https://tracker.ceph.com/issues/43384 > > > > My guess is that is comes from calculating: > > > > 1 - Max Avail / (Used + Max avail) = 0.93 > > > > > > Kind Regards, > > > > David Majchrzak > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Hope I could clarify some things and thanks for your feedback :) BTW this problem currently still exists as there wasn't any change to these mentioned lines after the nautilus release. Stephan

1 0

rbd images inaccessible for a longer period of time
by yveskretzschmar＠web.de 19 Dec '19

19 Dec '19

Hi, yesterday I had to power off some vm (proxmox) backed by rbd images for maintenance. After the VMs were off, I tried to create a Snapshot which didn't finish even after half an hour. Because it was maintenance I rebooted all VM nodes an all ceph nodes - nothing changed. Powering on the VM was impossible, kvm exited with timeout. This happened to two of about 15 VM. Two of three Images of one VM still had locks, which I did remove but still unable to power on. I tried to access the Image by mapping it with rbd-nbd, which was unsuccessful and logged this: [ 8601.746971] block nbd0: Connection timed out [ 8601.747648] block nbd0: shutting down sockets [ 8601.747653] block nbd0: Connection timed out [...] [ 8601.750419] block nbd0: Connection timed out [ 8601.750831] print_req_error: 121 callbacks suppressed [ 8601.750832] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 8601.751261] buffer_io_error: 182 callbacks suppressed [ 8601.751262] Buffer I/O error on dev nbd0, logical block 0, async page read [ 8601.751678] blk_update_request: I/O error, dev nbd0, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [...] [ 8601.760283] ldm_validate_partition_table(): Disk read failed. [ 8601.760344] Dev nbd0: unable to read RDB block 0 [ 8601.760985] nbd0: unable to read partition table [ 8601.761282] nbd0: detected capacity change from 0 to 375809638400 [ 8601.761382] ldm_validate_partition_table(): Disk read failed. [ 8601.761461] Dev nbd0: unable to read RDB block 0 [ 8601.762145] nbd0: unable to read partition table The rbd-nbd process kept existing and had to be killed Same thing with qemu-nbd. Exporting the Image via rbd export worked fine, also a rbd copy. Any other operation on the Image (feature dis / enable) took forever so I had to abort it. It seems that every operation leaves a lock on the image. Because it was in the middle of the night, I stopped working on it. Today Morning one of the images was accessible again, the others not. Anybody any hint? Some system information below. Regards, Yves ceph version 14.2.5 (3ce7517553bdd5195b68a6ffaf0bd7f3acad1647) nautilus (stable) Primary Cluster with a backup cluster (rbd mirror) [global] auth client required = none auth cluster required = none auth service required = none auth supported = none cephx_sign_messages = false cephx require signatures = False cluster_network = 172.16.230.0/24 debug asok = 0/0 debug auth = 0/0 debug bdev = 0/0 debug bluefs = 0/0 debug bluestore = 0/0 debug buffer = 0/0 debug civetweb = 0/0 debug client = 0/0 debug compressor = 0/0 debug context = 0/0 debug crush = 0/0 debug crypto = 0/0 debug dpdk = 0/0 debug eventtrace = 0/0 debug filer = 0/0 debug filestore = 0/0 debug finisher = 0/0 debug fuse = 0/0 debug heartbeatmap = 0/0 debug javaclient = 0/0 debug journal = 0/0 debug journaler = 0/0 debug kinetic = 0/0 debug kstore = 0/0 debug leveldb = 0/0 debug lockdep = 0/0 debug mds = 0/0 debug mds balancer = 0/0 debug mds locker = 0/0 debug mds log = 0/0 debug mds log expire = 0/0 debug mds migrator = 0/0 debug memdb = 0/0 debug mgr = 0/0 debug mgrc = 0/0 debug mon = 0/0 debug monc = 0/00 debug ms = 0/0 debug none = 0/0 debug objclass = 0/0 debug objectcacher = 0/0 debug objecter = 0/0 debug optracker = 0/0 debug osd = 0/0 debug paxos = 0/0 debug perfcounter = 0/0 debug rados = 0/0 debug rbd = 0/0 debug rbd mirror = 0/0 debug rbd replay = 0/0 debug refs = 0/0 debug reserver = 0/0 debug rgw = 0/0 debug rocksdb = 0/0 debug striper = 0/0 debug throttle = 0/0 debug timer = 0/0 debug tp = 0/0 debug xio = 0/0 fsid = 27fdf1bb-22a1-4d5e-9729-780cbdcd33fe mon_allow_pool_delete = true mon_host = 172.16.230.142 172.16.230.144 172.16.230.146 mon_osd_down_out_subtree_limit = host osd_backfill_scan_max = 16 osd_backfill_scan_min = 4 osd_deep_scrub_interval = 1209600 osd_journal_size = 5120 osd_max_backfills = 1 osd_max_trimming_pgs = 1 osd_pg_max_concurrent_snap_trims = 1 osd_pool_default_min_size = 2 osd_pool_default_size = 3 osd_recovery_max_active = 1 osd_recovery_max_single_start = 1 osd_recovery_op_priority = 1 osd_recovery_threads = 1 osd_scrub_begin_hour = 19 osd_scrub_chunk_max = 1 osd_scrub_chunk_min = 1 osd_scrub_during_recovery = false osd_scrub_end_hour = 6 osd_scrub_priority = 1 osd_scrub_sleep = 0.5 osd_snap_trim_priority = 1 osd_snap_trim_sleep = 0.005 osd_srub_max_interval = 1209600 public_network = 172.16.230.0/24 max open files = 131072 osd objectstore = bluestore osd op threads = 2 osd crush update on start = true Currently inaccessible image: rbd image 'vm-29009-disk-2': size 200 GiB in 51200 objects order 22 (4 MiB objects) snapshot_count: 2 id: 1abd04da8b9a4d block_name_prefix: rbd_data.1abd04da8b9a4d format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Tue Jul 9 13:07:36 2019 access_timestamp: Thu Dec 19 01:35:34 2019 modify_timestamp: Thu Dec 19 00:19:32 2019 journal: 1abd04da8b9a4d mirroring state: enabled mirroring global id: c71ec81f-18be-4d0b-93ed-0cebe3e619bb mirroring primary: true

2 2