- ceph-users - lists.ceph.io

Avast Battery Saver
by Avaknights14＠gmail.com 04 Aug '20

04 Aug '20

If your system runs out of battery very rapidly then avast is the best solution for you. It is one of the most trusted battery savers that stops applications that you are not using, will speed up your operating device and will extend your device’s battery life. It also tells how much time you are left with, on your device’s battery. To know more about how to extend your Android’s battery life with Avast Battery Saver either visit our site or contact our customer support. https://www.avast-support.net/avast-battery-saver/

1 0

Talk to a Cash App Representative to revive a shut account?
by mary smith 04 Aug '20

04 Aug '20

Here, you will get a bit of uplifting news, that weather this application deactivated your Cash app account because of any reason, so you can easily revive your shut account as well as you can also reactivate your Cash Card (Visa charge card). In this way, here are many features that make it popular overall. On the off chance that you any inquiries, please go to Talk to a Cash App Representative. https://www.customercare-email.com/customer-service/cash-app.html

1 0

Epson printer making weird sound? Reach tech support by contacting Epson Customer Service.
by mary smith 04 Aug '20

04 Aug '20

Sometimes an issue can arise when you might hear weird sounds coming from the printer due to some tech glitch. If that happens, then you can get the help by going to the tech help sites or you can call the Epson Customer Service to get the problem resolved. https://www.epsonprintersupportpro.net/

1 0

RadosGW/Keystone intergration issues
by Vladimir Sigunov 04 Aug '20

04 Aug '20

Hello community, Im trying to integrate ceph RadosGW with OpenStack Keystone. Everything is working as expected, but when I try to reach public buckets with public link generated in Horizon, I get a permanent error ‘NoSuchBucket’. However, this bucket and all it’s content does exists: I can access it as authenticated user in Horizon, I can access it as authenticated user via S3 browser/aws cli, I can see it with radosgw- admin bucket list --bucket <bucket>. We are running OpenStack Rocky and this issue appears to be with Ceph Octopus 15.2.4 (there was no issues with RGW on Nautilus and Luminous). Here is my configuration file: <...> [client.rgw.ceph-hdd-9.rgw0] host = ceph-hdd-9 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-hdd-9.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-hdd-9.rgw0.log rgw frontends = beast endpoint=10.10.200.179:8080 rgw thread pool size = 512 rgw zone = default rgw keystone api version = 3 rgw keystone url = https://<keystone url>:13000 rgw keystone accepted roles = admin, _member_, Member, member, creator, swiftoperator rgw keystone accepted admin roles = admin, _member_ #rgw keystone token cache size = 0 #rgw keystone revocation interval = 0 rgw keystone implicit tenants = true rgw keystone admin domain = default rgw keystone admin project = service rgw keystone admin user = swift rgw keystone admin password = swift_osp_password rgw s3 auth use keystone = true rgw s3 auth order = local, external rgw user default quota max size = -1 rgw swift account in url = true rgw dynamic resharding = false rgw bucket resharding = false rgw enable usage log = true rgw usage log tick interval = 30 rgw usage log flush threshold = 1024 rgw usage max shards = 32 rgw usage max user shards = 1 rgw verify ssl = false Please advise. Thank you in advance for your help, Vladimir

2 2

LDAP integration
by jhamster＠chem.byu.edu 03 Aug '20

03 Aug '20

Hello, I am looking into connecting my rados gateway to LDAP and found the following documentation. https://docs.ceph.com/docs/master/radosgw/ldap-auth/ I would like to allow an LDAP group to have access to create and manage buckets. The questions I still have are the following: -Do the LDAP users need to log in to some sort of portal before their corresponding ceph user is created? If so, where do they go to do so? Or does the creation of ceph users and keys happen automatically? -How can you access the ldap users key and secret after they are integrated? Thanks in advance for any information you can provide. Regards, Jared

1 0

HEALTH_WARN crush map has legacy tunables (require firefly, min is hammer)
by Mike Garza 03 Aug '20

03 Aug '20

Hi, I've been tasked with moving Jewel clusters to Nautilus. After the final upgrade Ceph Health warns about legacy tunables. On clusters running SSD's I enabled the optimal flag. Which took weeks to chug through remappings. My remaining clusters run HDD's. Does anyone have experience with using the legacy flag? I'd like to clear up the health warning without outright silencing it. But, I also do not want to kick off any remapping either. Does anyone have experience with pushing this change down the road? Mike

1 0

Re: Ceph does not recover from OSD restart
by Frank Schilder 03 Aug '20

03 Aug '20

Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Ceph does not recover from OSD restart
by Frank Schilder 03 Aug '20

03 Aug '20

Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

mimic: much more raw used than reported
by Frank Schilder 03 Aug '20

03 Aug '20

Dear fellow cephers, I observe a wired problem on our mimic-13.2.8 cluster. We have an EC RBD pool backed by HDDs. These disks are not in any other pool. I noticed that the total capacity (=USED+MAX AVAIL) reported by "ceph df detail" has shrunk recently from 300TiB to 200TiB. Part but by no means all of this can be explained by imbalance of the data distribution. When I compare the output of "ceph df detail" and "ceph osd df tree", I find 69TiB raw capacity used but not accounted for; see calculations below. These 69TiB raw are equivalent to 20% usable capacity and I really need it back. Together with the imbalance, we loose about 30% capacity. What is using these extra 69TiB and how can I get it back? Some findings: These are the 5 largest images in the pool, accounting for a total of 97TiB out of 119TiB usage: # rbd du : NAME PROVISIONED USED one-133 25 TiB 14 TiB NAME PROVISIONED USED one-153@222 40 TiB 14 TiB one-153@228 40 TiB 357 GiB one-153@235 40 TiB 797 GiB one-153@241 40 TiB 509 GiB one-153@242 40 TiB 43 GiB one-153@243 40 TiB 16 MiB one-153@244 40 TiB 16 MiB one-153@245 40 TiB 324 MiB one-153@246 40 TiB 276 MiB one-153@247 40 TiB 96 MiB one-153@248 40 TiB 138 GiB one-153@249 40 TiB 1.8 GiB one-153@250 40 TiB 0 B one-153 40 TiB 204 MiB <TOTAL> 40 TiB 16 TiB NAME PROVISIONED USED one-391@3 40 TiB 432 MiB one-391@9 40 TiB 26 GiB one-391@15 40 TiB 90 GiB one-391@16 40 TiB 0 B one-391@17 40 TiB 0 B one-391@18 40 TiB 0 B one-391@19 40 TiB 0 B one-391@20 40 TiB 3.5 TiB one-391@21 40 TiB 5.4 TiB one-391@22 40 TiB 5.8 TiB one-391@23 40 TiB 8.4 TiB one-391@24 40 TiB 1.4 TiB one-391 40 TiB 2.2 TiB <TOTAL> 40 TiB 27 TiB NAME PROVISIONED USED one-394@3 70 TiB 1.4 TiB one-394@9 70 TiB 2.5 TiB one-394@15 70 TiB 20 GiB one-394@16 70 TiB 0 B one-394@17 70 TiB 0 B one-394@18 70 TiB 0 B one-394@19 70 TiB 383 GiB one-394@20 70 TiB 3.3 TiB one-394@21 70 TiB 5.0 TiB one-394@22 70 TiB 5.0 TiB one-394@23 70 TiB 9.0 TiB one-394@24 70 TiB 1.6 TiB one-394 70 TiB 2.5 TiB <TOTAL> 70 TiB 31 TiB NAME PROVISIONED USED one-434 25 TiB 9.1 TiB The large 70TiB images one-391 and one-394 are currently copied to with ca. 5TiB per day. Output of "ceph df detail" with some columns removed: NAME ID USED %USED MAX AVAIL OBJECTS RAW USED sr-rbd-data-one-hdd 11 119 TiB 58.45 84 TiB 31286554 158 TiB Pool is EC 6+2. USED is correct: 31286554*4MiB=119TiB. RAW USED is correct: 119*8/6=158TiB. Most of this data is freshly copied onto large RBD images. Compression is enabled on this pool (aggressive,snappy). However, when looking at "deph osd df tree", I get The combined raw capacity of OSDs backing this pool is 406.8TiB (sum over SIZE). Summing up column USE over all OSDs gives 227.5TiB. This gives a difference of 69TiB (=227-158) that is not accounted for. Here the output of "ceph osd df tree limited" to the drives backing the pool: ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME 84 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 180 MiB 16 GiB 3.9 TiB 56.43 1.72 103 osd.84 145 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.6 TiB 144 MiB 14 GiB 4.3 TiB 51.37 1.57 87 osd.145 156 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.1 TiB 173 MiB 16 GiB 3.8 TiB 57.91 1.77 100 osd.156 168 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 164 MiB 16 GiB 3.9 TiB 56.31 1.72 98 osd.168 181 hdd 8.90999 1.00000 8.9 TiB 5.5 TiB 5.4 TiB 121 MiB 17 GiB 3.5 TiB 61.26 1.87 105 osd.181 74 hdd 8.90999 1.00000 8.9 TiB 4.2 TiB 4.2 TiB 148 MiB 13 GiB 4.7 TiB 46.79 1.43 85 osd.74 144 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 106 MiB 15 GiB 4.2 TiB 53.17 1.62 94 osd.144 157 hdd 8.90999 1.00000 8.9 TiB 5.8 TiB 5.8 TiB 192 MiB 18 GiB 3.1 TiB 65.02 1.99 111 osd.157 169 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 172 MiB 16 GiB 3.8 TiB 56.99 1.74 102 osd.169 180 hdd 8.90999 1.00000 8.9 TiB 5.8 TiB 5.8 TiB 131 MiB 18 GiB 3.1 TiB 65.04 1.99 111 osd.180 60 hdd 8.90999 1.00000 8.9 TiB 4.5 TiB 4.5 TiB 155 MiB 14 GiB 4.4 TiB 50.40 1.54 93 osd.60 146 hdd 8.90999 1.00000 8.9 TiB 4.8 TiB 4.8 TiB 139 MiB 15 GiB 4.1 TiB 53.70 1.64 92 osd.146 158 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.5 TiB 183 MiB 17 GiB 3.4 TiB 62.30 1.90 109 osd.158 170 hdd 8.90999 1.00000 8.9 TiB 5.7 TiB 5.6 TiB 205 MiB 18 GiB 3.2 TiB 63.53 1.94 112 osd.170 182 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.6 TiB 105 MiB 14 GiB 4.3 TiB 52.27 1.60 92 osd.182 63 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 156 MiB 15 GiB 4.2 TiB 52.74 1.61 98 osd.63 148 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.1 TiB 119 MiB 16 GiB 3.8 TiB 57.82 1.77 100 osd.148 159 hdd 8.90999 1.00000 8.9 TiB 4.0 TiB 4.0 TiB 89 MiB 12 GiB 4.9 TiB 44.61 1.36 79 osd.159 172 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 173 MiB 16 GiB 3.8 TiB 57.22 1.75 98 osd.172 183 hdd 8.90999 1.00000 8.9 TiB 6.0 TiB 6.0 TiB 135 MiB 19 GiB 2.9 TiB 67.35 2.06 118 osd.183 229 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.6 TiB 127 MiB 15 GiB 4.3 TiB 52.05 1.59 93 osd.229 232 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.2 TiB 158 MiB 17 GiB 3.7 TiB 58.22 1.78 101 osd.232 235 hdd 8.90999 1.00000 8.9 TiB 4.1 TiB 4.1 TiB 103 MiB 13 GiB 4.8 TiB 45.96 1.40 79 osd.235 238 hdd 8.90999 1.00000 8.9 TiB 5.4 TiB 5.4 TiB 120 MiB 17 GiB 3.5 TiB 60.47 1.85 104 osd.238 259 hdd 10.91399 1.00000 11 TiB 6.2 TiB 6.2 TiB 140 MiB 19 GiB 4.7 TiB 56.54 1.73 120 osd.259 231 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 114 MiB 16 GiB 3.8 TiB 56.90 1.74 101 osd.231 233 hdd 8.90999 1.00000 8.9 TiB 5.5 TiB 5.5 TiB 123 MiB 17 GiB 3.4 TiB 61.78 1.89 106 osd.233 236 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 114 MiB 16 GiB 3.8 TiB 57.53 1.76 101 osd.236 239 hdd 8.90999 1.00000 8.9 TiB 4.2 TiB 4.2 TiB 95 MiB 13 GiB 4.7 TiB 47.41 1.45 86 osd.239 263 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.3 TiB 178 MiB 17 GiB 5.6 TiB 48.73 1.49 102 osd.263 228 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 113 MiB 16 GiB 3.8 TiB 57.10 1.74 96 osd.228 230 hdd 8.90999 1.00000 8.9 TiB 4.9 TiB 4.9 TiB 144 MiB 16 GiB 4.0 TiB 55.20 1.69 99 osd.230 234 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.6 TiB 164 MiB 18 GiB 3.3 TiB 63.29 1.93 109 osd.234 237 hdd 8.90999 1.00000 8.9 TiB 4.8 TiB 4.8 TiB 110 MiB 15 GiB 4.1 TiB 54.33 1.66 97 osd.237 260 hdd 10.91399 1.00000 11 TiB 5.4 TiB 5.4 TiB 152 MiB 17 GiB 5.5 TiB 49.35 1.51 104 osd.260 0 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.2 TiB 157 MiB 16 GiB 3.7 TiB 58.28 1.78 102 osd.0 2 hdd 8.90999 1.00000 8.9 TiB 5.3 TiB 5.2 TiB 122 MiB 16 GiB 3.6 TiB 59.05 1.80 106 osd.2 72 hdd 8.90999 1.00000 8.9 TiB 4.4 TiB 4.4 TiB 145 MiB 14 GiB 4.5 TiB 49.89 1.52 89 osd.72 76 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 178 MiB 16 GiB 3.8 TiB 56.89 1.74 102 osd.76 86 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.5 TiB 155 MiB 14 GiB 4.3 TiB 51.18 1.56 94 osd.86 1 hdd 8.90999 1.00000 8.9 TiB 4.9 TiB 4.9 TiB 141 MiB 15 GiB 4.0 TiB 54.73 1.67 95 osd.1 3 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 156 MiB 15 GiB 4.2 TiB 52.40 1.60 94 osd.3 73 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 4.9 TiB 146 MiB 16 GiB 3.9 TiB 55.68 1.70 102 osd.73 85 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.5 TiB 192 MiB 18 GiB 3.3 TiB 62.46 1.91 109 osd.85 87 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 189 MiB 16 GiB 3.9 TiB 55.91 1.71 102 osd.87 Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 12

Problems with long taking deep-scrubbing processes causing PG_NOT_DEEP_SCRUBBED
by Carsten Grommel - Profihost AG 03 Aug '20

03 Aug '20

Hi, we are having problems with really long taking deep-scrubb processes causing PG_NOT_DEEP_SCRUBBED and ceph HEALTH_WARN. One ph is waiting since 2020-05-18 for the deep-scrubb. Is there any way to speed up the deep-scrubbing? Ceph-Version: ceph version 14.2.8-3-gc6b8eedb77 (c6b8eedb771089fe3b0a95da93158ec4144758f3) nautilus (stable)

3 4