September 2020 - ceph-users

How HP Printer Support help for troubleshooting?

by mary smith

To troubleshoot with the hp printer issue, you are required to download the HP Print and Scan Doctor from its official website. After that, you have to run HPPdr.exe from the download location on your system. And then, hit on the Start link and select your printer accordingly. Now, click fix printing. If there is an issue, use HP Printer Support. https://www.amiytech.com/hp-printer-support/

3 years, 7 months

1
0
0 0

ceph-volume lvm cannot zap???

by Marc Roos

[@~]# ceph-volume lvm zap /dev/sdi --> Zapping: /dev/sdi --> --destroy was not specified, but zapping a whole device will remove the partition table stderr: wipefs: error: /dev/sdi: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition stderr: wipefs: error: /dev/sdi: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition I can see where it is busy, at least not in lsof

3 years, 7 months

3
3
0 0

[nautilus] ceph tell hanging

by Nico Schottelius

Hello, recently we wanted to re-adjust rebalancing speed in one cluster with ceph tell osd.* injectargs '--osd-max-backfills 4' ceph tell osd.* injectargs '--osd-recovery-max-active 4' The first osds responded and after about 6-7 osds ceph tell stopped progressing, just after it encountered a dead osd (osd.10). We have since then removed osd.10 and all osds in the cluster are up. However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states. It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active. We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands. We are mostly on 14.2.9, besides the rgw: [16:44:47] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own? Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

3 years, 7 months

2
6
0 0

Unknown PGs after osd move

by Nico Schottelius

Hello, after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs: cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_WARN noscrub flag(s) set 2 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 241 pgs inactive 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops. services: mon: 3 daemons, quorum black1,black2,black3 (age 97m) mgr: black2(active, since 96m), standbys: black1, black3 osd: 85 osds: 85 up, 82 in; 118 remapped pgs flags noscrub rgw: 1 daemon active (admin) data: pools: 12 pools, 3000 pgs objects: 33.96M objects, 129 TiB usage: 388 TiB used, 159 TiB / 548 TiB avail pgs: 8.033% pgs unknown 409151/101874117 objects misplaced (0.402%) 2634 active+clean 241 unknown 107 active+remapped+backfill_wait 11 active+remapped+backfilling 7 active+clean+scrubbing+deep io: client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s This used to be around 700+ unknown, however these 241 are stuck in this state for more than 1h. Below is a sample of pgs from "ceph pg dump all | grep unknown" 2.7f7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c2 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7ab 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.78b 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.788 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.76e 0 Using ceph pg 2.7f7 query hangs. We checked and one server did have an incorrect MTU setting (9204 instead of the correct 9000), but that was fixed some hours ago. Does anyone have a hint on how to find those unknown osds? Version wise this is 14.2.9: [20:42:20] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } From ceph health detail: [20:42:58] black2.place6:~# ceph health detail HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops, oldest one blocked for 1986 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops. OSDMAP_FLAGS noscrub flag(s) set OSD_NEARFULL 2 nearfull osd(s) osd.36 is near full osd.54 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'ssd' is nearfull PG_AVAILABILITY Reduced data availability: 241 pgs inactive pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting [] pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting [] ... pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting [] pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting [] As can be seen, multiple pools are affected even though most missing pgs are from pool 2. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

3 years, 7 months

3
10
0 0

Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs

by Marc Roos

At least ceph thought you the essence of doing first proper testing ;) Because if you test your use case you either get a positive or negative result and not a problem. However I do have to admit that ceph could be more transparent with publishing testing and performance results. I have already discussed this with them on such a ceph day. It does not make sense to have to do everything yourself eg the luks overhead and putting the db/wal on ssd, rbd performance on hdds etc. Those can quickly show if ceph can be a candidate or not. -----Original Message----- From: Kevin Myers [mailto:response@ifastnet.com] Cc: Janne Johansson; Marc Roos; ceph-devel; ceph-users Subject: Re: [ceph-users] Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs Tbh ceph caused us more problems than it tried to fix ymmv good luck > On 22 Sep 2020, at 13:04, tri(a)postix.net wrote: > > The key is stored in the ceph cluster config db. It can be retrieved > by > > KEY=`/usr/bin/ceph --cluster ceph --name > client.osd-lockbox.${OSD_FSID} --keyring $OSD_PATH/lockbox.keyring > config-key get dm-crypt/osd/$OSD_FSID/luks` > > September 22, 2020 2:25 AM, "Janne Johansson" <icepic.dz(a)gmail.com> wrote: > >> Den mån 21 sep. 2020 kl 16:15 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: >> >>> When I create a new encrypted osd with ceph volume[1] >>> >>> Q4: Where is this luks passphrase stored? >> >> I think the OSD asks the mon for it after auth:ing, so "in the mon DBs" >> somewhere. >> >> -- >> May the most significant bit of your life be positive. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

3 years, 7 months

1
0
0 0

Ceph MDS stays in "up:replay" for hours. MDS failover takes 10-15 hours.

by heilig.oleg＠gmail.com

Hi there, We have 9 nodes Ceph cluster. Ceph version is 15.2.5. The cluster has 175 OSD (HDD) + 3 NVMe for cache tier for "cephfs_data" pool. CephFS pools info: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 350 TiB 179.53M 350 TiB 66.93 87 TiB cephfs_metadata 3 3.1 TiB 17.69M 3.1 TiB 1.77 87 TiB We use multiple active MDS instances: 3 "active" and 3 "standby". Each MDS server has 128GB RAM, "mds cache memory limit" = 64GB. Failover to a standby MDS instance takes 10-15 hours! CephFS is unreachable for the clients all this time. The MDS instance just stays in "up:replay" state for all this time. It looks like MDS demon checking all of the folders: 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EOpen.replay 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay 3 dirlumps by unknown.0 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay dir 0x300000041c5 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x300000041c5 /repository/files/14/ [2,head] auth v=2070324 cv=0/0 state=1610612737|complete f(v0 m2020-09-10T13:05:29.297254-0700 515=0+515) n(v46584 rc2020-09-21T20:38:49.071043-0700 b3937793650802 1056114=601470+454644) hs=515+0,ss=0+0 dirty=75 | child=1 subtree=0 dirty=1 0x55d4c9359b80] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay for [2,head] had [dentry #0x1/repository/files/14/14119 [2,head] auth (dversion lock) v=2049516 ino=0x30000812e2f state=1073741824 | inodepin=1 0x55db2463a1c0] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay for [2,head] had [inode 0x30000812e2f [...2,head] /repository/files/14/14119/ auth fragtree_t(*^3) v2049516 f(v0 m2020-09-18T10:17:53.379121-0700 13498=0+13498) n(v6535 rc2020-09-19T05:52:25.035403-0700 b272027384385 112669=81992+30677) (iversion lock) | dirfrag=8 0x55db24643000] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay dir 0x30000812e2f.000* 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x30000812e2f.000* /repository/files/14/14119/ [2,head] auth v=77082 cv=0/0 state=1073741824 f(v0 m2020-09-18T10:17:53.371122-0700 1636=0+1636) n(v6535 rc2020-09-19T05:51:18.063949-0700 b33321023818 13707=9986+3721) hs=885+0,ss=0+0 | child=1 0x55db845bf080] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/repository/files/14/14119/39823 [2,head] auth NULL (dversion lock) v=0 ino=(nil) state=1073741888|bottomlru 0x55d82061a900] We tried standby-replay and it helps but doesn't eliminate the root cause. We have millions folders with millions of small files. When the folders/subfolders scan is done, CephFS is active again. I believe 10 hours downtime is unexpected behaviour. Is there any way to force MDS to change status to active and run all of the required directory checks in the background? How can I localise the root cause?

3 years, 7 months

2
1
0 0

one-liner getting block device from mounted osd

by Marc Roos

I have a optimize script that I run after the reboot of a ceph node. It sets among other things /sys/block/sdg/queue/read_ahead_kb and /sys/block/sdg/queue/nr_requests of block devices being used for osd's. Normally I am using the mount command to discover these but with the tmpfs and ceph-volume this does not work. Is anyone able to share a simple onliner getting from osd's the used block devices? Eg from both types at the same time. Or do I really need to traverse this block -> /dev/mapper/... ?

3 years, 7 months

1
1
0 0

Re: rgw.none vs quota

by Jean-Sebastien Landry

Manuel & Konstantin, thank you to confirm this. I should upgrade to Nautilus in the next few weeks. I just live with it for now. Thanks!

3 years, 7 months

1
0
0 0

Slow cluster and incorrect peers

by Nico Schottelius

Hello again, following up on the previous mail, one cluster gets rather slow at the moment and we have spotted something "funny": When checking ceph pg dump we see some osds have HB peers with osds that they should not have any pg in common with. When restarting one of the effected osds, we get the following message: mon_cmd_maybe_osd_create fail: 'osd.12 has already bound to class 'xxx-ssd', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy When checking the output of ceph osd tree, it seems to be in the correct class: 12 xxx-ssd 0.21767 osd.12 up 1.00000 1.00000 Is it possible that the osd has "multiple" classes / that the cluster remebers a class that was set to osd.12 when it used to be an HDD? The output of ceph pg dump includes at the bottom this OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 12 150 GiB 72 GiB 151 GiB 223 GiB [3,11,13,25,36,43,54,64,71,82] 128 35 which is wrong, because osd.12 should only peer with osd.3 and osd.25, which are the only ones in the same pool that has the replicated rule set to match on xxx-ssd. And the obvious question: how do we fix this? At the moment we see around 75 pgs in peering and 39 activating, most of them which are in a pool with slower SSDs, but it seems that these peerings affect another pool that should have faster SSDs. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

3 years, 7 months

1
0
0 0

Mount CEPH-FS on multiple hosts with concurrent access to the same data objects?

by René Bartsch

I'm new on the list, so a "Hello" to all! :-) We're planning a Proxmox-Cluster. The data-center operator advised to use a virtual machine with NFS on top of a single CEPH-FS instance to mount the shared CEPH-FS storage on multiple hosts/VMs. As this NFS/CEPH-FS-VM could be a bottle-neck I was wondering if CEPH- FS is capable to manage concurrent access and locking itself. Is it possible to mount CEPH-FS instances on multiple hosts (e.g. /srv) all accessing the same data objects without data-loss or dead-locks by concurrent access? Will this perform better than a single NFS/CEPH-FS instance (VM)? Thanx for any hint Renne

3 years, 7 months

5
7
0 0

2024

2023

2022

2021

2020

2019

ceph-users September 2020