- ceph-users - lists.ceph.io

Cephadm: Upgrade 15.2.5 -> 15.2.9 stops on non existing OSD
by Kai Stian Olstad 11 Mar '21

11 Mar '21

Before I started the upgrade the cluster was healthy but one OSD(osd.355) was down, can't remember if it was in or out. Upgrade was started with ceph orch upgrade start --image goharbor.example.com/library/ceph/ceph:v15.2.9 The upgrade started but when Ceph tried to upgrade osd.355 it paused with the following messages: 2021-03-11T09:15:35.638104+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Target is goharbor.example.com/library/ceph/ceph:v15.2.9 with id dfc48307963697ff48acd9dd6fda4a7a24017b9d8124f86c2 a542b0802fe77ba 2021-03-11T09:15:35.639882+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking mgr daemons... 2021-03-11T09:15:35.644170+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: All mgr daemons are up to date. 2021-03-11T09:15:35.644376+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking mon daemons... 2021-03-11T09:15:35.647669+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: All mon daemons are up to date. 2021-03-11T09:15:35.647866+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking crash daemons... 2021-03-11T09:15:35.652035+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Setting container_image for all crash... 2021-03-11T09:15:35.653683+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: All crash daemons are up to date. 2021-03-11T09:15:35.653896+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking osd daemons... 2021-03-11T09:15:36.273345+0000 mgr.pech-mon-2.cjeiyc [INF] It is presumed safe to stop ['osd.355'] 2021-03-11T09:15:36.273504+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: It is presumed safe to stop ['osd.355'] 2021-03-11T09:15:36.273887+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Redeploying osd.355 2021-03-11T09:15:36.276673+0000 mgr.pech-mon-2.cjeiyc [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.355 on host pech-hd-009 failed. One of the first ting the upgrade did was to upgrade mon, so they are restarted and now the osd.355 no longer exist $ ceph osd info osd.355 Error EINVAL: osd.355 does not exist But if I run a resume ceph orch upgrade resume it still tries to upgrade osd.355, same message as above. I tried to stop and start the upgrade again with ceph orch upgrade stop ceph orch upgrade start --image goharbor.example.com/library/ceph/ceph:v15.2.9 it still tries to upgrade osd.355, with the same message as above. Looking at the source code it looks like it get daemons to upgrade from mgr cache, so I restarted both mgr but still it tries to upgrade osd.355. Does anyone know how I can get the upgrade to continue? -- Kai Stian Olstad

2 4

NVME pool creation time :: OSD services strange state
by Adrian Sevcenco 11 Mar '21

11 Mar '21

Hi! So, after i selected the tags to add 2 nvme ssds i declared a replicated n=2 pool .. and for the last 30 min the progress shown in notification is 0% and iotop shows around 100K/s for 2 (???) ceph-mon processes and that all ... and in my service list the osd services look somehow empty: https://prntscr.com/10iwwbh what did i miss? Thanks a lot! Adrian

1 2

Openstack rbd image Error deleting problem
by Norman.Kern 11 Mar '21

11 Mar '21

Hi Guys, I have used Ceph rbd for Openstack for sometime, I met a problem while destroying a VM. The Openstack tried to delete rbd image but failed. I have a test deleting a image by rbd command, it costs lots of time(image size 512G or more). Anyone met the same problem with me? Thanks, Norman

2 3

cephadm (curl master)/15.2.9:: how to add orchestration
by Adrian Sevcenco 11 Mar '21

11 Mar '21

Hi! After an initial bumpy bootstrapping (IMHO the defaults should be whatever is already defined in .ssh of the user and custom values setup with cli arguments) now i'm stuck adding any service/hosts/osds because apparently i lack orchestration .. the the documentation show a big "Page does not exist" see https://docs.ceph.com/en/latest/docs/octopus/mgr/orchestrator so, what is it and what options do i have? to set up it seems that is as easy as: ceph orch set backend I just started with ceph and i just want to start a ceph service (i cannot call it a cluster) on my desktop (with 2 dedicated osds) to get familiar also with usage. Thanks a lot! Adrian

4 4

how to tell balancer to balance
by Boris Behrens 11 Mar '21

11 Mar '21

Hi, I know this topic seems to be handled a lot (as far as I can see), but I reached the end of my google_foo. * We have OSDs that are near full, but there are also OSDs that are only loaded with 50%. * We have 4,8,16 TB rotating disks in the cluster. * The disks that get packed are 4TB disks and very empty disks are also 4TB * The OSD nodes are all around the same total disk space (51 - 59) * The balancer tells me that it can not find further optimization, or that pg_num is decreasin. How can I debug further before the cluster goes into a bad state? [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | head -n 30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME MIN/MAX VAR: 0.75/1.23 STDDEV: 6.96 TOTAL 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 TiB 70.39 -3 58.49872 - 58 TiB 39 TiB 36 TiB 8.2 GiB 85 GiB 19 TiB 67.40 0.96 - host s3db2 -4 58.49872 - 58 TiB 40 TiB 35 TiB 35 GiB 81 GiB 19 TiB 67.91 0.96 - host s3db3 -11 50.94173 - 51 TiB 35 TiB 35 TiB 3.5 GiB 94 GiB 16 TiB 68.00 0.97 - host s3db10 -10 51.28331 - 51 TiB 35 TiB 35 TiB 4.6 GiB 93 GiB 16 TiB 69.16 0.98 - host s3db9 -6 58.89636 - 59 TiB 41 TiB 40 TiB 2.4 GiB 102 GiB 18 TiB 69.15 0.98 - host s3db5 -12 50.99052 - 51 TiB 36 TiB 36 TiB 1.8 GiB 93 GiB 15 TiB 69.99 0.99 - host s3db11 -2 58.20561 - 58 TiB 41 TiB 37 TiB 9.6 GiB 96 GiB 17 TiB 70.00 0.99 - host s3db1 -1 673.44452 - 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 TiB 70.39 1.00 - root default -5 58.49872 - 58 TiB 42 TiB 35 TiB 7.0 GiB 94 GiB 17 TiB 71.06 1.01 - host s3db4 -8 58.89636 - 59 TiB 42 TiB 42 TiB 3.6 GiB 108 GiB 17 TiB 71.91 1.02 - host s3db7 -7 58.89636 - 59 TiB 43 TiB 42 TiB 15 GiB 120 GiB 16 TiB 72.69 1.03 - host s3db6 -37 58.55478 - 59 TiB 43 TiB 43 TiB 4.4 GiB 117 GiB 16 TiB 73.18 1.04 - host s3db12 -9 51.28331 - 51 TiB 38 TiB 38 TiB 4.9 GiB 103 GiB 13 TiB 74.18 1.05 - host s3db8 15 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 2.1 GiB 0 B 1.7 TiB 52.87 0.75 45 up osd.15 6 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 1.7 GiB 0 B 1.7 TiB 52.90 0.75 46 up osd.6 12 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 570 MiB 0 B 1.7 TiB 53.04 0.75 41 up osd.12 81 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 895 MiB 0 B 1.7 TiB 54.26 0.77 51 up osd.81 27 hdd 3.73630 1.00000 3.7 TiB 2.1 TiB 2.0 TiB 6.8 MiB 5.8 GiB 1.6 TiB 56.12 0.80 47 up osd.27 3 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.6 TiB 510 MiB 0 B 1.6 TiB 57.04 0.81 51 up osd.3 5 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 431 MiB 0 B 1.5 TiB 57.88 0.82 49 up osd.5 80 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 1.8 GiB 0 B 1.5 TiB 58.31 0.83 51 up osd.80 25 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 4.1 MiB 6.1 GiB 1.5 TiB 58.91 0.84 39 up osd.25 0 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 83 MiB 6.2 GiB 1.5 TiB 60.03 0.85 46 up osd.0 79 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.4 TiB 1.8 GiB 0 B 1.4 TiB 62.53 0.89 47 up osd.79 61 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 1.1 GiB 12 GiB 2.7 TiB 62.80 0.89 101 up osd.61 67 hdd 7.27739 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 557 MiB 13 GiB 2.7 TiB 63.29 0.90 96 up osd.67 72 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 107 MiB 11 GiB 2.7 TiB 63.36 0.90 87 up osd.72 [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail 51 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 724 MiB 14 GiB 1.7 TiB 76.34 1.08 105 up osd.51 71 hdd 3.68750 1.00000 3.7 TiB 2.8 TiB 2.8 TiB 3.7 MiB 7.8 GiB 867 GiB 77.04 1.09 47 up osd.71 82 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 839 GiB 628 MiB 0 B 839 GiB 77.48 1.10 45 up osd.82 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 777 GiB 18 GiB 0 B 777 GiB 79.14 1.12 59 up osd.14 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 752 GiB 826 MiB 0 B 752 GiB 79.80 1.13 53 up osd.4 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 523 MiB 8.2 GiB 757 GiB 79.95 1.14 53 up osd.75 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 237 MiB 9.2 GiB 668 GiB 82.30 1.17 50 up osd.76 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 380 MiB 8.5 GiB 671 GiB 82.46 1.17 57 up osd.33 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 464 MiB 8.4 GiB 605 GiB 84.18 1.20 60 up osd.34 35 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 352 MiB 8.7 GiB 515 GiB 86.55 1.23 53 up osd.35 [root@s3db1 ~]# ceph balancer status { "last_optimize_duration": "0:00:00.020142", "plans": [], "mode": "upmap", "active": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "last_optimize_started": "Thu Mar 11 13:42:32 2021" } [root@s3db1 ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 673 TiB 199 TiB 474 TiB 474 TiB 70.41 TOTAL 673 TiB 199 TiB 474 TiB 474 TiB 70.41 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 0 64 0 B 0 0 B 0 19 TiB .rgw.root 1 64 98 KiB 118 98 KiB 0 19 TiB eu-central-1.rgw.control 2 64 0 B 8 0 B 0 19 TiB eu-central-1.rgw.data.root 3 64 1022 KiB 3.02k 1022 KiB 0 19 TiB eu-central-1.rgw.gc 4 64 84 MiB 32 84 MiB 0 19 TiB eu-central-1.rgw.log 5 64 220 MiB 564 220 MiB 0 19 TiB eu-central-1.rgw.users.uid 6 64 2.8 MiB 6.89k 2.8 MiB 0 19 TiB eu-central-1.rgw.users.keys 7 64 262 KiB 6.70k 262 KiB 0 19 TiB eu-central-1.rgw.meta 8 64 384 KiB 1k 384 KiB 0 19 TiB eu-central-1.rgw.users.email 9 64 40 B 1 40 B 0 19 TiB eu-central-1.rgw.buckets.index 10 64 10 GiB 67.54k 10 GiB 0.02 19 TiB eu-central-1.rgw.buckets.data 11 1024 151 TiB 106.48M 151 TiB 72.67 19 TiB eu-central-1.rgw.buckets.non-ec 12 64 268 MiB 13.03k 268 MiB 0 19 TiB eu-central-1.rgw.usage 13 64 501 MiB 32 501 MiB 0 19 TiB eu-msg-1.rgw.control 56 64 0 B 8 0 B 0 19 TiB eu-msg-1.rgw.data.root 57 64 71 KiB 221 71 KiB 0 19 TiB eu-msg-1.rgw.gc 58 64 60 KiB 32 60 KiB 0 19 TiB eu-msg-1.rgw.log 59 64 835 KiB 242 835 KiB 0 19 TiB eu-msg-1.rgw.users.uid 60 64 56 KiB 107 56 KiB 0 19 TiB eu-msg-1.rgw.usage 61 64 36 MiB 25 36 MiB 0 19 TiB eu-msg-1.rgw.users.keys 62 64 3.8 KiB 97 3.8 KiB 0 19 TiB eu-msg-1.rgw.meta 63 64 600 KiB 1.58k 600 KiB 0 19 TiB eu-msg-1.rgw.buckets.index 64 64 46 MiB 112 46 MiB 0 19 TiB eu-msg-1.rgw.users.email 65 64 0 B 0 0 B 0 19 TiB eu-msg-1.rgw.buckets.data 66 64 2.8 TiB 1.14M 2.8 TiB 4.76 19 TiB eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB 353 2.2 MiB 0 19 TiB default.rgw.control 69 32 0 B 8 0 B 0 19 TiB default.rgw.data.root 70 32 0 B 0 0 B 0 19 TiB default.rgw.gc 71 32 0 B 0 0 B 0 19 TiB default.rgw.log 72 32 0 B 0 0 B 0 19 TiB default.rgw.users.uid 73 32 0 B 0 0 B 0 19 TiB fra-1.rgw.control 74 32 0 B 8 0 B 0 19 TiB fra-1.rgw.meta 75 32 0 B 0 0 B 0 19 TiB fra-1.rgw.log 76 32 50 B 28 50 B 0 19 TiB whitespace-again-2021-03-10 77 64 111 MiB 363.94k 111 MiB 0 19 TiB whitespace-again-2021-03-10_2 78 32 18 KiB 59 18 KiB 0 19 TiB whitespace-again-2021-03-10_3 79 32 11 KiB 36 11 KiB 0 19 TiB -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: how smart is ceph recovery?
by Marc 11 Mar '21

11 Mar '21

> > > > 2. If a down host comes up again and it's osd are started. Is data > still being copied, or does ceph see that checksums(?) > > PG or RADOS object epoch I think. So if data hasn’t changed, the > recovery completes without having anything to do. > > > are the same and just sets a pointer(?) > back to the old location? Yes I mean the osdmap having the old osd on the node that was down. Hmmm, I currently have pg's in 'active+remapped+backfill_wait' of a pool rbd.backup. Of which I know nothing in it has changed. And those osd's listed[1] are already up. Especially when you are getting at the 'end' of recovery where osd_max_backfills=X has less effect and recovery takes longer. It would be nice to have the "no work needed pg's" be processed quickly. [1] [18,25,41]p18 [18,41,17] [5,0,25]p5 [5,0,4

1 0

Unpurgeable rbd image from trash
by Enrico Bocchi 11 Mar '21

11 Mar '21

Hello everyone, We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration. When trying to delete manually, it says there are some watchers, but this is actually not the case: # rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed. # rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc # We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before? Many thanks! Cheers, Enrico -- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland

2 4

A practical approach to efficiently store 100 billions small objects in Ceph
by Loïc Dachary 11 Mar '21

11 Mar '21

Bonjour, In the past weeks a few mailing list threads[0][1][2] explored the problem of storing billions of small objects in Ceph. There was great feedback (I learned at lot) and it turns out the solution is a rather simple aggregation of the ideas that were suggested during these discussions. It is described in detail here: https://wiki.softwareheritage.org/wiki/A_practical_approach_to_efficiently_… The next step will be to write and run benchmarks[3]. Although I'm convinced it is a good solution, I've been wrong before and confirmation is required :-) Many thanks to all the participants in the discussions on behalf of the Software Heritage project[4]. Cheers [0] Storing 20 billions of immutable objects in Ceph, 75% <16KB https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JSG2TXKNXPX… [1] Small RGW objects and RADOS 64KB minimun size https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF… [2] Using RBD to pack billions of small files https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX… [3] https://forge.softwareheritage.org/T3054 [4] https://www.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre

4 8

RadosGW unable to start resharding
by Ansgar Jazdzewski 11 Mar '21

11 Mar '21

Hi Folks, We are running ceph 14.2.16 and I like to reshard a bucket because I have a large object warning! so I did: radosgw-admin bucket reshard --tenant="..." --bucket="..." --uid="..." --num-shards=512 but I got receive an error: ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time `radosgw-admin reshard list` is empty so I assume I have to delete some leftovers from the old resharding!? did someone has had this before? thanks for your input, Ansgar

2 3

Bluestore OSD Layout - WAL, DB, Journal
by Dave Hall 10 Mar '21

10 Mar '21

Hello, I'm in the process of doubling the number of OSD nodes in my Nautilus cluster - from 3 to 6. Based on answers receive from earlier posts to this list, the new nodes have more NVMe that the old nodes. More to the point, on the original nodes the amount of NVMe allocated to each OSD was about 120GB, so the RocksDB was limited to 30GB. However, for my workload 300GB is probably recommended. As I prepare to lay out the NVMe on these new nodes, I'm still trying to understand how to size the DB and WAL for my OSDs and whether Journal is even needed. According to https://docs.ceph.com/en/nautilus/ceph-volume/lvm/prepare/ <https://docs.ceph.com/en/nautilus/ceph-volume/lvm/prepare/> > Bluestore supports the following configurations: > > * A block device, a block.wal, and a block.db device > * A block device and a block.wal device > * A block device and a block.db device > * A single block device > First question: On my first nodes I managed to get a DB, but no WAL. My current perception is that WAL and DB occupy separate physical/logical partitions. By specifying a WAL size and a DB size, ceph-volume will create the corresponding logical volumes on the NVMe. Is this correct? It is also possible to lay these out as basic logical partitions? Second question: How do I decide whether I need WAL, DB, or both? Third question: Once I answer the above WAL/DB question, what are the guidelines for sizing them? Thanks. -Dave -- Dave Hall Binghamton University

2 1