May 2021 - ceph-users - lists.ceph.io

Re: after upgrade to 16.2.3 16.2.4 and after adding few hdd's OSD's started to fail 1 by 1.
by Neha Ojha 14 May '21

14 May '21

You are welcome! We still need to get to the bottom of this, I will update the tracker to make a note of this occurrence. Thanks, Neha On Fri, May 14, 2021 at 12:25 PM Andrius Jurkus <andrius.jurkus(a)unishop.lt> wrote: > > Big thanks, Much appreciated help. > > It probably is same bug. > > bluestore_allocator = bitmap > > by setting this parameter all failed OSD started. > > Thanks again! > > On 2021-05-14 21:09, Neha Ojha wrote: > > On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus > > <andrius.jurkus(a)unishop.lt> wrote: > >> > >> Hello, I will try to keep it sad and short :) :( PS sorry if this > >> dublicate I tried post it from web also. > >> > >> Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. > >> After data migration for few hours, 1 SSD failed, then another and > >> another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same > >> host has SSD and HDD, but only SSD's are failing so I think this has > >> to > >> be balancing refiling or something bug and probably not upgrade bug. > >> > >> Cluster has been in pause for 4 hours and no more OSD's are failing. > >> > >> full trace > >> https://pastebin.com/UxbfFYpb > > > > This looks very similar to https://tracker.ceph.com/issues/50656. > > Adding Igor for more ideas. > > > > Neha > > > >> > >> Now I m googling and learning but, Is there a way how to easily test > >> lets say 15.2.XX version on osd without losing anything? > >> > >> Any help would be appreciated. > >> > >> Error start like this > >> > >> May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for > >> 4e01640b-951b-4f75-8dca-0bad4faf1b11... > >> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > >> 16:58:53.057836433 +0000 UTC m=+0.454352919 container create > >> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > >> GIT_BRANCH=HEAD, maintainer=D > >> May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun > >> container. > >> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > >> 16:58:53.3394116 +0000 UTC m=+0.735928098 container init > >> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > >> maintainer=Dimitri Savineau <dsav > >> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > >> 16:58:53.446921192 +0000 UTC m=+0.843437626 container start > >> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > >> GIT_BRANCH=HEAD, org.label-sch > >> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > >> 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach > >> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > >> org.label-schema.name=CentOS > >> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > >> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 > >> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > >> /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev > >> /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 > >> --path /var/lib/ceph/osd/ceph-2 --no-mon-config > >> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > >> /usr/bin/ln -snf > >> /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 > >> /var/lib/ceph/osd/ceph-2/block > >> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > >> /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block > >> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > >> /usr/bin/chown -R ceph:ceph /dev/dm-1 > >> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: > >> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 > >> May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm > >> activate successful for osd ID: 2 > >> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 > >> 16:58:53.8147653 +0000 UTC m=+1.211281741 container died > >> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) > >> May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 > >> 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove > >> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, > >> CEPH_POINT_RELEASE=-16.2.4, R > >> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 > >> 16:58:55.594265612 +0000 UTC m=+0.369978347 container create > >> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, > >> org.label-schema.build-d > >> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 > >> 16:58:55.864589286 +0000 UTC m=+0.640302021 container init > >> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, > >> org.label-schema.schema-version=1.0, GIT > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 > >> (ceph:ceph) > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 > >> (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process > >> ceph-osd, pid 2 > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore > >> empty > >> --pid-file > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 > >> /var/lib/ceph/osd/ceph-2/block) open path > >> /var/lib/ceph/osd/ceph-2/block > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 > >> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, > >> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 > >> bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size > >> 3221225472 meta 0.45 kv 0.45 data 0.06 > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 > >> /var/lib/ceph/osd/ceph-2/block) open path > >> /var/lib/ceph/osd/ceph-2/block > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 > >> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, > >> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device > >> bdev > >> 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB > >> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 > >> /var/lib/ceph/osd/ceph-2/block) close > >> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 > >> 16:58:55.972267166 +0000 UTC m=+0.747979911 container start > >> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > >> (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, > >> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, > >> GIT_REPO=https://github.com/ > >> May 14 16:58:55 dragon-ball-radar bash[113558]: > >> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da > >> May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for > >> 4e01640b-951b-4f75-8dca-0bad4faf1b11. > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 > >> /var/lib/ceph/osd/ceph-2/block) close > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data > >> /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 > >> address in networks '10.0.199.0/24' interfaces '' > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 > >> address in networks '172.16.199.0/24' interfaces '' > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc > >> load: isa > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 > >> /var/lib/ceph/osd/ceph-2/block) open path > >> /var/lib/ceph/osd/ceph-2/block > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 > >> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, > >> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 > >> bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size > >> 3221225472 meta 0.45 kv 0.45 data 0.06 > >> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug > >> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 > >> /var/lib/ceph/osd/ceph-2/block) close > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> >

1 0

Re: "No space left on device" when deleting a file
by Mark Schouten 14 May '21

14 May '21

[Resent because of incorrect ceph-users@ address..] On Tue, Mar 26, 2019 at 05:19:24PM +0000, Toby Darling wrote: > Hi Dan > > Thanks! > > ceph tell mds.ceph1 config set mds_bal_fragment_size_max 200000 > > got us running again. This helped me too. However, should I see num_strays decrease again? I'm running a `find -ls` over my CephFS tree.. -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

1 2

cephadm stalled after adjusting placement
by Bryan Stillwell 14 May '21

14 May '21

I'm looking for help in figuring out why cephadm isn't making any progress after I told it to redeploy an mds daemon with: ceph orch daemon redeploy mds.cephfs.aladdin.kgokhr ceph/ceph:v15.2.12 The output from 'ceph -W cephadm' just says: 2021-05-14T16:24:46.628084+0000 mgr.paris.glbvov [INF] Schedule redeploy daemon mds.cephfs.aladdin.kgokhr However, the mds never gets redeployed. I do see this warning in 'ceph health detail' which might have something to do with it: Module 'cephadm' has failed: 'NoneType' object has no attribute 'target_id' What steps can I do to figure out why cephadm is hung? Thanks, Bryan

1 0

Ceph 16.2.3 issues during upgrade from 15.2.10 with cephadm/lvm list
by David Orman 14 May '21

14 May '21

Hi, We are seeing the mgr attempt to apply our OSD spec on the various hosts, then block. When we investigate, we see the mgr has executed cephadm calls like so, which are blocking: root 1522444 0.0 0.0 102740 23216 ? S 17:32 0:00 \_ /usr/bin/python3 /var/lib/ceph/XXXXX/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90 --image docker.io/ceph/ceph@sha256:694ba9cdcbe6cb7d25ab14b34113c42c2d1af18d4c79c7ba4d1f62cf43d145fe ceph-volume --fsid XXXXX -- lvm list --format json This occurs on all hosts in the cluster, following starting/restarting/failing over a manager. It's blocking an in-progress upgrade post-manager updates on one cluster, currently. Looking at the cephadm logs on the host(s) in question, we see the last entry appears to be truncated, like: 2021-05-10 17:32:06,471 INFO /usr/bin/podman: "ceph.db_uuid": "1n2f5v-EEgO-1Kn6-hQd2-v5QF-AN9o-XPkL6b", 2021-05-10 17:32:06,471 INFO /usr/bin/podman: "ceph.encrypted": "0", 2021-05-10 17:32:06,471 INFO /usr/bin/podman: "ceph.osd_fsid": "XXXX", 2021-05-10 17:32:06,471 INFO /usr/bin/podman: "ceph.osd_id": "205", 2021-05-10 17:32:06,471 INFO /usr/bin/podman: "ceph.osdspec_affinity": "osd_spec", 2021-05-10 17:32:06,471 INFO /usr/bin/podman: "ceph.type": "block", The previous entry looks like this: 2021-05-10 17:32:06,469 INFO /usr/bin/podman: "ceph.db_uuid": "TMTPD5-MLqp-06O2-raqp-S8o5-TfRG-hbFmpu", 2021-05-10 17:32:06,469 INFO /usr/bin/podman: "ceph.encrypted": "0", 2021-05-10 17:32:06,469 INFO /usr/bin/podman: "ceph.osd_fsid": "XXXX", 2021-05-10 17:32:06,469 INFO /usr/bin/podman: "ceph.osd_id": "195", 2021-05-10 17:32:06,470 INFO /usr/bin/podman: "ceph.osdspec_affinity": "osd_spec", 2021-05-10 17:32:06,470 INFO /usr/bin/podman: "ceph.type": "block", 2021-05-10 17:32:06,470 INFO /usr/bin/podman: "ceph.vdo": "0" 2021-05-10 17:32:06,470 INFO /usr/bin/podman: }, 2021-05-10 17:32:06,470 INFO /usr/bin/podman: "type": "block", 2021-05-10 17:32:06,470 INFO /usr/bin/podman: "vg_name": "ceph-ffd1a4a7-316c-4c85-acde-06459e26f2c4" 2021-05-10 17:32:06,470 INFO /usr/bin/podman: } 2021-05-10 17:32:06,470 INFO /usr/bin/podman: ], We'd like to get to the bottom of this, please let us know what other information we can provide. Thank you, David

2 4

mon vanished after cephadm upgrade
by Ashley Merrick 14 May '21

14 May '21

I had a 3 mon CEPH cluster, after updating from 15.2.x to 16.2.x one of my mon's is showing as a stopped state in the Ceph Dashboard.And checking the cephadm logs on the server in question I can see "/usr/bin/docker: Error: No such object: ceph-30449cba-44e4-11eb-ba64-dda10beff041-mon.sn-m01"There is a few OSD services running on the same physical server and they all are starting/running fine via docker.I tried to do a cephadm apply mon to push a new mon to the same host, but it seems to not do anything, nothing shows in the same log file on sn-m01Also ceph -s shows full health and no errors and has no trace of the "failed" mon (not sure if this is expected), only in the ceph dashboard under services can I see the stopped not running mon. Sent via MXlogin

2 2

RGW segmentation fault on Pacific 16.2.1 with multipart upload
by Daniel Iwan 14 May '21

14 May '21

Hi I have started to see segfaults during multiplart upload to one of the buckets File is about 60MB in size Upload of the same file to a brand new bucket works OK Command used aws --profile=tester --endpoint=$HOST_S3_API --region="" s3 cp ./pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack s3://tester-bucket/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack For some reason log shows upload to tester-bucket-2 ??? Bucket tester-bucket-2 is owned by the same user TESTER. I'm using Ceph 16.2.1 (recently upgraded from Octopus). Installed with cephadm in Docker OS Ubuntu 18.04.5 LTS Logs show as below May 11 11:00:46 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:46.891+0000 7ffb0e25e700 1 ====== starting new request req=0x7ffa8e15d620 ===== May 11 11:00:46 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:46.907+0000 7ffb0b258700 1 ====== req done req=0x7ffa8e15d620 op status=0 http_status=200 latency=0.011999841s ====== May 11 11:00:46 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:46.907+0000 7ffb0b258700 1 beast: 0x7ffa8e15d620: 11.1.150.14 - TESTER [11/May/2021:11:00:46.891 +0000] "POST /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploads HTTP/1.1" 200 296 - "aws-cli/2.1.23 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.18 p May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.055+0000 7ffb09254700 1 ====== starting new request req=0x7ffa8e15d620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.355+0000 7ffb51ae5700 1 ====== starting new request req=0x7ffa8e0dc620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.355+0000 7ffb4eadf700 1 ====== starting new request req=0x7ffa8e05b620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.355+0000 7ffb46acf700 1 ====== starting new request req=0x7ffa8df59620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.355+0000 7ffb44acb700 1 ====== starting new request req=0x7ffa8ded8620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.355+0000 7ffb3dabd700 1 ====== starting new request req=0x7ffa8dfda620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.359+0000 7ffb1d27c700 1 ====== starting new request req=0x7ffa8de57620 ===== May 11 11:00:47 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:47.359+0000 7ffb22a87700 1 ====== starting new request req=0x7ffa8ddd6620 ===== May 11 11:00:48 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:48.275+0000 7ffb2d29c700 1 ====== req done req=0x7ffa8e15d620 op status=0 http_status=200 latency=1.219983697s ====== May 11 11:00:48 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:48.275+0000 7ffb2d29c700 1 beast: 0x7ffa8e15d620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.055 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=8 HTTP/1.1" 200 2485288 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:00:54 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:54.695+0000 7ffad89f3700 1 ====== req done req=0x7ffa8ddd6620 op status=0 http_status=200 latency=7.335902214s ====== May 11 11:00:54 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:54.695+0000 7ffad89f3700 1 beast: 0x7ffa8ddd6620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.359 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=6 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:00:56 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:56.871+0000 7ffb11a65700 1 ====== req done req=0x7ffa8e0dc620 op status=0 http_status=200 latency=9.515872955s ====== May 11 11:00:56 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:56.871+0000 7ffb11a65700 1 beast: 0x7ffa8e0dc620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.355 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=7 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:00:59 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:59.491+0000 7ffac89d3700 1 ====== req done req=0x7ffa8dfda620 op status=0 http_status=200 latency=12.135838509s ====== May 11 11:00:59 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:00:59.491+0000 7ffac89d3700 1 beast: 0x7ffa8dfda620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.355 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=2 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:01:02 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:02.891+0000 7ffb68312700 1 ====== req done req=0x7ffa8e05b620 op status=0 http_status=200 latency=15.535793304s ====== May 11 11:01:02 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:02.891+0000 7ffb68312700 1 beast: 0x7ffa8e05b620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.355 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=4 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.299+0000 7ffb70b23700 1 ====== req done req=0x7ffa8df59620 op status=0 http_status=200 latency=15.943787575s ====== May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.299+0000 7ffb70b23700 1 beast: 0x7ffa8df59620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.355 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=3 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.647+0000 7ffb8534c700 1 ====== req done req=0x7ffa8ded8620 op status=0 http_status=200 latency=16.291782379s ====== May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.647+0000 7ffb8534c700 1 beast: 0x7ffa8ded8620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.355 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=1 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.835+0000 7ffabe9bf700 1 ====== req done req=0x7ffa8de57620 op status=0 http_status=200 latency=16.475780487s ====== May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.835+0000 7ffabe9bf700 1 beast: 0x7ffa8de57620: 11.1.150.14 - TESTER [11/May/2021:11:00:47.359 +0000] "PUT /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF&partNumber=5 HTTP/1.1" 200 8388608 - "aws-cli/2.1.23 Python/3.7.3 Linux May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.875+0000 7ffabf1c0700 1 ====== starting new request req=0x7ffa8de57620 ===== May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.895+0000 7ffaa0983700 1 ====== req done req=0x7ffa8de57620 op status=0 http_status=200 latency=0.019999731s ====== May 11 11:01:03 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:03.895+0000 7ffaa0983700 1 beast: 0x7ffa8de57620: 11.1.150.14 - TESTER [11/May/2021:11:01:03.875 +0000] "POST /tester-bucket-2/pack-a9201afb4682b74c7c5a5d6070e661662bdfea1a.pack?uploadId=2~JhGavMwngl_FH6-LcE2vFxMRjcf4qTF HTTP/1.1" 200 400 - "aws-cli/2.1.23 Python/3.7.3 Linux/4.19.128-micros May 11 11:01:06 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:06.147+0000 7ffac31c8700 1 failed to read header: The socket was closed due to a timeout May 11 11:01:06 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:06.147+0000 7ffac31c8700 1 ====== req done http_status=400 ====== May 11 11:01:16 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:16.667+0000 7ffab51ac700 1 failed to read header: The socket was closed due to a timeout May 11 11:01:16 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:16.667+0000 7ffab51ac700 1 ====== req done http_status=400 ====== May 11 11:01:17 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:17.687+0000 7ffaa598d700 1 failed to read header: The socket was closed due to a timeout May 11 11:01:17 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:17.687+0000 7ffaa598d700 1 ====== req done http_status=400 ====== May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:18.179+0000 7ffa9e97f700 1 ====== starting new request req=0x7ffbd40b8620 ===== May 11 11:01:18 ceph-om-vm-node1 bash[27881]: *** Caught signal (Segmentation fault) ** May 11 11:01:18 ceph-om-vm-node1 bash[27881]: in thread 7ffac89d3700 thread_name:radosgw May 11 11:01:18 ceph-om-vm-node1 bash[27881]: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ffbc8558b20] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7ffbd3393403] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 3: (rgw::sal::RGWObject::get_obj() const+0x20) [0x7ffbd33c1bb0] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 4: (RGWInitMultipart::verify_permission(optional_yield)+0x6c) [0x7ffbd36abf4c] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x898) [0x7ffbd3373f58] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 7: /lib64/libradosgw.so.2(+0x49510d) [0x7ffbd32ca10d] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 8: /lib64/libradosgw.so.2(+0x496b74) [0x7ffbd32cbb74] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 9: /lib64/libradosgw.so.2(+0x496dde) [0x7ffbd32cbdde] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 10: make_fcontext() May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug 2021-05-11T11:01:18.183+0000 7ffac89d3700 -1 *** Caught signal (Segmentation fault) ** May 11 11:01:18 ceph-om-vm-node1 bash[27881]: in thread 7ffac89d3700 thread_name:radosgw May 11 11:01:18 ceph-om-vm-node1 bash[27881]: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ffbc8558b20] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7ffbd3393403] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 3: (rgw::sal::RGWObject::get_obj() const+0x20) [0x7ffbd33c1bb0] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 4: (RGWInitMultipart::verify_permission(optional_yield)+0x6c) [0x7ffbd36abf4c] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x898) [0x7ffbd3373f58] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 7: /lib64/libradosgw.so.2(+0x49510d) [0x7ffbd32ca10d] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 8: /lib64/libradosgw.so.2(+0x496b74) [0x7ffbd32cbb74] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 9: /lib64/libradosgw.so.2(+0x496dde) [0x7ffbd32cbdde] May 11 11:01:18 ceph-om-vm-node1 bash[27881]: 10: make_fcontext() May 11 11:01:18 ceph-om-vm-node1 bash[27881]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 11 11:01:18 ceph-om-vm-node1 bash[27881]: --- begin dump of recent events --- May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2732> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command assert hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2731> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command abort hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2730> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command leak_some_memory hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2729> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perfcounters_dump hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2728> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command 1 hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2727> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perf dump hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2726> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perfcounters_schema hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2725> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perf histogram dump hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2724> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command 2 hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2723> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perf schema hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2722> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perf histogram schema hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2721> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command perf reset hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2720> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command config show hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2719> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command config help hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2718> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command config set hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2717> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command config unset hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2716> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command config get hook 0x55f96fdf4580 May 11 11:01:18 ceph-om-vm-node1 bash[27881]: debug -2715> 2021-05-11T10:57:05.234+0000 7ffbd4103440 5 asok(0x55f96fef0000) register_command config diff hook 0x55f96fdf4580 Regards Daniel

2 2

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Limit memory of ceph-mgr
by mabi 14 May '21

14 May '21

Hello, I just noticed on my small Octopus cluster that the ceph-mgr on a mgr/mon node uses 3.6GB of resident memory (RES) as you can see below from the top output: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2704 167 20 0 5030528 3.6g 35796 S 6.6 47.2 23:08.18 ceph-mgr 2699 167 20 0 1291504 884796 23672 S 4.6 11.1 13:23.63 ceph-mon Is there a way to limit the memory usage of ceph-mgr just like one can do with ceph OSD (osd_memory_target)? I tried something like mgr_memory_target but that parameter does not exist. Thanks, Mabi

1 0

v14.2.21 Nautilus released
by David Galloway 14 May '21

14 May '21

This is a hotfix release addressing a number of security issues and regressions. We recommend all users update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-21-nautilus-released Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-14.2.21.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 5ef401921d7a88aea18ec7558f7f9374ebd8f5a6

3 2

How to "out" a mon/mgr node with orchestrator
by mabi 14 May '21

14 May '21

Hello, I need to re-install one node of my Octopus cluster (installed with cephadm) which is a mon/mgr node and did not find in the documentation how to do that with the new ceph orchestrator commands. So my question would be what are the "ceph orch" commands I need to run in order to "out" nicely the mgr and mon services from that specific node? I have a standby manager and 3 mons in total so from the redundancy it should be no problem to take that one node out for re-installing it. Best regards, Mabi

1 0

DNS and /etc/hosts in Pacific Release
by Paul Cuzner 14 May '21

14 May '21

Hi, One of the recent changes that Ceph Pacific has introduced is the removal of support for /etc/hosts on the ceph cluster nodes, that use *podman* as the container engine (CentOS8+) This means that name resolution from within the ceph containers now relies on either DNS, or the host to ip mapping that is created when adding a host to the cluster with the orchestrator CLI (orch host add). The exclusion of /etc/hosts has been implemented using a --no-hosts setting on the "podman run" command. Installations that use docker are unaffected. So if you're planning to use Ceph Pacific with podman *and* need /etc/hosts to work, it would be great to hear from you! Cheers, Paul Cuzner

1 0

Osd can not goto up/in status on arm64
by 赵贺东 14 May '21

14 May '21

Hi ceph-users, I deploy ceph on arm64 by cephadm. Mon/mgr seems work well. But for osd, it can not work well, that means I can add osd to my cluster but osd can not goto up&in status. I can not find helpful info from ceph log. Can anyone help me, or how can I find the reason why osd can not goto up/in status. Before add osd, raw disk is available. Add osd by "ceph orch apply osd --all-available-devices” Hardware: NXP LS1043A processor，64bit OS: Ubuntu 18.04.5 LTS (GNU/Linux 4.19.26 aarch64) Ceph:15.2.9/15.2.11/16.2.3

1 0

v15.2.12 Octopus released
by David Galloway 14 May '21

14 May '21

This is a hotfix release addressing a number of security issues and regressions. We recommend all users update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-12-octopus-released Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-15.2.12.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ce065eabfa5ce81323b009786bdf5bb03127cbe1

1 0

Ceph Octopus 15.2.11 - rbd diff --from-snap lists all objects
by David Herselman 13 May '21

13 May '21

Hi, Has something change with 'rbd diff' in Octopus or have I hit a bug? I am no longer able to obtain the list of objects that have changed between two snapshots of an image, it always lists all allocated regions of the RBD image. This behaviour however only occurs when I add the '--whole-object' switch. Using KRBD client with kernel 5.11.7 and Ceph Octopus 15.2.11 as part of Proxmox PVE 6.4 which is based on Debian 10. Images have the following features and I've performed offline object map checks and rebuilds (no errors reported). To reproduce my issue I first create a new RBD image (default features are 63), map it using KRBD, write some data, create first snapshot, write a single object (4 MiB), create a second snapshot and then list the differences: [admin@kvm1a ~]# rbd create rbd_hdd/test --size 40G [admin@kvm1a ~]# rbd info rbd_hdd/test rbd image 'test': size 40 GiB in 10240 objects order 22 (4 MiB objects) snapshot_count: 0 id: 73363f8443987b block_name_prefix: rbd_data.73363f8443987b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Wed May 12 23:01:11 2021 access_timestamp: Wed May 12 23:01:11 2021 modify_timestamp: Wed May 12 23:01:11 2021 [admin@kvm1a ~]# rbd map rbd_hdd/test /dev/rbd18 [admin@kvm1a ~]# dd if=/dev/zero of=/dev/rbd18 bs=64M count=1 1+0 records in 1+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 0.668701 s, 100 MB/s [admin@kvm1a ~]# sync [admin@kvm1a ~]# rbd snap create rbd_hdd/test@snap1 [admin@kvm1a ~]# dd if=/dev/zero of=/dev/rbd18 bs=4M count=1 1+0 records in 1+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.265691 s, 15.8 MB/s [admin@kvm1a ~]# sync [admin@kvm1a ~]# rbd snap create rbd_hdd/test@snap2 [admin@kvm1a ~]# rbd diff --from-snap snap1 rbd_hdd/test@snap2 --format=json [{"offset":0,"length":4194304,"exists":"true"}] [admin@kvm1b ~]# rbd diff --from-snap snap1 rbd_hdd/test@snap2 --format=json --whole-object [{"offset":0,"length":4194304,"exists":"true"},{"offset":4194304,"length":4194304,"exists":"true"},{"offset":8388608,"length":4194304,"exists":"true"},{"offset":12582912,"length":4194304,"exists":"true"},{"offset":16777216,"length":4194304,"exists":"true"},{"offset":20971520,"length":4194304,"exists":"true"},{"offset":25165824,"length":4194304,"exists":"true"},{"offset":29360128,"length":4194304,"exists":"true"},{"offset":33554432,"length":4194304,"exists":"true"},{"offset":37748736,"length":4194304,"exists":"true"},{"offset":41943040,"length":4194304,"exists":"true"},{"offset":46137344,"length":4194304,"exists":"true"},{"offset":50331648,"length":4194304,"exists":"true"},{"offset":54525952,"length":4194304,"exists":"true"},{"offset":58720256,"length":4194304,"exists":"true"},{"offset":62914560,"length":4194304,"exists":"true"}] [admin@kvm1a ~]# rbd du rbd_hdd/test NAME PROVISIONED USED test@snap1 40 GiB 64 MiB test@snap2 40 GiB 64 MiB test 40 GiB 4 MiB <TOTAL> 40 GiB 132 MiB My tests appear to confirm that adding the 'whole-object' option to rbd diff results in it listing every allocated extend instead of only the changes... Regards David Herselman

1 1

monitor connection error
by Tuffli, Chuck 13 May '21

13 May '21

Hi I'm new to ceph and have been following the Manual Deployment document [1]. The process seems to work correctly until step 18 ("Verify that the monitor is running"): [centos@cnode-01 ~]$ uname -a Linux cnode-01 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux [centos@cnode-01 ~]$ ceph -v ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) [centos@cnode-01 ~]$ sudo ceph --cluster es-c1 -s [errno 2] RADOS object not found (error connecting to the cluster) [centos@cnode-01 ~]$ What is this error trying to tell me? TIA [1] https://docs.ceph.com/en/latest/install/manual-deployment/

2 9

rgw bug adding null characters in multipart object names and in Etags
by Scheurer François 13 May '21

13 May '21

Dear All We are trying to remove old multipart uploads but get in trouble with some of them having null characters: rados -p zh-1.rgw.buckets.index rmomapkey .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 '_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC.25' rados -p zh-1.rgw.buckets.index rmomapkey .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 $(echo -ne '_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC\0.25') -bash: warning: command substitution: ignored null byte in input rados -p zh-1.rgw.buckets.index listomapkeys .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 | grep -a '_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC' | cat -A _multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC^@.25$ # <= not deleted ! It is not working, as the Null Char is stripped off. Any Idea how to proceed? This bucket was created on luminous. But this specific object was created after our upgrade to nautilus. Apparently some bugs have added NullChars at the end of MPU object names, between uploadid and suffix. Output from 'radosgw-admin bi list' (see the \u0000 NullChars): { "type": "plain", "idx": "_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC\u0000.25", "entry": { "name": "_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC\u0000.25", "instance": "", "ver": { "pool": 6, "epoch": 852938 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 157286400, "mtime": "2020-12-25 23:39:20.019898Z", "etag": "a126c2f0d439c44176a5d07bd5841575", "storage_class": "", "owner": "40eb21a9092c4948bcf94386f6042f94", "owner_display_name": "amsler1", "content_type": "", "accounted_size": 157286400, "user_data": "", "appendable": "false" }, "tag": "_vMx_4vu-E5nWf7kCHJIQCFPGEHRiUAG", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, On the same bucket, we also see NullChars at the end of some Etags when we using 'radosgw-admin bucket list --bucket' but not with 'radosgw-admin object stat': object='MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml' radosgw-admin object stat --bucket="$bucket" --object="$object" | jq -c '{name, size, etag, tag, obj_size: .manifest.obj_size, marker:.manifest.tail_placement.bucket.marker, bucket_id:.manifest.tail_placement.bucket.bucket_id}' | cat -A {"name":"MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml","size":39372,"etag":"0e73d594032900acb74d3f06b230aeb9","tag":"_xhNKxuWrfxDO5XfYs8Llq8vLTUYqtmm","obj_size":39372,"marker":"cb1594b3-a782-49d0-a19f-68cd48870a63.19334234.139","bucket_id":"cb1594b3-a782-49d0-a19f-68cd48870a63.20382694.169"}$ # <= no NullChar radosgw-admin bucket list --bucket "${bucket}" --allow-unordered --max-entries 20000000 | jq -c 'sort_by(.bucket) | .[] | {name, accounted_size: .meta.accounted_size, etag: .meta.etag}' | fgrep -a 0e73d594032900acb74d3f06b230aeb9 | cat -A {"name":"MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml","accounted_size":39372,"etag":"0e73d594032900acb74d3f06b230aeb9\u0000"}$ # <= no NullChar rados -p zh-1.rgw.buckets.data stat 'cb1594b3-a782-49d0-a19f-68cd48870a63.19334234.139_'"$object" zh-1.rgw.buckets.data/cb1594b3-a782-49d0-a19f-68cd48870a63.19334234.139_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml mtime 2020-04-21 14:21:27.000000, size 39372 This bucket was causing multi-site rgw sync to crash every minute when using rgw_sync_obj_etag_verify = true. These Etag NullChars may be the cause of this bug: * https://tracker.ceph.com/issues/49955 It may also be related to: * https://tracker.ceph.com/issues/23939 So we would be glad to know how to remove these NullChars from the Etags and how to remove the MPU's with NullChars in the object names... These both issues seem to be the cause of many weird behaviors: 1. rgw sync crashes (with rgw_sync_obj_etag_verify = true) 2. radosgw-admin bucket sync status --bucket "$bucket" --source-zone ch-zh1-az2 => reports "bucket is caught up with source" but when most of the objects are missing 3. radosgw-admin bucket list --bucket "$bucket" --allow-unordered --max-entries 99000000 => returns an imcomplete list 4. radosgw-admin bucket stats --bucket "$bucket" => returns wrong number of objects and utilized size The only reliable outputs is from bi list: * radosgw-admin bi list --bucket=$bucket | jq -cr 'map(select(.type == "plain" or .type == "instance") | .entry' Do you know if following commands may help and are safe in multi-site? * radosgw-admin bucket check --bucket $bucket --fix --check-objects * radosgw-admin bucket rewrite --bucket $bucket --min-rewrite-size 0 Or maybe only a dedicated tool need to be developped to deal with these NullChars? Many thanks in advance. Cheers Francois Scheurer -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

1 3

RGW federated user cannot access created bucket
by Daniel Iwan 13 May '21

13 May '21

Hi all Scenario is as follows Federated user assumes a role via AssumeRoleWithWebIdentity, which gives permission to create a bucket. User creates a bucket and becomes an owner (this is visible in Ceph's web ui as Owner $oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b). User cannot list the content of the bucket however, because role's policy does not give access to the bucket. Later on when user re-authenticates and assumes the same role again. At this point user cannot access a bucket it owns for the reason as above I'm assuming. Bucket's ACL after creation radosgw-admin policy --bucket my-bucket { "acl": { "acl_user_map": [ { "user": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b", "acl": 15 } ], "acl_group_map": [], "grant_map": [ { "id": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b", "grant": { "type": { "type": 0 }, "id": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b", "email": "", "permission": { "flags": 15 }, "name": "", "group": 0, "url_spec": "" } } ] }, "owner": { "id": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b", "display_name": "" } } This seems inconsistent with buckets created by regular users Is this expected behaviour? Regards Daniel

2 3

Using ID of a federated user in a bucket policy in RGW
by Daniel Iwan 13 May '21

13 May '21

Hi all I'm working on the following scenario User is authenticated with OIDC and tries to access a bucket which it does not own. How to specify user ID etc. to give access to such a user? By trial and error I found out that principal can be specified as "Principal": {"Federated":["arn:aws:sts:::assumed-role/MySession"]}, but I want to use shadow user ID or something similar as the principal Docs https://docs.ceph.com/en/latest/radosgw/STS/ states: 'A shadow user is created corresponding to every federated user. The user id is derived from the ‘sub’ field of the incoming web token. The user is created in a separate namespace - ‘oidc’ such that the user id doesn’t clash with any other user ids in rgw. The format of the user id is - <tenant>$<user-namespace>$<sub> where user-namespace is ‘oidc’ for users that authenticate with oidc providers.' I see a shadow user in Web UI as e.g. 7f71c7c5-c24f-418e-87ac-aa8fe271289b but I cannot work out the syntax of a user id, I was expecting something like "arn:aws:iam:::user/$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b" but when trying to list content of a bucket I get AccessDenied. If bucket policy has Principal "*" the my authenticated user can access the bucket Is this possible? Regards Daniel

2 3

"ceph orch ls", "ceph orch daemon rm" fail with exception "'KeyError: 'not'" on 15.2.10
by Erkki Seppala 13 May '21

13 May '21

Hi, How I got here -------------- Yesterday evening I added an OSD to my hobby system most likely using the command: # ceph-volume raw prepare --bluestore --data /dev/bcache0 # cephadm adopt --style legacy --name osd.20 I also used the command (after not having much luck with that, but I don't have the specifics): % ceph orch daemon add osd tutu:/tmp/bcache0 per https://docs.ceph.com/en/latest/cephadm/osd/#creating-new-osds ..which I think resulted in new osd.18, putting the bcache0 inside its own VG and its own LV. I don't have actual log of the used command available, but I did end up with new osds 18 and 20. First time using these command as well, my previous ways to achieve the same were a bit more long-winded.. According to my monitoring my main issue appeared around the same time. In this post I don't worry about the state of the OSD but only about management. Actual issue ------------ So when I now issue "ceph orch ls" I get the following output: % ceph orch ls Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1204, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 320, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 102, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 91, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 503, in _list_services raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 642, in raise_if_exception raise e AssertionError: not ("ceph orch ps" works fine.) Similarly the output of "ceph -s" is: % ceph -s ... health: HEALTH_ERR Module 'cephadm' has failed: 'not' ... The relevant log from the manager, as per the mgr web interface, is: _Promise failed Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 294, in _finalize next_result = self._on_complete(self._value) File "/usr/share/ceph/mgr/cephadm/module.py", line 107, in <lambda> return CephadmCompletion(on_complete=lambda _: f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1333, in describe_service hosts=[dd.hostname] File "/lib/python3.6/site-packages/ceph/deployment/service_spec.py", line 429, in __init__ assert service_type in ServiceSpec.KNOWN_SERVICE_TYPES, service_type AssertionError: not I also noticed this seemingly highly relevant bit in my ceph orch ps: NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID not.osd.20 tutu stopped 13h ago 14h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> I'm not quite sure how I ended up with that, but I wouldn't exclude operator error :) such as entering "cephadm adopt --style legacy --name not.osd.20" (but WHY..). Sure enough, there is no such docker container running in the host and the job ceph-3046312a-e453-11ea-b1f5-b42e993e47fc(a)osd.20.service has failed with "RuntimeError: could not find osd.20 with osd_fsid 212c336a-9516-4818-aeaf-2d0c24c4ca65" (this error makes sense, as both osds 18 and 20 try to use the same bcache0, but the actual bluestore filesystem is inside vg/lv as used by 18, whereas 20 tries to use bcache0 directly), but as I said I won't worry about the OSD at the moment. I tried the command "ceph orch daemon rm not.osd.20", however I'm not sure if it even should work. It nevertheless fails the same way: % ceph orch daemon rm not.osd.20 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1204, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 320, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 102, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 91, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1061, in _daemon_rm raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 642, in raise_if_exception raise e KeyError: 'not' with the following entries in the mgr log: 5/13/21 1:26:06 PM[ERR]_Promise failed Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 294, in _finalize next_result = self._on_complete(self._value) File "/usr/share/ceph/mgr/cephadm/module.py", line 107, in <lambda> return CephadmCompletion(on_complete=lambda _: f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1515, in remove_daemons return self._remove_daemons(args) File "/usr/share/ceph/mgr/cephadm/utils.py", line 65, in forall_hosts_wrapper return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get raise self._value File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar return list(map(*args)) File "/usr/share/ceph/mgr/cephadm/utils.py", line 58, in do_work return f(self, *arg) File "/usr/share/ceph/mgr/cephadm/module.py", line 1804, in _remove_daemons return self._remove_daemon(name, host) File "/usr/share/ceph/mgr/cephadm/module.py", line 1818, in _remove_daemon self.cephadm_services[daemon_type].pre_remove(daemon) KeyError: 'not' 5/13/21 1:26:06 PM[ERR]executing _remove_daemons((<cephadm.module.CephadmOrchestrator object at 0x7f1f4fec2bd0>, [('not.osd.20', 'tutu')])) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 58, in do_work return f(self, *arg) File "/usr/share/ceph/mgr/cephadm/module.py", line 1804, in _remove_daemons return self._remove_daemon(name, host) File "/usr/share/ceph/mgr/cephadm/module.py", line 1818, in _remove_daemon self.cephadm_services[daemon_type].pre_remove(daemon) KeyError: 'not' I tried also that "ceph orch daemon rm foo.bar.42" gives the error "Error EINVAL: Unable to find daemon(s) ['foo.bar.42']", so it seems it processes the actual command fine in part. Thanks for any assistance! -- _____________________________________________________________________ / __// /__ ____ __ Erkki Seppälä\ \ / /_ / // // /\ \/ / \ / /_/ /_/ \___/ /_/\_\(a)inside.org http://www.inside.org/~flux/

1 0

OSD lost: firmware bug in Kingston SSDs?
by Frank Schilder 13 May '21

13 May '21

Hi all, I lost 2 OSDs deployed on a single Kingston SSD in a rather strange way and am wondering if anyone has made similar observations or is aware of a firmware bug with these disks. Disk model: KINGSTON SEDC500M3840G (it ought to be a DC grade model with super capacitors) Smartctl does not report any drive errors. Performance per TB is as expected, OSDs are "ceph-volume lvm batch" bluestore deployed, everything collocated. Short version: I disable volatile write cache on all OSD disks, but the Kingston disks seem to behave as if this cache is *not* disabled. Smartctl and hdparm report wcache=off though. The OSD loss looks like what unflushed write cache during power loss would result in. I'm afraid now that our cluster might be vulnerable to power loss. Long version: Our disks are on Dell HBA330 Mini controllers and are in state "non-raid". The controller itself has no cache and is HBA-mode only. Log entry: The iDRAC log shows that the disk was removed from a drive group: --- PDR5 Disk 6 in Backplane 2 of Integrated Storage Controller 1 is removed. Detailed Description: A physical disk has been removed from the disk group. This alert can also be caused by loose or defective cables or by problems with the enclosure. --- The iDRAC did not report the disk as failed and neither as "removed from drive bay". I reseated the disk and it came back as healthy. I assume it was a problem with connectivity to the back-plane (chassis). If I now try to start up the OSDs on this disk, I get the error: starting osd.581 at - osd_data /var/lib/ceph/osd/ceph-581 /var/lib/ceph/osd/ceph-581/journal starting osd.580 at - osd_data /var/lib/ceph/osd/ceph-580 /var/lib/ceph/osd/ceph-580/journal 2021-05-06 09:23:47.160 7fead5a1fb80 -1 bluefs mount failed to replay log: (5) Input/output error 2021-05-06 09:23:47.160 7fead5a1fb80 -1 bluestore(/var/lib/ceph/osd/ceph-581) _open_db failed bluefs mount: (5) Input/output error 2021-05-06 09:23:47.630 7fead5a1fb80 -1 osd.581 0 OSD:init: unable to mount object store 2021-05-06 09:23:47.630 7fead5a1fb80 -1 ** ERROR: osd init failed: (5) Input/output error I have removed disks of active OSDs before without any bluestore corruption happening. While it is very well possible that this particular "disconnect" event may lead to a broken OSD, there is also another observation where the Kingston disks stick out compared with other SSD OSDs, which make me suspicious of this being a disk cache firmware problem: The I/O indicator LED lights up with significantly lower frequency than for all other SSD types on the same pool even though we have 2 instead of 1 OSD deployed on the Kingstons (the other disks are 2TB Micron Pro). While this could be due to a wiring difference I'm starting to suspect that this might be an indication of volatile caching. Does anyone using Kingston DC-M-SSDs have similar or contradicting experience? How did these disks handle power outages? Any recommendations? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

Manager carries wrong information until killing it
by Nico Schottelius 12 May '21

12 May '21

Hello, we have a recurring, funky problem with managers on Nautilus (and probably also earlier versions): the manager displays incorrect information. This is a recurring pattern and it also breaks the prometheus graphs, as the I/O is described insanely incorrectly: "recovery: 43 TiB/s, 3.62k keys/s, 11.40M objects/s" - which basically changes the scale of any related graph to unusable. The latest example from today shows slow ops for an OSD that has been down for 17h: -------------------------------------------------------------------------------- [09:50:31] black2.place6:~# ceph -s cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_WARN 18 slow ops, oldest one blocked for 975 sec, osd.53 has slow ops services: mon: 5 daemons, quorum server9,server2,server8,server6,server4 (age 2w) mgr: server2(active, since 2w), standbys: server8, server4, server9, server6, ciara3 osd: 108 osds: 107 up (since 17h), 107 in (since 17h) data: pools: 4 pools, 2624 pgs objects: 42.52M objects, 162 TiB usage: 486 TiB used, 298 TiB / 784 TiB avail pgs: 2616 active+clean 8 active+clean+scrubbing+deep io: client: 522 MiB/s rd, 22 MiB/s wr, 8.18k op/s rd, 689 op/s wr -------------------------------------------------------------------------------- Killing the manager on server2 changes the status to another temporary incorrect status, because the rebalance finished hours ago, paired with the incorrect rebalance speed that we see from time to time: -------------------------------------------------------------------------------- [09:51:59] black2.place6:~# ceph -s cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_OK services: mon: 5 daemons, quorum server9,server2,server8,server6,server4 (age 2w) mgr: server8(active, since 11s), standbys: server4, server9, server6, ciara3 osd: 108 osds: 107 up (since 17h), 107 in (since 17h) data: pools: 4 pools, 2624 pgs objects: 42.52M objects, 162 TiB usage: 486 TiB used, 298 TiB / 784 TiB avail pgs: 2616 active+clean 8 active+clean+scrubbing+deep io: client: 214 TiB/s rd, 54 TiB/s wr, 4.86G op/s rd, 1.06G op/s wr recovery: 43 TiB/s, 3.62k keys/s, 11.40M objects/s progress: Rebalancing after osd.53 marked out [========================......] -------------------------------------------------------------------------------- Then a bit later, the status on the newly started manager is correct: -------------------------------------------------------------------------------- [09:52:18] black2.place6:~# ceph -s cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_OK services: mon: 5 daemons, quorum server9,server2,server8,server6,server4 (age 2w) mgr: server8(active, since 47s), standbys: server4, server9, server6, server2, ciara3 osd: 108 osds: 107 up (since 17h), 107 in (since 17h) data: pools: 4 pools, 2624 pgs objects: 42.52M objects, 162 TiB usage: 486 TiB used, 298 TiB / 784 TiB avail pgs: 2616 active+clean 8 active+clean+scrubbing+deep io: client: 422 MiB/s rd, 39 MiB/s wr, 7.91k op/s rd, 752 op/s wr -------------------------------------------------------------------------------- Question: is this a know bug, is anyone else seeing it or are we doing something wrong? Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

2 2

May 10 Upstream Lab Outage
by David Galloway 12 May '21

12 May '21

Hi all, I wanted to provide an RCA for the outage you may have been affected by yesterday. Some services that went down: - All CI/testing - quay.ceph.io - telemetry.ceph.com (your cluster may have gone into HEALTH_WARN if you report telemetry data) - lists.ceph.io (so all mailing lists) All of our critical infra is running in a Red Hat Virtualization (RHV) instance backed by Red Hat Gluster Storage (RHGS) as the storage. Before you go, "wait.. Gluster?" Yes, this cluster was set up before Ceph was supported as backend storage for RHV/RHEV. The root cause for the outage is the Gluster volumes got 100% full. Once no writes were possible, RHV paused all the VMs. Why didn't monitoring catch this? I honestly don't know. # grep ssdstore01 nagios-05-*2021* | grep Disk nagios-05-01-2021-00.log:[1619740800] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-02-2021-00.log:[1619827200] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-03-2021-00.log:[1619913600] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-04-2021-00.log:[1620000000] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-05-2021-00.log:[1620086400] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-06-2021-00.log:[1620172800] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-07-2021-00.log:[1620259200] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-08-2021-00.log:[1620345600] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-09-2021-00.log:[1620432000] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-10-2021-00.log:[1620518400] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now nagios-05-11-2021-00.log:[1620604800] CURRENT SERVICE STATE: ssdstore01;Disk Space;OK;HARD;1;Disks are OK now Yet RHV knew we were running out of space. I don't have e-mail notifications set up in RHV, however. # zgrep "disk space" engine*202105*.gz | cut -d ',' -f4 | head -n 10 Low disk space. hosted_storage domain has 24 GB of free space. Low disk space. hosted_storage domain has 24 GB of free space. Low disk space. hosted_storage domain has 23 GB of free space. Low disk space. hosted_storage domain has 23 GB of free space. Low disk space. hosted_storage domain has 23 GB of free space. Low disk space. hosted_storage domain has 23 GB of free space. Low disk space. hosted_storage domain has 23 GB of free space. Low disk space. hosted_storage domain has 21 GB of free space. Low disk space. hosted_storage domain has 20 GB of free space. Low disk space. hosted_storage domain has 11 GB of free space. Our nagios instances runs this to check disk space: https://github.com/ceph/ceph-cm-ansible/blob/master/roles/common/files/libe… You can ignore the comment about it only working for EXT2. [root@ssdstore01 ~]# /usr/libexec/diskusage.pl 90 95 Disks are OK now I ran this manually on one of the storage hosts and intentionally set the WARN level to a number lower than the current usage percentage. [root@ssdstore01 ~]# df -h | grep 'Size\|gluster' Filesystem Size Used Avail Use% Mounted on /dev/md124 8.8T 6.7T 2.1T 77% /gluster [root@ssdstore01 ~]# /usr/libexec/diskusage.pl 95 70 /gluster is at 77% [root@ssdstore01 ~]# echo $? 2 When I logged in to the storage hosts yesterday morning, the /gluster mount was at 100%. So nagios should have known. How'd it get fixed? I happened to have some large capacity drives that fit the storage nodes lying around. They're being installed in a different project soon. However, I was able to add these drives, add "bricks" to the Gluster storage, then rebalance the data. Once that was done, I was able to restart all the VMs and delete old VMs and snapshots I no longer needed. How do we keep this from happening again? Well, as you may have been able to deduce... we were running out of space at a rate of 1-10 GB/day. As you can see now, the Gluster volume has 2.1TB of space left. So even if we grew by 10GB/day again, we'd be okay for 200ish days. I aim to have some (if not all) of these services moved off this platform and into an Openshift cluster backed by Ceph this year. Sadly, I just don't think I have enough logging enabled to nail down exactly what happened. -- David Galloway Senior Systems Administrator Ceph Engineering

1 0

Write Ops on CephFS Increasing exponentially
by Kyle Dean 12 May '21

12 May '21

Hi, hoping someone could help me get to the bottom of this particular issue I'm having. I have ceph octopus installed using ceph-ansible. Currently, I have 3 MDS servers running, and one client connected to the active MDS. I'm currently storing a very large encrypted container on the CephFS file system, 8TB worth, and I'm writing data into it from the client host. recently I have noticed a severe impact on performance, and the time take to do processing on file within the container has increased from 1 minute to 11 minutes. in the ceph dashboard, when I take a look at the performance tab on the file system page, the Write Ops are increasing exponentially over time. At the end of April around the 22nd I had 49 write Ops on the performance page for the MDS deamons. This is now at 266467 Write Ops and increasing. Also the client requests have gone from 14 to 67 to 117 and is now at 283 would someone be able to help me make sense of why the performance has decreased and what is going on with the client requests and write operations. Kind regards, kyle

2 2

Re: Ceph Month June 2021 Event
by Mike Perez 12 May '21

12 May '21

Hi everyone, Today is the last day to get your proposal in for the Ceph June Month event! The types of talks include: * Lightning talk - 5 minutes * Presentation - 20 minutes with q/a * Unconference (Bof) - 40 minutes We will be confirming with speakers for the date/time by May 16th. https://ceph.io/events/ceph-month-june-2021/cfp On Wed, Apr 21, 2021 at 6:30 AM Mike Perez <thingee(a)redhat.com> wrote: > > Hi everyone, > > We're looking for presentations, lightning talks, and BoFs to schedule > for Ceph Month in June 2021. Please submit your proposals before May > 12th: > > https://ceph.io/events/ceph-month-june-2021/cfp > > On Wed, Apr 14, 2021 at 12:35 PM Mike Perez <thingee(a)redhat.com> wrote: > > > > Hi everyone, > > > > In June 2021, we're hosting a month of Ceph presentations, lightning > > talks, and unconference sessions such as BOFs. There is no > > registration or cost to attend this event. > > > > The CFP is now open until May 12th. > > > > https://ceph.io/events/ceph-month-june-2021/cfp > > > > Speakers will receive confirmation that their presentation is accepted > > and further instructions for scheduling by May 16th. > > > > The schedule will be available on May 19th. > > > > Join the Ceph community as we discuss how Ceph, the massively > > scalable, open-source, software-defined storage system, can radically > > improve the economics and management of data storage for your > > enterprise. > > > > -- > > Mike Perez

1 0

Ceph stretch mode enabling
by Felix O 12 May '21

12 May '21

Hello, I'm trying to deploy my test ceph cluster and enable stretch mode ( https://docs.ceph.com/en/latest/rados/operations/stretch-mode/) My problem is enabling the stretch mode. ---------------------------------------------------- $ ceph mon enable_stretch_mode ceph-node-05 stretch_rule datacenter Error EINVAL: Could not find location entry for datacenter on monitor ceph-node-05 ---------------------------------------------------- ceph-node-5 is the tiebreaker monitor I tried to create the third datacenter and put the tiebreaker there but got the following error: ---------------------------------------------------- root@ceph-node-01:/home/clouduser# ceph mon enable_stretch_mode ceph-node-05 stretch_rule datacenter Error EINVAL: there are 3datacenter's in the cluster but stretch mode currently only works with 2! ---------------------------------------------------- An additional info: ---------------------------------------------------- Setup method: cephadm (https://docs.ceph.com/en/latest/cephadm/install/) # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.03998 root default -11 0.01999 datacenter site1 -5 0.00999 host ceph-node-01 0 hdd 0.00999 osd.0 up 1.00000 1.00000 -3 0.00999 host ceph-node-02 1 hdd 0.00999 osd.1 up 1.00000 1.00000 -12 0.01999 datacenter site2 -9 0.00999 host ceph-node-03 3 hdd 0.00999 osd.3 up 1.00000 1.00000 -7 0.00999 host ceph-node-04 2 hdd 0.00999 osd.2 up 1.00000 1.00000 stretch_rule is added to the crush # ceph mon set_location ceph-node-01 datacenter=site1 # ceph mon set_location ceph-node-02 datacenter=site1 # ceph mon set_location ceph-node-03 datacenter=site2 # ceph mon set_location ceph-node-04 datacenter=site2 # ceph versions { "mon": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 2 }, "osd": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 4 }, "mds": {}, "overall": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 11 } } Thank you for your support. -- Best regards,

2 1

cephfs mds issues
by Mazzystr 12 May '21

12 May '21

I did a simple os update and reboot. Now mds is stuck in replay. I'm running octapus debug mds = 20 shows some pretty lame logs # tail -f ceph-mds.bridge.log 2021-05-11T18:24:04.859-0700 7f41314a1700 20 mds.0.cache upkeep thread waiting interval 1s 2021-05-11T18:24:05.860-0700 7f41314a1700 10 mds.0.cache cache not ready for trimming 2021-05-11T18:24:05.860-0700 7f41314a1700 20 mds.0.cache upkeep thread waiting interval 1s 2021-05-11T18:24:06.859-0700 7f4133ca6700 20 mds.0.2898629 get_task_status 2021-05-11T18:24:06.859-0700 7f4133ca6700 20 mds.0.2898629 send_task_status: updating 1 status keys 2021-05-11T18:24:06.859-0700 7f4133ca6700 20 mds.0.2898629 schedule_update_timer_task 2021-05-11T18:24:06.859-0700 7f41314a1700 10 mds.0.cache cache not ready for trimming 2021-05-11T18:24:06.859-0700 7f41314a1700 20 mds.0.cache upkeep thread waiting interval 1s 2021-05-11T18:24:07.859-0700 7f41314a1700 10 mds.0.cache cache not ready for trimming 2021-05-11T18:24:07.859-0700 7f41314a1700 20 mds.0.cache upkeep thread waiting interval 1s # cephfs-journal-tool event recover_dentries summary gets stuck on an object and stays stuck. I tried to run rados -p cephfs_metadata_pool rmomapkey per https://tracker.ceph.com/issues/38452 but the cmd ran for hours and never completes. # cephfs-journal-tool --rank cephfs:0 journal reset 2021-05-11T18:31:26.860-0700 7f2e9c2a9700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2021-05-11T18:31:26.860-0700 7f2f2989ba80 4 waiting for MDS map... 2021-05-11T18:31:26.860-0700 7f2f2989ba80 4 Got MDS map 2898629 2021-05-11T18:31:26.861-0700 7f2f2989ba80 10 main: JournalTool::main 2021-05-11T18:31:26.861-0700 7f2f2989ba80 4 main: JournalTool: connecting to RADOS... 2021-05-11T18:31:26.863-0700 7f2f2989ba80 4 main: JournalTool: resolving pool 1 2021-05-11T18:31:26.863-0700 7f2f2989ba80 4 main: JournalTool: creating IoCtx.. 2021-05-11T18:31:26.863-0700 7f2f2989ba80 4 main: Executing for rank 0 2021-05-11T18:31:26.864-0700 7f2edc2aa700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2021-05-11T18:31:26.864-0700 7f2f2989ba80 4 waiting for MDS map... 2021-05-11T18:31:26.865-0700 7f2f2989ba80 4 Got MDS map 2898629 2021-05-11T18:31:26.865-0700 7f2f2989ba80 4 client.2024650.journalpointer Reading journal pointer '400.00000000' 2021-05-11T18:31:26.865-0700 7f2f2989ba80 1 client.2024650.journaler.resetter(ro) recover start 2021-05-11T18:31:26.865-0700 7f2f2989ba80 1 client.2024650.journaler.resetter(ro) read_head 2021-05-11T18:31:26.865-0700 7f291c293700 1 client.2024650.journaler.resetter(ro) _finish_read_head loghead(trim 14172553216, expire 14174788378, write 14400838791, stream_format 1). probing for end of log (from 14400838791)... 2021-05-11T18:31:26.865-0700 7f291c293700 1 client.2024650.journaler.resetter(ro) probing for end of the log I've been stuck here for hours # strace -f -p 10357 [pid 10360] <... sendmsg resumed>) = 9 [pid 10361] read(14, <unfinished ...> [pid 10360] epoll_wait(7, <unfinished ...> [pid 10361] <... read resumed>0x55e95d982000, 4096) = -1 EAGAIN (Resource temporarily unavailable) [pid 10360] <... epoll_wait resumed>[{EPOLLIN, {u32=16, u64=16}}, {EPOLLIN, {u32=18, u64=18}}], 5000, 30000) = 2 [pid 10361] epoll_wait(10, <unfinished ...> [pid 10360] read(16, "\23\1\10\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\354^\340;"..., 4096) = 57 [pid 10360] read(16, 0x55e95d9a8000, 4096) = -1 EAGAIN (Resource temporarily unavailable) [pid 10360] read(18, "\17\264R\233`\327\275\222+", 4096) = 9 [pid 10360] read(18, 0x55e95d9f4000, 4096) = -1 EAGAIN (Resource temporarily unavailable) [pid 10360] epoll_wait(7, ^X <unfinished ...> [pid 10370] <... futex resumed>) = -1 ETIMEDOUT (Connection timed out) [pid 10381] <... futex resumed>) = -1 ETIMEDOUT (Connection timed out) [pid 10370] clock_gettime(CLOCK_REALTIME, <unfinished ...> [pid 10389] <... futex resumed>) = -1 ETIMEDOUT (Connection timed out) [pid 10381] clock_gettime(CLOCK_REALTIME, <unfinished ...> [pid 10370] <... clock_gettime resumed>{tv_sec=1620791989, tv_nsec=731038214}) = 0 [pid 10389] clock_gettime(CLOCK_REALTIME, <unfinished ...> [pid 10381] <... clock_gettime resumed>{tv_sec=1620791989, tv_nsec=731105584}) = 0 [pid 10389] <... clock_gettime resumed>{tv_sec=1620791989, tv_nsec=731125991}) = 0 [pid 10370] clock_gettime(CLOCK_REALTIME, <unfinished ...> [pid 10381] clock_gettime(CLOCK_REALTIME, <unfinished ...> [pid 10389] clock_gettime(CLOCK_REALTIME, <unfinished ...> [pid 10370] <... clock_gettime resumed>{tv_sec=1620791989, tv_nsec=731162065}) = 0 [pid 10389] <... clock_gettime resumed>{tv_sec=1620791989, tv_nsec=731184311}) = 0 [pid 10381] <... clock_gettime resumed>{tv_sec=1620791989, tv_nsec=731174345}) = 0 [pid 10370] futex(0x55e95d97c2d8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> [pid 10381] futex(0x55e95d8a5320, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> [pid 10370] <... futex resumed>) = 0 [pid 10389] futex(0x55e95d97fad8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> [pid 10381] <... futex resumed>) = 0 [pid 10370] futex(0x55e95d97c31c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 17805, {tv_sec=1620791990, tv_nsec=731161399}, 0xffffffff <unfinished ...> [pid 10389] <... futex resumed>) = 0 [pid 10381] futex(0x55e95d8a5364, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 17805, {tv_sec=1620791990, tv_nsec=731173986}, 0xffffffff <unfinished ...> [pid 10389] futex(0x55e95d97fb1c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 17805, {tv_sec=1620791990, tv_nsec=731183618}, 0xffffffff^Cstrace: Process 10357 detached Any help would be great. Thanks, /C

1 1

DocuBetter Meeting -- 12 May 2021 1730 UTC
by John Zachary Dover 11 May '21

11 May '21

There will be a DocuBetter Meeting held on 12 May 2021 at 1730 UTC. This is the monthly DocuBetter Meeting that is more convenient for European and North American Ceph contributors than the other meeting, which is convenient for people in Australia and Asia (and which is very rarely attended). I plan to discuss at this meeting the continuing cleaning of the cephadm documentation, and to discuss an ambitious plan of virtually Alexandrian hubris to create a 10ish-page Ceph Overview document (a long-term, tedious project that will involve a dozen people, so don't get too excited about it). Bring your docs complaints and requests to this meeting. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

Re: radosgw-admin user create takes a long time (with failed to distribute cache message)
by Boris Behrens 11 May '21

11 May '21

Good call. I just restarted the whole cluster, but the problem still persists. I don't think it is a problem with the rados, but with the radosgw. But I still struggle to pin the issue. Am Di., 11. Mai 2021 um 10:45 Uhr schrieb Thomas Schneider < Thomas.Schneider-q2p(a)ruhr-uni-bochum.de>: > Hey all, > > we had slow RGW access when some OSDs were slow due to an (to us) unknown > OSD bug that made PG access either slow or impossible. (It showed itself > through slowness of the mgr as well, but nothing other than that). > We restarted all OSDs that held RGW data and the problem was gone. > I have no good way to debug the problem since it never occured again after > we restarted the OSDs. > > Kind regards, > Thomas > > > Am 11. Mai 2021 08:47:06 MESZ schrieb Boris Behrens <bb(a)kervyn.de>: > >Hi Amit, > > > >I just pinged the mons from every system and they are all available. > > > >Am Mo., 10. Mai 2021 um 21:18 Uhr schrieb Amit Ghadge < > amitg.b14(a)gmail.com>: > > > >> We seen slowness due to unreachable one of them mgr service, maybe here > >> are different, you can check monmap/ ceph.conf mon entry and then verify > >> all nodes are successfully ping. > >> > >> > >> -AmitG > >> > >> > >> On Tue, 11 May 2021 at 12:12 AM, Boris Behrens <bb(a)kervyn.de> wrote: > >> > >>> Hi guys, > >>> > >>> does someone got any idea? > >>> > >>> Am Mi., 5. Mai 2021 um 16:16 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > >>> > >>> > Hi, > >>> > since a couple of days we experience a strange slowness on some > >>> > radosgw-admin operations. > >>> > What is the best way to debug this? > >>> > > >>> > For example creating a user takes over 20s. > >>> > [root@s3db1 ~]# time radosgw-admin user create --uid test-bb-user > >>> > --display-name=test-bb-user > >>> > 2021-05-05 14:08:14.297 7f6942286840 1 robust_notify: If at first > you > >>> > don't succeed: (110) Connection timed out > >>> > 2021-05-05 14:08:14.297 7f6942286840 0 ERROR: failed to distribute > >>> cache > >>> > for eu-central-1.rgw.users.uid:test-bb-user > >>> > 2021-05-05 14:08:24.335 7f6942286840 1 robust_notify: If at first > you > >>> > don't succeed: (110) Connection timed out > >>> > 2021-05-05 14:08:24.335 7f6942286840 0 ERROR: failed to distribute > >>> cache > >>> > for eu-central-1.rgw.users.keys:**** > >>> > { > >>> > "user_id": "test-bb-user", > >>> > "display_name": "test-bb-user", > >>> > .... > >>> > } > >>> > real 0m20.557s > >>> > user 0m0.087s > >>> > sys 0m0.030s > >>> > > >>> > First I thought that rados operations might be slow, but adding and > >>> > deleting objects in rados are fast as usual (at least from my > >>> perspective). > >>> > Also uploading to buckets is fine. > >>> > > >>> > We changed some things and I think it might have to do with this: > >>> > * We have a HAProxy that distributes via leastconn between the 3 > >>> radosgw's > >>> > (this did not change) > >>> > * We had three times a daemon with the name "eu-central-1" running > (on > >>> the > >>> > 3 radosgw's) > >>> > * Because this might have led to our data duplication problem, we > have > >>> > split that up so now the daemons are named per host > (eu-central-1-s3db1, > >>> > eu-central-1-s3db2, eu-central-1-s3db3) > >>> > * We also added dedicated rgw daemons for garbage collection, because > >>> the > >>> > current one were not able to keep up. > >>> > * So basically ceph status went from "rgw: 1 daemon active > >>> (eu-central-1)" > >>> > to "rgw: 14 daemons active (eu-central-1-s3db1, eu-central-1-s3db2, > >>> > eu-central-1-s3db3, gc-s3db12, gc-s3db13...) > >>> > > >>> > > >>> > Cheers > >>> > Boris > >>> > > >>> > >>> > >>> -- > >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > im > >>> groÃƒ¼en Saal. > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >> > > > > -- > Thomas Schneider > IT.SERVICES > Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 > Bochum > Telefon: +49 234 32 23939 > http://www.it-services.rub.de/ > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

MonSession vs TCP connection
by Jan Pekař - Imatic 11 May '21

11 May '21

Hi all, I would like to "pair" MonSession with TCP connection to get real process, which is using that session. I need it to identify processes with old ceph features. MonSession looks like MonSession(client.84324148 [..IP...]:0/3096235764 is open allow *, features 0x27018fb86aa42ada (jewel)) What does client.NUMBER and 0/3096235764 means? How can I resolve client.NUMBER or that /NUMBER with certain TCP session. I have many processes on that server (on that IP) with different features. Thank you -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

1 0

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 11 May '21

11 May '21

On Mon, May 3, 2021 at 12:24 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 11:22 schrieb Ilya Dryomov: > > There is a 6th osd directory on both machines, but it's empty > > [root@s0 osd]# ll > total 0 > drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-1 > drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-3 > drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-4 > drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-5 > drwxr-xr-x. 2 ceph ceph 6 3. Apr 19:50 ceph-8 <=== > drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-9 > [root@s0 osd]# pwd > /var/lib/ceph/osd > > [root@s1 osd]# ll > total 0 > drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-0 > drwxr-xr-x. 2 ceph ceph 6 Mar 13 17:54 ceph-1 <=== > drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-2 > drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-6 > drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-7 > drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-8 > [root@s1 osd]# pwd > /var/lib/ceph/osd > > The bogus directories are empty and they are > used on the other machine for a real osd! > > How is that? > > Should I remove them and restart ceph.target? I don't think empty directories matter at this point. You may not have had 12 OSDs at any point in time, but the max_osd value appears to have gotten bumped when you were replacing those disks. Note that max_osd being greater than the number of OSDs is not a big problem by itself. The osdmap is going to be larger and require more memory but that's it. You can test by setting it back to 12 and trying to mount -- it should work. The issue is specific to how to those OSDs were replaced -- something went wrong and the osdmap somehow ended up with rather bogus addrvec entries. Not sure if it's ceph-deploy's fault, something weird in ceph.conf (back then) or a an actual ceph bug. Thanks, Ilya

2 3

Which EC-code for 6 servers?
by Szabo, Istvan (Agoda) 11 May '21

11 May '21

Hi, Thinking to have 2:2 so I can tolerate 2 hosts loss, but if I just want to tolerate 1 host loss, which one better, 3:2 or 4:1? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Re: radosgw-admin user create takes a long time (with failed to distribute cache message)
by Boris Behrens 11 May '21

11 May '21

Hi Amit, it is the same physical interface but different VLANs. I checked all IP adresses from all systems and everything is direct connected, without any gateway hops. Am Di., 11. Mai 2021 um 10:59 Uhr schrieb Amit Ghadge <amitg.b14(a)gmail.com>: > I hope you are using a single network interface for the public and cluster? > > On Tue, May 11, 2021 at 2:15 PM Thomas Schneider < > Thomas.Schneider-q2p(a)ruhr-uni-bochum.de> wrote: > >> Hey all, >> >> we had slow RGW access when some OSDs were slow due to an (to us) unknown >> OSD bug that made PG access either slow or impossible. (It showed itself >> through slowness of the mgr as well, but nothing other than that). >> We restarted all OSDs that held RGW data and the problem was gone. >> I have no good way to debug the problem since it never occured again >> after we restarted the OSDs. >> >> Kind regards, >> Thomas >> >> >> Am 11. Mai 2021 08:47:06 MESZ schrieb Boris Behrens <bb(a)kervyn.de>: >> >Hi Amit, >> > >> >I just pinged the mons from every system and they are all available. >> > >> >Am Mo., 10. Mai 2021 um 21:18 Uhr schrieb Amit Ghadge < >> amitg.b14(a)gmail.com>: >> > >> >> We seen slowness due to unreachable one of them mgr service, maybe here >> >> are different, you can check monmap/ ceph.conf mon entry and then >> verify >> >> all nodes are successfully ping. >> >> >> >> >> >> -AmitG >> >> >> >> >> >> On Tue, 11 May 2021 at 12:12 AM, Boris Behrens <bb(a)kervyn.de> wrote: >> >> >> >>> Hi guys, >> >>> >> >>> does someone got any idea? >> >>> >> >>> Am Mi., 5. Mai 2021 um 16:16 Uhr schrieb Boris Behrens <bb(a)kervyn.de >> >: >> >>> >> >>> > Hi, >> >>> > since a couple of days we experience a strange slowness on some >> >>> > radosgw-admin operations. >> >>> > What is the best way to debug this? >> >>> > >> >>> > For example creating a user takes over 20s. >> >>> > [root@s3db1 ~]# time radosgw-admin user create --uid test-bb-user >> >>> > --display-name=test-bb-user >> >>> > 2021-05-05 14:08:14.297 7f6942286840 1 robust_notify: If at first >> you >> >>> > don't succeed: (110) Connection timed out >> >>> > 2021-05-05 14:08:14.297 7f6942286840 0 ERROR: failed to distribute >> >>> cache >> >>> > for eu-central-1.rgw.users.uid:test-bb-user >> >>> > 2021-05-05 14:08:24.335 7f6942286840 1 robust_notify: If at first >> you >> >>> > don't succeed: (110) Connection timed out >> >>> > 2021-05-05 14:08:24.335 7f6942286840 0 ERROR: failed to distribute >> >>> cache >> >>> > for eu-central-1.rgw.users.keys:**** >> >>> > { >> >>> > "user_id": "test-bb-user", >> >>> > "display_name": "test-bb-user", >> >>> > .... >> >>> > } >> >>> > real 0m20.557s >> >>> > user 0m0.087s >> >>> > sys 0m0.030s >> >>> > >> >>> > First I thought that rados operations might be slow, but adding and >> >>> > deleting objects in rados are fast as usual (at least from my >> >>> perspective). >> >>> > Also uploading to buckets is fine. >> >>> > >> >>> > We changed some things and I think it might have to do with this: >> >>> > * We have a HAProxy that distributes via leastconn between the 3 >> >>> radosgw's >> >>> > (this did not change) >> >>> > * We had three times a daemon with the name "eu-central-1" running >> (on >> >>> the >> >>> > 3 radosgw's) >> >>> > * Because this might have led to our data duplication problem, we >> have >> >>> > split that up so now the daemons are named per host >> (eu-central-1-s3db1, >> >>> > eu-central-1-s3db2, eu-central-1-s3db3) >> >>> > * We also added dedicated rgw daemons for garbage collection, >> because >> >>> the >> >>> > current one were not able to keep up. >> >>> > * So basically ceph status went from "rgw: 1 daemon active >> >>> (eu-central-1)" >> >>> > to "rgw: 14 daemons active (eu-central-1-s3db1, eu-central-1-s3db2, >> >>> > eu-central-1-s3db3, gc-s3db12, gc-s3db13...) >> >>> > >> >>> > >> >>> > Cheers >> >>> > Boris >> >>> > >> >>> >> >>> >> >>> -- >> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend >> im >> >>> groÃƒ¼en Saal. >> >>> _______________________________________________ >> >>> ceph-users mailing list -- ceph-users(a)ceph.io >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>> >> >> >> > >> >> -- >> Thomas Schneider >> IT.SERVICES >> Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 >> Bochum >> Telefon: +49 234 32 23939 >> http://www.it-services.rub.de/ >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

CephFS Subvolume Snapshot data corruption?
by Andras Sali 11 May '21

11 May '21

Hi, We experienced a strange issue with a CephFS snapshot becoming partially unreadable. The snapshot was created about 2 months ago and we started a read operation from it. For a while everything was working fine with all directories accessible, however after some point clients (FUSE, v15.2.9) started complaining about I/O error on directories that were working fine previously. When listing the top level contents of the snapshot, the directories that show I/O error (e.g. home) are reporting missing metadata when listing via 'ls': d????????? ? ? ? ? ? home Creating new snapshots is working properly and otherwise the whole cluster (Ceph v15.2.9) is reporting a healthy status. Has anyone experienced an issue like this before? We tried to restart the MDS servers, however this didn't solve the issue. As additional context, during the read time from the snapshot other snaptrim operations (for other subvolumes) were ongoing, but not for the one affected by the data error. Any insight into what might cause this and how to avoid / recover from such a situation would be much appreciated. Thank you and kind regards, Andras

1 0

one ODS out-down after upgrade to v16.2.3
by Milosz Szewczak 11 May '21

11 May '21

Hi, I updated cluster as usual to newest version 16.2.3 from 16.2.1 ceph orch upgrade start --ceph-version 16.2.3 all went fine.... except on of OSD.54 stop working on host i see in logs ( after try to manual start ) ``` root@ceph-nvme01:/var/lib/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1/osd.54# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init --name ceph-77fc6eb4-7146-11eb-aa58-55847fcdb1f1-osd.54-activate -e CONTAINER_IMAGE=docker.io/ceph/ceph@sha256:c820cef23fb93518d5b35683d6301bae36511e52e0f8cd1495fd58805b849383 -e NODE_NAME=ceph-nvme01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1:/var/run/ceph:z -v /var/log/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1:/var/log/ceph:z -v /var/lib/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1/osd.54:/var/lib/ceph/osd/ceph-54:z -v /var/lib/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1/osd.54/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm docker.io/ceph/ceph@sha256:c820cef23fb93518d5b35683d6301bae36511e52e0f8cd1495fd58805b849383 lvm activate 54 07b257ee-bb09-4c4e-9f70-9f0da56253c7 --no-systemd Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-54 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-e5ddebd9-52f5-481d-9470-fb2c94fe79e3/osd-block-07b257ee-bb09-4c4e-9f70-9f0da56253c7 --path /var/lib/ceph/osd/ceph-54 --no-mon-config stderr: failed to read label for /dev/ceph-e5ddebd9-52f5-481d-9470-fb2c94fe79e3/osd-block-07b257ee-bb09-4c4e-9f70-9f0da56253c7: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1 ``` ----- ``` root@ceph-nvme01:/var/lib/ceph/77fc6eb4-7146-11eb-aa58-55847fcdb1f1/osd.54# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop2 7:2 0 70.4M 1 loop /snap/lxd/19647 loop3 7:3 0 55.5M 1 loop /snap/core18/1997 loop4 7:4 0 32.3M 1 loop /snap/snapd/11588 loop5 7:5 0 32.3M 1 loop /snap/snapd/11402 loop6 7:6 0 67.6M 1 loop /snap/lxd/20326 sda 8:0 1 447.1G 0 disk ├─sda1 8:1 1 512M 0 part └─sda2 8:2 1 445G 0 part └─md0 9:0 0 445G 0 raid1 / sdb 8:16 1 447.1G 0 disk ├─sdb1 8:17 1 512M 0 part /boot/efi └─sdb2 8:18 1 445G 0 part └─md0 9:0 0 445G 0 raid1 / nvme1n1 259:0 0 2.9T 0 disk └─ceph--257763b1--ef44--43e8--a8a7--f046f5a69fd4-osd--block--1086e697--7949--417b--9ce2--1459db52bbe2 253:0 0 2.9T 0 lvm nvme0n1 259:1 0 2.9T 0 disk └─ceph--7488e6b6--5883--4208--8951--0e3e5cc1c903-osd--block--aea644af--7a01--49c9--8169--cb6ec24d8b78 253:1 0 2.9T 0 lvm nvme2n1 259:5 0 2.9T 0 disk └─ceph--de73e328--c694--4f46--9a5f--0a7e356cab85-osd--block--b3a93422--57c0--4be3--ad61--836bb7763fae 253:4 0 2.9T 0 lvm nvme4n1 259:6 0 2.9T 0 disk └─ceph--08d2cbac--cb88--40cf--ad6e--88fc5ca3491c-osd--block--35ec77e0--db35--4a43--bb25--4666ce0ea756 253:2 0 2.9T 0 lvm nvme3n1 259:7 0 2.9T 0 disk └─ceph--f28ba826--b16c--4e4e--b0de--f556274bbc07-osd--block--399337e3--f2f8--4561--8e02--2e9830b31ec5 253:3 0 2.9T 0 lvm nvme7n1 259:8 0 2.9T 0 disk └─ceph--0c92fca4--5ff6--49f6--b994--694253ac9225-osd--block--973a39c9--f330--46f4--afc9--fb11f2862d5d 253:9 0 2.9T 0 lvm nvme10n1 259:9 0 2.9T 0 disk └─ceph--e5ddebd9--52f5--481d--9470--fb2c94fe79e3-osd--block--07b257ee--bb09--4c4e--9f70--9f0da56253c7 253:6 0 2.9T 0 lvm nvme6n1 259:10 0 2.9T 0 disk └─ceph--caf0ef69--b335--4b6e--a560--ecd4cd74fc9b-osd--block--f0bfe9f0--26f8--4dd4--95eb--3ef5a728cbf9 253:8 0 2.9T 0 lvm nvme8n1 259:11 0 2.9T 0 disk └─ceph--a4da070c--9db2--408e--9fae--b7b61023177f-osd--block--66b6ce4e--8ce3--4960--980d--e617c7378763 253:10 0 2.9T 0 lvm nvme9n1 259:12 0 2.9T 0 disk └─ceph--21562262--bc8b--4002--8575--7b6729ffef84-osd--block--9c427785--3dbe--4def--ab30--a19579fa63ce 253:11 0 2.9T 0 lvm nvme11n1 259:13 0 2.9T 0 disk └─ceph--2de30cc9--9d2a--4f61--818e--ce91a90c3489-osd--block--c4bc16fb--a645--42a6--a2d2--b09c812843e7 253:7 0 2.9T 0 lvm nvme5n1 259:15 0 2.9T 0 disk └─ceph--aebd9d97--b693--404e--9a81--55df782a15af-osd--block--2fffb8f0--f41d--423c--becd--88339631de13 253:5 0 2.9T 0 lvm ``` Additional i saw in GUI that all other OSD have *ceph_version* ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable) but failied one have *ceph_version* ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) so look like that it didn't updated well. How to fix this error ? Milosz

1 0

Re: radosgw-admin user create takes a long time (with failed to distribute cache message)
by Boris Behrens 11 May '21

11 May '21

Hi Amit, I just pinged the mons from every system and they are all available. Am Mo., 10. Mai 2021 um 21:18 Uhr schrieb Amit Ghadge <amitg.b14(a)gmail.com>: > We seen slowness due to unreachable one of them mgr service, maybe here > are different, you can check monmap/ ceph.conf mon entry and then verify > all nodes are successfully ping. > > > -AmitG > > > On Tue, 11 May 2021 at 12:12 AM, Boris Behrens <bb(a)kervyn.de> wrote: > >> Hi guys, >> >> does someone got any idea? >> >> Am Mi., 5. Mai 2021 um 16:16 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: >> >> > Hi, >> > since a couple of days we experience a strange slowness on some >> > radosgw-admin operations. >> > What is the best way to debug this? >> > >> > For example creating a user takes over 20s. >> > [root@s3db1 ~]# time radosgw-admin user create --uid test-bb-user >> > --display-name=test-bb-user >> > 2021-05-05 14:08:14.297 7f6942286840 1 robust_notify: If at first you >> > don't succeed: (110) Connection timed out >> > 2021-05-05 14:08:14.297 7f6942286840 0 ERROR: failed to distribute >> cache >> > for eu-central-1.rgw.users.uid:test-bb-user >> > 2021-05-05 14:08:24.335 7f6942286840 1 robust_notify: If at first you >> > don't succeed: (110) Connection timed out >> > 2021-05-05 14:08:24.335 7f6942286840 0 ERROR: failed to distribute >> cache >> > for eu-central-1.rgw.users.keys:**** >> > { >> > "user_id": "test-bb-user", >> > "display_name": "test-bb-user", >> > .... >> > } >> > real 0m20.557s >> > user 0m0.087s >> > sys 0m0.030s >> > >> > First I thought that rados operations might be slow, but adding and >> > deleting objects in rados are fast as usual (at least from my >> perspective). >> > Also uploading to buckets is fine. >> > >> > We changed some things and I think it might have to do with this: >> > * We have a HAProxy that distributes via leastconn between the 3 >> radosgw's >> > (this did not change) >> > * We had three times a daemon with the name "eu-central-1" running (on >> the >> > 3 radosgw's) >> > * Because this might have led to our data duplication problem, we have >> > split that up so now the daemons are named per host (eu-central-1-s3db1, >> > eu-central-1-s3db2, eu-central-1-s3db3) >> > * We also added dedicated rgw daemons for garbage collection, because >> the >> > current one were not able to keep up. >> > * So basically ceph status went from "rgw: 1 daemon active >> (eu-central-1)" >> > to "rgw: 14 daemons active (eu-central-1-s3db1, eu-central-1-s3db2, >> > eu-central-1-s3db3, gc-s3db12, gc-s3db13...) >> > >> > >> > Cheers >> > Boris >> > >> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> groÃƒ¼en Saal. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Building ceph clusters with 8TB SSD drives?
by Matt Larson 10 May '21

10 May '21

Is anyone trying Ceph clusters containing larger (4-8TB) SSD drives? 8TB SSDs are described here ( https://www.anandtech.com/show/16136/qlc-8tb-ssd-review-samsung-870-qvo-sab… ) and make use QLC NAND flash memory to reach the costs and capacity. Currently, the 8TB Samsung 870 SSD is $800/ea at some online retail stores. SATA form-factor SSDs can reach read/write rates of 560/520 MB/s, while not as great as nVME drives is still a multiple faster than 7200 RPM drives. SSDs now appear to have much lower failure rates than HDs in 2021 ( https://www.techspot.com/news/89590-backblaze-latest-storage-reliability-fi… ). Are there any major caveats to considering working with larger SSDs for data pools? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

5 4

How to deploy ceph with ssd ?
by codignotto 10 May '21

10 May '21

I'm deploying 6 ceph servers with 128GB of memory each, 12 SSDs of 1 Tb on each server, 10Gb network cards connected to 10Gb port switches. I'm following this documentation https://docs.ceph.com/en/octopus/cephadm/install/ But I don't know if this is the best way to get the most out of the disks, I will use it with RBD only and deliver it to a proxmox cluster. Do you have any more complete documentation? Some tuning tips for the best SSD speed process? Many Tks

1 0

v16.2.2 Pacific released
by David Galloway 10 May '21

10 May '21

This is the second backport release in the Pacific stable series. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v16-2-2-pacific-released Notable Changes --------------- * Cephadm now supports an *ingress* service type that provides load balancing and HA (via haproxy and keepalived on a virtual IP) for RGW service. The experimental *rgw-ha* service has been removed. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-16.2.2.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: e8f22dde28889481f4dda2beb8a07788204821d3

4 3

Host crash undetected by ceph health check
by Frank Schilder 10 May '21

10 May '21

Dear cephers, today it seems I observed an impossible event for the first time: an OSD host crashed, but the ceph health monitoring did not recognise the crash. Not a single OSD was marked down and IO simply stopped, waiting for the crashed OSDs to respond. All that was reported was slow ops, slow meta data IO, MDS behind on trimming, but no OSD fail. I have rebooted these machines a lot of times and have never seen the health check fail to recognise that instantly. The only difference I see is that these were clean shut-downs, not crashes (I believe the OSDs mark themselves as down). For debugging this problem, can anyone provide me with a pointer when this could be the result of a misconfiguration? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

radosgw-admin user create takes a long time (with failed to distribute cache message)
by Boris Behrens 10 May '21

10 May '21

Hi, since a couple of days we experience a strange slowness on some radosgw-admin operations. What is the best way to debug this? For example creating a user takes over 20s. [root@s3db1 ~]# time radosgw-admin user create --uid test-bb-user --display-name=test-bb-user 2021-05-05 14:08:14.297 7f6942286840 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-05-05 14:08:14.297 7f6942286840 0 ERROR: failed to distribute cache for eu-central-1.rgw.users.uid:test-bb-user 2021-05-05 14:08:24.335 7f6942286840 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-05-05 14:08:24.335 7f6942286840 0 ERROR: failed to distribute cache for eu-central-1.rgw.users.keys:**** { "user_id": "test-bb-user", "display_name": "test-bb-user", .... } real 0m20.557s user 0m0.087s sys 0m0.030s First I thought that rados operations might be slow, but adding and deleting objects in rados are fast as usual (at least from my perspective). Also uploading to buckets is fine. We changed some things and I think it might have to do with this: * We have a HAProxy that distributes via leastconn between the 3 radosgw's (this did not change) * We had three times a daemon with the name "eu-central-1" running (on the 3 radosgw's) * Because this might have led to our data duplication problem, we have split that up so now the daemons are named per host (eu-central-1-s3db1, eu-central-1-s3db2, eu-central-1-s3db3) * We also added dedicated rgw daemons for garbage collection, because the current one were not able to keep up. * So basically ceph status went from "rgw: 1 daemon active (eu-central-1)" to "rgw: 14 daemons active (eu-central-1-s3db1, eu-central-1-s3db2, eu-central-1-s3db3, gc-s3db12, gc-s3db13...) Cheers Boris

1 1

Weird PG Acting Set
by Lazuardi Nasution 09 May '21

09 May '21

Hi, Suddenly we have a recovery_unfound situation. I find that PG acting set is missing some OSDs which are up. Why can't OSD 3 and 71 on following PG query result be members of PG acting set? Currently, we use v15.2.8. How to recover from this situation? { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "active+forced_recovery+recovery_unfound+undersized+degraded+remapped", "epoch": 237505, "up": [ 3, 237, 71, 132, 115, 56 ], "acting": [ 2147483647, 237, 2147483647, 132, 115, 56 ], "backfill_targets": [ "3(0)", "71(2)" ], "acting_recovery_backfill": [ "3(0)", "56(5)", "71(2)", "115(4)", "132(3)", "237(1)" ], Best regards.

2 3

Performance compare between CEPH multi replica and EC
by zp_8483 09 May '21

09 May '21

Hi all, How much EC performance will degrade compared to multi replica when using the same hardware configuration. Is there any offical data?

2 1

Re: RGW failed to start after upgrade to pacific
by 胡玮文 08 May '21

08 May '21

Hi Casey, I still see this error after upgrade to 16.2.3 > 在 2021年4月7日，00:54，Casey Bodley <cbodley(a)redhat.com> 写道： > > thanks for the details. this is a regression from changes to the > datalog storage for multisite - this -5 error is coming from the new > 'fifo' backend. as a workaround, you can set the new > 'rgw_data_log_backing' config variable back to 'omap' > > Adam has fixes already merged to the pacific branch; be aware that the > first pacific point release will change the name of > 'rgw_data_log_backing' to 'rgw_default_data_log_backing' and default > back to 'fifo' > >> On Tue, Apr 6, 2021 at 2:37 AM Martin Verges <martin.verges(a)croit.io> wrote: >> Hello, >> we see same problems. Deleting all the pools and redeploy rgw solved it on >> that test cluster, however that is no solution for production ;) >> systemd[1]: Started Ceph rados gateway. >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 deferred set >> uid:gid to 167:167 (ceph:ceph) >> radosgw[7171]: failed to chown /dev/null: (30) Read-only file system >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 ceph version >> 16.2.0-31-g5922b2b9c1 (5922b2b9c17f0877f84b0b3f2557ab72a628cbfe) pacific >> (stable), process radosgw, pid 7171 >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework: >> beast >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf >> key: ssl_port, val: 443 >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf >> key: port, val: 80 >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf >> key: ssl_certificate, val: /etc/ceph/rgwcert.pem >> radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 1 radosgw_Main >> not setting numa affinity >> radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int >> rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, >> std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, >> optional_yield, std::optional<rados::cls::fifo::objv>, >> std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 >> create_meta failed: r=-5 >> radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int >> rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, >> std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, >> optional_yield, std::optional<rados::cls::fifo::objv>, >> std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 >> create_meta failed: r=-5 >> radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int >> RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, >> librados::v14_2_0::Rados*): Error when starting backend: Input/output error >> radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed >> to start datalog_rados service ((5) Input/output error >> radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int >> RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, >> librados::v14_2_0::Rados*): Error when starting backend: Input/output error >> radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed >> to init services (ret=(5) Input/output error) >> radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init >> storage provider (RADOS) >> radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init >> storage provider (RADOS) >> systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Main process >> exited, code=exited, status=5/NOTINSTALLED >> systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Unit entered >> failed state. >> systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Failed with >> result 'exit-code'. >> -- >> Martin Verges >> Managing director >> Mobile: +49 174 9335695 >> E-Mail: martin.verges(a)croit.io >> Chat: https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft.me%2FM… >> croit GmbH, Freseniusstr. 31h, 81247 Munich >> CEO: Martin Verges - VAT-ID: DE310638492 >> Com. register: Amtsgericht Munich HRB 231263 >> Web: https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io… >> YouTube: https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgoo.gl%2… >> -- >> Martin Verges >> Managing director >> Mobile: +49 174 9335695 >> E-Mail: martin.verges(a)croit.io >> Chat: https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft.me%2FM… >> croit GmbH, Freseniusstr. 31h, 81247 Munich >> CEO: Martin Verges - VAT-ID: DE310638492 >> Com. register: Amtsgericht Munich HRB 231263 >> Web: https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io… >> YouTube: https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgoo.gl%2… >> On Mon, 5 Apr 2021 at 19:59, Robert Sander <r.sander(a)heinlein-support.de> >> wrote: >>> Hi, >>>> Am 04.04.21 um 15:22 schrieb 胡玮文: >>>> bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static >>> int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, >>> std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, >>> optional_yield, std::optional<rados::cls::fifo::objv>, >>> std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 >>> create_meta failed: r=-5 >>>> bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int >>> RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, >>> librados::v14_2_0::Rados*): Error when starting backend: Input/output error >>>> bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: >>> failed to start datalog_rados service ((5) Input/output error >>>> bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: >>> failed to init services (ret=(5) Input/output error) >>> I see the same issues on an upgraded cluster. >>> Regards >>> -- >>> Robert Sander >>> Heinlein Consulting GmbH >>> Schwedter Str. 8/9b, 10119 Berlin >>> https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.heinl… >>> Tel: 030 / 405051-43 >>> Fax: 030 / 405051-19 >>> Zwangsangaben lt. §35a GmbHG: >>> HRB 93818 B / Amtsgericht Berlin-Charlottenburg, >>> Geschäftsführer: Peer Heinlein -- Sitz: Berlin >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Building ceph clusters with 8TB SSD drives?
by Erik Lindahl 08 May '21

08 May '21

The main reason for SSDs is typically to improve IOPS for small writes, but for that usage most (all) consumer SSDs we have tested perform badly in Ceph. The reason for this is that Ceph requires SYNC writes, and since consumer SSDs (and now even some cheap datacenter ones) don't have capacitors for power-loss-protection they cannot use the volatile caches that give them (semi-fake) good performance on desktops. If that sounds bad, you should be even more careful of you shop around until you find a cheap drive that performs well - because there have historically been consumer drives that lie and acknowledge a sync even if the data is just in volatile memory rather than safe :-) Samsung PM883 is likely one of the cheapest drives that you can still fully trust - at least if your application is not highly write-intensive. Now, having said that, we have had pretty good experience with a way to partly cheat around these limitations: since we have large servers with mixed HDDs we also have 2-3 NVMe samsung PM983 M.2 drives per server on PCIe cards for the DB/wal. It seems to work remarkably well to do this for consumer SSDs to, I.e. let each 4TB el cheapo SATA SSD (we used Samsung 860) use a ~100GB db/wal partition on an NVMe drive. This gives very nice low latencies in rados benchmarks, although they are still ~50% higher than with proper enterprise SSDs. Caveats: - Think about balancing IOPS. If you have 10 SSD OSDs share a single NVMe WAL device you will likely be limited by the NVMe. - if the NVMe drive dies, all the corresponding OSDs die. - This might work for read-intensive applications, but if you try it for write-intensive applications you will wear out the consumer SSDs (check their write endurance). - You will still see latency/bandwidth go up/down and periodically throttle for consumer SSDs. In comparison, even the relatively cheap pm883 "just works" at constant high bandwidth close to the bus limit, and the latency is a constant low fraction of a millisecond in ceph. In summary, while somewhat possible, I don't think it's worth the hassle/risk/complex setup with consumer drives, but if I absolutely had to i would at least avoid the absolutely cheapest QVO models - and if you don't put the WAL on a better device I predict you'll regret it once you start doing benchmarks in RADOS. Cheers, Erik

1 0

x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
by David Orman 07 May '21

07 May '21

Hi, Is there any way to log the x-amz-request-id along with the request in the rgw logs? We're using beast and don't see an option in the configuration documentation to add headers to the request lines. We use centralized logging and would like to be able to search all layers of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. Right now, all we see is this: debug 2021-04-01T15:55:31.105+0000 7f54e599b700 1 beast: 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+0000] "PUT /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; amd64)" - We've also tried this: ceph config set global rgw_enable_ops_log true ceph config set global rgw_ops_log_socket_path /tmp/testlog After doing this, inside the rgw container, we can socat - UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that we want, but there has to be a better way to do this, where the logs are emitted like the request logs above by beast, so that we can handle it using journald. If there's an alternative that would accomplish the same thing, we're very open to suggestions. Thank you, David

3 6

Upgrade and lost osds Operation not permitted
by Behzad Khoshbakhti 07 May '21

07 May '21

Hi all, As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages: root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2 Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory* root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd(a)2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE) Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2# root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~# root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2# any help is much appreciated -- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)

6 14

Natutilus - not unmapping
by Joe Comeau 07 May '21

07 May '21

Nautilus cluster is not unmapping ceph 14.2.16 ceph report |grep "osdmap_.*_committed" report 1175349142 "osdmap_first_committed": 285562, "osdmap_last_committed": 304247, we've set osd_map_cache_size = 20000 but its is slowly growing to that difference as well OSD map first committed is not changing for some strange reason Cluster has been around and upgraded since either firefly or jewel I have seen a few other with this problem to no solution to it Any suggestions ? Thanks Joe

2 2

How to trim RGW sync errors
by by morphin 07 May '21

07 May '21

Hello. I was have multisite RGW (14.2.16 nautilus) setup and some of the bucket couldn't finish bucket sync due to overfill buckets, There was different needs and the sync started purpose of migration. I made the secondary zone the master and removed the old master zone from zonegroup. Now I still have sync errors and sync error trim do not work. radosgw-admin --id radosgw.srv1 sync error list | grep name | wc -l 32000 Thats a lot of errors. Sync error trim does nothing. When I run period update commit I saw sync status field has a lot of records as below. radosgw-admin --id radosgw.srv1 period update --commit { "id": "e5d30f8f", "epoch": 7, "predecessor_uuid": "1d0b7132", "sync_status": [ "1_1611733356.499643_1448979853.1", "1_1611225916.734727_865381974.1", "1_1611648125.876993_1659659292.1", "1_1608194415.061001_737663090.1", "1_1605880458.143435_1259922694.1", "1_1611225999.087089_1887995199.1", "1_1586035175.626619_488028.1", "", "", "1_1611057887.910246_973493243.1", "1_1612180963.822684_807349060.1", "", "", "1_1612180818.328001_807344892.1", "1_1611058156.662721_1887884194.1", "1_1611057588.159455_1887883796.1", "1_1611647015.874625_1129837262.1", "1_1586035175.602419_753756.1", "", "1_1606215091.912960_988474411.1", "", "1_1600418137.932356_1027064325.1", "1_1609926537.036681_832230841.1", "", "", "1_1611057624.857485_1658280806.1", "1_1600419671.553723_365405366.1", "", "1_1611057662.014628_859134308.1", "1_1611057665.933662_843443436.1", "1_1605879154.805811_700811071.1", "1_1602509494.904964_696294030.1", "", "1_1611057618.891024_1150752303.1", "1_1611440831.055432_1458827253.1", "1_1611451128.857514_806931659.1", "", "1_1611057597.877068_1785564634.1", "1_1611057860.565465_1785564826.1", "1_1585821684.950844_61616.1", "", "", "", "1_1601647994.988107_511440126.1", "", "1_1608194424.578834_777512349.1", "1_1605879126.845904_958578574.1", "", "1_1590061636.162223_183644368.1", "1_1609834839.884870_1076396513.1", "", "1_1612430017.546386_612493167.1", "1_1605879158.230856_1635059634.1", "", "1_1612420115.322098_1468865033.1", "1_1611057731.182423_817020944.1", "1_1611225026.887795_806142997.1", "1_1612188490.428048_1152864210.1", "1_1612187913.914410_861646554.1", "1_1609393942.952120_574675578.1", "1_1611733086.223927_861322773.1", "1_1605880394.928467_759903023.1", "1_1600418082.175862_556536400.1", "1_1605879150.320951_1210709666.1" ], "period_map": { "id": "e5d30f8f", "zonegroups": [ { "id": "667afef", "name": "xy", "api_name": "xy", "is_master": "true", "endpoints": [ "http://dns:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "fe8ee939", "zones": [ { "id": "fe8ee939", "name": "prod", "endpoints": [ "http://dns:80" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 101, "read_only": "false", "tier_type": "", "sync_from_all": "false", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "234837df" } ], "short_zone_ids": [ { "key": "fe8ee939", "val": 2970845644 } ] }, "master_zonegroup": "667afefc", "master_zone": "fe8ee939", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }, "realm_id": "234837df", "realm_name": "rep", "realm_epoch": 3 } I need to clean these errors before re-add the secondary zone to zonegroup. Do you have any opinion? If I delete old periods what will happen?

1 0

[v15.2.11] radosgw / RGW crash at start, Segmentation Fault
by Gilles Mocellin 07 May '21

07 May '21

Hello, Since I upgrade to Ceph Octopus v15.2.11, on Ubuntu 18.04, Radosgw crash straight at start. On Two clusters, one Lab, and some test on a production cluster, shows the same crash for radosgw. As I don't find any similar bug in the Tracker, neither in this mailing list... Am I alone ? The logs are : May 07 14:04:59 fidcl-mrs4-sto-sds-07 systemd[1]: Started Ceph rados gateway. May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: *** Caught signal (Segmentation fault) ** May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: in thread 7ff556655140 thread_name:radosgw May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 1: (()+0x3f040) [0x7ff554e19040] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 2: (std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x55aadc0f5c54] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_trait May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 6: (()+0x40ce0a) [0x7ff555847e0a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 7: (radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 8: (__libc_start_main()+0xe7) [0x7ff554dfbbf7] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 9: (_start()+0x2a) [0x55aadc0a836a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 2021-05-07T14:05:00.309+0200 7ff556655140 -1 *** Caught signal (Segmentation fault) ** May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: in thread 7ff556655140 thread_name:radosgw May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 1: (()+0x3f040) [0x7ff554e19040] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 2: (std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x55aadc0f5c54] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_trait May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 6: (()+0x40ce0a) [0x7ff555847e0a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 7: (radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 8: (__libc_start_main()+0xe7) [0x7ff554dfbbf7] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 9: (_start()+0x2a) [0x55aadc0a836a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 0> 2021-05-07T14:05:00.309+0200 7ff556655140 -1 *** Caught signal (Segmentation fault) ** May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: in thread 7ff556655140 thread_name:radosgw May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 1: (()+0x3f040) [0x7ff554e19040] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 2: (std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x55aadc0f5c54] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_trait May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 6: (()+0x40ce0a) [0x7ff555847e0a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 7: (radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 8: (__libc_start_main()+0xe7) [0x7ff554dfbbf7] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 9: (_start()+0x2a) [0x55aadc0a836a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: -3> 2021-05-07T14:05:00.309+0200 7ff556655140 -1 *** Caught signal (Segmentation fault) ** May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: in thread 7ff556655140 thread_name:radosgw May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 1: (()+0x3f040) [0x7ff554e19040] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 2: (std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x55aadc0f5c54] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_trait May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 6: (()+0x40ce0a) [0x7ff555847e0a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 7: (radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 8: (__libc_start_main()+0xe7) [0x7ff554dfbbf7] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 9: (_start()+0x2a) [0x55aadc0a836a] May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: ceph-radosgw(a)rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Main process exited, code=killed, status=11/SEGV May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: ceph-radosgw(a)rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Failed with result 'signal'. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: ceph-radosgw(a)rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Service hold-off time over, scheduling restart. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: ceph-radosgw(a)rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Scheduled restart job, restart counter is at 653. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: Stopped Ceph rados gateway. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: ceph-radosgw(a)rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Start request repeated too quickly. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: ceph-radosgw(a)rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Failed with result 'signal'. May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: Failed to start Ceph rados gateway.

2 2

Monitor gets removed from monmap when host down
by contact＠maelgui.fr 07 May '21

07 May '21

Hi everyone, I'm new to ceph, and I'm currently doing some tests with cephadm and a few virtual machines. I've deployed ceph with cephadm on 5 VMs, each one have a 10GB virtual disk attached, and everything is working perfectly. (So 5 osd and 5 monitor in my cluster) However when I turn down a node, wait a few minutes and turn it up again, I was expecting the services to be running again automatically, but this is not happening... The monitor service does not restart. It looks like the monmap gets changed when a node is offline, causing the monitor service to refuse to restart. I need to remove the monitor service with `ceph orch daemon rm <name>` so that a new service is automatically deploy again on this node. Is that the expected behaviour ? Best regards, Maël

1 0

orch upgrade mgr starts too slow and is terminated?
by Kai Börnert 07 May '21

07 May '21

Hi all, upon updating to 16.2.2 via cephadm the upgrade is being stuck on the first mgr Looking into this via docker logs I see that it is still loading modules when it is apparently terminated and restarted in a loop When pausing the update, the mgr succeeds to start with the new version, however when resuming the update, it seems to try to update it again even tho it already has the new version, leading to the exact same loop. Is there some setting or workaround to increase the time before it is attempted to be redeployed, or can this behavior be caused by something else? Greetings, Kai

3 5

Re: fixing future rctimes
by David Rivera 07 May '21

07 May '21

Hi guys, Did anyone ever figure out how to fix rctime? I had a directory that was robocopied from a windows host that contained files with modified times in the future. Now the directory tree up to the root will not update rctime. Thanks, David

1 0

Ceph cluster not recover after OSD down
by Andres Rojas Guerrero 07 May '21

07 May '21

Hi, I have a Nautilus cluster version 14.2.6 , and I have noted that when some OSD go down the cluster doesn't start recover. I have checked that the option noout is unset. What could be the reason for this behavior? -- ******************************************************* Andrés Rojas Guerrero Unidad Sistemas Linux Area Arquitectura Tecnológica Secretaría General Adjunta de Informática Consejo Superior de Investigaciones Científicas (CSIC) Pinar 19 28006 - Madrid Tel: +34 915680059 -- Ext. 990059 email: a.rojas(a)csic.es ID comunicate.csic.es: @50852720l:matrix.csic.es *******************************************************

7 27

Slow performance and many slow ops
by codignotto 06 May '21

06 May '21

Hello, I have 6 hosts with 12 SSD disks on each host for a total of 72 OSD, I am using CEPH Octopos in its latest version, the deployment was done using ceph admin and containers according to the dosing, we are having some problems with performance of the cluster, I mount it on a proxmox cluster and on windows VMs I have the problem of the disks being 100% occupied with a simple browser opening, when I switch to another NFS storage for example everything goes back to normal, I have the CEPH cluster now mounted and with only 1 VM inside it, and we have the problem of slowness and slow ops, the network speed between the hosts in the cluster is 25Gb tested with iperf, between ceph and proxmox is 25Gb per host, someone already passed that? Many Tks

2 2

v16.2.3 Pacific released
by David Galloway 06 May '21

06 May '21

This is the third backport release in the Pacific series. We recommend all users update to this release. Notable Changes --------------- * This release fixes a cephadm upgrade bug that caused some systems to get stuck in a loop restarting the first mgr daemon. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-16.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 381b476cb3900f9a92eb95d03b4850b953cfd79a

1 0

How to find out why osd crashed with cephadm/podman containers?
by mabi 06 May '21

06 May '21

Hello, I have a small 6 nodes Octopus 15.2.11 cluster installed on bare metal with cephadm and I added a second OSD to one of my 3 OSD nodes. I started then copying data to my ceph fs mounted with kernel mount but then both OSDs on that specific nodes crashed. To this topic I have the following questions: 1) How can I find out why the two OSD crashed? because everything is in podman containers I don't know where are the logs to find out the reason why this happened. From the OS itself everything looks ok, there was no out of memory error. 2) I would assume the two OSD container would restart on their own but this is not the case it looks like. How can I restart manually these 2 OSD containers on that node? I believe this should be a "cephadm orch" command? The health of the cluster right now is: CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) PG_DEGRADED: Degraded data redundancy: 132518/397554 objects degraded (33.333%), 65 pgs degraded, 65 pgs undersized Thank your for your hints. Best regards, Mabi

2 2

Upgrade problem with cephadm
by fcid 06 May '21

06 May '21

Hello ceph community, I'm trying to upgrade a pacific (v16.2.0) cluster to the last version, but the upgrading process seems to be stuck. The mgr log (debug level) does not show any significant message regarding the upgrade, other than when it is started/paused/resumed/stopped. 2021-05-06T14:29:59.294725+0000 mgr.hostc.riclju (mgr.3935983) 35645 : cephadm [INF] Upgrade: Started with target docker.io/ceph/ceph:v16.2.2 2021-05-06T14:49:55.710023+0000 mgr.hostc.riclju (mgr.3935983) 36285 : cephadm [INF] Paused 2021-05-06T14:50:24.444742+0000 mgr.hostc.riclju (mgr.3935983) 36302 : cephadm [INF] Resumed 2021-05-06T14:51:36.888269+0000 mgr.hostc.riclju (mgr.3935983) 36349 : cephadm [INF] Upgrade: Paused upgrade to docker.io/ceph/ceph:v16.2.2 2021-05-06T14:51:50.411779+0000 mgr.hostc.riclju (mgr.3935983) 36357 : cephadm [INF] Upgrade: Resumed upgrade to docker.io/ceph/ceph:v16.2.2 2021-05-06T14:52:01.660682+0000 mgr.hostc.riclju (mgr.3935983) 36365 : cephadm [INF] Upgrade: Stopped It may be worth mentioning that last week I had trouble trying to deploy RGWs. It was not possible to deploy de RGWs using this command ceph orch apply rgw orbyta --realm=realma --zone=zonea --placement="2" So the following were used ceph orch daemon add rgw zonea --placement hostb ceph orch daemon add rgw zonea --placement hosta After those commands were issued the orchestrator would still not deploy the RGWs, unless the current MGR failed over to another standby MGR. After that, the RGWs where depoyed. Another problem I have is the refresh parameter of the orchestrator. The last time the daemons listed in ceph orch ps where refreshed is the last time a MGR was set to failed, and issuing ceph orch ps --refresh does not seem to update It looks like all those symptoms are related somehow, but I don't know how to dig further into the internals of the orchestrator to get more information. I greatly appreciate if you can point me in the right direction. Thank you, kind regards. -- AltaVoz <https://www.altavoz.net/> Fernando Cid Ingeniero de Operaciones www.altavoz.net <https://www.altavoz.net/> Ubicación AltaVoz Viña del Mar: 2 Poniente 355 of 53 <https://www.altavoz.net/altavoz/contacto> | +56 32 276 8060 <tel:+56322768060> Santiago: Antonio Bellet 292 of 701 <https://www.altavoz.net/altavoz/contacto> | +56 2 2585 4264 <tel:+562225854264>

1 0

Out of Memory after Upgrading to Nautilus
by Christoph Adomeit 06 May '21

06 May '21

I manage a historical cluster of severak ceph nodes with each 128 GB Ram and 36 OSD each 8 TB size. The cluster ist just for archive purpose and performance is not so important. The cluster was running fine for long time using ceph luminous. Last week I updated it to Debian 10 and Ceph Nautilus. Now I can see that the memory usage of each osd grows slowly to 4 GB each and once the system has no memory left it will oom-kill processes I have already configured osd_memory_target = 1073741824 . This helps for some hours but then memory usage will grow from 1 GB to 4 GB per OSD. Any ideas what I can do to further limit osd memory usage ? It would be good to keep the hardware running some more time without upgrading RAM on all OSD machines. Any Ideas ? Thanks Christoph

5 6

RGW Beast SSL version
by Glen Baars 06 May '21

06 May '21

Hello Ceph, Can you set the SSL min version? Such as TLS1.2? Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0

Re: How to set bluestore_rocksdb_options_annex
by ceph＠elchaka.de 05 May '21

05 May '21

Hello Anthony, it was introduced in octopus 15.2.10 See: https://docs.ceph.com/en/latest/releases/octopus/ Do you know how you would set it in pacific? :) Guess, there shouldnt be much difference... Thank you Mehmet Am 28. April 2021 19:21:19 MESZ schrieb Anthony D'Atri <anthony.datri(a)gmail.com>: >I think that’s new with Pacific. > >> On Apr 28, 2021, at 1:26 AM, ceph(a)elchaka.de wrote: >> >> >> >> Hello, >> >> I have an octopus cluster and want to change some values - but i >cannot find any documentation on how to set values(multiple) with >> >> bluestore_rocksdb_options_annex >> >> Could someone give me some examples. >> I would like to do this like ceph config set ... >> >> Thanks in advice >> Mehmet >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5

pgremapper released
by Josh Baergen 05 May '21

05 May '21

Hello all, I just wanted to let you know that DigitalOcean has open-sourced a tool we've developed called pgremapper. Originally inspired by CERN's upmap exception table manipulation scripts, pgremapper is a CLI written in Go which exposes a number of upmap-based algorithms for backfill-related usecases: Canceling backfill (like CERN's upmap-remapped.py, but with some extra tricks up its sleeve), draining PGs off of an OSD, undoing upmaps in a controlled and concurrent manor, and more. If you're interested, please read the details in the repo's README: https://github.com/digitalocean/pgremapper Josh

2 1

Call For Submissions IO500 ISC21 List
by IO500 Committee 05 May '21

05 May '21

https://io500.org/cfs Stabilization Period: 05 - 14 May 2021 AoE Submission Deadline: 11 June 2021 AoE The IO500 is now accepting and encouraging submissions for the upcoming 8th IO500 list. Once again, we are also accepting submissions to the 10 Node Challenge to encourage the submission of small scale results. The new ranked lists will be announced via live-stream at a virtual session. We hope to see many new results. What's New Starting with ISC'21, the IO500 now follows a two-staged approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly. During this period the benchmark will be updated based upon feedback from the community. The final benchmark will then be released on Monday, May 1st. We expect that runs compliant with the rules made during the stabilization period are valid as the final submission unless a significant defect is found. We are now creating a more detailed schema to describe the hardware and software of the system under test and provide the first set of tools to ease capturing of this information for inclusion with the submission. Further details will be released on the submission page. Background The benchmark suite is designed to be easy to run and the community has multiple active support channels to help with any questions. Please note that submissions of all sizes are welcome; the site has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. Additionally, the list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. More details below. Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown exponentially since then. The need for such an initiative has long been known within High-Performance Computing; however, defining appropriate benchmarks had long been challenging. Despite this challenge, the community, after long and spirited discussion, finally reached consensus on a suite of benchmarks and a metric for resolving the scores into a single ranking. The multi-fold goals of the benchmark suite are as follows: Maximizing simplicity in running the benchmark suite Encouraging optimization and documentation of tuning parameters for performance Allowing submitters to highlight their "hero run" performance numbers Forcing submitters to simultaneously report performance for challenging IO patterns. Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly prescribed parameters in an attempt to determine a lower-bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Submitters are encouraged to share their tuning insights for publication. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators 10 Node I/O Challenge The 10 Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with, e.g., any number of servers. When submitting for the IO500 list, you can opt-in for "Participate in the 10 compute node challenge only", then we will not include the results into the ranked list. Other 10-node node submissions will be included in the full list and in the ranked list. We will announce the result in a separate derived list and in the full list but not on the ranked IO500 list at io500.org. Birds-of-a-Feather Once again, we encourage you to submit to join our community, and to attend our virtual BoF "The IO500 and the Virtual Institute of I/O" at ISC 2021, (time to be announced), where we will announce the new IO500 and 10 node challenge lists. The current list includes results from BeeGFS, CephFS, DAOS, DataWarp, GekkoFS, GFarm, IME, Lustre, MadFS, Qumulo, Spectrum Scale, Vast, WekaIO, and YRCloudFile. We hope that the upcoming list grows even more. -- The IO500 Committee

1 0

dashboard connecting to the object gateway
by Fabrice Bacchella 05 May '21

05 May '21

I'm still trying to understand how the manager and dashboard connect to different object gateway, and I don’t really understand how it works. Initially, I wanted to have each gateway listen only to localhost, on http: [client.radosgw.<%= $id%>] rgw_frontends = beast endpoint=127.0.0.1:9080 It fails, but in a strange way. It was indeed connecting to the port 9080, so the mgr reads that line, but using the host IP. I can easily understand why binding only to 127.0.0.1 was a bad idea, but I don't understand how it choose the IP to connect to. When looking at the configuration using the dasbhoard, I see a field called hostname. This value seems to be read-only, as set it it in ceph.conf to a obvious wrong value (www.google.com <http://www.google.com/>) does not change it. Is it that field that is used ? It's important because if I want to have SSL verification, I must know the value: IP, hostname, FQDN ? that I will put in the Subject Alternative Name of the certificate.

1 0

14.2.20: Strange monitor problem eating 100% CPU
by Rainer Krienke 05 May '21

05 May '21

Hello, I am playing around with a test ceph 14.2.20 cluster. The cluster consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1, vceph2 and vceph3 are monitors. vceph1 is also mgr. What I did was quite simple. The cluster is in the state HEALTHY: vceph2: systemctl stop ceph-osd@2 # let ceph repair until ceph -s reports cluster is healthy again vceph2: systemctl start ceph-osd@2 # @ 15:39:15, for the logs # cluster reports in cephs -s that 8 OSDs are up and in, then # starts rebalance osd.2 vceph2: ceph -s # hangs forever also if executed on vceph3 or 4 # mon on vceph1 eats 100% CPU permanently, the other mons ~0 %CPU vceph1: systemctl stop ceph-mon@vceph1 # wait ~30 sec to terminate vceph1: systemctl start ceph-mon@vceph1 # Everything is OK again I posted the mon-log to: https://cloud.uni-koblenz.de/s/t8tWjWFAobZb5Hy Strange enough if I set "debug mon 20" before starting the experiment this bug does not show up. I also tried the very same procedure on the same cluster updated to 15.2.11 but I was unable to reproduce this bug in this ceph version. Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312

3 6

Certificat format for the SSL dashboard
by Fabrice Bacchella 04 May '21

04 May '21

Once activated the dashboard, I try to import certificates, but it fails: $ ceph dashboard set-ssl-certificate-key -i /data/ceph/conf/ceph.key Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1337, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 389, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/dashboard/module.py", line 385, in set_ssl_certificate_key self.set_store('key', inbuf.decode()) AttributeError: 'str' object has no attribute 'decode' $ ceph dashboard set-ssl-certificate -i /data/ceph/conf/ceph.crt Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1337, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 389, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/dashboard/module.py", line 372, in set_ssl_certificate self.set_store('crt', inbuf.decode()) AttributeError: 'str' object has no attribute 'decode' They are both PEM encoded files: file /data/ceph/conf/ceph.key /data/ceph/conf/ceph.crt /data/ceph/conf/ceph.key: PEM RSA private key /data/ceph/conf/ceph.crt: PEM certificate What format does this command expect ? That error happens on Centos 8.3.2011 with ceph-mgr-16.2.1-0.el8.x86_64, downloaded directly from ceph.

2 3

Where is the MDS journal written to?
by mabi 04 May '21

04 May '21

Hello, I have a small Octopus cluster (3 mon/mgr nodes, 3 osd nodes) installed with cephadm and hence running inside podman containers on Ubuntu 20.04. I want to use CephFS so I created a fs volume and saw that two MDS containers have been automatically deployed on two of my OSD nodes. Now I saw in the documentation (https://docs.ceph.com/en/latest/cephfs/) that the MDS first writes to a journal before writing into the meta pool so I was wondering where exactly is this MDS journal located? Because my MDS service is running inside a container, does this mean the journal gets written inside the container itself? I am asking because my containers are running from the OS hard disk which is not as performant as the OSD hard disk. So if this is the case it would be better for performance to have a dedicated hard disk for MDS? is this correct? And if yes how do I specify to cephadm that my MDS container should be writing its journal at another location? Finally I read (https://docs.ceph.com/en/latest/start/hardware-recommendations/) that the Ceph MDS service only needs 1 MB storage for its journal, is that all? Best regards, Mabi

2 2

Failed cephadm Upgrade - ValueError
by Ashley Merrick 04 May '21

04 May '21

Hello All,I was running 15.2.8 via cephadm on docker Ubuntu 20.04I just attempted to upgrade to 16.2.1 via the automated method, it successfully upgraded the mon/mgr/mds and some OSD's, however it then failed on an OSD and hasn't been able to pass even after stopping and restarting the upgrade.It reported the following ""message": "Error: UPGRADEREDEPLOYDAEMON: Upgrading daemon osd.35 on host sn-s01 failed.""If I run 'ceph health detail' I get lot's of the following error : "ValueError: not enough values to unpack (expected 2, got 1)" throughout the detail reportUpon googling, it looks like I am hitting something along the lines of https://158.69.68.89/issues/48924 & https://tracker.ceph.com/issues/49522What do I need to do to either get around this bug, or a way I can manually upgrade the remaining ceph OSD's to 16.2.1, currently my cluster is working but the last OSD it failed to upgrade is currently offline (I guess as no image attached to it now as it failed to pull it), and I have a cluster with OSD's from not 15.2.8 and 16.2.1Thanks Sent via MXlogin

2 4

possible bug in radosgw-admin bucket radoslist
by Rob Haverkamp 04 May '21

04 May '21

Hi there, I think I found a bug in the radosgw-admin bucket radoslist command. I'm not 100% sure so would like to check here first before I fill a bug report. I have a bucket called bucket3. If I do a multipart upload and stop it halfway for example and start a new upload with the same name and abort it, I'll get duplicate lines in the radoslist output. For example: root@alpha:~# radosgw-admin bucket radoslist --bucket bucket3 root@alpha:~# root@alpha:~# s3cmd put -P 100MB.bin s3://bucket3/multipart-obj-fail upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 1 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 35.14 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 2 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 33.59 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 3 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 33.88 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 4 of 7, 15MB] [1 of 1] 65536 of 15728640 0% in 0s 806.10 KB/s^CERROR: Upload of '100MB.bin' part 4 failed. Use /usr/bin/s3cmd abortmp s3://bucket3/multipart-obj-fail 2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX to abort the upload, or /usr/bin/s3cmd --upload-id 2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX put ... to continue the If I now run a radoslist it looks fine: 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_3 However, if I do a second upload with the same name: root@alpha:~# s3cmd put -P 100MB.bin s3://bucket3/multipart-obj-fail upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 1 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 23.18 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 2 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 1s 13.10 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 3 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 29.93 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 4 of 7, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 39.53 MB/s done upload: '100MB.bin' -> 's3://bucket3/multipart-obj-fail' [part 5 of 7, 15MB] [1 of 1] ^CERROR: Upload of '100MB.bin' part 5 failed. Use /usr/bin/s3cmd abortmp s3://bucket3/multipart-obj-fail 2~yS7Tzru_FSP6rkg4yeO28os207nDtTw to abort the upload, or /usr/bin/s3cmd --upload-id 2~yS7Tzru_FSP6rkg4yeO28os207nDtTw put ... to continue the upload. See ya! Note that the IDs are unique, run1: 2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX run2: 2~yS7Tzru_FSP6rkg4yeO28os207nDtTw But if we look at the radoslist output again: 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3_3 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4_1 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4_2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__shadow_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4_3 the duplicates are(output from radoslist | sort | uniq -c): 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4 If I again do a new upload and stop it I get a new upload ID 2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX but again with duplicate entries(output from radoslist | sort | uniq -c): 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.1 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.2 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.3 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~0GMvfYOGO5yhFppCWjYrUqBAOqQscoX.4 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.1 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.2 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.3 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~iNQha2hAznnKdSLukUAbJT1-4nXoWUy.4 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.1 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.2 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.3 2 646e346c-2355-49df-973f-d8ac2c6349f9.74148.1__multipart_multipart-obj-fail.2~yS7Tzru_FSP6rkg4yeO28os207nDtTw.4 I'm running Ceph Octopus deployed by cephadm: root@alpha:~# ceph versions { "overall": { "ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)": 12 } } Is this expected behaviour or should I make a bug report for this? Kind regards, Rob

1 0

OSD id 241 != my id 248: conversion from "ceph-disk" to "ceph-volume simple" destroys OSDs
by Frank Schilder 04 May '21

04 May '21

Hi all, this is a follow-up on "reboot breaks OSDs converted from ceph-disk to ceph-volume simple". I converted a number of ceph-disk OSDs to ceph-volume using "simple scan" and "simple activate". Somewhere along the way, the OSDs meta-data gets rigged and the prominent symptom is that the symlink block is changes from a part-uuid target to an unstable device name target like: before conversion: block -> /dev/disk/by-partuuid/9123be91-7620-495a-a9b7-cc85b1de24b7 after conversion: block -> /dev/sdj2 This is a huge problem as the "after conversion" device names are unstable. I have now a cluster that I cannot reboot servers on due to this problem. OSDs randomly re-assigned devices will refuse to start with: 2021-03-02 15:56:21.709 7fb7c2549b80 -1 OSD id 241 != my id 248 Please help me with getting out of this mess. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 7

using ec pool with rgw
by Marco Savoca 03 May '21

03 May '21

Hi, I’m currently deploying a new cluster for cold storage with rgw. Is there actually a more elegant method to get the bucket data on an erasure coding pool other than moving the pool or creating the bucket.data pool prior to data upload? Thanks, Marco Savoca

2 1

Spam from Chip Cox
by Frank Schilder 03 May '21

03 May '21

Does anyone else receive unsolicited replies from sender "Chip Cox <chip(a)softiron.com>" to e-mails posted on this list? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

[ Ceph MDS MON Config Variables ] Failover Delay issue
by Lokendra Rathour 03 May '21

03 May '21

Hi Team, I was setting up the ceph cluster with - Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second. T*est Case: Power-off any active MDS Node for failover to happen* *Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs: 1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2(a)1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. * *Query:* 1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1 We need to tune this setup to get the failover duration as low as 5-7 seconds. Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration. -- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

5 11

How can I get tail information a parted rados object
by by morphin 03 May '21

03 May '21

Hello. I'm trying to export objects from rados with rados get. Some objects bigger than 4M and they have tails. Is there any easy way to get tail information an object? For example this is an object: - c106b26b.3_Img/2017/12/im034113.jpg These are the objet parts: - c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1 - c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_1 - c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_2 - c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.2 As you can see the object has 2 multipart and 2 shadow object. This jpg only works when I get all the parts and make it one with the order. order: "cat 9.1 9.1_1 9.1_2 9.2 > im034113.jpg" I'm trying to write a code and the code gonna read objects from a list and find all the parts, bring it together with the order... But I couldn't find a good way to get part information. I followed the link https://www.programmersought.com/article/31497869978/ and I get the object manifest with getxattr and decode it with "ceph-dencoder type RGWBucketEnt decode dump_json" But in the manifest I can not find a path to code it. It's not useful. Is there any different place that I can take the part information an object? Or better! Is there any tool to export an object with its tails? btw: these objects created by RGW using s3. RGW can not access these files. Because of that I'm trying to export it from rados and send it to different RGW.

2 2

Troubleshoot MDS failure
by Alessandro Piazza 03 May '21

03 May '21

Dear all, I'm having a hard time troubleshooting a file-system failure on my 3 node cluster (deployed with cephadm + docker). After moving some files between folders, the cluster became laggy and Metadata Servers started failing and got stuck in rejoin state. Of course I already tried to restart the cluster multiple times. The mds units are now in a failed state because of too many restarts, the file-system is degraded and cannot be mounted because no mds is up. I think the data pool is ok because I can get files using rados. I can trigger the standby mds to become the "major" with ceph orch daemon rm mds <mds-in-error-id> or deploy a new one but the new "major" mds go again in error state. I don't find the mds logs really helpful but you can find one in the attachments for someone more expert than me. I am hesitant to follow the guide https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ because of the warnings and because the ceph-journal-tool is poorly documented. The following might be useful seppia:~# ceph fs status starfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin(laggy) starfs.polposition.njarir 539 25 17 0 POOL TYPE USED AVAIL cephfs.starfs.meta metadata 9900M 1027G cephfs.starfs.data data 12.1T 1027G MDS version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) seppia:~ # ceph health detail HEALTH_WARN 2 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available; 7 pgs not deep-scrubbed in time [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon mds.starfs.polposition.njarir on polposition.starfleet.sns.it is in error state daemon mds.starfs.seppia.wdwrho on seppia.starfleet.sns.it is in error state [WRN] FS_DEGRADED: 1 filesystem is degraded fs starfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [WRN] PG_NOT_DEEP_SCRUBBED: 7 pgs not deep-scrubbed in time pg 3.a8 not deep-scrubbed since 2021-04-20T20:07:48.346677+0000 pg 3.a2 not deep-scrubbed since 2021-04-21T08:10:55.220263+0000 pg 3.7 not deep-scrubbed since 2021-04-21T07:24:20.073569+0000 pg 2.0 not deep-scrubbed since 2021-04-21T05:01:18.439456+0000 pg 9.1a not deep-scrubbed since 2021-04-21T05:18:20.171151+0000 pg 3.1cb not deep-scrubbed since 2021-04-20T21:54:38.251349+0000 pg 3.1ef not deep-scrubbed since 2021-04-21T07:07:18.842132+0000 Thanks for any suggestions, Alessandro Piazza

1 0

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 03 May '21

03 May '21

On Mon, May 3, 2021 at 12:00 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 11:22 schrieb Ilya Dryomov: > > max_osd 12 > > I never had more then 10 osds on the two osd nodes of this cluster. > > I was running a 3 osd-node cluster earlier with more than 10 > osds, but the current cluster has been setup from scratch and > I definitely don't remember having ever more than 10 osds! > Very strange! > > I had to replace 2 disks because of DOA-Problems, but for that > I removed 2 osds and created new ones. > > I used ceph-deploy do create new osds. > > To delete osd.8 I used: > > # take it out > ceph osd out 8 > > # wait for rebalancing to finish > > systemctl stop ceph-osd@8 > > # wait for a healthy cluster > > ceph osd purge 8 --yes-i-really-mean-it > > # edit ceph.conf and remove osd.8 > > ceph-deploy --overwrie-conf admin s0 s1 > > # Add the new disk and: > ceph-deploy osd create --data /dev/sdc s0 > ... > > it get's created with the next free osd num (8) because purge releases 8 for reuse It would be nice to track it down, but for the immediate issue of kernel 5.11 not working, "ceph osd setmaxosd 10" should fix it. Thanks, Ilya

1 0

Cannot create issue in bugtracker
by Tobias Urdin 03 May '21

03 May '21

Hello, Is it only me that's getting Internal error when trying to create issues in the bugtracker for some day or two? https://tracker.ceph.com/issues/new Best regards

2 2

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 03 May '21

03 May '21

On Mon, May 3, 2021 at 9:20 AM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 00:44 schrieb Ilya Dryomov: > > On Sun, May 2, 2021 at 11:15 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Hi, > > I know there is a thread about problems with mounting cephfs with 5.11 kernels. > > ... > > Hi Magnus, > > What is the output of "ceph config dump"? > > Instead of providing those lines, can you run "ceph osd getmap 64281 -o > osdmap.64281" and attach osdmap.64281 file? > > Thanks, > > Ilya > > Hi Ilya, > > [root@s1 ~]# ceph config dump > WHO MASK LEVEL OPTION VALUE RO > global basic device_failure_prediction_mode local > global advanced ms_bind_ipv4 false > mon advanced auth_allow_insecure_global_id_reclaim false > mon advanced mon_lease 8.000000 > mgr advanced mgr/devicehealth/enable_monitoring true > > getmap output is attached, I see the problem, but I don't understand the root cause yet. It is related to the two missing OSDs: > May 02 22:54:05 islay kernel: libceph: no match of type 1 in addrvec > May 02 22:54:05 islay kernel: libceph: corrupt full osdmap (-2) epoch 64281 off 3154 (00000000a90fe1d7 of 000000000083f4bd-00000000c03bdc9b) > max_osd 12 > osd.0 up in ... [v2:192.168.200.141:6804/3027,v1:192.168.200.141:6805/3027] ... exists,up 631bc170-45fd-4948-9a5e-4c278569c0bc > osd.1 up in ... [v2:192.168.200.140:6811/3066,v1:192.168.200.140:6813/3066] ... exists,up 660a762c-001d-4160-a9ee-d0acd078e776 > osd.2 up in ... [v2:192.168.200.141:6815/3008,v1:192.168.200.141:6816/3008] ... exists,up e4d94d3a-ec58-46a1-b61c-c47dd39012ed > osd.3 up in ... [v2:192.168.200.140:6800/3067,v1:192.168.200.140:6801/3067] ... exists,up 26d25060-fd99-4d15-a1b2-ebb77646671e > osd.4 up in ... [v2:192.168.200.140:6804/3049,v1:192.168.200.140:6806/3049] ... exists,up 238f197d-ecbc-4588-8a99-6a63c9bb1a17 > osd.5 up in ... [v2:192.168.200.140:6816/3073,v1:192.168.200.140:6817/3073] ... exists,up a9dcb26f-0f1c-4067-a26b-a29939285e0b > osd.6 up in ... [v2:192.168.200.141:6808/3020,v1:192.168.200.141:6809/3020] ... exists,up f399b47d-063f-4b2f-bd93-289377dc9945 > osd.7 up in ... [v2:192.168.200.141:6800/3023,v1:192.168.200.141:6801/3023] ... exists,up 3557ceca-7bd8-401e-abd3-59bee168e8f6 > osd.8 up in ... [v2:192.168.200.141:6812/3017,v1:192.168.200.141:6813/3017] ... exists,up 7f9cad3f-163d-4bb7-85b2-fffd46982fff > osd.9 up in ... [v2:192.168.200.140:6805/3053,v1:192.168.200.140:6807/3053] ... exists,up c543b12a-f9bf-4b83-af16-f6b8a3926e69 The kernel client is failing to parse addrvec entries for non-existent osd10 and osd11. It is probably being too stringent, but before fixing it I'd like to understand what happened to those OSDs. It looks like they were removed but not completely. What let to their removal? What commands were used? Thanks, Ilya

1 0

global multipart lc policy in radosgw
by Boris Behrens 02 May '21

02 May '21

Hi, I have a lot of multipart uploads that look like they never finished. Some of them date back to 2019. Is there a way to clean them up when they didn't finish in 28 days? I know I can implement a LC policy per bucket, but how do I implement it cluster wide? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Magnus Harlander 02 May '21

02 May '21

Hi, I know there is a thread about problems with mounting cephfs with 5.11 kernels. I tried everything that's mentioned there, but I still can not mount a cephfs from an octopus node. I verified: - I can not mount with 5.11 client kernels (fedora 33 and ubuntu 21.04) - I can mount with 5.10 client kernels - It is not due to ipv4/ipv6. I'm not using ipv6 - I'm using a cluster network on a private network segment. Because this was mentioned as a possible cause for the problems (next to ipv6) I removed the cluster network and now I'm using the same network for osd syncs and client connections. It did not help. - mount returns with a timeout and error after about 1 minute - I tried the ms_mode=legacy (and others) mount options. Nothing helped - I tried to use IP:PORT:/fs to mount to exclude DNS as the cause. Didn't help. - I did setup a similar test cluster on a few VMs and did not have a problem with mouting. Even used cluster networks, which also worked fine. I'm running out of ideas? Any help would be appreciated. \Magnus My Setup: SERVER OS: ========== [root@s1 ~]# hostnamectl Static hostname: s1.harlan.de Icon name: computer-desktop Chassis: desktop Machine ID: 3a0a6308630842ffad6b9bb8be4c7547 Boot ID: ffb2948d3934419dafceb0990316d9fd Operating System: CentOS Linux 8 CPE OS Name: cpe:/o:centos:centos:8 Kernel: Linux 4.18.0-240.22.1.el8_3.x86_64 Architecture: x86-64 CEPH VERSION: ============= ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) CLIENT OS: ========== [root@islay ~]# hostnamectl Static hostname: islay Icon name: computer-laptop Chassis: laptop Machine ID: 6de7b27dfd864e9ea52b8b0cff47cdfc Boot ID: 6d8d8bb36f274458b2b761b0a046c8ad Operating System: Fedora 33 (Workstation Edition) CPE OS Name: cpe:/o:fedoraproject:fedora:33 Kernel: Linux 5.11.16-200.fc33.x86_64 Architecture: x86-64 CEPH VERSION: ============= [root@islay harlan]# ceph version ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) [root@s1 ~]# ceph version ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) FSTAB ENTRY: ============ cfs0,cfs1:/fs /data/fs ceph rw,_netdev,name=admin,secretfile=/etc/ceph/fs.secret 0 0 IP CONFIG MON/OSD NODE (s1) ======================= [root@s1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp4s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 98:de:d0:04:26:86 brd ff:ff:ff:ff:ff:ff 3: enp5s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether a8:a1:59:18:e7:ea brd ff:ff:ff:ff:ff:ff 4: vmbr: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 98:de:d0:04:26:86 brd ff:ff:ff:ff:ff:ff inet 192.168.200.111/24 brd 192.168.200.255 scope global noprefixroute vmbr valid_lft forever preferred_lft forever inet 192.168.200.141/24 brd 192.168.200.255 scope global secondary noprefixroute vmbr valid_lft forever preferred_lft forever inet 192.168.200.101/24 brd 192.168.200.255 scope global secondary vmbr valid_lft forever preferred_lft forever inet6 fe80::be55:705d:7c9e:eaa4/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr state UP group default qlen 1000 link/ether 98:de:d0:04:26:86 brd ff:ff:ff:ff:ff:ff 6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:32:ea:2f brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:32:ea:2f brd ff:ff:ff:ff:ff:ff 8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr state UNKNOWN group default qlen 1000 link/ether fe:54:00:67:4d:15 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe67:4d15/64 scope link valid_lft forever preferred_lft forever CEPH STATUS: ============ [root@s1 ~]# ceph -s cluster: id: 86bbd6c5-ae96-4c78-8a5e-50623f0ae524 health: HEALTH_OK services: mon: 4 daemons, quorum s0,mbox,s1,r1 (age 6h) mgr: s1(active, since 6h), standbys: s0 mds: fs:1 {0=s1=up:active} 1 up:standby osd: 10 osds: 10 up (since 6h), 10 in (since 6h) data: pools: 6 pools, 289 pgs objects: 1.75M objects, 1.6 TiB usage: 3.3 TiB used, 13 TiB / 16 TiB avail pgs: 289 active+clean io: client: 0 B/s rd, 245 KiB/s wr, 0 op/s rd, 4 op/s wr CEPH OSD TREE: ============== [root@s1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 16.99994 root default -9 8.39996 host s0 1 hdd 4.00000 osd.1 up 1.00000 1.00000 5 hdd 1.79999 osd.5 up 1.00000 1.00000 9 hdd 1.79999 osd.9 up 1.00000 1.00000 3 ssd 0.50000 osd.3 up 1.00000 1.00000 4 ssd 0.29999 osd.4 up 1.00000 1.00000 -12 8.59998 host s1 6 hdd 1.79999 osd.6 up 1.00000 1.00000 7 hdd 1.79999 osd.7 up 1.00000 1.00000 8 hdd 4.00000 osd.8 up 1.00000 1.00000 0 ssd 0.50000 osd.0 up 1.00000 1.00000 2 ssd 0.50000 osd.2 up 1.00000 1.00000 CEPH MON STAT: ============== [root@s1 ~]# ceph mon stat e19: 4 mons at {mbox=[v2:192.168.200.5:3300/0,v1:192.168.200.5:6789/0],r1=[v2:192.168.200.113:3300/0,v1:192.168.200.113:6789/0],s0=[v2:192.168.200.110:3300/0,v1:192.168.200.110:6789/0],s1=[v2:192.168.200.111:3300/0,v1:192.168.200.111:6789/0]}, election epoch 8618, leader 0 s0, quorum 0,1,2,3 s0,mbox,s1,r1 CEPH FS DUMP: ============= [root@s1 ~]# ceph fs dump dumped fsmap epoch 15534 e15534 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 2 Filesystem 'fs' (2) fs_name fs epoch 15534 flags 12 created 2021-02-02T18:47:25.306744+0100 modified 2021-05-02T16:33:36.738341+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client 0 (unknown) last_failure 0 last_failure_osd_epoch 64252 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=54782953} failed damaged stopped data_pools [10] metadata_pool 11 inline_data disabled balancer standby_count_wanted 1 [mds.s1{0:54782953} state up:active seq 816 addr [v2:192.168.200.111:6800/1895356761,v1:192.168.200.111:6801/1895356761]] Standby daemons: [mds.s0{-1:54958514} state up:standby seq 1 addr [v2:192.168.200.110:6800/297471268,v1:192.168.200.110:6801/297471268]] CEPH CONF: ========== [root@s1 ~]# cat /etc/ceph/ceph.conf [global] fsid = 86bbd6c5-ae96-4c78-8a5e-50623f0ae524 mon_initial_members = s0, s1, mbox, r1 mon_host = 192.168.200.110,192.168.200.111,192.168.200.5,192.168.200.113 ms_bind_ipv4 = true ms_bind_ipv6 = false auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public network = 192.168.200.0/24 [osd] public network = 192.168.200.0/24 osd_memory_target = 2147483648 osd crush update on start = false [osd.1] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.3] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.4] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.5] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.9] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.0] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.2] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.6] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.7] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.8] public addr = 192.168.200.141 osd_memory_target = 2147483648 CEPH FS STAT ============ [root@s1 ~]# ceph fs status fs - 0 clients == RANK STATE MDS ACTIVITY DNS INOS 0 active s1 Reqs: 0 /s 0 0 POOL TYPE USED AVAIL cfs_md metadata 2365M 528G cfs data 2960G 4967G STANDBY MDS s0 VERSION DAEMONS None s1 ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) s0 CLIENT JOURNALCTL WHEN MOUNTING =============================== May 02 22:54:04 islay kernel: FS-Cache: Loaded May 02 22:54:05 islay kernel: Key type ceph registered May 02 22:54:05 islay kernel: libceph: loaded (mon/osd proto 15/24) May 02 22:54:05 islay kernel: FS-Cache: Netfs 'ceph' registered for caching May 02 22:54:05 islay kernel: ceph: loaded (mds proto 32) May 02 22:54:05 islay kernel: libceph: mon1 (1)192.168.200.111:6789 session established May 02 22:54:05 islay kernel: libceph: mon1 (1)192.168.200.111:6789 socket closed (con state OPEN) May 02 22:54:05 islay kernel: libceph: mon1 (1)192.168.200.111:6789 session lost, hunting for new mon May 02 22:54:05 islay kernel: libceph: mon0 (1)192.168.200.5:6789 session established May 02 22:54:05 islay kernel: libceph: no match of type 1 in addrvec May 02 22:54:05 islay kernel: libceph: corrupt full osdmap (-2) epoch 64281 off 3154 (00000000a90fe1d7 of 000000000083f4bd-00000000c03bdc9b) May 02 22:54:05 islay kernel: osdmap: 00000000: 08 07 4f 24 00 00 09 01 9e 12 00 00 86 bb d6 c5 ..O$............ May 02 22:54:05 islay kernel: osdmap: 00000010: ae 96 4c 78 8a 5e 50 62 3f 0a e5 24 19 fb 00 00 ..Lx.^Pb?..$.... May 02 22:54:05 islay kernel: osdmap: 00000020: 54 f0 53 5d 3a fd ae 0e 1b 07 8f 60 b3 8e d2 2f T.S]:......`.../ May 02 22:54:05 islay kernel: osdmap: 00000030: 06 00 00 00 02 00 00 00 00 00 00 00 1d 05 44 01 ..............D. May 02 22:54:05 islay kernel: osdmap: 00000040: 00 00 01 02 02 02 20 00 00 00 20 00 00 00 00 00 ...... ... ..... May 02 22:54:05 islay kernel: osdmap: 00000050: 00 00 00 00 00 00 5e fa 00 00 2e 04 00 00 00 00 ......^......... May 02 22:54:05 islay kernel: osdmap: 00000060: 00 00 5e fa 00 00 00 00 00 00 00 00 00 00 00 00 ..^............. ..... many more lines, i can provide if they are useful. CEPH OSDMAP: ============ epoch 64281 fsid 86bbd6c5-ae96-4c78-8a5e-50623f0ae524 created 2019-08-14T13:28:20.246349+0200 modified 2021-05-02T22:10:03.802328+0200 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 140 full_ratio 0.92 backfillfull_ratio 0.9 nearfull_ratio 0.88 require_min_compat_client jewel min_compat_client jewel require_osd_release octopus pool 2 'vms' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 64094 lfor 0/62074/62072 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'ssdpool' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 61436 lfor 0/61436/61434 flags hashpspool stripe_width 0 pool 9 'hddpool' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 61413 lfor 0/61413/61411 flags hashpspool stripe_width 0 pool 10 'cfs' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 63328 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 11 'cfs_md' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 63332 flags hashpspool stripe_width 0 application cephfs pool 12 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 64255 flags hashpspool stripe_width 0 application mgr_devicehealth max_osd 12 osd.0 up in weight 1 up_from 64236 up_thru 64263 down_at 64233 last_clean_interval [64211,64231) [v2:192.168.200.141:6804/3027,v1:192.168.200.141:6805/3027] [v2:192.168.200.111:6806/3027,v1:192.168.200.111:6807/3027] exists,up 631bc170-45fd-4948-9a5e-4c278569c0bc osd.1 up in weight 1 up_from 64259 up_thru 64260 down_at 64249 last_clean_interval [64223,64248) [v2:192.168.200.140:6811/3066,v1:192.168.200.140:6813/3066] [v2:192.168.200.110:6813/3066,v1:192.168.200.110:6815/3066] exists,up 660a762c-001d-4160-a9ee-d0acd078e776 osd.2 up in weight 1 up_from 64236 up_thru 64266 down_at 64233 last_clean_interval [64211,64231) [v2:192.168.200.141:6815/3008,v1:192.168.200.141:6816/3008] [v2:192.168.200.111:6816/3008,v1:192.168.200.111:6817/3008] exists,up e4d94d3a-ec58-46a1-b61c-c47dd39012ed osd.3 up in weight 1 up_from 64256 up_thru 64264 down_at 64249 last_clean_interval [64221,64248) [v2:192.168.200.140:6800/3067,v1:192.168.200.140:6801/3067] [v2:192.168.200.110:6802/3067,v1:192.168.200.110:6803/3067] exists,up 26d25060-fd99-4d15-a1b2-ebb77646671e osd.4 up in weight 1 up_from 64256 up_thru 64264 down_at 64249 last_clean_interval [64221,64248) [v2:192.168.200.140:6804/3049,v1:192.168.200.140:6806/3049] [v2:192.168.200.110:6806/3049,v1:192.168.200.110:6807/3049] exists,up 238f197d-ecbc-4588-8a99-6a63c9bb1a17 osd.5 up in weight 1 up_from 64260 up_thru 64260 down_at 64249 last_clean_interval [64226,64248) [v2:192.168.200.140:6816/3073,v1:192.168.200.140:6817/3073] [v2:192.168.200.110:6818/3073,v1:192.168.200.110:6819/3073] exists,up a9dcb26f-0f1c-4067-a26b-a29939285e0b osd.6 up in weight 1 up_from 64240 up_thru 64260 down_at 64233 last_clean_interval [64218,64231) [v2:192.168.200.141:6808/3020,v1:192.168.200.141:6809/3020] [v2:192.168.200.111:6810/3020,v1:192.168.200.111:6811/3020] exists,up f399b47d-063f-4b2f-bd93-289377dc9945 osd.7 up in weight 1 up_from 64238 up_thru 64260 down_at 64233 last_clean_interval [64214,64231) [v2:192.168.200.141:6800/3023,v1:192.168.200.141:6801/3023] [v2:192.168.200.111:6802/3023,v1:192.168.200.111:6803/3023] exists,up 3557ceca-7bd8-401e-abd3-59bee168e8f6 osd.8 up in weight 1 up_from 64242 up_thru 64260 down_at 64233 last_clean_interval [64216,64231) [v2:192.168.200.141:6812/3017,v1:192.168.200.141:6813/3017] [v2:192.168.200.111:6814/3017,v1:192.168.200.111:6815/3017] exists,up 7f9cad3f-163d-4bb7-85b2-fffd46982fff osd.9 up in weight 1 up_from 64257 up_thru 64257 down_at 64249 last_clean_interval [64229,64248) [v2:192.168.200.140:6805/3053,v1:192.168.200.140:6807/3053] [v2:192.168.200.110:6808/3053,v1:192.168.200.110:6809/3053] exists,up c543b12a-f9bf-4b83-af16-f6b8a3926e69 blacklist 192.168.200.110:0/3803039218 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.111:6800/3725740504 expires 2021-05-03T15:37:38.953040+0200 blacklist 192.168.200.110:6822/3464419 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.110:6801/838484672 expires 2021-05-03T15:56:13.108594+0200 blacklist 192.168.200.110:6800/838484672 expires 2021-05-03T15:56:13.108594+0200 blacklist 192.168.200.111:6841/159804987 expires 2021-05-03T14:54:05.413130+0200 blacklist 192.168.200.111:6840/159804987 expires 2021-05-03T14:54:05.413130+0200 blacklist 192.168.200.111:6801/3725740504 expires 2021-05-03T15:37:38.953040+0200 blacklist 192.168.200.110:6807/453197 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.5:6801/3078236863 expires 2021-05-03T14:38:57.694004+0200 blacklist 192.168.200.110:0/1948864559 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.111:6800/3987205903 expires 2021-05-03T15:32:12.633802+0200 blacklist 192.168.200.111:6800/2342337613 expires 2021-05-03T14:46:57.936272+0200 blacklist 192.168.200.110:0/3020995128 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.5:6800/3078236863 expires 2021-05-03T14:38:57.694004+0200 blacklist 192.168.200.110:0/2607867017 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.111:6801/3987205903 expires 2021-05-03T15:32:12.633802+0200 blacklist 192.168.200.110:0/3159222459 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.110:6806/453197 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.110:6823/3464419 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.111:6801/2342337613 expires 2021-05-03T14:46:57.936272+0200 blacklist 192.168.200.111:6800/2205788037 expires 2021-05-03T14:56:56.448631+0200 blacklist 192.168.200.111:6801/2205788037 expires 2021-05-03T14:56:56.448631+0200 -- Dr. Magnus Harlander Mail: harlan(a)harlan.de Web: www.harlan.de Stiftung: www.harlander-stiftung.de Ceterum censeo bitcoin esse delendam!

2 1

How radosgw works ?
by Fabrice Bacchella 02 May '21

02 May '21

I'm trying to understand what and where radosgw listen ? There is a lot of contradictory or redundant informations about that. First about the contradictory informations for the socket. At https://docs.ceph.com/en/pacific/radosgw/config-ref/ <https://docs.ceph.com/en/pacific/radosgw/config-ref/>, it says rgw_socket_path, but at https://docs.ceph.com/en/pacific/man/8/radosgw/ <https://docs.ceph.com/en/pacific/man/8/radosgw/> is says 'rgw socket path' That problem is quite common in the ceph documentation. Are both value accepted ? Next about some naming, or binding IP. Where it's defined, and how ? You have: rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 port=443 ..." rgw_host = rgw_port = rgw_dns_name = That's a lot of redundancy, or contradictory informations. What is the purpose of each one ? What is the difference between rgw_frontends = ".. port = ..." and rgw_port = ? Or rgw_host and rgw_dns_name. What is the difference ? The documentation provides no help at all: rgw_dns_name Description: The DNS name of the served domain. See also the hostnames setting within regions. The description says nothing new, it just repeat the field name. Is one of them used by the manager for communication ? I already had the problem for the entry in the certificate used by the frontend, it used an IP coming from nowhere. If a fcgi is used, how the manager find the endpoint ?

2 2

Big OSD add, long backfill, degraded PGs, deep-scrub backlog, OSD restarts
by Dave Hall 01 May '21

01 May '21

Hello, I recently added 2 OSD nodes to my Nautilus cluster, increasing the OSD count from 32 to 48 - all 12TB HDDs with NVMe for db. I generally keep an ssh session open where I can run 'watch cepf -s'. My observations are mostly based on what I saw from watching this. Even with 10GB networking, rebalancing 529 pgs took 10 days, during which there were always a few PGs undersized+degraded, frequent flashes of slow ops, occasional OSD restarts, and the scrub and deep-scrub backlog steadily increased. When the backfills completed I had 24 missed deep-scrubs and 10 missed scrubs. I suspect that this is because of some settings that I had fiddled with, so this post may be an advertisement for what not to do to your cluster. However, I'd like to know if my understanding is accurate. I believe that my settings resulted In short, I think I had my config set up so there was contention due to too many processes trying to do things to some OSDs all at once: - osd_scrub_during_recovery: I think I had this set to true for the first 9 days, but set it to false when I started to realize that it might be causing contention - osd_max_scrubs: I had this set high - global:30 osd:10. At some earlier time when I had a scrub backlog I thought that these were counts for simultaneous scrubs across all OSDs rather than 'per OSD' - Now I see why the default is 1. - Assumption: on an HDD multiple competing scrubs cause excessive seeking and thus compound impacts to scrub progress - osd_max_backfills: I had bumped this up as well - global:30 osd:10, thinking it would speed up the rebalancing of my PGs onto my new OSDs. - Now, the same thinking as for osd_max_scrubs: compounding contention, further compounded by the scrub acivity that should have been inhibited by osd_scrub_during_recovery:false. I believe that all of this also resulted in my EC pgs (8 + 2) becoming degraded. My assumption here is that collisions between deep-scrubs and backfills sometimes locked the backfill process out of a piece of an EC PG, causing backfil to rebuild instead of copy. The good news is that I haven't lost and data and, other than the scrub backlog things seem to be working smoothly. It seems like with 1 or 2 scrubs (deep or regular) running they are taking about 2 hours per scrub. As the scrubs progress, more scrub deadlines are missed, so it's not a steady march to zero. Please feel free to comment. I'd be glad to know if I'm on the right track as we expect the cluster to double in size over the next 12 to 18 months. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Best distro to run ceph.
by Peter Childs 01 May '21

01 May '21

I'm trying to set up a new ceph cluster, and I've hit a bit of a blank. I started off with centos7 and cephadm. Worked fine to a point, except I had to upgrade podman but it mostly worked with octopus. Since this is a fresh cluster and hence no data at risk, I decided to jump straight into Pacific when it came out and upgrade. Which is where my trouble began. Mostly because Pacific needs a version on lvm later than what's in centos7. I can't upgrade to centos8 as my boot drives are not supported by centos8 due to the way redhst disabled lots of disk drivers. I think I'm looking at Ubuntu or debian. Given cephadm has a very limited set of depends it would be good to have a supported matrix, it would also be good to have a check in cephadm on upgrade, that says no I won't upgrade if the version of lvm2 is too low on any host and let's the admin fix the issue and try again. I was thinking to upgrade to centos8 for this project anyway until I relised that centos8 can't support my hardware I've inherited. But currently I've got a broken cluster unless I can workout some way to upgrade lvm in centos7. Peter.

3 3