- ceph-users - lists.ceph.io

ceph 18.2.4 on el8?
by Christopher Durham 01 Aug '24

01 Aug '24

Hi, I see that 18.2.4 is out, in rpm for el9 at: http://download.ceph.com/rpm-18.2.4/ Are there any plans for an '8' version? One of my clusters is not yet ready to update to Rocky 9. We will update to 9 moving forward but this itme around it would be good to have a Rocky 8 version. Thanks! cbd

2 2

Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
by Bartosz Bezak 31 Jul '24

31 Jul '24

Hi, Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308. Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga. I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine. With rgw_swift_account_in_url = true and proper endpoints: "https://rgw.test/swift/v1/AUTH_%(project_id)s" ticking public access in horizon properly sets ACL on the bucket according to swift client: swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 however still getting 404 NoSuchBucket error Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you. Best regards, Bartosz Bezak

3 3

ceph-mgr memory problems 16.2.15
by Тимур Мухаметов 31 Jul '24

31 Jul '24

I have faced a similar problem that is described here https://tracker.ceph.com/issues/59580#change-270387 upon update to 16.2.15. I have a question for developers. Are you planning to fix this problem? Thank you in advance.

1 0

Ceph on Ubuntu 24.04 - Arm64
by Daniel Brown 31 Jul '24

31 Jul '24

Is there any workable solution for running Ceph on Ubuntu 24.04 on Arm64? I’ve tried about every package install method I could think of, short of compiling it myself. I’m aiming for a “cephadm” install using docker containers. It be super nice if there was a working “noble” install of reef 18.2.4 for Arm64 made available on download.ceph.com Ubuntu suggests using their “noble-proposed” in this thread: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2063456?comments=all But it ends up trying to pull down an amd64 container for my arm64 machine: sudo cephadm shell ceph -s Inferring fsid b7105258-638a-11ee-b38c-bf980c3a0ec9 Inferring config /etc/ceph/ceph.conf This is a development version of cephadm. For information regarding the latest stable release: https://docs.ceph.com/docs/reef/cephadm/install Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:main -e NODE_NAME=hc-36c0ddecd544cb8d quay.ceph.io/ceph-ci/ceph:main -c %u %g /var/lib/ceph stat: stderr Unable to find image 'quay.ceph.io/ceph-ci/ceph:main' locally stat: stderr main: Pulling from ceph-ci/ceph stat: stderr 43735fbce921: Pulling fs layer stat: stderr 7679c4b05544: Pulling fs layer stat: stderr 43735fbce921: Verifying Checksum stat: stderr 43735fbce921: Download complete stat: stderr 43735fbce921: Pull complete stat: stderr 7679c4b05544: Verifying Checksum stat: stderr 7679c4b05544: Download complete stat: stderr 7679c4b05544: Pull complete stat: stderr Digest: sha256:e7e652f9de2825d155a993aecfce4872bbb0c497a4342127081b529aec0528f5 stat: stderr Status: Downloaded newer image for quay.ceph.io/ceph-ci/ceph:main stat: stderr WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested stat: stderr [FATAL tini (7)] exec stat failed: Exec format error ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:main -e NODE_NAME=hc-36c0ddecd544cb8d quay.ceph.io/ceph-ci/ceph:main -c %u %g /var/lib/ceph

3 2

[RGW] Resharding in multi-zonegroup env cause data loss
by Huy Nguyen 31 Jul '24

31 Jul '24

Hi, My lab setup two Ceph clusters as 2 zonegroups in a realm. Ceph version 16.2.13 As I know, if a bucket is created on secondary zonegroup, the master zonegroup only know about its metadata, not data. So after resharding command is run on master (to avoid inconsistent metadata), the bucket is synced back to secondary which mean it will have no data, just like in master. Is this a bug or I'm doing it wrong? Is there any other way to reshard a bucket in non-master zonegroup? Thanks

2 1

Please guide us in identifying the cause of the data miss in EC pool
by wu_chulin＠qq.com 31 Jul '24

31 Jul '24

Dear Ceph team:     On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list && --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker: https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com

1 0

Re: 18.2.4 regression: 'diskprediction_local' has failed: No module named 'sklearn'
by thymus_03fumbler＠icloud.com 31 Jul '24

31 Jul '24

Same issue here

1 0

RBD Stuck Watcher
by Reid Guyett 30 Jul '24

30 Jul '24

Hi, I have a small script in a Docker container we use for a type of CRUD test to monitor availability. The script uses Python librbd/librados and is launched by Telegraf input.exec. It does the following: 1. Creates an rbd image 2. Writes a small amount of data to the rbd 3. Reads the data from the rbd 4. Deletes the rbd 5. Closes connections It works great for 99% of the time but there is a small chance that something happens and the script takes too long (1 min) to complete and it is killed. I don't have logging to know which step it happens at yet but will be adding some. Regardless when the script is killed, sometimes the watcher on the rbd isn't going away. I use the same RBD name for each test and try to clean up the rbd if it exists prior to starting the next test but when the watcher is stuck, it can't. The only way to cleanup the watcher is to restart the primary osd for the rbd_header. Blocklist and restarting the container free the watcher. When I look at the status of the image I can see the watcher. # rbd -p pool status crud-image Watchers: watcher=<ipaddr>:0/3587274006 client.1053762394 cookie=140375838755648 Lookup up primary OSD # rbd -p pool info crud-image | grep id id: cf235ae95099cb # ceph osd map pool rbd_header.cf235ae95099cb osdmap e332984 pool 'pool' (1) object 'rbd_header.cf235ae95099cb' -> pg 1.a76f353e (1.53e) -> up ([7,66,176], p7) acting ([7,66,176], p7) Checking watchers on primary OSD does NOT list rbd_header.cf235ae95099cb # ceph tell osd.7 dump_watchers [ { "namespace": "", "object": "rbd_header.70fa4f9b5c2cf8", "entity_name": { "type": "client", "num": 998139266 }, "cookie": 140354859197312, "timeout": 30, "entity_addr_t": { "type": "v1", "addr": "<ipaddr>:0", "nonce": 2665188958 } } ] Is this a bug somewhere? I expect that if my script is killed it's watcher should die out within a minute. New runs of the script would result in new watcher/client/cookie ids. Thanks! Reid

2 2

Re: A problem with the CEPH PG state getting stuck
by Eugen Block 30 Jul '24

30 Jul '24

Hi, your cli output is barely readable, although it probably is not that relevant here. Apparently, it's an EC pool you're referring to? A pg repair tries to repair inconsistent objects, see [1] for more details. I don't really know how to explain "repeer", I'm also not a dev, so maybe someone from the ceph team can explain it better. But from how I understand it, a temporary new mapping for the primary OSD is created and that would trigger something like a refresh, I guess: // map to just primary; it will map back to what it wants pending_inc.new_pg_temp[pgid] = { primary } So this doesn't really affect your PG content wise, I think, it's just a refresh. Why your PGs get stuck (is it always the same? Is a specific OSD involved in all of the cases?) is difficut to answer without knowing so little about your cluster. Is the overall cluster health status okay? Are your OSDs (slow or fast drives?) highly utilized? [1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/#m… Zitat von 苏察哈尔灿 <2644294460(a)qq.com>: > My ceph cluster sometimes gets stuck in the active+clean+snaptrim > state when doing regular snapshot deletion, and the corresponding pg > does not change for a long time. As follows: > > > 27.7c     14350        >  0          0      >   0  38073876582            > 0           0   2864  >     3000        > active+clean+snaptrim     9h  43777'677286  > 43777:1018248    [52,55,20,36,29,91,9,63,14,2]p52  >   [52,55,20,36,29,91,9,63,14,2]p52  > 2024-07-21T13:57:12.602903+0000  > 2024-07-20T11:02:06.953985+0000          >          52  queued for scrub  >                   >                   >           > 27.c1     14055        >  0          0      >   0  37408096205            > 0           0   2754  >     3000        > active+clean+snaptrim     9h  43777'676887  >  43777:952875    >  [0,21,39,62,26,58,41,86,66,2]p0    >  [0,21,39,62,26,58,41,86,66,2]p0  > 2024-07-22T02:26:43.470883+0000  > 2024-07-20T05:09:23.763918+0000          >          44  periodic scrub scheduled > @ 2024-07-23T07:46:50.084316+0000      > 27.19b    14711         0  >         0        0  > 38926389125            0    >        0   2765      > 3000        active+clean+snaptrim    >  9h  43777'698478   43777:849574    >   [5,7,29,14,61,10,22,0,19,37]p5      > [5,7,29,14,61,10,22,0,19,37]p5  > 2024-07-22T03:28:39.036189+0000  > 2024-07-17T14:00:43.082766+0000          >          40  periodic scrub scheduled > @ 2024-07-23T12:30:57.488823+0000      > 27.1a3    14323         0  >         0        0  > 37918899397            0    >        0   2930      > 3000        active+clean+snaptrim    >  9h  43777'675713   43777:943324  >  [52,4,10,46,20,69,49,39,44,30]p52  >  [52,4,10,46,20,69,49,39,44,30]p52  > 2024-07-21T11:15:16.442506+0000  > 2024-07-20T04:10:45.391550+0000          >          50  queued for scrub  >                 > > > No change for 9 hours. Then, after I typed the command "ceph pg > repeer 27.7c", the corresponding pg state was restored to normal. I > don't know what this "repeer" command is for, will it have any > effect on pg? What's the difference between "repeer" and "ceph pg > repair"? Then, why does the pg get stuck so often? Thanks for your > help! > > > My ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > quincy (stable) > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Is it possible to replace the md5 etags with the blake3 etags?
by jmusial 30 Jul '24

30 Jul '24

Answer very much appreciated. Thank you, Jan Musial

1 0