- ceph-users - lists.ceph.io

Resize RBD - New size not compatible with object map
by Torkil Svensgaard 06 Aug '24

06 Aug '24

Hi [ceph: root@ceph-flash1 /]# rbd info rbd_ec/projects rbd image 'projects': size 750 TiB in 196608000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 15a979db61dda7 data_pool: rbd_ec_data block_name_prefix: rbd_data.10.15a979db61dda7 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Thu Jul 7 10:57:13 2022 access_timestamp: Thu Jul 7 10:57:13 2022 modify_timestamp: Thu Jul 7 10:57:13 2022 We wanted to resize it to 1PB but that failed: [ceph: root@ceph-flash1 /]# rbd resize rbd_ec/projects --size 1024T Resizing image: 0% complete...failed. rbd: shrinking an image is only allowed with the --allow-shrink flag 2024-08-06T08:42:01.053+0000 7fc996492580 -1 librbd::Operations: New size not compatible with object map We can do 800T though: [ceph: root@ceph-flash1 /]# rbd resize rbd_ec/projects --size 800T Resizing image: 100% complete...done. A problem with the --1024T notation? Or we hitting some sort of size limit for RBD? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 2

OSD data corruption after node reboot in Rook
by Reza Bakhshayeshi 05 Aug '24

05 Aug '24

Hello, Whenever a node reboots in the cluster I get some corrupted OSDs, is there any config I should set to prevent this from happening that I am not aware of? Here is the error log: # kubectl logs rook-ceph-osd-1-5dcbd99cc7-2l5g2 -c expand-bluefs ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f969977ce15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f969977cfdb] 3: (BlueStore::expand_devices(std::ostream&)+0x5ff) [0x55ce89d1f3ff] 4: main() 5: __libc_start_main() 6: _start() 0> 2024-07-31T08:39:19.840+0000 7f969b1c0980 -1 *** Caught signal (Aborted) ** in thread 7f969b1c0980 thread_name:ceph-bluestore- ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f969843fd20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f969977ce6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f969977cfdb] 6: (BlueStore::expand_devices(std::ostream&)+0x5ff) [0x55ce89d1f3ff] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2 1

Cephadm Offline Bootstrapping Issue
by Alex Hussein-Kershaw (HE/HIM) 05 Aug '24

05 Aug '24

Hi, I'm hitting an issue doing an offline install of Ceph 18.2.2 using cephadm. Long output below... any advice is appreciated. Looks like we don't managed to add admin labels (but also trying with --skip-admin results in a similar health warning). Subsequently trying to add an OSD fails quietly, I assume because cephadm is unhappy. Thanks, Alex $ sudo cephadm --image "ceph/ceph:v18.2.2" --docker bootstrap --mon-ip `hostname -I` --skip-pull --ssh-user qs-admin --ssh-private-key /home/qs-admin/.ssh/id_rsa --ssh-public-key /home/qs-admin/.ssh/id_rsa.pub --skip-dashboard Verifying ssh connectivity using standard pubkey authentication ... Adding key to qs-admin@localhost authorized_keys... key already in qs-admin@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 65bee110-3ae6-11ef-a1de-005056013d88 Verifying IP 10.235.22.8 port 3300 ... Verifying IP 10.235.22.8 port 6789 ... Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20` Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Ceph version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting public_network to 10.235.16.0/20 in mon config section Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 0.0.0.0:9283 ... Verifying port 0.0.0.0:8765 ... Verifying port 0.0.0.0:8443 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Using provided ssh keys... Adding key to qs-admin@localhost authorized_keys... key already in qs-admin@localhost authorized_keys... Adding host starlight-1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling client.admin keyring and conf on hosts with "admin" label Non-zero exit code 5 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=ceph/ceph:v18.2.2 -e NODE_NAME=starlight-1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/65bee110-3ae6-11ef-a1de-005056013d88:/var/log/ceph:z -v /tmp/ceph-tmpxbngx708:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp94g7iyn2:/etc/ceph/ceph.conf:z ceph/ceph:v18.2.2 orch client-keyring set client.admin label:_admin /usr/bin/ceph: stderr Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: ContainerInspectInfo(image_id='3c937764e6f5de1131b469dc69f0db09f8bd55cf6c983482cde518596d3dd0e5', ceph_version='ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)', repo_digests=['']) Unable to set up "admin" label; assuming older version of Ceph Saving cluster configuration to /var/lib/ceph/65bee110-3ae6-11ef-a1de-005056013d88/config directory Enabling autotune for osd_memory_target You can access the Ceph CLI as following in case of multi-cluster or non-default config: sudo /usr/sbin/cephadm shell --fsid 65bee110-3ae6-11ef-a1de-005056013d88 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Or, if you are only running a single cluster on this host: sudo /usr/sbin/cephadm shell Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/en/latest/mgr/telemetry/ Bootstrap complete. ]$ sudo docker exec ceph-1b19e642-3ae5-11ef-b4e4-005056013d88-mon-starlight-1 ceph -s cluster: id: 1b19e642-3ae5-11ef-b4e4-005056013d88 health: HEALTH_ERR Module 'cephadm' has failed: ContainerInspectInfo(image_id='3c937764e6f5de1131b469dc69f0db09f8bd55cf6c983482cde518596d3dd0e5', ceph_version='ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)', repo_digests=['']) OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum starlight-1 (age 2m) mgr: starlight-1.yhqrry(active, since 107s) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:

4 5

RGW sync gets stuck every day
by Olaf Seibert 05 Aug '24

05 Aug '24

Hi all, we have some Ceph clusters with RGW replication between them. It seems that in the last month at least, it gets stuck at around the same time ~every day. Not 100% the same time, and also not 100% of the days, but in the more recent days seem to happen more, and for longer. With "stuck" I mean that the "oldest incremental change not applied" is getting 5 or more minutes old. In the past this seemed to resolve itself in a short time, but recently it didn't. It remained stuck at the same place for several hours. The output on one end looks like this (I redacted out some of the data because I don't know how much of the naming would be sensitive information): root@zone2:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) current time 2024-08-04T10:22:00Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [30,90,95] oldest incremental change not applied: 2024-08-04T10:05:54.015403+0000 [30] while on the other side it looks ok (not more than half a minute behind): root@zone1:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) current time 2024-08-04T10:23:05Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [89,92,95,98] oldest incremental change not applied: 2024-08-04T10:22:53.175975+0000 [95] With some experimenting, we found that redeploying the RGWs on this side resolves the situation: "ceph orch redeploy rgw.zone1-backup". The shards go into "Recovering" state and after a short time it is "caught up with source" as well. Redeploying stuff seems like a much too big hammer to get things going again. Surely there must be something more reasonable? Also, any ideas about how we can find out what is causing this? It may be that some customer has some job running every 24 hours, but that shouldn't cause the replication to get stuck. Thanks in advance, -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

1 0

Ceph MDS failing because of corrupted dentries in lost+found after update from 17.2.7 to 18.2.0
by Justin Lee 03 Aug '24

03 Aug '24

After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log. debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep(a)0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy. We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc. Has anyone seen this type of issue or know how to solve it? Thanks!

2 4

Re: Please guide us in identifying the cause of the data miss in EC pool
by Frédéric Nass 03 Aug '24

03 Aug '24

Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. ________________________________ De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:     On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list && --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker: https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph orchestrator upgrade quincy to reef, missing ceph-exporter
by Frank de Bot (lists) 02 Aug '24

02 Aug '24

Hi, When upgrading a cephadm deployed quincy cluster to reef, there will be no ceph-exporter service launched. Being new in reef (from release notes: ceph-exporter: Now the performance metrics for Ceph daemons are exported by ceph-exporter, which deploys on each daemon rather than using prometheus exporter. This will reduce performance bottlenecks. ), metrics will be missing when the ceph-exporter service is not applied after the upgrade. Is it intentional it's not added after an upgrade by orchestrator? A ceph orch apply ceph-exporter solves it, but I think it could be included in the upgraded since it may not be clear at first why metrics in grafana are missing. Regards, Frank de Bot

2 1

RGW bucket notifications stop working after a while and blocking requests
by Florian Schwab 02 Aug '24

02 Aug '24

Hi, we just set up 2 new ceph clusters (using rook). To do some processing of the user activity we configured a topic that sends events to Kafka. After 5-12 hours this stops working with a 503 SlowDown response: debug 2024-08-02T09:17:58.205+0000 7ff4359ad700 1 req 13681579273117692719 0.005000019s ERROR: failed to reserve notification on queue: private.rgw. error: -28 First thought would be that the queue is full but up to this point see messages coming into Kafka and without much activity on the RGW itself (only a few requests against the S3 API) so it can’t be a load issue. What helps is to remove the notification configuration on the buckets (put-bucket-notification-configuration). If we directly re-add the previous notification configuration it also continuous working for a few hours before failing again with the same error/behaviour. We haven’t been able to reproduce this if we disable persistence for the topic so it looks like it is related to the persistence option - otherwise there would be also no queuing of the event for sending to Kafka. This also suggests that the issue is not with Kafka - this is also what we suspected first e.g. it can’t handle the amount of messages etc. Does anyone else have or had this issue and found the cause or a suggestion on how to best continue debugging? Are there detailed metrics etc. on the size and usage of the event queue? Here is the configuration for the topic and for a bucket: $ radosgw-admin topic list { "topics": [ { "user": "", "name": "private.rgw", "dest": { "push_endpoint": "kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512", "push_endpoint_args": "OpaqueData=&Version=2010-03-31&kafka-ack-level=broker&persistent=false&push-endpoint=kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512&use-ssl=true&verify-ssl=true", "push_endpoint_topic": "private.rgw", "stored_secret": true, "persistent": true }, "arn": "arn:aws:sns:ceph-objectstore::private.rgw", "opaqueData": "" } ] } $ aws s3api get-bucket-notification-configuration --bucket=XXX { "TopicConfigurations": [ { "Id": “my-id", "TopicArn": "arn:aws:sns:ceph-objectstore::private.rgw", "Events": [ "s3:ObjectCreated:*", "s3:ObjectRemoved:*" ] } ] } Thank you for any input to solve this! Cheers, Florian

1 0

Error When Replacing OSD - Please Help
by duluxoz 02 Aug '24

02 Aug '24

Hi All, I'm trying to replace an OSD in our cluster. This is on Reef 18.2.2 on Rocky 9.4. I performed the following steps (from this page of the Ceph Doco: https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/) 1. Make sure that it is safe to destroy the OSD: `while!cephosdsafe-to-destroyosd.{id};dosleep10;done` 2. Destroy the OSD: `cephosddestroy0--yes-i-really-mean-it` 3. Replaced the HDD 4. Prepare the disk for replacement by using the ID of the OSD that was destroyed in previous steps: `ceph-volumelvmprepare--osd-id0--data/dev/sd3` However, at this point I get the following errors: ~~~ Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json stderr: 2024-08-02T13:52:58.812+1000 7ff26e904640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory stderr: 2024-08-02T13:52:58.812+1000 7ff26e904640 -1 AuthRegistry(0x7ff268063e88) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx stderr: 2024-08-02T13:52:58.817+1000 7ff26e904640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-08-02T13:52:58.817+1000 7ff26e904640 -1 AuthRegistry(0x7ff268063e88) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-08-02T13:52:58.819+1000 7ff26e904640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-08-02T13:52:58.819+1000 7ff26e904640 -1 AuthRegistry(0x7ff268065b00) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-08-02T13:52:58.821+1000 7ff26e904640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-08-02T13:52:58.821+1000 7ff26e904640 -1 AuthRegistry(0x7ff26e9030c0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: [errno 2] RADOS object not found (error connecting to the cluster) --> RuntimeError: Unable check if OSD id exists: 0 ~~~ I can see a `client.bootstap-osd` user in the Ceph Dashboard Ceph User List, so I'm not sure what's going on. Any help is greatly appreciated - thanks.

2 1

ceph pg stuck active+remapped+backfilling
by Jorge Garcia 02 Aug '24

02 Aug '24

We were having an OSD reporting lots of errors, so I tried to remove it by doing: ceph orch osd rm 139 --zap It started moving all the data. Eventually, we got to the point that there's only 1 pg backfilling, but that seems to be stuck now. I think it may be because, in the process, another OSD (103) started reporting errors, too. The pool is erasure k:5 m:2, so it should still be OK. I don't see any progress happening on the backfill, and ceph -s has been reporting "69797/925734949 objects misplaced (0.008%)" for days now. How can I get it to finish the backfilling, or at least find out why it's not working? ceph version: quincy Thanks! Jorge

2 1