August 2024 - ceph-users

Recovering from total mon loss and backing up lockbox secrets
by Boris 06 Aug '24

06 Aug '24

Hi, I am in the process of creating disaster recovery documentation and I have two topics where I am not sure how to do it or even if it is possible. Is it possible to recover from a 100% mon data loss? Like all mons fail and the actual mon data is not recoverable. In my head I would thing that I can just create new mons with the same cluster ID and then start everything. The OSDs still have their PGs and data and after some period of time everything will be ok again. But then I thought that we use dmcrypt in ceph and I would need to somehow backup all the keys to some offsite location. So here are my questions: - How do I backup the lockbox secrets? - Do I need to backup the whole mon data, and if so how can I do it? Cheers Boris

2 1

Osds going down/flapping after Luminous to Nautilus upgrade part 1
by Mark Kirkwood 06 Aug '24

06 Aug '24

We have upgraded one of our Ceph clusters to Nautilus. We have run into 2 issues that are causing osds to flap. I'll cover the 1st one here, this one we solved but it raises an interesting question that might bear on the 2nd one (will post that next). After upgrading deb packages to Nautilus and restarting the mons and mgrs we worked through restarting the osds. We started to see some of them flap and saw this in the osd log (many times): 2024-07-31 11:03:33.264 7f22ab6e0700 0 --1- [v2:[2404:130:8020:5::73]:6820/220732,v1:[2404:130:8020:5::73]:6821/220732] >> v1:[2404:130:8020:5::103]:6909/2987374 conn(0x555a5bbbb180 0x555a4172c800 :-1 s=OPENED pgs=144 cs=3 l=0).fault initiating reconnect And later (usually a single line): 2404:130:8020:5::135]:6903/2933993 conn(0x555ab7b03200 0x555a099fa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=154 cs=4 l=0).handle_connect_reply_2 connect got BADAUTHORIZER Examining the code showed: markir@zmori:/download/ceph/src/ceph$ find . -type f -exec grep -l "initiating reconnect" {} \; ./src/msg/simple/Pipe.cc ./src/msg/async/ProtocolV1.cc ./src/msg/async/ProtocolV2.cc markir@zmori:/download/ceph/src/ceph$ vi src/msg/async/ProtocolV2.cc markir@zmori:/download/ceph/src/ceph$ find . -type f -exec grep -l "got BADAUTHORIZER" {} \; ./src/msg/simple/Pipe.cc ./src/msg/async/ProtocolV1.cc Which led us to suspect that the osds were using the v1 msgr protocol (ceph osd dump seems to validate this). We hoped that once we enabled the v2 msgr that this error would vanish. This appears to have happened. So my question is this: looks like there is something wrong with communications via v1 protocol post upgrade - is that expected? Regards Mark

2 1

Resize RBD - New size not compatible with object map
by Torkil Svensgaard 06 Aug '24

06 Aug '24

Hi [ceph: root@ceph-flash1 /]# rbd info rbd_ec/projects rbd image 'projects': size 750 TiB in 196608000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 15a979db61dda7 data_pool: rbd_ec_data block_name_prefix: rbd_data.10.15a979db61dda7 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Thu Jul 7 10:57:13 2022 access_timestamp: Thu Jul 7 10:57:13 2022 modify_timestamp: Thu Jul 7 10:57:13 2022 We wanted to resize it to 1PB but that failed: [ceph: root@ceph-flash1 /]# rbd resize rbd_ec/projects --size 1024T Resizing image: 0% complete...failed. rbd: shrinking an image is only allowed with the --allow-shrink flag 2024-08-06T08:42:01.053+0000 7fc996492580 -1 librbd::Operations: New size not compatible with object map We can do 800T though: [ceph: root@ceph-flash1 /]# rbd resize rbd_ec/projects --size 800T Resizing image: 100% complete...done. A problem with the --1024T notation? Or we hitting some sort of size limit for RBD? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 2

OSD data corruption after node reboot in Rook
by Reza Bakhshayeshi 05 Aug '24

05 Aug '24

Hello, Whenever a node reboots in the cluster I get some corrupted OSDs, is there any config I should set to prevent this from happening that I am not aware of? Here is the error log: # kubectl logs rook-ceph-osd-1-5dcbd99cc7-2l5g2 -c expand-bluefs ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f969977ce15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f969977cfdb] 3: (BlueStore::expand_devices(std::ostream&)+0x5ff) [0x55ce89d1f3ff] 4: main() 5: __libc_start_main() 6: _start() 0> 2024-07-31T08:39:19.840+0000 7f969b1c0980 -1 *** Caught signal (Aborted) ** in thread 7f969b1c0980 thread_name:ceph-bluestore- ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f969843fd20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f969977ce6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f969977cfdb] 6: (BlueStore::expand_devices(std::ostream&)+0x5ff) [0x55ce89d1f3ff] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2 1

Cephadm Offline Bootstrapping Issue
by Alex Hussein-Kershaw (HE/HIM) 05 Aug '24

05 Aug '24

Hi, I'm hitting an issue doing an offline install of Ceph 18.2.2 using cephadm. Long output below... any advice is appreciated. Looks like we don't managed to add admin labels (but also trying with --skip-admin results in a similar health warning). Subsequently trying to add an OSD fails quietly, I assume because cephadm is unhappy. Thanks, Alex $ sudo cephadm --image "ceph/ceph:v18.2.2" --docker bootstrap --mon-ip `hostname -I` --skip-pull --ssh-user qs-admin --ssh-private-key /home/qs-admin/.ssh/id_rsa --ssh-public-key /home/qs-admin/.ssh/id_rsa.pub --skip-dashboard Verifying ssh connectivity using standard pubkey authentication ... Adding key to qs-admin@localhost authorized_keys... key already in qs-admin@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 65bee110-3ae6-11ef-a1de-005056013d88 Verifying IP 10.235.22.8 port 3300 ... Verifying IP 10.235.22.8 port 6789 ... Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20` Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Ceph version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting public_network to 10.235.16.0/20 in mon config section Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 0.0.0.0:9283 ... Verifying port 0.0.0.0:8765 ... Verifying port 0.0.0.0:8443 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Using provided ssh keys... Adding key to qs-admin@localhost authorized_keys... key already in qs-admin@localhost authorized_keys... Adding host starlight-1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling client.admin keyring and conf on hosts with "admin" label Non-zero exit code 5 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=ceph/ceph:v18.2.2 -e NODE_NAME=starlight-1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/65bee110-3ae6-11ef-a1de-005056013d88:/var/log/ceph:z -v /tmp/ceph-tmpxbngx708:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp94g7iyn2:/etc/ceph/ceph.conf:z ceph/ceph:v18.2.2 orch client-keyring set client.admin label:_admin /usr/bin/ceph: stderr Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: ContainerInspectInfo(image_id='3c937764e6f5de1131b469dc69f0db09f8bd55cf6c983482cde518596d3dd0e5', ceph_version='ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)', repo_digests=['']) Unable to set up "admin" label; assuming older version of Ceph Saving cluster configuration to /var/lib/ceph/65bee110-3ae6-11ef-a1de-005056013d88/config directory Enabling autotune for osd_memory_target You can access the Ceph CLI as following in case of multi-cluster or non-default config: sudo /usr/sbin/cephadm shell --fsid 65bee110-3ae6-11ef-a1de-005056013d88 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Or, if you are only running a single cluster on this host: sudo /usr/sbin/cephadm shell Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/en/latest/mgr/telemetry/ Bootstrap complete. ]$ sudo docker exec ceph-1b19e642-3ae5-11ef-b4e4-005056013d88-mon-starlight-1 ceph -s cluster: id: 1b19e642-3ae5-11ef-b4e4-005056013d88 health: HEALTH_ERR Module 'cephadm' has failed: ContainerInspectInfo(image_id='3c937764e6f5de1131b469dc69f0db09f8bd55cf6c983482cde518596d3dd0e5', ceph_version='ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)', repo_digests=['']) OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum starlight-1 (age 2m) mgr: starlight-1.yhqrry(active, since 107s) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:

4 5

RGW sync gets stuck every day
by Olaf Seibert 05 Aug '24

05 Aug '24

Hi all, we have some Ceph clusters with RGW replication between them. It seems that in the last month at least, it gets stuck at around the same time ~every day. Not 100% the same time, and also not 100% of the days, but in the more recent days seem to happen more, and for longer. With "stuck" I mean that the "oldest incremental change not applied" is getting 5 or more minutes old. In the past this seemed to resolve itself in a short time, but recently it didn't. It remained stuck at the same place for several hours. The output on one end looks like this (I redacted out some of the data because I don't know how much of the naming would be sensitive information): root@zone2:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) current time 2024-08-04T10:22:00Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [30,90,95] oldest incremental change not applied: 2024-08-04T10:05:54.015403+0000 [30] while on the other side it looks ok (not more than half a minute behind): root@zone1:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) current time 2024-08-04T10:23:05Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [89,92,95,98] oldest incremental change not applied: 2024-08-04T10:22:53.175975+0000 [95] With some experimenting, we found that redeploying the RGWs on this side resolves the situation: "ceph orch redeploy rgw.zone1-backup". The shards go into "Recovering" state and after a short time it is "caught up with source" as well. Redeploying stuff seems like a much too big hammer to get things going again. Surely there must be something more reasonable? Also, any ideas about how we can find out what is causing this? It may be that some customer has some job running every 24 hours, but that shouldn't cause the replication to get stuck. Thanks in advance, -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

1 0

Ceph MDS failing because of corrupted dentries in lost+found after update from 17.2.7 to 18.2.0
by Justin Lee 03 Aug '24

03 Aug '24

After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log. debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep(a)0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy. We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc. Has anyone seen this type of issue or know how to solve it? Thanks!

2 4

Re: Please guide us in identifying the cause of the data miss in EC pool
by Frédéric Nass 03 Aug '24

03 Aug '24

Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. ________________________________ De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:     On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list && --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker: https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph orchestrator upgrade quincy to reef, missing ceph-exporter
by Frank de Bot (lists) 02 Aug '24

02 Aug '24

Hi, When upgrading a cephadm deployed quincy cluster to reef, there will be no ceph-exporter service launched. Being new in reef (from release notes: ceph-exporter: Now the performance metrics for Ceph daemons are exported by ceph-exporter, which deploys on each daemon rather than using prometheus exporter. This will reduce performance bottlenecks. ), metrics will be missing when the ceph-exporter service is not applied after the upgrade. Is it intentional it's not added after an upgrade by orchestrator? A ceph orch apply ceph-exporter solves it, but I think it could be included in the upgraded since it may not be clear at first why metrics in grafana are missing. Regards, Frank de Bot

2 1

RGW bucket notifications stop working after a while and blocking requests
by Florian Schwab 02 Aug '24

02 Aug '24

Hi, we just set up 2 new ceph clusters (using rook). To do some processing of the user activity we configured a topic that sends events to Kafka. After 5-12 hours this stops working with a 503 SlowDown response: debug 2024-08-02T09:17:58.205+0000 7ff4359ad700 1 req 13681579273117692719 0.005000019s ERROR: failed to reserve notification on queue: private.rgw. error: -28 First thought would be that the queue is full but up to this point see messages coming into Kafka and without much activity on the RGW itself (only a few requests against the S3 API) so it can’t be a load issue. What helps is to remove the notification configuration on the buckets (put-bucket-notification-configuration). If we directly re-add the previous notification configuration it also continuous working for a few hours before failing again with the same error/behaviour. We haven’t been able to reproduce this if we disable persistence for the topic so it looks like it is related to the persistence option - otherwise there would be also no queuing of the event for sending to Kafka. This also suggests that the issue is not with Kafka - this is also what we suspected first e.g. it can’t handle the amount of messages etc. Does anyone else have or had this issue and found the cause or a suggestion on how to best continue debugging? Are there detailed metrics etc. on the size and usage of the event queue? Here is the configuration for the topic and for a bucket: $ radosgw-admin topic list { "topics": [ { "user": "", "name": "private.rgw", "dest": { "push_endpoint": "kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512", "push_endpoint_args": "OpaqueData=&Version=2010-03-31&kafka-ack-level=broker&persistent=false&push-endpoint=kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512&use-ssl=true&verify-ssl=true", "push_endpoint_topic": "private.rgw", "stored_secret": true, "persistent": true }, "arn": "arn:aws:sns:ceph-objectstore::private.rgw", "opaqueData": "" } ] } $ aws s3api get-bucket-notification-configuration --bucket=XXX { "TopicConfigurations": [ { "Id": “my-id", "TopicArn": "arn:aws:sns:ceph-objectstore::private.rgw", "Events": [ "s3:ObjectCreated:*", "s3:ObjectRemoved:*" ] } ] } Thank you for any input to solve this! Cheers, Florian

1 0

Error When Replacing OSD - Please Help
by duluxoz 02 Aug '24

02 Aug '24

Hi All, I'm trying to replace an OSD in our cluster. This is on Reef 18.2.2 on Rocky 9.4. I performed the following steps (from this page of the Ceph Doco: https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/) 1. Make sure that it is safe to destroy the OSD: `while!cephosdsafe-to-destroyosd.{id};dosleep10;done` 2. Destroy the OSD: `cephosddestroy0--yes-i-really-mean-it` 3. Replaced the HDD 4. Prepare the disk for replacement by using the ID of the OSD that was destroyed in previous steps: `ceph-volumelvmprepare--osd-id0--data/dev/sd3` However, at this point I get the following errors: ~~~ Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json stderr: 2024-08-02T13:52:58.812+1000 7ff26e904640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory stderr: 2024-08-02T13:52:58.812+1000 7ff26e904640 -1 AuthRegistry(0x7ff268063e88) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx stderr: 2024-08-02T13:52:58.817+1000 7ff26e904640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-08-02T13:52:58.817+1000 7ff26e904640 -1 AuthRegistry(0x7ff268063e88) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-08-02T13:52:58.819+1000 7ff26e904640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-08-02T13:52:58.819+1000 7ff26e904640 -1 AuthRegistry(0x7ff268065b00) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-08-02T13:52:58.821+1000 7ff26e904640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-08-02T13:52:58.821+1000 7ff26e904640 -1 AuthRegistry(0x7ff26e9030c0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: [errno 2] RADOS object not found (error connecting to the cluster) --> RuntimeError: Unable check if OSD id exists: 0 ~~~ I can see a `client.bootstap-osd` user in the Ceph Dashboard Ceph User List, so I'm not sure what's going on. Any help is greatly appreciated - thanks.

2 1

ceph pg stuck active+remapped+backfilling
by Jorge Garcia 02 Aug '24

02 Aug '24

We were having an OSD reporting lots of errors, so I tried to remove it by doing: ceph orch osd rm 139 --zap It started moving all the data. Eventually, we got to the point that there's only 1 pg backfilling, but that seems to be stuck now. I think it may be because, in the process, another OSD (103) started reporting errors, too. The pool is erasure k:5 m:2, so it should still be OK. I don't see any progress happening on the backfill, and ceph -s has been reporting "69797/925734949 objects misplaced (0.008%)" for days now. How can I get it to finish the backfilling, or at least find out why it's not working? ceph version: quincy Thanks! Jorge

2 1

Please Reply my Mail Sent to you ON (24th July 2024)
by Mrs. Hanana Shrawi 02 Aug '24

02 Aug '24

Dear Greetings I have the pleasure to communicate with you via this medium. My intent is to discuss investment business matters with you so as to be able to learn of the available investment opportunities in your region. I have attech my Personal Documents for you to see, Do you remember me, Looking forward to hearing from you. Best regards, Mrs. Hanan Ashrawi

1 0

Difficulty importing bluestore OSDs from the old cluster (bad fsid) - OSD does not start
by Vinícius Barreto 02 Aug '24

02 Aug '24

Hello everybody! I am fascinated by Ceph, but now I live in moments of terror and despair. I'm using ceph 18.2.2 (reef) and at the moment We need to import 4 OSDs from an old cluster (which was removed by accident). In short, we suspect that the cause and solution for this case lies in the information in the OSD log below: (all OSD daemons repeat the same information) After importing I receive the following information in the "journalctl -u ceph-osd@2" log: --- Jul 17 20:09:55 pxm3 ceph-osd[647313]: in thread 702342c006c0 thread_name:ms_dispatch Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) Jul 17 20:09:55 pxm3 ceph-osd[647313]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x70235785b050] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x7023578a9e2c] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 3: gsignal() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 4: abort() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18a) [0x62edaeab47> Jul 17 20:09:55 pxm3 ceph-osd[647313]: 6: (OSD::handle_osd_map(MOSDMap*)+0x384a) [0x62edaec0aeca] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 7: (OSD::ms_dispatch(Message*)+0x62) [0x62edaec0b332] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc1) [0x62edaf5eea51] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 9: (DispatchQueue::entry()+0x6cf) [0x62edaf5ed53f] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x62edaf40fd5d] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7023578a8134] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x7023579287dc] Jul 17 20:09:55 pxm3 ceph-osd[647313]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 17 20:09:55 pxm3 ceph-osd[647313]: -142> 2024-07-17T20:09:54.845-0300 702356d616c0 -1 osd.2 71 log_to_monitors true Jul 17 20:09:55 pxm3 ceph-osd[647313]: -2> 2024-07-17T20:09:55.362-0300 702342c006c0 -1 osd.2 71 ERROR: bad fsid? i have 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 and inc has f4466e33-b57d-4d68-9909-346> Jul 17 20:09:55 pxm3 ceph-osd[647313]: -1> 2024-07-17T20:09:55.366-0300 702342c006c0 -1 ./src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 702342c006c0 time 2024-07-17T20:09:5> Jul 17 20:09:55 pxm3 ceph-osd[647313]: ./src/osd/OSD.cc: 8098: ceph_abort_msg("bad fsid") Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) --- Above in the OSDs daemon log, we see the "bad fsid" error which indicates that the OSDs are trying to connect to a different Ceph cluster than the one they were initially configured for. Each Ceph cluster has a unique identifier, the FSID, and if the OSDs detect a different FSID, they cannot connect correctly. Note: I believe this is the root cause of the problem and the solution. Problem: OSD has fsid from the old cluster. This would be the solution, change the ceph_fsid (fsid) of the OSDs to the fsid of the new cluster, or recreate a new cluster using the fsid that is recorded in the OSD metadata, that is, recreate the cluster with the fsid of the old cluster? would it be possible?) I also opened a topic for this case on the proxmox forum and inserted a lot of data about this scenario there: https://forum.proxmox.com/threads/ceph-cluster-rebuild-import-bluestore-osd… ----- ~# cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 192.168.0.2/24 fsid = f4466e33-b57d-4d68-9909-3468afd9e5c2 mon_allow_pool_delete = true mon_host = 192.168.0.2 192.168.0.3 192.168.0.1 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.0.0/24 [client] # keyring = /etc/pve/priv/$cluster.$name.keyring keyring = /etc/pve/priv/ceph.client.admin.keyring #[client.crash] # keyring = /etc/pve/ceph/$cluster.$name.keyring [client.crash] key = AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== [client.admin] key = AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mon.pxm1] public_addr = 192.168.0.1 [mon.pxm2] public_addr = 192.168.0.2 [mon.pxm3] public_addr = 192.168.0.3 --- root@pxm3:~# ceph fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 --- ~# ceph -s cluster: id: f4466e33-b57d-4d68-9909-3468afd9e5c2 health: HEALTH_WARN mon pxm1 is low on available space 4 osds down 2 hosts (4 osds) down 1 root (4 osds) down 170 daemons have recently crashed services: mon: 3 daemons, quorum pxm2,pxm3,pxm1 (age 3h) mgr: pxm2(active, since 3h), standbys: pxm1, pxm3 osd: 4 osds: 0 up, 4 in (since 2h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: --- ~# cephadm ls [ { "style": "legacy", "name": "osd.0", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@0", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.1", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@1", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" } ] root@pxm3:~# cephadm ls [ { "style": "legacy", "name": "osd.3", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@3", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.2", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@2", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" } ] --- To import the OSDs, we first increased the epoch of the new cluster, repeating the commands below until the cluster's epoch count number was greater than the OSDs' epoch number: Bash: ceph osd set noin ceph osd set noout ceph osd set noup ceph osd set nodown ceph osd set norebalance ceph osd set nobackfill ceph osd unset noin ceph osd unset noout ceph osd unset noup ceph osd unset nodown ceph osd unset norebalance ceph osd unset nobackfill --- OSD volumes: Code: pxm2: -> OSD.0: /dev/ceph-1740d41a-2ae7-4c4d-820f-ec3702e3ba90/osd-block-39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa -> OSD.1: /dev/ceph-ad425d70-4aa3-419a-997f-f3a4082c9904/osd-block-bb4df480-2b9b-4604-a44d-6151d5c0cb33 pxm3: -> OSD.2: /dev/ceph-94682b88-d09c-4eab-9170-c6d31eac79e6/osd-block-3f6756d6-e64b-4c60-9ac2-305c0e71cc51 -> OSD.3: /dev/ceph-d5ffd027-8289-4a1c-9378-6687d9f950ad/osd-block-eece9fc9-44d6-460b-aced-572c79a98be8 --- ceph-bluestore-tool show-label: -> osb.0: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-0/block": { "osd_uuid": "39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa", "size": 2000397795328, "btime": "2024-07-07T21:32:30.861509-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:32.459102Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg==", "ready": "ready", "require_osd_release": "18", "whoami": "0" } } -> osd.1: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "bb4df480-2b9b-4604-a44d-6151d5c0cb33", "size": 2000397795328, "btime": "2024-07-07T21:32:43.729638-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:45.577456Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA==", "ready": "ready", "require_osd_release": "18", "whoami": "1" } } -> osd.2 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-2 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-2/block": { "osd_uuid": "3f6756d6-e64b-4c60-9ac2-305c0e71cc51", "size": 2000397795328, "btime": "2024-07-07T21:33:07.812888-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:09.404317Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg==", "ready": "ready", "require_osd_release": "18", "whoami": "2" } } -> osd.3 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-3 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-3/block": { "osd_uuid": "eece9fc9-44d6-460b-aced-572c79a98be8", "size": 2000397795328, "btime": "2024-07-07T21:33:25.725294-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:27.323085Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw==", "ready": "ready", "require_osd_release": "18", "whoami": "3" } } --- root@pxm2:~# cat /var/lib/ceph/osd/ceph-0/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 --- ~# ceph daemon osd.0 status no valid command found; 10 closest matches: 0 1 2 abort assert bluefs debug_inject_read_zeros bluefs files list bluefs stats bluestore allocator dump block bluestore allocator fragmentation block admin_socket: invalid command --- ~# ceph osd info osd.0 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa --- ~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 0 0 0 0 0 0 exists 1 0 0 0 0 0 0 exists 2 0 0 0 0 0 0 exists 3 0 0 0 0 0 0 exists root@pxm3:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.27759 root default -2 3.63879 host pxm2 0 1.81940 osd.0 down 1.00000 1.00000 1 1.81940 osd.1 down 1.00000 1.00000 -5 3.63879 host pxm3 3 1.81940 osd.3 down 1.00000 1.00000 2 ssd 1.81940 osd.2 down 1.00000 1.00000 --- ~# ceph osd dump epoch 164 fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 created 2024-07-14T13:04:53.354056-0300 modified 2024-07-17T20:24:17.550123-0300 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 63 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release reef stretch_mode_enabled false max_osd 4 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa osd.1 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists bb4df480-2b9b-4604-a44d-6151d5c0cb33 osd.2 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 3f6756d6-e64b-4c60-9ac2-305c0e71cc51 osd.3 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists eece9fc9-44d6-460b-aced-572c79a98be8 blocklist 192.168.0.1:0/1377959162 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6800/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:0/3150227222 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1759809784 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.1:0/3747372149 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6801/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1472036822 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:0/571633622 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6801/10543 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6800/10543 expires 2024-07-18T00:56:24.333969-0300 --- ~# ceph osd metadata [ { "id": 0 }, { "id": 1 }, { "id": 2 }, { "id": 3 } ] --- ~# cat /var/lib/ceph/osd/ceph-0/keyring [osd.0] key = AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg== root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/keyring [osd.1] key = AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA== root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/keyring [osd.2] key = AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/keyring [osd.3] key = AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw== --- ~# ceph auth list osd.0 key: AQCKQJhmMYWeEhAAosrE8Ff+1kZbKcroi22TvQ== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQCTQJhm2uLvAxAAC3uIcRk9d0sxLJgIxcivtw== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== caps: [mon] allow profile osd caps: [osd] allow * osd.3 key: AQBYMZhmmlaFDRAAtoy4//XweaI94OvrPV1aiQ== caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQAl95NmWuofFRAAhV7/f1PykW/KlcB8Rede9w== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQAl95NmmPQfFRAAG2cQNiWQgdx0Rvr7ZFmuhg== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQAl95Nmr/0fFRAA4TahG5PCbZIsltFgkRNEgA== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQAl95Nm0gYgFRAAWBvqfphKk62InqY9x0ijHg== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rbd-mirror key: AQAl95Nmxg8gFRAAomFkp299Ca04NwGpfbSZRg== caps: [mon] allow profile bootstrap-rbd-mirror client.bootstrap-rgw key: AQAl95Nm3RwgFRAA/QoIemXPFv5Gs1/PWfJuYw== caps: [mon] allow profile bootstrap-rgw client.crash key: AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== caps: [mgr] profile crash caps: [mon] profile crash mgr.pxm1 key: AQDm+JNmtJ3CCRAAl/wZaT6z12LCfAghvm6s4w== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm2 key: AQAm95Nm6+dmABAA6405D8ROtNpJf6iVhMegQA== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm3 key: AQDp95Nm3ZbLDxAAWc1zLaVOhE0wMxLJCL5IGg== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow *

2 1

rbd-mirror keeps crashing
by armsby＠armsby.dk 02 Aug '24

02 Aug '24

Hi everyone, I've been running rbd-mirror between my old Ceph system (16.2.10) and my new system (18.2.2). I'm using journaling mode on a pool that contains 7,500 images. Everything was running perfectly until it processed about 5,608 images. Now, it keeps crashing with the following message: 2024-07-19T05:49:32.425+0000 7f582b3fd6c0 0 set uid:gid to 167:167 (ceph:ceph) 2024-07-19T05:49:32.425+0000 7f582b3fd6c0 0 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), process rbd-mirror, pid 7 2024-07-19T05:49:32.429+0000 7f582b3fd6c0 1 mgrc service_daemon_register rbd-mirror.3606956688 metadata {arch=x86_64, ceph_release=pacific, ceph_version=ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), ceph_version_short=16.2.10, container_hostname=mon-001, container_image=quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd, cpu=Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz, distro=centos, distro_description=CentOS Stream 8, distro_version=8, hostname=mon-001, id=mon-001.lcqrti, instance_id=3606956688, kernel_description=#1 SMP Mon Jul 18 17:42:52 UTC 2022, kernel_version=4.18.0-408.el8.x86_64, mem_swap_kb=4194300, mem_total_kb=131393360, os=Linux} 2024-07-19T05:50:28.305+0000 7f5812582700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f5812582700 time 2024-07-19T05:50:28.303536+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/Thread.cc: 165: FAILED ceph_assert(ret == 0) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58218b6de8] 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f58218b7002] 3: /usr/lib64/ceph/libceph-common.so.2(+0x362fd7) [0x7f58219a2fd7] 4: (CommonSafeTimer<std::mutex>::init()+0x1fe) [0x7f58219a963e] 5: (journal::Journaler::Threads::Threads(ceph::common::CephContext*)+0x2fc) [0x55c9b33c6ddc] 6: (journal::Journaler::Journaler(librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, journal::Settings const&, journal::CacheManagerHandler*)+0x50) [0x55c9b33c6f10] 7: (librbd::Journal<librbd::ImageCtx>::get_tag_owner(librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, librbd::asio::ContextWQ*, Context*)+0x19f) [0x55c9b2fa65af] 8: (librbd::mirror::GetInfoRequest<librbd::ImageCtx>::get_journal_tag_owner()+0x210) [0x55c9b31869f0] 9: (librbd::mirror::GetInfoRequest<librbd::ImageCtx>::handle_get_mirror_image(int)+0x8c8) [0x55c9b3189d78] 10: /lib64/librados.so.2(+0xa8546) [0x7f582aedb546] 11: /lib64/librados.so.2(+0xc17e5) [0x7f582aef47e5] 12: /lib64/librados.so.2(+0xc3742) [0x7f582aef6742] 13: /lib64/librados.so.2(+0xc914a) [0x7f582aefc14a] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f581fb03ba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5820cec1ca] 16: clone() Has anyone encountered a similar problem or have any insight into what might be causing this crash? Thanks in advance for your help.

2 1

Help with osd spec needed
by Kristaps Cudars 02 Aug '24

02 Aug '24

3 nodes each: 3 hdd – 21G 1 ssd – 80G Create osd containing block_data with block_db size 15G located on ssd- This par works Create block_data osd on remaining space 35G in ssd- This part is not working ceph orch apply osd -i /path/to/osd_spec.yml service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' spec: block_db_size: 15G db_slots: 3 data_devices: rotational: 1 db_devices: rotational: 0 --- service_type: osd service_id: odd_spec_ssd placement: host_pattern: '*' spec: data_devices: rotational: 0

2 1

Ceph Quarterly #5 - April 2024 to June 2024
by Zac Dover 02 Aug '24

02 Aug '24

The fifth issue of "Ceph Quarterly" is linked below in this email. Ceph Quarterly (or "CQ") is an overview of the past three months of upstream Ceph development. CQ#5 covers the period from April 2024 to June 2024. https://ceph.io/en/community/cq/Ceph_Quarterly_005_A4.pdf https://ceph.io/en/community/cq/Ceph_Quarterly_005_Letter.pdf Zac Dover Head of Documentation Ceph Foundation

1 0

Debian package for 18.2.4 broken
by Burkhard Linke 01 Aug '24

01 Aug '24

Hi, the Debian bookworm packages for 18.2.4 are completely broken and unusable: 1. missing dependencies, e.g. python3-packaging: root@cc-r3-ceph-2:~# ceph-volume lvm list Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 33, in <module> sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/sbin/ceph-volume", line 25, in importlib_load_entry_point return next(matches).load() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1206, in _gcd_import File "<frozen importlib._bootstrap>", line 1178, in _find_and_load File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 690, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 940, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 9, in <module> from ceph_volume import log, devices, configuration, conf, exceptions, terminal, inventory, drive_group, activate File "/usr/lib/python3/dist-packages/ceph_volume/devices/__init__.py", line 1, in <module> from . import lvm, simple, raw # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/__init__.py", line 1, in <module> from .main import LVM # noqa ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 4, in <module> from . import activate File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/activate.py", line 9, in <module> from ceph_volume.util import encryption as encryption_utils File "/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py", line 9, in <module> from packaging import version ModuleNotFoundError: No module named 'packaging' 2. After installing missing dependencies activating encrypted OSDs fails: root@cc-r3-ceph-2:~# ceph-volume lvm activate --all --> Activating OSD ID 875 FSID 06296c01-4c59-4fde-afe4-0e1e4f078ba9 --> InvalidVersion: Invalid version: 'cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI ' Downgrade to 18.2.2 resolves the problem. Did anyone test the packages before release? Regards, Burkhard

2 1

Problems with crash and k8sevents modules
by Massimo Sgaravatto 01 Aug '24

01 Aug '24

Dear all Today I realized that the syslog on the mon/mgr modules was complaining about missing credentials for the crash client So. following what is reported in: https://docs.ceph.com/en/quincy/mgr/crash/ I did: root@ceph-mon-01 ~]# ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash' [client.crash] key = AQAqNKtmny3yNRAALoVRCgPju2Epc+BPfqOSdw== and on the 3 mon/mgr nodes I created the file /etc/ceph/ceph.client.crash.keyring root@ceph-mon-01 ~]# ll /etc/ceph/ceph.client.crash.keyring -rw-r--r-- 1 ceph ceph 64 Aug 1 09:21 /etc/ceph/ceph.client.crash.keyring [root@ceph-mon-01 ~]# cat /etc/ceph/ceph.client.crash.keyring [client.crash] key = AQAqNKtmny3yNRAALoVRCgPju2Epc+BPfqOSdw== But it is still complaining in the syslog [*] Do I need to also create a client.crash.$hostname ? I have to admit that the doc is not fully clear to me .... At any rate now ceph shows: [root@ceph-mon-01 ~]# ceph health detail HEALTH_WARN 8 mgr modules have recently crashed [WRN] RECENT_MGR_MODULE_CRASH: 8 mgr modules have recently crashed mgr module k8sevents crashed in daemon mgr.ceph-mon-01 on host ceph-mon-01.cloud.pd.infn.it at 2024-07-23T08:53:52.543451Z mgr module k8sevents crashed in daemon mgr.ceph-mon-01 on host ceph-mon-01.cloud.pd.infn.it at 2024-07-23T09:09:17.703402Z mgr module k8sevents crashed in daemon mgr.ceph-mon-02 on host ceph-mon-02.cloud.pd.infn.it at 2024-07-22T13:44:18.779525Z mgr module k8sevents crashed in daemon mgr.ceph-mon-02 on host ceph-mon-02.cloud.pd.infn.it at 2024-07-22T13:57:24.207309Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-07-22T10:42:42.002880Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-07-22T11:09:21.639576Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-08-01T06:02:01.139099Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-08-01T07:45:16.249664Z but that module is disabled [**]. So I am a bit confused Thanks, Massimo [**] [root@ceph-mon-01 ~]# ceph mgr module ls MODULE balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) dashboard on restful on alerts - cephadm - influx - insights - iostat - k8sevents - localpool - mds_autoscaler - mirroring - nfs - osd_perf_query - osd_support - prometheus - rgw - rook - selftest - snap_schedule - stats - telegraf - test_orchestrator - zabbix - [root@ceph-mon-01 ~]# ceph mgr module disable k8sevents module 'k8sevents' is already disabled [root@ceph-mon-01 ~]# [*] Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-07-23T08:53:52.543451Z_da7be91d-5025-4bd0-b6a9-8fe2a1d3b71d as client.crash.ceph-mon-01.cloud.pd.infn.it failed: 2024-08-01T09:26:39.599+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.599+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44b4060ac0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44b408c3d0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44ba8ad0c0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: [errno 2] RADOS object not found (error connecting to the cluster) Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-07-23T09:09:17.703402Z_911fe89d-8f28-4f0e-8280-09b2e8a2e342 as client.crash.ceph-mon-01.cloud.pd.infn.it failed: 2024-08-01T09:26:40.277+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.277+0200 7fdfa4314640 -1 AuthRegistry(0x7fdf9c060ac0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.282+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.282+0200 7fdfa4314640 -1 AuthRegistry(0x7fdf9c08c3d0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.283+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.283+0200 7fdfa4314640 -1 AuthRegistry(0x7fdfa43130c0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: [errno 2] RADOS object not found (error connecting to the cluster)

1 0

Re: 18.2.4 regression: 'diskprediction_local' has failed: No module named 'sklearn'
by Devender Singh 01 Aug '24

01 Aug '24

What is the fix for above error. My OS is : ubuntu 20.04 and ceph version is 18.2.4 Currently I disabled the prediction. : ceph mgr module disable diskprediction_local Regards Dev

1 0

ceph-mgr memory problems 16.2.15
by timureh＠gmail.com 01 Aug '24

01 Aug '24

I have faced a similar problem that is described here https://tracker.ceph.com/issues/59580#change-270387 upon update to 16.2.15. I have a question for developers. Are you planning to fix this problem? Thank you in advance.

1 0

Osds going down/flapping after Luminous to Nautilus upgrade part 2
by Mark Kirkwood 01 Aug '24

01 Aug '24

This 2nd post is about the next type of flapping osds we encountered after upgrading. We started to see osds going down with this in 'ceph -w': 2024-08-01 12:02:57.437135 mon.cat-hlz-stor001 [INF] osd.479 marked down after no beacon for 902.637005 seconds 2024-08-01 12:02:57.468372 mon.cat-hlz-stor001 [WRN] Health check failed: 1 osds down (OSD_DOWN) We have the beacon interval set to 300. To fix this we tried: - restarting osds - restarting mons - ntp tidyup - restarting mgrs However it is still happening. Poking around in the osd and mon logs we did see some lines that hinted that the mon might be listening for beacons using v1 - which could be broken (see part 1). Hence restarting them again. This did not have any effect. Apart from enabling the v2 msgr we have not altered our Luminous config for Nautilus, are we missing something? Regards Mark

1 1