- ceph-users - lists.ceph.io

IT Consulting Firms with Ceph and Hashicorp Vault expertise?
by Michael Worsham 11 Jul '24

11 Jul '24

I am in need of a list of IT consulting firms that can set up high-availability Vault and also configure the Ceph Object Gateway to use SSE-S3 with Vault. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

2 2

High RAM usage for OSDs
by Work Ceph 11 Jul '24

11 Jul '24

Hello guys, We are running Ceph Octopus on Ubuntu 18.04. We noticed that some OSDs are using more than 16GiB of RAM. However, the option "osd_memory_target" is set to 4GiB. The OSDs are SSDs and have 2TiB in size each. Have you guys seen such behavior? Are we missing some other configuration or parameter? Or, maybe, is it some memory leak?

1 0

[RGW][Lifecycle][Versioned Buckets][Reef] Although LC deletes non-current
by Oguzhan Ozmen (BLOOMBERG/ 120 PARK) 10 Jul '24

10 Jul '24

This is similar to an old thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FXVEWDU6NY… but I don't see any responses there so opening this one. PROBLEM DESCRIPTION * Issue is seen on versioned buckets. * Using extended logging (debug level 5), we can see that LC deletes expired (non-current) versions of objects but those versions are still to be found in the bucket: both listed in the bucket index and accessible to the user. AN EXAMPLE OBJECT/VERSION * Bucket: bald * Object: adc/certs/injr-f5lb01b-vadc04.json * Versions: It currently has 6 versions $ aws s3api list-object-versions --bucket=bald --prefix=adc/certs/injr-f5lb01b-vadc04.json jq -r '.Versions[] | [.Key, .VersionId, .LastModified] | @tsv' adc/certs/injr-f5lb01b-vadc04.json nBHrRDYZzuIrA0hORAIzh6QG8rzRF14 2024-06-28T21:13:00.014Z adc/certs/injr-f5lb01b-vadc04.json YGgH7VmZDq4M-j8qIrKq.4Valvvuoh4 2024-06-26T20:58:09.835Z adc/certs/injr-f5lb01b-vadc04.json qefseb1l.6WJyDNhH5buqX-qcZV2GAJ 2024-06-18T21:32:02.304Z adc/certs/injr-f5lb01b-vadc04.json s4YG598JEQC9A5jaJuI5S4XkCh1NRpN 2024-06-10T21:37:16.074Z adc/certs/injr-f5lb01b-vadc04.json z96LISOi8jBYHCrnbqHgNAAsnpAqbXm 2024-06-07T01:15:21.802Z adc/certs/injr-f5lb01b-vadc04.json Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp 2024-05-30T19:45:19.726Z Looking at the oldest version with ID Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp: LC actually deletes (or tries to delete?) it at several different occasions -- LC runs daily at midnight UTC. For example, . . . 2024-07-04T00:00:02.711+0000 7fd30f989700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 1, 0 . . 2024-07-08T00:00:04.989+0000 7f199a7ac700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 2, 3 . . 2024-07-09T00:00:02.671+0000 7f5a23cea700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 0, 4 However, as seen in the above in aws-cli output, this version is still there. Below is the output when we retrieve this exact version: $ aws s3api get-object --bucket=bald --key=adc/certs/injr-f5lb01b-vadc04.json --version-id=Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp /tmp/outfile { "AcceptRanges": "bytes", "Expiration": "expiry-date=\"Sat, 01 Jun 2024 00:00:00 GMT\", rule-id=\"delete-prior-versions\"", "LastModified": "Thu, 30 May 2024 19:45:19 GMT", "ContentLength": 1299, "ETag": "\"d9c9ff538f4e2f1435746d16cd9e62c8\"", "VersionId": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ContentType": "binary/octet-stream", "Metadata": {} } and in radosgw-admin bucket list: { "name": "adc/certs/injr-f5lb01b-vadc04.json", "instance": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ver": { "pool": 15, "epoch": 563937 }, "locator": "", "exists": true, "meta": { "category": 1, "size": 1299, "mtime": "2024-05-30T19:45:19.726701Z", "etag": "d9c9ff538f4e2f1435746d16cd9e62c8", "storage_class": "", ... "content_type": "", "accounted_size": 1299, "user_data": "", "appendable": false }, "tag": "1eeb7b2c-aaab-4dff-be19-be27acab9e85.1228454711.3959044872362449539", "flags": 1, "pending_map": [], "versioned_epoch": 85 }, SOME MORE NOTES ON THIS BUCKET & OBJECT * The current object is not deleted: no delete-marker. * No object locking is configured for this bucket. * I don't see any trace of this bucket or object in gc list. * The bucket has 101 shards and each shard has around ~30K objects so there's no noticeable skewness in distribution of objects across bucket index. However, I see below ERROR lines streaming when listing the bucket. Not sure it'd be relevant to the LC issue. ... 2024-07-09T18:28:32.470+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=4, prev_marker=lb_summary.lock[s2BIhE0HnXnj.yONcP6.T-dkwU-aWhn], cur_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS] 2024-07-09T18:28:32.530+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS], cur_marker=lb_summary.lock[GDUSDyTB4nGsjT0GfDYlnB5zrB8UnSV] 2024-07-09T18:28:32.546+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=3, prev_marker=lb_summary.lock[U4cpWSv2b5bI8rm1vsDw.kcXmXrDYuV], cur_marker=lb_summary.lock[2NCtXN7KbO0ypy4CSmJCk1gGZEnhWfL] ... * We ran "bucket check --fix" on the bucket a few days ago but it didn't resolve the LC issue or "failed to make forward progress" error stream during bucket listing. * Bucket stats for reference: $ radosgw-admin bucket stats --bucket=bald | jq '. | .bucket, [.num_shards, .usage]' "bald" [ 101, { "rgw.main": { "size": 13065121441, "size_actual": 16259469312, "size_utilized": 13065121441, "size_kb": 12758908, "size_kb_actual": 15878388, "size_kb_utilized": 12758908, "num_objects": 1233984 } } ] QUESTIONS * Is this by any chance a known issue? I searched the tracker but couldn't find a duplicate. * Any ideas why the deletes initiated by LC might fail silently? I don't see any indication of gc queue getting full around the time. * Any ideas on debugging this issue further? Would log-level 20 be helpful and/or any other log lines to look for?

1 0

ceph_fill_inode BAD symlink
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hi, we noticed the following ceph errors in the kernel messages (dmesg -T): [Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode 10003683698.fffffffffffffffe BAD symlink size 0 Is this something that we should be worried about? We are currently trying to identify the inode/file using: # find /data -inum 10003683698 -printf "%i %p\n" Could deleting the inode fix the issue? If not what can we do? Thanks Dietmar

2 3

Re: [EXTERN] ceph_fill_inode BAD symlink
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hi, On 7/10/24 12:07, Loïc Tortay wrote: > On 10/07/2024 11:20, Dietmar Rieder wrote: >> Hi, >> >> Am 9. Juli 2024 13:37:58 MESZ schrieb Dietmar Rieder >> <dietmar.rieder(a)i-med.ac.at>: >>> Hi, >>> >>> we noticed the following ceph errors in the kernel messages (dmesg -T): >>> >>> [Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode >>> 10003683698.fffffffffffffffe BAD symlink size 0 >>> >>> Is this something that we should be worried about? >>> >>> We are currently trying to identify the inode/file using: >>> >>> # find /data -inum 10003683698 -printf "%i %p\n" >>> >>> Could deleting the inode fix the issue? If not what can we do? >> >> find did not return anything. The error keeps showing up , has someone >> an idea? >> > Hello, > The inode number in the error message is in hexadecimal while "find" > expects a decimal number. > > You can ask the MDS instead of using "find": > ceph tell 'mds.*' dump inode 10003683698 | grep '"path"' > > There is a more complicated way to find the name if the inode is not in > the MDS cache > (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MSB5TIG42XA…) Thanks a lot again. The method described in the linked posting worked. We deleted the symlink, we will keep an eye on it. Cheers Dietmar

1 0

CephFS MDS crashing during replay with standby MDSes crashing afterwards
by Ivan Clayson 10 Jul '24

10 Jul '24

Hello, We have been experiencing a serious issue with our CephFS backup cluster running quincy (version 17.2.7) on a RHEL8-derivative Linux kernel (Alma8.9, 4.18.0-513.9.1 kernel) where our MDSes for our filesystem are constantly in a "replay" or "replay(laggy)" state and keep crashing. We have a single MDS filesystem called "ceph_backup" with 2 standby MDSes along with a 2nd unused filesystem "ceph_archive" (this holds little to no data) where we are using our "ceph_backup" filesystem to backup our data and this is the one which is currently broken. The Ceph health outputs currently are: root@pebbles-s1 14:05 [~]: ceph -s cluster: id: e3f7535e-d35f-4a5d-88f0-a1e97abcd631 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available 1319 pgs not deep-scrubbed in time 1054 pgs not scrubbed in time services: mon: 4 daemons, quorum pebbles-s1,pebbles-s2,pebbles-s3,pebbles-s4 (age 36m) mgr: pebbles-s2(active, since 36m), standbys: pebbles-s4, pebbles-s3, pebbles-s1 mds: 2/2 daemons up osd: 1380 osds: 1380 up (since 29m), 1379 in (since 3d); 37 remapped pgs data: volumes: 1/2 healthy, 1 recovering pools: 7 pools, 2177 pgs objects: 3.55G objects, 7.0 PiB usage: 8.9 PiB used, 14 PiB / 23 PiB avail pgs: 83133528/30006841533 objects misplaced (0.277%) 2090 active+clean 47 active+clean+scrubbing+deep 29 active+remapped+backfilling 8 active+remapped+backfill_wait 2 active+clean+scrubbing 1 active+clean+snaptrim io: recovery: 1.9 GiB/s, 719 objects/s root@pebbles-s1 14:09 [~]: ceph fs status ceph_backup - 0 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) pebbles-s3 0 0 0 0 POOL TYPE USED AVAIL mds_backup_fs metadata 1255G 2780G ec82_primary_fs_data data 0 2780G ec82pool data 8442T 3044T ceph_archive - 2 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active pebbles-s2 Reqs: 0 /s 13.4k 7105 118 2 POOL TYPE USED AVAIL mds_archive_fs metadata 5184M 2780G ec83_primary_fs_data data 0 2780G ec83pool data 138T 2767T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) root@pebbles-s1 14:09 [~]: ceph health detail | head HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1319 pgs not deep-scrubbed in time; 1054 pgs not scrubbed in time [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceph_backup is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more When our cluster first ran after a reboot, Ceph ran through the 2 standby MDSes, crashing them all, until it reached the final MDS and is now stuck in this "replay(laggy)" state. Putting our MDSes into debugging mode, we can see that this MDS crashed when replaying the journal for a particular inode (this is the same for all the MDSes and they all crash on the same object): ... 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay for [521,head] had [inode 0x1005ba89481 [...539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/ auth fragtree_t(*^2 00*^3 00000*^ 4 00001*^3 00010*^4 00011*^4 00100*^4 00101*^4 00110*^4 00111*^4 01*^3 01000*^4 01001*^3 01010*^4 01011*^3 01100*^4 01101*^4 01110*^4 01111*^4 10*^3 10000*^4 10001*^4 10010*^4 10011*^4 10100*^4 10101*^3 10110*^4 10111*^4 11*^6) v10880645 f(v0 m2024-06-22 T05:41:10.213700+0100 1281276=1281276+0) n(v12 rc2024-06-22T05:41:10.213700+0100 b1348251683896 1281277=1281276+1) old_inodes=8 (iversion lock) | dirfrag=416 dirty=1 0x55770a2bdb80] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay dir 0x1005ba89481.011011000* 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x1005ba89481.011011000* /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/ [2,head] auth v=436385 cv=0/0 state=107374182 4 f(v0 m2024-06-22T05:41:10.213700+0100 2502=2502+0) n(v12 rc2024-06-22T05:41:10.213700+0100 b2120744220 2502=2502+0) hs=32+33,ss=0+0 dirty=65 | child=1 0x55770ebcda80] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fracti ons_ave_Z124.mrc.teberet7.partial [539,head] auth NULL (dversion lock) v=436384 ino=(nil) state=1610612800|bottomlru | dirty=1 0x557710444500] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay added [inode 0x1005cd4fe35 [539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_ 005006_fractions_ave_Z124.mrc.teberet7.partial auth v436384 s=0 n(v0 1=1+0) (iversion lock) cr={99995144=0-4194304@538} 0x557710438680] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.cache.ino(0x1005cd4fe35) mark_dirty_parent 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay noting opened inode [inode 0x1005cd4fe35 [539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_2762 6130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial auth v436384 DIRTYPARENT s=0 n(v0 1=1+0) (iversion lock) cr={99995144=0-4194304@538} | dirtyparent=1 dirty=1 0x557710438680] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay inotable tablev 3112837 <= table 3112837 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay sessionmap v 1560540883, table 1560540882 prealloc [] used 0x1005cd4fe35 2024-06-24T13:44:55.563+0100 7f8811c40700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/include/interval_set.h: I n function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f8811c40700 time 2024-06-24T13:44:55.564315+0100 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8821e814a3] 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f8821e81669] 3: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x5576f9bb2885] 4: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4377) [0x5576f9eb77b7] 5: (EUpdate::replay(MDSRank*)+0x61) [0x5576f9ebbbd1] 6: (MDLog::_replay_thread()+0x7bb) [0x5576f9e4254b] 7: (MDLog::ReplayThread::entry()+0x11) [0x5576f9af5041] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f8820e6f1ca] 9: clone() I've only included a short section of the crash (this is the first trace in the log with regards to the crash with a 10/20 debug_mds option). We tried deleting the 0x1005cd4fe35 object from the object store using the "rados" command but this did not allow our MDS to successfully replay. From my understanding the journal seems okay as we didn't run out of space for example on our metadata pool and "cephfs-journal-tool journal inspect" doesn't seem to think there is any damage: root@pebbles-s1 13:58 [~]: cephfs-journal-tool --rank=ceph_backup:0 journal inspect Overall journal integrity: OK root@pebbles-s1 14:04 [~]: cephfs-journal-tool --rank=ceph_backup:0 event get --inode 1101069090357 summary Events by type: OPEN: 1 UPDATE: 3 Errors: 0 root@pebbles-s1 14:05 [~]: cephfs-journal-tool --rank=ceph_backup:0 event get --inode 1101069090357 list 2024-06-22T05:41:10.214635+0100 0x51f97d4cfe35 UPDATE: (openc) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:11.203312+0100 0x51f97d59c848 UPDATE: (check_inode_max_size) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:15.484871+0100 0x51f97e7344cc OPEN: () FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:15.484921+0100 0x51f97e73493b UPDATE: (rename) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc I was wondering whether anyone had any advice for us on how we should proceed forward? We were thinking about manually applying these events (via "event apply") where failing that we could erase this problematic event with "cephfs-journal-tool --rank=ceph_backup:0 event splice --inode 1101069090357". Is this a good idea? We would rather not rebuild the entire metadata pool if we could avoid it (once was enough for us) as this cluster has ~9 PB of data on it. Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

3 16

Urgent help with degraded filesystem needed
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hello cephers, we have a degraded filesystem on our ceph 18.2.2 cluster and I'd need to get it up again. We have 6 MDS daemons and (3 active, each pinned to a subtree, 3 standby) It started this night, I got the first HEALTH_WARN emails saying: HEALTH_WARN --- New --- [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 === Full health status === [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 then it went on with: HEALTH_WARN --- New --- [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded --- Cleared --- [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 === Full health status === [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded Then one after another MDS was going to error state: HEALTH_WARN --- Updated --- [WARN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in error state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state === Full health status === [WARN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in error state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WARN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more In the morning then I tried to restart the MDS in error state but the kept failing. I then reduced the number of active MDS to 1 ceph fs set cephfs max_mds 1 And set the filesystem down ceph fs set cephfs down true I tried to restart the MDS again but now I'm stuck at the following status: [root@ceph01-b ~]# ceph -s cluster: id: aae23c5c-a98b-11ee-b44d-00620b05cac4 health: HEALTH_WARN 4 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 3 daemons, quorum cephmon-01,cephmon-03,cephmon-02 (age 2w) mgr: cephmon-01.dsxcho(active, since 11w), standbys: cephmon-02.nssigg, cephmon-03.rgefle mds: 3/3 daemons up osd: 336 osds: 336 up (since 11w), 336 in (since 3M) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 6401 pgs objects: 284.69M objects, 623 TiB usage: 889 TiB used, 3.1 PiB / 3.9 PiB avail pgs: 6186 active+clean 156 active+clean+scrubbing 59 active+clean+scrubbing+deep [root@ceph01-b ~]# ceph health detail HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in unknown state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [root@ceph01-b ~]# [root@ceph01-b ~]# ceph fs status cephfs - 40 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve default.cephmon-02.nyfook 12.3k 11.8k 3228 0 1 replay(laggy) default.cephmon-02.duujba 0 0 0 0 2 resolve default.cephmon-01.pvnqad 15.8k 3541 1409 0 POOL TYPE USED AVAIL ssd-rep-metadata-pool metadata 295G 63.5T sdd-rep-data-pool data 10.2T 84.6T hdd-ec-data-pool data 808T 1929T MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) The end log file of the replay(laggy) default.cephmon-02.duujba shows: [...] -11> 2024-06-19T07:12:38.980+0000 7f90fd117700 1 mds.1.journaler.pq(ro) _finish_probe_end write_pos = 8673820672 (header had 8623488918). recovered. -10> 2024-06-19T07:12:38.980+0000 7f90fd117700 4 mds.1.purge_queue operator(): open complete -9> 2024-06-19T07:12:38.980+0000 7f90fd117700 4 mds.1.purge_queue operator(): recovering write_pos -8> 2024-06-19T07:12:39.015+0000 7f9104926700 10 monclient: get_auth_request con 0x55a93ef42c00 auth_method 0 -7> 2024-06-19T07:12:39.025+0000 7f9105928700 10 monclient: get_auth_request con 0x55a93ef43400 auth_method 0 -6> 2024-06-19T07:12:39.038+0000 7f90fd117700 4 mds.1.purge_queue _recover: write_pos recovered -5> 2024-06-19T07:12:39.038+0000 7f90fd117700 1 mds.1.journaler.pq(ro) set_writeable -4> 2024-06-19T07:12:39.044+0000 7f9105127700 10 monclient: get_auth_request con 0x55a93ef43c00 auth_method 0 -3> 2024-06-19T07:12:39.113+0000 7f9104926700 10 monclient: get_auth_request con 0x55a93ed97000 auth_method 0 -2> 2024-06-19T07:12:39.123+0000 7f9105928700 10 monclient: get_auth_request con 0x55a93e903c00 auth_method 0 -1> 2024-06-19T07:12:39.236+0000 7f90fa912700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f90fa912700 time 2024-06-19T07:12:39.235633+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f910c722e15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f910c722fdb] 3: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x55a93c0de9a5] 4: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4207) [0x55a93c3e76e7] 5: (EUpdate::replay(MDSRank*)+0x61) [0x55a93c3e9f81] 6: (MDLog::_replay_thread()+0x6c9) [0x55a93c3701d9] 7: (MDLog::ReplayThread::entry()+0x11) [0x55a93c01e2d1] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f910b4c81ca] 9: clone() 0> 2024-06-19T07:12:39.236+0000 7f90fa912700 -1 *** Caught signal (Aborted) ** in thread 7f90fa912700 thread_name:md_log_replay ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f910b4d2d20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f910c722e6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f910c722fdb] 6: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x55a93c0de9a5] 7: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4207) [0x55a93c3e76e7] 8: (EUpdate::replay(MDSRank*)+0x61) [0x55a93c3e9f81] 9: (MDLog::_replay_thread()+0x6c9) [0x55a93c3701d9] 10: (MDLog::ReplayThread::entry()+0x11) [0x55a93c01e2d1] 11: /lib64/libpthread.so.0(+0x81ca) [0x7f910b4c81ca] 12: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f90fa912700 / md_log_replay 7f90fb914700 / 7f90fc115700 / MR_Finisher 7f90fd117700 / PQ_Finisher 7f90fe119700 / ms_dispatch 7f910011d700 / ceph-mds 7f9102121700 / ms_dispatch 7f9103123700 / io_context_pool 7f9104125700 / admin_socket 7f9104926700 / msgr-worker-2 7f9105127700 / msgr-worker-1 7f9105928700 / msgr-worker-0 7f910d8eab00 / ceph-mds max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.default.cephmon-02.duujba.log --- end dump of recent events --- I have no idea how to resolve this and would be grateful for any help. Dietmar

7 28

squid 19.1.0 RC QE validation status
by Yuri Weinstein 10 Jul '24

10 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/66756#note-1 Release Notes - TBD LRC upgrade - TBD (Reruns were not done yet.) Seeking approvals/reviews for: smoke rados - Radek, Laura rgw- Casey fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad perf-basic - Yaarit, Laura crimson-rados - Samuel ceph-volume - Guillaume Pls let me know if any tests were missed from this list.

9 17

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 10 Jul '24

10 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

[RGW][Lifecycle][Versioned Buckets][Reef] Although LC deletes non-current versions, they still exist
by oguzhan ozmen 10 Jul '24

10 Jul '24

This is similar to an old thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FXVEWDU6NY… but I don't see any responses there so opening this one. PROBLEM DESCRIPTION * Issue is seen on versioned buckets. * Using extended logging (debug level 5), we can see that LC deletes expired (non-current) versions of objects but those versions are still found in the bucket: both seen in bucket index and accessible to the user. AN EXAMPLE OBJECT/VERSION * Bucket: bald * Object: adc/certs/injr-f5lb01b-vadc04.json * Versions: It currently has 6 versions $ aws s3api list-object-versions --bucket=bald --prefix=adc/certs/injr-f5lb01b-vadc04.json jq -r '.Versions[] | [.Key, .VersionId, .LastModified] | @tsv' adc/certs/injr-f5lb01b-vadc04.json nBHrRDYZzuIrA0hORAIzh6QG8rzRF14 2024-06-28T21:13:00.014Z adc/certs/injr-f5lb01b-vadc04.json YGgH7VmZDq4M-j8qIrKq.4Valvvuoh4 2024-06-26T20:58:09.835Z adc/certs/injr-f5lb01b-vadc04.json qefseb1l.6WJyDNhH5buqX-qcZV2GAJ 2024-06-18T21:32:02.304Z adc/certs/injr-f5lb01b-vadc04.json s4YG598JEQC9A5jaJuI5S4XkCh1NRpN 2024-06-10T21:37:16.074Z adc/certs/injr-f5lb01b-vadc04.json z96LISOi8jBYHCrnbqHgNAAsnpAqbXm 2024-06-07T01:15:21.802Z adc/certs/injr-f5lb01b-vadc04.json Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp 2024-05-30T19:45:19.726Z Looking at the oldest version with ID Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp ... LC actually deletes (or tries to delete?) it at several different occasions -- LC runs daily at midnight UTC. For example, . . . 2024-07-04T00:00:02.711+0000 7fd30f989700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 1, 0 . . 2024-07-08T00:00:04.989+0000 7f199a7ac700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 2, 3 . . 2024-07-09T00:00:02.671+0000 7f5a23cea700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 0, 4 However, as seen in the above in aws-cli output, this version is still there. Below is the output when we retrieve this exact version: $ aws s3api get-object --bucket=bald --key=adc/certs/injr-f5lb01b-vadc04.json --version-id=Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp /tmp/outfile { "AcceptRanges": "bytes", "Expiration": "expiry-date=\"Sat, 01 Jun 2024 00:00:00 GMT\", rule-id=\"delete-prior-versions\"", "LastModified": "Thu, 30 May 2024 19:45:19 GMT", "ContentLength": 1299, "ETag": "\"d9c9ff538f4e2f1435746d16cd9e62c8\"", "VersionId": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ContentType": "binary/octet-stream", "Metadata": {} } and in radosgw-admin bucket list: { "name": "adc/certs/injr-f5lb01b-vadc04.json", "instance": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ver": { "pool": 15, "epoch": 563937 }, "locator": "", "exists": true, "meta": { "category": 1, "size": 1299, "mtime": "2024-05-30T19:45:19.726701Z", "etag": "d9c9ff538f4e2f1435746d16cd9e62c8", "storage_class": "", ... "content_type": "", "accounted_size": 1299, "user_data": "", "appendable": false }, "tag": "1eeb7b2c-aaab-4dff-be19-be27acab9e85.1228454711.3959044872362449539", "flags": 1, "pending_map": [], "versioned_epoch": 85 }, SOME MORE NOTES ON THIS BUCKET & OBJECT * The current object is not deleted: no delete-marker. * No object locking is configured for this bucket. * I don't see any trace of this bucket or object in gc list. * The bucket has 101 shards and each shard has around ~30K objects so there's no noticeable skewness in distribution of objects across bucket index. However, I see below ERROR lines streaming when listing the bucket. Not sure it'd be relevant to the LC issue. ... 2024-07-09T18:28:32.470+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=4, prev_marker=lb_summary.lock[s2BIhE0HnXnj.yONcP6.T-dkwU-aWhn], cur_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS] 2024-07-09T18:28:32.530+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS], cur_marker=lb_summary.lock[GDUSDyTB4nGsjT0GfDYlnB5zrB8UnSV] 2024-07-09T18:28:32.546+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=3, prev_marker=lb_summary.lock[U4cpWSv2b5bI8rm1vsDw.kcXmXrDYuV], cur_marker=lb_summary.lock[2NCtXN7KbO0ypy4CSmJCk1gGZEnhWfL] ... * We ran "bucket check --fix" on the bucket a few days ago but it didn't resolve the LC issue or "failed to make forward progress" error stream during bucket listing. * Bucket stats for reference: $ radosgw-admin bucket stats --bucket=bald | jq '. | .bucket, [.num_shards, .usage]' "bald" [ 101, { "rgw.main": { "size": 13065121441, "size_actual": 16259469312, "size_utilized": 13065121441, "size_kb": 12758908, "size_kb_actual": 15878388, "size_kb_utilized": 12758908, "num_objects": 1233984 } } ] QUESTIONS * Is this by any chance a known issue? I searched the tracker but couldn't find a duplicate. * Any ideas why the deletes initiated by LC might fail silently? I don't see any indication of gc queue getting full around the time. * Any ideas on debugging this issue further? Would log-level 20 be helpful and/or any other log lines to look for?

1 0