July 2024 - ceph-users - lists.ceph.io

[RGW][Lifecycle][Versioned Buckets][Reef] Although LC deletes non-current
by Oguzhan Ozmen (BLOOMBERG/ 120 PARK) 10 Jul '24

10 Jul '24

This is similar to an old thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FXVEWDU6NY… but I don't see any responses there so opening this one. PROBLEM DESCRIPTION * Issue is seen on versioned buckets. * Using extended logging (debug level 5), we can see that LC deletes expired (non-current) versions of objects but those versions are still to be found in the bucket: both listed in the bucket index and accessible to the user. AN EXAMPLE OBJECT/VERSION * Bucket: bald * Object: adc/certs/injr-f5lb01b-vadc04.json * Versions: It currently has 6 versions $ aws s3api list-object-versions --bucket=bald --prefix=adc/certs/injr-f5lb01b-vadc04.json jq -r '.Versions[] | [.Key, .VersionId, .LastModified] | @tsv' adc/certs/injr-f5lb01b-vadc04.json nBHrRDYZzuIrA0hORAIzh6QG8rzRF14 2024-06-28T21:13:00.014Z adc/certs/injr-f5lb01b-vadc04.json YGgH7VmZDq4M-j8qIrKq.4Valvvuoh4 2024-06-26T20:58:09.835Z adc/certs/injr-f5lb01b-vadc04.json qefseb1l.6WJyDNhH5buqX-qcZV2GAJ 2024-06-18T21:32:02.304Z adc/certs/injr-f5lb01b-vadc04.json s4YG598JEQC9A5jaJuI5S4XkCh1NRpN 2024-06-10T21:37:16.074Z adc/certs/injr-f5lb01b-vadc04.json z96LISOi8jBYHCrnbqHgNAAsnpAqbXm 2024-06-07T01:15:21.802Z adc/certs/injr-f5lb01b-vadc04.json Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp 2024-05-30T19:45:19.726Z Looking at the oldest version with ID Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp: LC actually deletes (or tries to delete?) it at several different occasions -- LC runs daily at midnight UTC. For example, . . . 2024-07-04T00:00:02.711+0000 7fd30f989700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 1, 0 . . 2024-07-08T00:00:04.989+0000 7f199a7ac700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 2, 3 . . 2024-07-09T00:00:02.671+0000 7f5a23cea700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 0, 4 However, as seen in the above in aws-cli output, this version is still there. Below is the output when we retrieve this exact version: $ aws s3api get-object --bucket=bald --key=adc/certs/injr-f5lb01b-vadc04.json --version-id=Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp /tmp/outfile { "AcceptRanges": "bytes", "Expiration": "expiry-date=\"Sat, 01 Jun 2024 00:00:00 GMT\", rule-id=\"delete-prior-versions\"", "LastModified": "Thu, 30 May 2024 19:45:19 GMT", "ContentLength": 1299, "ETag": "\"d9c9ff538f4e2f1435746d16cd9e62c8\"", "VersionId": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ContentType": "binary/octet-stream", "Metadata": {} } and in radosgw-admin bucket list: { "name": "adc/certs/injr-f5lb01b-vadc04.json", "instance": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ver": { "pool": 15, "epoch": 563937 }, "locator": "", "exists": true, "meta": { "category": 1, "size": 1299, "mtime": "2024-05-30T19:45:19.726701Z", "etag": "d9c9ff538f4e2f1435746d16cd9e62c8", "storage_class": "", ... "content_type": "", "accounted_size": 1299, "user_data": "", "appendable": false }, "tag": "1eeb7b2c-aaab-4dff-be19-be27acab9e85.1228454711.3959044872362449539", "flags": 1, "pending_map": [], "versioned_epoch": 85 }, SOME MORE NOTES ON THIS BUCKET & OBJECT * The current object is not deleted: no delete-marker. * No object locking is configured for this bucket. * I don't see any trace of this bucket or object in gc list. * The bucket has 101 shards and each shard has around ~30K objects so there's no noticeable skewness in distribution of objects across bucket index. However, I see below ERROR lines streaming when listing the bucket. Not sure it'd be relevant to the LC issue. ... 2024-07-09T18:28:32.470+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=4, prev_marker=lb_summary.lock[s2BIhE0HnXnj.yONcP6.T-dkwU-aWhn], cur_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS] 2024-07-09T18:28:32.530+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS], cur_marker=lb_summary.lock[GDUSDyTB4nGsjT0GfDYlnB5zrB8UnSV] 2024-07-09T18:28:32.546+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=3, prev_marker=lb_summary.lock[U4cpWSv2b5bI8rm1vsDw.kcXmXrDYuV], cur_marker=lb_summary.lock[2NCtXN7KbO0ypy4CSmJCk1gGZEnhWfL] ... * We ran "bucket check --fix" on the bucket a few days ago but it didn't resolve the LC issue or "failed to make forward progress" error stream during bucket listing. * Bucket stats for reference: $ radosgw-admin bucket stats --bucket=bald | jq '. | .bucket, [.num_shards, .usage]' "bald" [ 101, { "rgw.main": { "size": 13065121441, "size_actual": 16259469312, "size_utilized": 13065121441, "size_kb": 12758908, "size_kb_actual": 15878388, "size_kb_utilized": 12758908, "num_objects": 1233984 } } ] QUESTIONS * Is this by any chance a known issue? I searched the tracker but couldn't find a duplicate. * Any ideas why the deletes initiated by LC might fail silently? I don't see any indication of gc queue getting full around the time. * Any ideas on debugging this issue further? Would log-level 20 be helpful and/or any other log lines to look for?

1 0

ceph_fill_inode BAD symlink
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hi, we noticed the following ceph errors in the kernel messages (dmesg -T): [Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode 10003683698.fffffffffffffffe BAD symlink size 0 Is this something that we should be worried about? We are currently trying to identify the inode/file using: # find /data -inum 10003683698 -printf "%i %p\n" Could deleting the inode fix the issue? If not what can we do? Thanks Dietmar

2 3

Re: [EXTERN] ceph_fill_inode BAD symlink
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hi, On 7/10/24 12:07, Loïc Tortay wrote: > On 10/07/2024 11:20, Dietmar Rieder wrote: >> Hi, >> >> Am 9. Juli 2024 13:37:58 MESZ schrieb Dietmar Rieder >> <dietmar.rieder(a)i-med.ac.at>: >>> Hi, >>> >>> we noticed the following ceph errors in the kernel messages (dmesg -T): >>> >>> [Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode >>> 10003683698.fffffffffffffffe BAD symlink size 0 >>> >>> Is this something that we should be worried about? >>> >>> We are currently trying to identify the inode/file using: >>> >>> # find /data -inum 10003683698 -printf "%i %p\n" >>> >>> Could deleting the inode fix the issue? If not what can we do? >> >> find did not return anything. The error keeps showing up , has someone >> an idea? >> > Hello, > The inode number in the error message is in hexadecimal while "find" > expects a decimal number. > > You can ask the MDS instead of using "find": > ceph tell 'mds.*' dump inode 10003683698 | grep '"path"' > > There is a more complicated way to find the name if the inode is not in > the MDS cache > (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MSB5TIG42XA…) Thanks a lot again. The method described in the linked posting worked. We deleted the symlink, we will keep an eye on it. Cheers Dietmar

1 0

CephFS MDS crashing during replay with standby MDSes crashing afterwards
by Ivan Clayson 10 Jul '24

10 Jul '24

Hello, We have been experiencing a serious issue with our CephFS backup cluster running quincy (version 17.2.7) on a RHEL8-derivative Linux kernel (Alma8.9, 4.18.0-513.9.1 kernel) where our MDSes for our filesystem are constantly in a "replay" or "replay(laggy)" state and keep crashing. We have a single MDS filesystem called "ceph_backup" with 2 standby MDSes along with a 2nd unused filesystem "ceph_archive" (this holds little to no data) where we are using our "ceph_backup" filesystem to backup our data and this is the one which is currently broken. The Ceph health outputs currently are: root@pebbles-s1 14:05 [~]: ceph -s cluster: id: e3f7535e-d35f-4a5d-88f0-a1e97abcd631 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available 1319 pgs not deep-scrubbed in time 1054 pgs not scrubbed in time services: mon: 4 daemons, quorum pebbles-s1,pebbles-s2,pebbles-s3,pebbles-s4 (age 36m) mgr: pebbles-s2(active, since 36m), standbys: pebbles-s4, pebbles-s3, pebbles-s1 mds: 2/2 daemons up osd: 1380 osds: 1380 up (since 29m), 1379 in (since 3d); 37 remapped pgs data: volumes: 1/2 healthy, 1 recovering pools: 7 pools, 2177 pgs objects: 3.55G objects, 7.0 PiB usage: 8.9 PiB used, 14 PiB / 23 PiB avail pgs: 83133528/30006841533 objects misplaced (0.277%) 2090 active+clean 47 active+clean+scrubbing+deep 29 active+remapped+backfilling 8 active+remapped+backfill_wait 2 active+clean+scrubbing 1 active+clean+snaptrim io: recovery: 1.9 GiB/s, 719 objects/s root@pebbles-s1 14:09 [~]: ceph fs status ceph_backup - 0 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) pebbles-s3 0 0 0 0 POOL TYPE USED AVAIL mds_backup_fs metadata 1255G 2780G ec82_primary_fs_data data 0 2780G ec82pool data 8442T 3044T ceph_archive - 2 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active pebbles-s2 Reqs: 0 /s 13.4k 7105 118 2 POOL TYPE USED AVAIL mds_archive_fs metadata 5184M 2780G ec83_primary_fs_data data 0 2780G ec83pool data 138T 2767T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) root@pebbles-s1 14:09 [~]: ceph health detail | head HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1319 pgs not deep-scrubbed in time; 1054 pgs not scrubbed in time [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceph_backup is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more When our cluster first ran after a reboot, Ceph ran through the 2 standby MDSes, crashing them all, until it reached the final MDS and is now stuck in this "replay(laggy)" state. Putting our MDSes into debugging mode, we can see that this MDS crashed when replaying the journal for a particular inode (this is the same for all the MDSes and they all crash on the same object): ... 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay for [521,head] had [inode 0x1005ba89481 [...539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/ auth fragtree_t(*^2 00*^3 00000*^ 4 00001*^3 00010*^4 00011*^4 00100*^4 00101*^4 00110*^4 00111*^4 01*^3 01000*^4 01001*^3 01010*^4 01011*^3 01100*^4 01101*^4 01110*^4 01111*^4 10*^3 10000*^4 10001*^4 10010*^4 10011*^4 10100*^4 10101*^3 10110*^4 10111*^4 11*^6) v10880645 f(v0 m2024-06-22 T05:41:10.213700+0100 1281276=1281276+0) n(v12 rc2024-06-22T05:41:10.213700+0100 b1348251683896 1281277=1281276+1) old_inodes=8 (iversion lock) | dirfrag=416 dirty=1 0x55770a2bdb80] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay dir 0x1005ba89481.011011000* 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x1005ba89481.011011000* /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/ [2,head] auth v=436385 cv=0/0 state=107374182 4 f(v0 m2024-06-22T05:41:10.213700+0100 2502=2502+0) n(v12 rc2024-06-22T05:41:10.213700+0100 b2120744220 2502=2502+0) hs=32+33,ss=0+0 dirty=65 | child=1 0x55770ebcda80] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fracti ons_ave_Z124.mrc.teberet7.partial [539,head] auth NULL (dversion lock) v=436384 ino=(nil) state=1610612800|bottomlru | dirty=1 0x557710444500] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay added [inode 0x1005cd4fe35 [539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_ 005006_fractions_ave_Z124.mrc.teberet7.partial auth v436384 s=0 n(v0 1=1+0) (iversion lock) cr={99995144=0-4194304@538} 0x557710438680] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.cache.ino(0x1005cd4fe35) mark_dirty_parent 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay noting opened inode [inode 0x1005cd4fe35 [539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_2762 6130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial auth v436384 DIRTYPARENT s=0 n(v0 1=1+0) (iversion lock) cr={99995144=0-4194304@538} | dirtyparent=1 dirty=1 0x557710438680] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay inotable tablev 3112837 <= table 3112837 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay sessionmap v 1560540883, table 1560540882 prealloc [] used 0x1005cd4fe35 2024-06-24T13:44:55.563+0100 7f8811c40700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/include/interval_set.h: I n function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f8811c40700 time 2024-06-24T13:44:55.564315+0100 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8821e814a3] 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f8821e81669] 3: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x5576f9bb2885] 4: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4377) [0x5576f9eb77b7] 5: (EUpdate::replay(MDSRank*)+0x61) [0x5576f9ebbbd1] 6: (MDLog::_replay_thread()+0x7bb) [0x5576f9e4254b] 7: (MDLog::ReplayThread::entry()+0x11) [0x5576f9af5041] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f8820e6f1ca] 9: clone() I've only included a short section of the crash (this is the first trace in the log with regards to the crash with a 10/20 debug_mds option). We tried deleting the 0x1005cd4fe35 object from the object store using the "rados" command but this did not allow our MDS to successfully replay. From my understanding the journal seems okay as we didn't run out of space for example on our metadata pool and "cephfs-journal-tool journal inspect" doesn't seem to think there is any damage: root@pebbles-s1 13:58 [~]: cephfs-journal-tool --rank=ceph_backup:0 journal inspect Overall journal integrity: OK root@pebbles-s1 14:04 [~]: cephfs-journal-tool --rank=ceph_backup:0 event get --inode 1101069090357 summary Events by type: OPEN: 1 UPDATE: 3 Errors: 0 root@pebbles-s1 14:05 [~]: cephfs-journal-tool --rank=ceph_backup:0 event get --inode 1101069090357 list 2024-06-22T05:41:10.214635+0100 0x51f97d4cfe35 UPDATE: (openc) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:11.203312+0100 0x51f97d59c848 UPDATE: (check_inode_max_size) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:15.484871+0100 0x51f97e7344cc OPEN: () FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:15.484921+0100 0x51f97e73493b UPDATE: (rename) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc I was wondering whether anyone had any advice for us on how we should proceed forward? We were thinking about manually applying these events (via "event apply") where failing that we could erase this problematic event with "cephfs-journal-tool --rank=ceph_backup:0 event splice --inode 1101069090357". Is this a good idea? We would rather not rebuild the entire metadata pool if we could avoid it (once was enough for us) as this cluster has ~9 PB of data on it. Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

3 16

Urgent help with degraded filesystem needed
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hello cephers, we have a degraded filesystem on our ceph 18.2.2 cluster and I'd need to get it up again. We have 6 MDS daemons and (3 active, each pinned to a subtree, 3 standby) It started this night, I got the first HEALTH_WARN emails saying: HEALTH_WARN --- New --- [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 === Full health status === [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 then it went on with: HEALTH_WARN --- New --- [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded --- Cleared --- [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 === Full health status === [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded Then one after another MDS was going to error state: HEALTH_WARN --- Updated --- [WARN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in error state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state === Full health status === [WARN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in error state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WARN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more In the morning then I tried to restart the MDS in error state but the kept failing. I then reduced the number of active MDS to 1 ceph fs set cephfs max_mds 1 And set the filesystem down ceph fs set cephfs down true I tried to restart the MDS again but now I'm stuck at the following status: [root@ceph01-b ~]# ceph -s cluster: id: aae23c5c-a98b-11ee-b44d-00620b05cac4 health: HEALTH_WARN 4 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 3 daemons, quorum cephmon-01,cephmon-03,cephmon-02 (age 2w) mgr: cephmon-01.dsxcho(active, since 11w), standbys: cephmon-02.nssigg, cephmon-03.rgefle mds: 3/3 daemons up osd: 336 osds: 336 up (since 11w), 336 in (since 3M) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 6401 pgs objects: 284.69M objects, 623 TiB usage: 889 TiB used, 3.1 PiB / 3.9 PiB avail pgs: 6186 active+clean 156 active+clean+scrubbing 59 active+clean+scrubbing+deep [root@ceph01-b ~]# ceph health detail HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in unknown state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [root@ceph01-b ~]# [root@ceph01-b ~]# ceph fs status cephfs - 40 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve default.cephmon-02.nyfook 12.3k 11.8k 3228 0 1 replay(laggy) default.cephmon-02.duujba 0 0 0 0 2 resolve default.cephmon-01.pvnqad 15.8k 3541 1409 0 POOL TYPE USED AVAIL ssd-rep-metadata-pool metadata 295G 63.5T sdd-rep-data-pool data 10.2T 84.6T hdd-ec-data-pool data 808T 1929T MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) The end log file of the replay(laggy) default.cephmon-02.duujba shows: [...] -11> 2024-06-19T07:12:38.980+0000 7f90fd117700 1 mds.1.journaler.pq(ro) _finish_probe_end write_pos = 8673820672 (header had 8623488918). recovered. -10> 2024-06-19T07:12:38.980+0000 7f90fd117700 4 mds.1.purge_queue operator(): open complete -9> 2024-06-19T07:12:38.980+0000 7f90fd117700 4 mds.1.purge_queue operator(): recovering write_pos -8> 2024-06-19T07:12:39.015+0000 7f9104926700 10 monclient: get_auth_request con 0x55a93ef42c00 auth_method 0 -7> 2024-06-19T07:12:39.025+0000 7f9105928700 10 monclient: get_auth_request con 0x55a93ef43400 auth_method 0 -6> 2024-06-19T07:12:39.038+0000 7f90fd117700 4 mds.1.purge_queue _recover: write_pos recovered -5> 2024-06-19T07:12:39.038+0000 7f90fd117700 1 mds.1.journaler.pq(ro) set_writeable -4> 2024-06-19T07:12:39.044+0000 7f9105127700 10 monclient: get_auth_request con 0x55a93ef43c00 auth_method 0 -3> 2024-06-19T07:12:39.113+0000 7f9104926700 10 monclient: get_auth_request con 0x55a93ed97000 auth_method 0 -2> 2024-06-19T07:12:39.123+0000 7f9105928700 10 monclient: get_auth_request con 0x55a93e903c00 auth_method 0 -1> 2024-06-19T07:12:39.236+0000 7f90fa912700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f90fa912700 time 2024-06-19T07:12:39.235633+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f910c722e15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f910c722fdb] 3: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x55a93c0de9a5] 4: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4207) [0x55a93c3e76e7] 5: (EUpdate::replay(MDSRank*)+0x61) [0x55a93c3e9f81] 6: (MDLog::_replay_thread()+0x6c9) [0x55a93c3701d9] 7: (MDLog::ReplayThread::entry()+0x11) [0x55a93c01e2d1] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f910b4c81ca] 9: clone() 0> 2024-06-19T07:12:39.236+0000 7f90fa912700 -1 *** Caught signal (Aborted) ** in thread 7f90fa912700 thread_name:md_log_replay ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f910b4d2d20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f910c722e6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f910c722fdb] 6: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x55a93c0de9a5] 7: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4207) [0x55a93c3e76e7] 8: (EUpdate::replay(MDSRank*)+0x61) [0x55a93c3e9f81] 9: (MDLog::_replay_thread()+0x6c9) [0x55a93c3701d9] 10: (MDLog::ReplayThread::entry()+0x11) [0x55a93c01e2d1] 11: /lib64/libpthread.so.0(+0x81ca) [0x7f910b4c81ca] 12: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f90fa912700 / md_log_replay 7f90fb914700 / 7f90fc115700 / MR_Finisher 7f90fd117700 / PQ_Finisher 7f90fe119700 / ms_dispatch 7f910011d700 / ceph-mds 7f9102121700 / ms_dispatch 7f9103123700 / io_context_pool 7f9104125700 / admin_socket 7f9104926700 / msgr-worker-2 7f9105127700 / msgr-worker-1 7f9105928700 / msgr-worker-0 7f910d8eab00 / ceph-mds max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.default.cephmon-02.duujba.log --- end dump of recent events --- I have no idea how to resolve this and would be grateful for any help. Dietmar

7 28

squid 19.1.0 RC QE validation status
by Yuri Weinstein 09 Jul '24

09 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/66756#note-1 Release Notes - TBD LRC upgrade - TBD (Reruns were not done yet.) Seeking approvals/reviews for: smoke rados - Radek, Laura rgw- Casey fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad perf-basic - Yaarit, Laura crimson-rados - Samuel ceph-volume - Guillaume Pls let me know if any tests were missed from this list.

9 17

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

[RGW][Lifecycle][Versioned Buckets][Reef] Although LC deletes non-current versions, they still exist
by oguzhan ozmen 09 Jul '24

09 Jul '24

This is similar to an old thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FXVEWDU6NY… but I don't see any responses there so opening this one. PROBLEM DESCRIPTION * Issue is seen on versioned buckets. * Using extended logging (debug level 5), we can see that LC deletes expired (non-current) versions of objects but those versions are still found in the bucket: both seen in bucket index and accessible to the user. AN EXAMPLE OBJECT/VERSION * Bucket: bald * Object: adc/certs/injr-f5lb01b-vadc04.json * Versions: It currently has 6 versions $ aws s3api list-object-versions --bucket=bald --prefix=adc/certs/injr-f5lb01b-vadc04.json jq -r '.Versions[] | [.Key, .VersionId, .LastModified] | @tsv' adc/certs/injr-f5lb01b-vadc04.json nBHrRDYZzuIrA0hORAIzh6QG8rzRF14 2024-06-28T21:13:00.014Z adc/certs/injr-f5lb01b-vadc04.json YGgH7VmZDq4M-j8qIrKq.4Valvvuoh4 2024-06-26T20:58:09.835Z adc/certs/injr-f5lb01b-vadc04.json qefseb1l.6WJyDNhH5buqX-qcZV2GAJ 2024-06-18T21:32:02.304Z adc/certs/injr-f5lb01b-vadc04.json s4YG598JEQC9A5jaJuI5S4XkCh1NRpN 2024-06-10T21:37:16.074Z adc/certs/injr-f5lb01b-vadc04.json z96LISOi8jBYHCrnbqHgNAAsnpAqbXm 2024-06-07T01:15:21.802Z adc/certs/injr-f5lb01b-vadc04.json Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp 2024-05-30T19:45:19.726Z Looking at the oldest version with ID Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp ... LC actually deletes (or tries to delete?) it at several different occasions -- LC runs daily at midnight UTC. For example, . . . 2024-07-04T00:00:02.711+0000 7fd30f989700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 1, 0 . . 2024-07-08T00:00:04.989+0000 7f199a7ac700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 2, 3 . . 2024-07-09T00:00:02.671+0000 7f5a23cea700 2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 0, 4 However, as seen in the above in aws-cli output, this version is still there. Below is the output when we retrieve this exact version: $ aws s3api get-object --bucket=bald --key=adc/certs/injr-f5lb01b-vadc04.json --version-id=Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp /tmp/outfile { "AcceptRanges": "bytes", "Expiration": "expiry-date=\"Sat, 01 Jun 2024 00:00:00 GMT\", rule-id=\"delete-prior-versions\"", "LastModified": "Thu, 30 May 2024 19:45:19 GMT", "ContentLength": 1299, "ETag": "\"d9c9ff538f4e2f1435746d16cd9e62c8\"", "VersionId": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ContentType": "binary/octet-stream", "Metadata": {} } and in radosgw-admin bucket list: { "name": "adc/certs/injr-f5lb01b-vadc04.json", "instance": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp", "ver": { "pool": 15, "epoch": 563937 }, "locator": "", "exists": true, "meta": { "category": 1, "size": 1299, "mtime": "2024-05-30T19:45:19.726701Z", "etag": "d9c9ff538f4e2f1435746d16cd9e62c8", "storage_class": "", ... "content_type": "", "accounted_size": 1299, "user_data": "", "appendable": false }, "tag": "1eeb7b2c-aaab-4dff-be19-be27acab9e85.1228454711.3959044872362449539", "flags": 1, "pending_map": [], "versioned_epoch": 85 }, SOME MORE NOTES ON THIS BUCKET & OBJECT * The current object is not deleted: no delete-marker. * No object locking is configured for this bucket. * I don't see any trace of this bucket or object in gc list. * The bucket has 101 shards and each shard has around ~30K objects so there's no noticeable skewness in distribution of objects across bucket index. However, I see below ERROR lines streaming when listing the bucket. Not sure it'd be relevant to the LC issue. ... 2024-07-09T18:28:32.470+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=4, prev_marker=lb_summary.lock[s2BIhE0HnXnj.yONcP6.T-dkwU-aWhn], cur_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS] 2024-07-09T18:28:32.530+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS], cur_marker=lb_summary.lock[GDUSDyTB4nGsjT0GfDYlnB5zrB8UnSV] 2024-07-09T18:28:32.546+0000 7f8ef5db0740 0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=3, prev_marker=lb_summary.lock[U4cpWSv2b5bI8rm1vsDw.kcXmXrDYuV], cur_marker=lb_summary.lock[2NCtXN7KbO0ypy4CSmJCk1gGZEnhWfL] ... * We ran "bucket check --fix" on the bucket a few days ago but it didn't resolve the LC issue or "failed to make forward progress" error stream during bucket listing. * Bucket stats for reference: $ radosgw-admin bucket stats --bucket=bald | jq '. | .bucket, [.num_shards, .usage]' "bald" [ 101, { "rgw.main": { "size": 13065121441, "size_actual": 16259469312, "size_utilized": 13065121441, "size_kb": 12758908, "size_kb_actual": 15878388, "size_kb_utilized": 12758908, "num_objects": 1233984 } } ] QUESTIONS * Is this by any chance a known issue? I searched the tracker but couldn't find a duplicate. * Any ideas why the deletes initiated by LC might fail silently? I don't see any indication of gc queue getting full around the time. * Any ideas on debugging this issue further? Would log-level 20 be helpful and/or any other log lines to look for?

1 0

Phantom hosts
by Tim Holloway 09 Jul '24

09 Jul '24

It's getting worse. As many may be aware, the venerable CentOS 7 OS is hitting end-of-life in a matter of days. The easiest way to upgrade my serves has been to simply create an alternate disk with the new OS, turn my provisioning system loose on it, yank the old OS system disk and jack in the new one. However, Ceph is another matter. For that part, the simplest thing to do is to destroy the Ceph node(s) on the affected box, do the OS upgrade, then re-create the nodes. But now I have even MORE strays. The OSD on my box lives on in Ceph in the dashboard host view even though the documented removal procedures were followed and the VM itself was destroyed. Further, this last node is an RGW node and I cannot remove it from the RGW configuration. It not only shows on the dashboard, it also lists as still active on the command line and as entries in the config database no matter what I do. I really need some solution to this, as it's a major chokepoint in the upgrade process Tim

3 2

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

CephFS snapshots and kernel cephfs continuous I/O
by caskd 08 Jul '24

08 Jul '24

Hello ceph users and devs, I've set up CephFS as a storage which ingests data for multiple hours and snap-schedule hourly. This however causes problems in the case of clients which cannot handle the EAGAIN (Resource busy/temporary unavailable) the kernel gives back during snapshotting. Is there any plans or option to change this behaviour to just wait until the snapshot is done and continue IO as nothing happened? Thanks in advance. -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

2 3

Slow osd ops on large arm cluster
by Adam Prycki 08 Jul '24

08 Jul '24

Hello, we are having issues with slow ops on our large ARM hpc ceph cluster. Cluster runs on 18.2.0 and ubutnu 20.04 MONs, MGRs and MDSs had to be moved to intel servers because of poor single core performance on our arm servers. Our main cephfs data pool is on 54 serwers in 9 racks with 1458 HDDs in total. (OSDs without block.db on ssd) Cephfs data pool is configured as erasure coded pool with k=6 m=2 and rack level replication. Pool has about 16k PGs with average pg per osd at ~90. We have had good experience with EC cephfs on 3,5 times smaller intel ceph cluster. But this arm deployment is becoming problematic. We started experiencing issues since one of the users started to generate sequential RW traffic at at about 5GiB/s. Single OSD with slow ops was enough to create laggy PG and crash application generating this traffic. We've even had issue where osd with slow ops was lagged for 6 hours and required manual restart. Now we are experiencing slow ops even at much lower read only traffic ~400MiB/s Here is an example of slow ops on OSD: { "ops": [ { "description": "osd_op(client.255949991.0:92728602 4.d22s0 4:44b3390a:::1000b640ddc.0000039b:head [read 3633152~8192] snapc 0=[] ondisk+read+known_if_redirected e1117246)", "initiated_at": "2024-07-08T10:19:58.469537+0000", "age": 507.242936848, "duration": 507.24298854800003, "type_data": { "flag_point": "started", "client_info": { "client": "client.255949991", "client_addr": "x.x.x.x:0/887459214", "tid": 92728602 }, "events": [ { "event": "initiated", "time": "2024-07-08T10:19:58.469537+0000", "duration": 0 }, { "event": "throttled", "time": "2024-07-08T10:19:58.469537+0000", "duration": 0 }, { "event": "header_read", "time": "2024-07-08T10:19:58.469535+0000", "duration": 4294967295.9999981 }, { "event": "all_read", "time": "2024-07-08T10:19:58.469571+0000", "duration": 3.5859999999999999e-05 }, { "event": "dispatched", "time": "2024-07-08T10:19:58.469573+0000", "duration": 2.08e-06 }, { "event": "queued_for_pg", "time": "2024-07-08T10:19:58.469586+0000", "duration": 1.2721000000000001e-05 }, { "event": "reached_pg", "time": "2024-07-08T10:19:58.485132+0000", "duration": 0.015546048999999999 }, { "event": "started", "time": "2024-07-08T10:19:58.485147+0000", "duration": 1.5160000000000001e-05 } ] } }, HDD with this OSD is not busy. Arm cores on these servers are slow but no process reaches full 100% core usage. I think we may have the same issue as one described here: https://www.mail-archive.com/ceph-users@ceph.io/msg13273.html I've tried to reduce osd_pool_default_read_lease_ratio form 0.8 to 0.2 I've tried to reduce osd_heartbeat_grace from 20 to 10. It should lower read_lease_interval from 16 to 2 but it didn't help. Still see a lot of slow ops. Could you give me tips what I could tune to fix this issue? Could this be an issue with large number of EC PGs on large cluster with weak CPUs? Best regards Adam Prycki

1 0

Re: pg's stuck activating on osd create
by Eugen Block 08 Jul '24

08 Jul '24

Hi, it depends a bit on the actual OSD layout on the node and your procedure, but there's a chance you might have hit the overdose. But I would expect it to be logged in the OSD logs, two years ago in a Nautilus cluster the message looked like this: > maybe_wait_for_max_pg withhold creation of pg ... According to github in 16.2.15 it could look like this: > maybe_wait_for_max_pg hit max pg, dropping ... But I'm not sure, I haven't seen that in newer clusters (yet). Regards, Eugen Zitat von Richard Bade <hitrich(a)gmail.com>: > Hi Everyone, > I had an issue last night when I was bringing online some osds that I > was rebuilding. When the osds created and came online 15pgs got stuck > in activating. The first osd (osd.112) seemed to come online ok, but > the second one (osd.113) triggered the issue. All the pgs in > activating included osd.112 in the pg map and I resolved it by doing > pg-upmap-items to map the pg back from osd.112 to where it currently > was but it was painful having 10min of stuck i/o os an rbd pool with > vm's running. > > Some details about the cluster: > Pacific 16.2.15, upgraded from Nautilus fairly recently and Luminos > back in the past. All osds were rebuilt on bluestore in Nautilus, as > were the mons. > The disks in question are Intel DC P4510 8TB nvme. I'm rebuilding them > as I had previously had 4x2TB osd's per disk and now wanted to > consolidate down to one osd per disk. > There's around 300 osd's in the pool with 16384 pgs which means that > the 2TB osds had 157pgs on them. However this means that the 8TB osds > have 615pgs on them and I'm wondering if this is maybe the cause of > the problem. > > There are no warnings about too many pgs per osd in the logs or ceph status. > I have the default value of 250 for mon_max_pg_per_osd and default > value of 3.0 for osd_max_pg_per_osd_hard_ratio. > > My plan is to reduce the number of pgs in the pool but I want to > understand and prove what happened here. > Is it likely I've hit pg overdose protection? If I have, how would I > tell as I can't see anything in the cluster logs. > > Thanks, > Rich > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Sanity check
by Adam Witwicki 08 Jul '24

08 Jul '24

Hello, Can someone please let me know what failure domain my erasure code pool is, osd or host? We thought we testing this by turning off 2 hosts, we have had one host offline recently and the cluster was still serving clients - did we get lucky? ceph osd pool get <pool> crush_rule crush_rule: ecpool ceph osd pool get <pool> erasure_code_profile erasure_code_profile: 6-2 rule ecpool { id 3 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } ceph osd erasure-code-profile get 6-2 crush-device-class=hdd crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 octopus Regards Adam

2 1

Pacific 16.2.15 `osd noin`
by Zakhar Kirpichenko 08 Jul '24

08 Jul '24

Hi, I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`: cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`: services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin # ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000 Is this expected behavior? Do I misunderstand the purpose of the `noin` option? Best regards, Zakhar

5 7

Re: CephFS constant high write I/O to the metadata pool
by Olli Rajala 04 Jul '24

04 Jul '24

Hi - mostly as a note to future me and if anyone else looking for the same issue... I finally solved this a couple of months ago. No idea what is wrong with Ceph but the root cause that was triggering this MDS issue was that I had several workstations and a couple servers where the updatedb of "locate" was getting run by daily cron exactly the same time every night causing high momentary strain on the MDS which then somehow screwed up the metadata caching and flushing creating this cumulative write io. The thing to note here is that there's a difference with "locate" and "mlocate" packages. The default config (on Ubuntu atleast) of updatedb for "mlocate" does skip scanning cephfs filesystems but not so for "locate" which happily ventures onto all of your cephfs mounts :| --------------------------- Olli Rajala - Lead TD Anima Vitae Ltd. www.anima.fi --------------------------- On Wed, Dec 14, 2022 at 7:41 PM Olli Rajala <olli.rajala(a)anima.fi> wrote: > Hi, > > One thing I now noticed in the mds logs is that there's a ton of entries > like this: > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d345,d346] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d345,d346] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d343,d344] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d343,d344] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d341,d342] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d341,d342] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d33f,d340] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d33f,d340] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d33d,d33e] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d33d,d33e] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > > ...and after dropping the caches considerably less of those - normal, > abnormal, typical, atypical? ...or is that just something that starts > happening after the cache gets filled? > > Tnx, > --------------------------- > Olli Rajala - Lead TD > Anima Vitae Ltd. > www.anima.fi > --------------------------- > > > On Sun, Dec 11, 2022 at 9:07 PM Olli Rajala <olli.rajala(a)anima.fi> wrote: > >> Hi, >> >> I'm still totally lost with this issue. And now lately I've had a couple >> of incidents where the write bw has suddenly jumped to even crazier levels. >> See the graph here: >> https://gist.github.com/olliRJL/3e97e15a37e8e801a785a1bd5358120d >> >> The points where it drops to something manageable again are when I have >> dropped the mds caches. Usually after the drop there is steady rise but now >> these sudden jumps are something new and even more scary :E >> >> Here's a fresh 2sec level 20 mds log: >> https://gist.github.com/olliRJL/074bec65787085e70db8af0ec35f8148 >> >> Any help and ideas greatly appreciated. Is there any tool or procedure to >> safely check or rebuild the mds data? ...if this behaviour could be caused >> by some hidden issue with the data itself. >> >> Tnx, >> --------------------------- >> Olli Rajala - Lead TD >> Anima Vitae Ltd. >> www.anima.fi >> --------------------------- >> >> >> On Fri, Nov 11, 2022 at 9:14 AM Venky Shankar <vshankar(a)redhat.com> >> wrote: >> >>> On Fri, Nov 11, 2022 at 3:06 AM Olli Rajala <olli.rajala(a)anima.fi> >>> wrote: >>> > >>> > Hi Venky, >>> > >>> > I have indeed observed the output of the different sections of perf >>> dump like so: >>> > watch -n 1 ceph tell mds.`hostname` perf dump objecter >>> > watch -n 1 ceph tell mds.`hostname` perf dump mds_cache >>> > ...etc... >>> > >>> > ...but without any proper understanding of what is a normal rate for >>> some number to go up it's really difficult to make anything from that. >>> > >>> > btw - is there some convenient way to capture this kind of temporal >>> output for others to view. Sure, I could just dump once a second to a file >>> or sequential files but is there some tool or convention that is easy to >>> look at and analyze? >>> >>> Not really - you'd have to do it yourself. >>> >>> > >>> > Tnx, >>> > --------------------------- >>> > Olli Rajala - Lead TD >>> > Anima Vitae Ltd. >>> > www.anima.fi >>> > --------------------------- >>> > >>> > >>> > On Thu, Nov 10, 2022 at 8:18 AM Venky Shankar <vshankar(a)redhat.com> >>> wrote: >>> >> >>> >> Hi Olli, >>> >> >>> >> On Mon, Oct 17, 2022 at 1:08 PM Olli Rajala <olli.rajala(a)anima.fi> >>> wrote: >>> >> > >>> >> > Hi Patrick, >>> >> > >>> >> > With "objecter_ops" did you mean "ceph tell mds.pve-core-1 ops" >>> and/or >>> >> > "ceph tell mds.pve-core-1 objecter_requests"? Both these show very >>> few >>> >> > requests/ops - many times just returning empty lists. I'm pretty >>> sure that >>> >> > this I/O isn't generated by any clients - I've earlier tried to >>> isolate >>> >> > this by shutting down all cephfs clients and this didn't have any >>> >> > noticeable effect. >>> >> > >>> >> > I tried to watch what is going on with that "perf dump" but to be >>> honest >>> >> > all I can see is some numbers going up in the different sections :) >>> >> > ...don't have a clue what to focus on and how to interpret that. >>> >> > >>> >> > Here's a perf dump if you or anyone could make something out of >>> that: >>> >> > https://gist.github.com/olliRJL/43c10173aafd82be22c080a9cd28e673 >>> >> >>> >> You'd need to capture this over a period of time to see what ops might >>> >> be going through and what the mds is doing. >>> >> >>> >> > >>> >> > Tnx! >>> >> > o. >>> >> > >>> >> > --------------------------- >>> >> > Olli Rajala - Lead TD >>> >> > Anima Vitae Ltd. >>> >> > www.anima.fi >>> >> > --------------------------- >>> >> > >>> >> > >>> >> > On Fri, Oct 14, 2022 at 8:32 PM Patrick Donnelly < >>> pdonnell(a)redhat.com> >>> >> > wrote: >>> >> > >>> >> > > Hello Olli, >>> >> > > >>> >> > > On Thu, Oct 13, 2022 at 5:01 AM Olli Rajala <olli.rajala(a)anima.fi> >>> wrote: >>> >> > > > >>> >> > > > Hi, >>> >> > > > >>> >> > > > I'm seeing constant 25-50MB/s writes to the metadata pool even >>> when all >>> >> > > > clients and the cluster is idling and in clean state. This >>> surely can't >>> >> > > be >>> >> > > > normal? >>> >> > > > >>> >> > > > There's no apparent issues with the performance of the cluster >>> but this >>> >> > > > write rate seems excessive and I don't know where to look for >>> the >>> >> > > culprit. >>> >> > > > >>> >> > > > The setup is Ceph 16.2.9 running in hyperconverged 3 node core >>> cluster >>> >> > > and >>> >> > > > 6 hdd osd nodes. >>> >> > > > >>> >> > > > Here's typical status when pretty much all clients are idling. >>> Most of >>> >> > > that >>> >> > > > write bandwidth and maybe fifth of the write iops is hitting the >>> >> > > > metadata pool. >>> >> > > > >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > root@pve-core-1:~# ceph -s >>> >> > > > cluster: >>> >> > > > id: 2088b4b1-8de1-44d4-956e-aa3d3afff77f >>> >> > > > health: HEALTH_OK >>> >> > > > >>> >> > > > services: >>> >> > > > mon: 3 daemons, quorum pve-core-1,pve-core-2,pve-core-3 >>> (age 2w) >>> >> > > > mgr: pve-core-1(active, since 4w), standbys: pve-core-2, >>> pve-core-3 >>> >> > > > mds: 1/1 daemons up, 2 standby >>> >> > > > osd: 48 osds: 48 up (since 5h), 48 in (since 4M) >>> >> > > > >>> >> > > > data: >>> >> > > > volumes: 1/1 healthy >>> >> > > > pools: 10 pools, 625 pgs >>> >> > > > objects: 70.06M objects, 46 TiB >>> >> > > > usage: 95 TiB used, 182 TiB / 278 TiB avail >>> >> > > > pgs: 625 active+clean >>> >> > > > >>> >> > > > io: >>> >> > > > client: 45 KiB/s rd, 38 MiB/s wr, 6 op/s rd, 287 op/s wr >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > >>> >> > > > Here's some daemonperf dump: >>> >> > > > >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > root@pve-core-1:~# ceph daemonperf mds.`hostname -s` >>> >> > > > >>> >> > > >>> ----------------------------------------mds----------------------------------------- >>> >> > > > --mds_cache--- ------mds_log------ -mds_mem- >>> -------mds_server------- >>> >> > > mds_ >>> >> > > > -----objecter------ purg >>> >> > > > req rlat fwd inos caps exi imi hifc crev cgra ctru cfsa >>> cfa hcc >>> >> > > hccd >>> >> > > > hccr prcr|stry recy recd|subm evts segs repl|ino dn |hcr >>> hcs hsr cre >>> >> > > > cat |sess|actv rd wr rdwr|purg| >>> >> > > > 40 0 0 767k 78k 0 0 0 1 6 1 0 >>> 0 5 5 >>> >> > > > 3 7 |1.1k 0 0 | 17 3.7k 134 0 |767k 767k| 40 5 >>> 0 0 >>> >> > > > 0 |110 | 4 2 21 0 | 2 >>> >> > > > 57 2 0 767k 78k 0 0 0 3 16 3 0 >>> 0 11 11 >>> >> > > > 0 17 |1.1k 0 0 | 45 3.7k 137 0 |767k 767k| 57 8 >>> 0 0 >>> >> > > > 0 |110 | 0 2 28 0 | 4 >>> >> > > > 57 4 0 767k 78k 0 0 0 4 34 4 0 >>> 0 34 33 >>> >> > > > 2 26 |1.0k 0 0 |134 3.9k 139 0 |767k 767k| 57 13 >>> 0 0 >>> >> > > > 0 |110 | 0 2 112 0 | 19 >>> >> > > > 67 3 0 767k 78k 0 0 0 6 32 6 0 >>> 0 22 22 >>> >> > > > 0 32 |1.1k 0 0 | 78 3.9k 141 0 |767k 768k| 67 4 >>> 0 0 >>> >> > > > 0 |110 | 0 2 56 0 | 2 >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > Any ideas where to look at? >>> >> > > >>> >> > > Check the perf dump output of the mds: >>> >> > > >>> >> > > ceph tell mds.<fs_name>:0 perf dump >>> >> > > >>> >> > > over a period of time to identify what's going on. You can also >>> look >>> >> > > at the objecter_ops (another tell command) for the MDS. >>> >> > > >>> >> > > -- >>> >> > > Patrick Donnelly, Ph.D. >>> >> > > He / Him / His >>> >> > > Principal Software Engineer >>> >> > > Red Hat, Inc. >>> >> > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>> >> > > >>> >> > > >>> >> > _______________________________________________ >>> >> > ceph-users mailing list -- ceph-users(a)ceph.io >>> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> > >>> >> >>> >> >>> >> -- >>> >> Cheers, >>> >> Venky >>> >> >>> >>> >>> -- >>> Cheers, >>> Venky >>> >>>

3 3

[RGW] Bucket synchronization in multi-zonegroup
by Huy Nguyen 04 Jul '24

04 Jul '24

Hi community, I tried "radosgw-admin sync group" but it seems to only work between zones within the same zonegroup. So does radosgw support sync a bucket between regions/zonegroups? Or could it be done in any other way? Thanks

1 0

multipart file in broken state
by Matthew Darwin 04 Jul '24

04 Jul '24

When trying to clean up multi-part files, I get the following error: $ rclone backend cleanup s3:bucket-name 2024/07/04 02:42:19 ERROR : S3 bucket bucket-name: failed to remove pending multipart upload for bucket "bucket-name" key "0a424a15dee6fecb241130e9e4e49d99ed120f05/outputs/0121490000-0121491000.output.zst" dated 2024-06-20 09:28:34.112 +0000 UTC (329h13m44.854458863s ago): NoSuchUpload: status code: 404, request id: tx00000bd994059ba3d77af-0066860c0a-ce710f2-s3-1, host id: Why do files get in this state? How should they be fixed? Any hints? I have multiple files in the same state. This is ceph quincy.

1 0

Cluster Alerts
by filip Mutterer 03 Jul '24

03 Jul '24

In my cluster I have old Alerts, how should solved Alerts be handled? Just wait until they disappear or Silence them? Whats the recommended way? filip

2 1

Ceph 16.2 vs 18.2 use case Docker/Swarm LXC
by filip Mutterer 02 Jul '24

02 Jul '24

Could I be missing any significant features when using Ceph 16.2 instead of 18.2 when using Docker/Swarm or LXC? I am asking, because I am struggling to set it up in Version 18.2 due to confusion with the debian 12 packages as some still stay in version 16.2 even after adding the sources for 18.2.

1 0

Ceph Leadership Team Meeting, 2024-07-01
by Ernesto Puerta 02 Jul '24

02 Jul '24

Hi Cephers, These are the topics that we discussed in our meeting: - [cbodley] rgw tech lead transitioning to Eric Ivancich and Adam Emerson - You may send your congrats to them! (offline) - [Zac] - last week's unfinished business - https://github.com/ceph/ceph/pull/58092 - [Zac] - https://github.com/ceph/ceph/pull/58323 - "run-tox-cephadm" is failing -- this is beyond Zac's art. He needs help if the Reef docs are going to update. - Adam King is taking care of it. - [Zac] - CQ#5 - We stop taking CQ-related requests one week from today. - https://pad.ceph.com/p/ ceph_quarterly_2024_07 - Still 1 week to go - [Neha] 19.1.0 - https://tracker.ceph.com/issues/66756 - release notes PR/release highlights for announcement: - Neha request leads to contribute to https://pad.ceph.com/p/squid_release_highlights - [David] Last Pacific build, connected to CentOS 8 and fs upgrades from Pacific-Quincy/Reef. - [Radek] Nightly QA runs are taking longer than usual: - https://pulpito.ceph.com/?branch=main&suite=rados - [Ilya]: we didn't previously have nightly runs, - [Patrick] They have lower priority than integration branches. - "They're helpful as a baseline"... "but do they need to be run nightly?" - [yuri] Run rados main tests on weekends with higher prio (manually?) - [Patrick]: https://github.com/ceph/ceph/blob/main/qa/crontab/teuthology-cronjobs - 1 component / day so it's ~ weekly Kind Regards, Ernesto Puerta

1 0

Re: Debian 12 + Ceph 18.2 + ARM64
by filip Mutterer 02 Jul '24

02 Jul '24

In the official guide for manual installation the command ceph -s shows cluster status. But having it in a different version gives me doubts if I am on the right way installing ceph. On 02.07.2024 02:03, Anthony D'Atri wrote: > I’m not familiar with a “Ceph” package. Might be a Debian meta package ? > > > >> On Jul 1, 2024, at 7:09 PM, filip Mutterer <filip(a)3dds.io> wrote: >> >> ⋚훲犉ꙺ筢鸊⺵ꥥꋘ庮쨮귇갆默幺睩槉᪁燪慶❧뛈棂齍듯洶㣜৭蚉୫謊眪⹶⬚湗궅渦ꜹ⫞ꚋ⊶諢立⣊讝ꉱ㛳佽똇궶អ궻ꝶ垝滨ꕴ멝盓渚랭딓漴S챾⥢꓋궵그ꖊ鷝盈ꣂ먭示砭ꈩꥥ৪懗춨鰷鮉꧵�궽誉障醨ṱ隊⒳૨鸌ⅺ睨詸᪦�檷ⅱ㌧Ầᥢ닗幮᭭ꛏｶ谧隆鵱犉뽵橺�ￛꊉア릦権삦�檷↭雖ꦶ잚귈屺顠귪痳斀䓒ങꩭ皇겞譡櫷骲횛闠謢⧭窷�笈궅욮똪➊笜긪涱잩蘊�꧚疫Ⅾ蚫盏ꛥ立婮垵땮騵�媖圞ꘙꂭ잩薧曾쭚湗겵ꛥ筟㛛囩ꍝ뻕ꥥ燪憚૝櫈审ꫝﻋ婮垬떦弶�嶾햩敱騊�줩귧扲�鹚ᱪ忬떦쭚湗뗳涵溚㗛陗Ầᦠ깏ⱺ뛏ꛥ立婮垵땮騵�媖圞ꘙꂮ樝멗걲諞ﻋ婮垬떦弶�嶾햩敱騊褿닖鮕榹廗춶햺棗澵橙屺顩꺉麶ឮ놩庮�닖鮕榹廗춶햺棗澵橙屺顯ꉛꙻשּׂ榹庲횛闭糛浛ꚍ盻嚥闇ꦅשּׁ藩旾쭚湗겵ꛥ筟㛛囩ꍝ뻕ꥥ燪慾쭨ꟻ⵩륞닖鮕�宦赶ﭖꖖ墛燪慾죚붯ꛥ立婮垵땮騵�媖婲똚⟝잩藊⚚觿닖鮕榹廗춶햺棗澵橙劢�귨겞蜞ꘚ婲䚠稩묢ꉽ糙뮭뷪觵ᒲ⧠橰ᄱ뚬秛㊲힦棺垦藪�쩚牆ꁺ욛ꋷ麝�橙䢲⭩ꋋ≮垭ꇘ걩륞똗ꊞ쭚陖궊觨簧ꦆ쨥痪橚牆ꁺ윞ꘛ걺묦権択ॢ닗Ầᮬ窻ᱺ顢ꄺ⺞쮛뇊鷖ꝺ暢雚ᱺ顮뇪秇ꦆ�

1 0

Re: Debian 12 + Ceph 18.2 + ARM64
by filip Mutterer 01 Jul '24

01 Jul '24

I tried by commenting out all other sources. But in the end package "ceph" didn't show up. On 02.07.2024 00:43, Anthony D'Atri wrote: > Did you disable the Debian OS repositories, so you don’t get their bundled build? > >> On Jul 1, 2024, at 6:21 PM, filip Mutterer <filip(a)3dds.io> wrote: >> >> Trying to install Ceph 18.2 on Debian 12 but version of package "ceph" looks wrong when doing apt search ceph. >> >> >> My ceph.list: >> >> deb https://download.ceph.com/debian-reef/ bookworm main >> >> >> Apt search result: >> >> apt search ceph|grep 18\.2 >> >> WARNING: apt does not have a stable CLI interface. Use with caution in scripts. >> >> ceph-grafana-dashboards/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-cephadm/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-dashboard/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-diskprediction-local/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-k8sevents/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-modules-core/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-rook/stable,stable 18.2.2-1~bpo12+1 all >> ceph-prometheus-alerts/stable,stable 18.2.2-1~bpo12+1 all >> ceph-volume/stable,stable 18.2.2-1~bpo12+1 all >> cephfs-shell/stable,stable 18.2.2-1~bpo12+1 all >> cephfs-top/stable,stable 18.2.2-1~bpo12+1 all >> libcephfs-java/stable,stable 18.2.2-1~bpo12+1 all >> python3-ceph-common/stable,stable 18.2.2-1~bpo12+1 all >> >> So there is no "ceph" package in version 18.2 but version 16. Is Using an ARM based system a Problem here as packages above end on "all"? >> >> Is it possible to disable the installation of Cephs older 16.2 packages? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Fwd: Debian 12 + Ceph 18.2 + ARM64
by filip Mutterer 01 Jul '24

01 Jul '24

I tried by commenting out all other sources. But in the end package "ceph" didn't show up. On 02.07.2024 00:43, Anthony D'Atri wrote: > Did you disable the Debian OS repositories, so you don’t get their > bundled build? > >> On Jul 1, 2024, at 6:21 PM, filip Mutterer <filip(a)3dds.io> wrote: >> >> Trying to install Ceph 18.2 on Debian 12 but version of package >> "ceph" looks wrong when doing apt search ceph. >> >> >> My ceph.list: >> >> deb https://download.ceph.com/debian-reef/ bookworm main >> >> >> Apt search result: >> >> apt search ceph|grep 18\.2 >> >> WARNING: apt does not have a stable CLI interface. Use with caution >> in scripts. >> >> ceph-grafana-dashboards/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-cephadm/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-dashboard/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-diskprediction-local/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-k8sevents/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-modules-core/stable,stable 18.2.2-1~bpo12+1 all >> ceph-mgr-rook/stable,stable 18.2.2-1~bpo12+1 all >> ceph-prometheus-alerts/stable,stable 18.2.2-1~bpo12+1 all >> ceph-volume/stable,stable 18.2.2-1~bpo12+1 all >> cephfs-shell/stable,stable 18.2.2-1~bpo12+1 all >> cephfs-top/stable,stable 18.2.2-1~bpo12+1 all >> libcephfs-java/stable,stable 18.2.2-1~bpo12+1 all >> python3-ceph-common/stable,stable 18.2.2-1~bpo12+1 all >> >> So there is no "ceph" package in version 18.2 but version 16. Is >> Using an ARM based system a Problem here as packages above end on "all"? >> >> Is it possible to disable the installation of Cephs older 16.2 packages? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Debian 12 + Ceph 18.2 + ARM64
by filip Mutterer 01 Jul '24

01 Jul '24

Trying to install Ceph 18.2 on Debian 12 but version of package "ceph" looks wrong when doing apt search ceph. My ceph.list: deb https://download.ceph.com/debian-reef/ bookworm main Apt search result: apt search ceph|grep 18\.2 WARNING: apt does not have a stable CLI interface. Use with caution in scripts. ceph-grafana-dashboards/stable,stable 18.2.2-1~bpo12+1 all ceph-mgr-cephadm/stable,stable 18.2.2-1~bpo12+1 all ceph-mgr-dashboard/stable,stable 18.2.2-1~bpo12+1 all ceph-mgr-diskprediction-local/stable,stable 18.2.2-1~bpo12+1 all ceph-mgr-k8sevents/stable,stable 18.2.2-1~bpo12+1 all ceph-mgr-modules-core/stable,stable 18.2.2-1~bpo12+1 all ceph-mgr-rook/stable,stable 18.2.2-1~bpo12+1 all ceph-prometheus-alerts/stable,stable 18.2.2-1~bpo12+1 all ceph-volume/stable,stable 18.2.2-1~bpo12+1 all cephfs-shell/stable,stable 18.2.2-1~bpo12+1 all cephfs-top/stable,stable 18.2.2-1~bpo12+1 all libcephfs-java/stable,stable 18.2.2-1~bpo12+1 all python3-ceph-common/stable,stable 18.2.2-1~bpo12+1 all So there is no "ceph" package in version 18.2 but version 16. Is Using an ARM based system a Problem here as packages above end on "all"? Is it possible to disable the installation of Cephs older 16.2 packages?

1 0

Ceph tracker broken?
by Frank Schilder 01 Jul '24

01 Jul '24

Hi all, hopefully someone on this list can help me out. I recently started to receive unsolicited e-mail from the ceph tracker and also certain merge/pull requests. The latest one is: [CephFS - Bug #66763] (New) qa: revert commit to unblock snap-schedule testing I have nothing to do with that and I have not subscribed to this tracker item (https://tracker.ceph.com/issues/66763) eithrt. Yet, I receive unrequested updates. Could someone please take a look and try to find out what the problem is? Thanks a lot! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

Viability of NVMeOF/TCP for VMWare
by Drew Weaver 01 Jul '24

01 Jul '24

Howdy, I recently saw that Ceph has a gateway which allows VMWare ESXi to connect to RBD. We had another gateway like this awhile back the ISCSI gateway. The ISCSI gateway ended up being... let's say problematic. Is there any reason to believe that NVMeOF will also end up on the floor and has anyone that uses VMWare extensively evaluated it's viability? Just curious! Thanks, -Drew

5 4

How can I increase or decrease the number of osd backfilling instantly
by Jaemin Joo 01 Jul '24

01 Jul '24

Hi All, I'd like to speed up or slow down osd recovery in ceph v18.2.1. According to the page ( https://www.suse.com/ko-kr/support/kb/doc/?id=000019693 ), I understand that osd_max_backfills, osd_recovery_max_active have to be increased or decreased. but It seems not to impact the number of osd backfilling instantly. so I tried to restart all of the osds even though I didn't need to restart osds. I wonder how long it takes to show that the number of backfilling is increased or decreased If the osds is not restarted. and Is there a way to increase or decrease the number of backfilling instantly after the configuration is changed.

2 2

ATTN: DOCS
by Lander Duncan 01 Jul '24

01 Jul '24

Good morning, Here are a few issues I noticed with the documentation: 1: Page : Hardware Recommendation ( https://docs.ceph.com/en/latest/start/hardware-recommendations/) Issue: Spelling on word "overprovsioning" 2: Page : Hardware Recommendation ( https://docs.ceph.com/en/latest/start/hardware-recommendations/) Issue: Missing word in sentence "You should also each host’s percentage of the cluster’s overall capacity." Potential fix: Add the word check between "also" and "each" 3: Page: https://docs.ceph.com/en/reef/cephfs/mount-prerequisites/ Issue: Broken Link: "You can use CephFS by mounting it to your local filesystem or by using [cephfs-shell]( https://docs.ceph.com/en/reef/cephfs/cephfs-shell)" 4: Page: https://docs.ceph.com/en/reef/rados/configuration/mon-lookup-dns/#looking-u… Issue: lacking information: As you can see in the code here: https://github.com/ceph/ceph/blob/cac7dcd63481ddbc4fe26f4b8d5918510e8be457/… in lines 790-793 and through experimentation you can use the "mon_dns_srv_name" directive to pass either the "ceph-mon" and use DNS search domains, as mentioned on this page or you can pass the name followed by an underscore to manually pass the search domain. For example: ceph-mon_example.com with records placed in _ceph-mon._example.com Have a lovely day

2 4