- ceph-users - lists.ceph.io

Monitors for two different cluster
by Michel Niyoyita 03 Oct '24

03 Oct '24

Hello Team, I have a running cluster deployed using ceph-ansible pacific version and ubuntu OS , with three mons and 3 osds servers , the cluster is well running , now I want to make another cluster wich will consist of 3 osds servers , can the new cluster be deployed using cephadm and using the existing Mons for the first cluster? Best regards Michel

3 3

MDS stuck in replay and continually crashing during replay
by Ivan Clayson 03 Oct '24

03 Oct '24

Hello everyone, We're having an issue with our backup filesystem running Reef (18.2.4) on an Alma9.4 (version 5.14 kernel) cluster where are our filesystem "ceph_spare" is in a constant degraded state: root@pebbles-n4 11:49 [~]: ceph fs status ceph_spare - 360 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay pebbles-s2 14.1M 6858k 240k 0 POOL TYPE USED AVAIL mds_spare_fs metadata 1451G 3216G ec82_spare_primary_fs_data data 0 3216G ec82pool_spare data 8820T 4626T STANDBY MDS pebbles-s1 pebbles-s2 MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@pebbles-n4 11:49 [~]: ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 large omap objects; 1 MDSs report oversized cache; 1 MDSs behind on trimming [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceph_spare is degraded [WRN] LARGE_OMAP_OBJECTS: 1 large omap objects 1 large objects found in pool 'mds_spare_fs' Search the cluster log for 'Large omap object found' for more details. [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.pebbles-s2(mds.0): MDS cache is too large (24GB/4GB); 0 inodes in use by clients, 0 stray files [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.pebbles-s2(mds.0): Behind on trimming (69293/128) max_segments: 128, num_segments: 69293 Our MDS is stuck in replay constantly and crashing before finishing despite a restart, reboot, and wiping of the client sessions ("ceph config set mds mds_wipe_sessions true"). When following the replay, we've noticed that the MDS always crashes just before the "journal_read_pos" reaches the "journal_write_pos" where the MDS logs with a 10/20 debug report the following: ... -14> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay dir 0x1002a7c9f3a.010001* -13> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x1002a7c9f3a.010001* /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_Sma llHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/ [2,head] auth v=283167 cv=0/0 state=1073741824 f(v0 m2024-09-30T02:20:11.052706+0100 4262=4262+0) n(v37 rc2024-09-30T02:20:11.052706+0100 b4168080204 4262=4262+0) hs=877+853,ss=3+0 dirty=1733 | child=1 0x5642e8d5d680] -12> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.cache.dir(0x1002a7c9f3a.010001*) lookup_exact_snap (head, 'FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks _plot_0012.ef980630.partial') -11> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.cache.dir(0x1002a7c9f3a.010001*) lookup_exact_snap (head, 'FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks _plot_0012.ef980630.partial') -10> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 12 mds.0.cache.dir(0x1002a7c9f3a.010001*) add_null_dentry [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallH ole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lo ck) pv=0 v=283167 ino=(nil) state=1073741824 0x564f219fca00] -9> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_P u_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lock) v=28316 6 ino=(nil) state=1610612736 | dirty=1 0x564f219fca00] -8> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 12 mds.0.cache.dir(0x1002a7c9f3a.010001*) link_primary_inode [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_Sma llHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_46164 89_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lock) v=283166 ino=(nil) state=1610612736 | dirty=1 0x564f219fca00] [inode 0x1002c5f1d10 [126,head] #1002c5f1d10 auth v283166 s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} 0x564f21a00000] -7> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay added [inode 0x1002c5f1d10 [126,head] /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial auth v283166 s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} 0x564f21a00000] -6> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.cache.ino(0x1002c5f1d10) mark_dirty_parent -5> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay noting opened inode [inode 0x1002c5f1d10 [126,head] /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial auth v283166 DIRTYPARENT s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} | dirtyparent=1 dirty=1 0x564f21a00000] -4> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay inotable tablev 1481899 <= table 1481899 -3> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay sessionmap v 746010300, table 746010299 prealloc [] used 0x1002c5f1d10 -2> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.journal (session prealloc [0x1002bdf9f74~0x22,0x1002bdfb51d~0x7c,0x1002c59def8~0xfb,0x1002c5eec0c~0x86,0x1002c5eee87~0x1f5]) -1> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fa8b6d95640 time 2024-10-02T15:20:52.495403+0100 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa8c416b04d] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fa8c416b20b] 3: /usr/bin/ceph-mds(+0x1ef0fe) [0x5642daf490fe] 4: /usr/bin/ceph-mds(+0x1ef145) [0x5642daf49145] 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bad) [0x5642db17b67d] 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5642db18477d] 7: (MDLog::_replay_thread()+0x75e) [0x5642db12d00e] 8: /usr/bin/ceph-mds(+0x13c561) [0x5642dae96561] 9: /lib64/libc.so.6(+0x89c02) [0x7fa8c3889c02] 10: /lib64/libc.so.6(+0x10ec40) [0x7fa8c390ec40] 0> 2024-10-02T15:20:52.495+0100 7fa8b6d95640 -1 *** Caught signal (Aborted) ** in thread 7fa8b6d95640 thread_name:md_log_replay ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa8c383e6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fa8c388b94c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa8c416b0a7] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fa8c416b20b] 7: /usr/bin/ceph-mds(+0x1ef0fe) [0x5642daf490fe] 8: /usr/bin/ceph-mds(+0x1ef145) [0x5642daf49145] 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bad) [0x5642db17b67d] 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5642db18477d] 11: (MDLog::_replay_thread()+0x75e) [0x5642db12d00e] 12: /usr/bin/ceph-mds(+0x13c561) [0x5642dae96561] 13: /lib64/libc.so.6(+0x89c02) [0x7fa8c3889c02] 14: /lib64/libc.so.6(+0x10ec40) [0x7fa8c390ec40] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 10/20 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7fa8b6d95640 / md_log_replay max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.pebbles-s3.log --- end dump of recent events --- Our MDS then starts at the beginning of the replay process and continually re-replays the journal until it crashes again at the same point. It looks like our journal has gotten corrupted at this file from what I understand and our journal (worryingly) is exceptionally large where we've had to use a 2 TiB machine just to try and export it. What is causing this issue? Can we do small modifications to the journal or similar to rectify this issue or move the faulty object in the journal out of the bulk object-store to fail (and thus skip) the transaction? We really do not want to go through disaster recovery again (https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#disaster-re…) as this is the 2nd time this has happened to this cluster in the last 4 months and it took over a month to recover the data last time Kindest regards, Ivan -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

1 0

Using XFS and LVM backends together on the same cluster and hosts
by Özkan Göksu 03 Oct '24

03 Oct '24

Hello folks! I hope you are doing well :) I have a general question about XFS and LVM backend OSD performance and possible effects if they are used together in the same pool. I built a cluster 5 years ago with Nautilus and I used the XFS backend for OSD's. After 5 years they reached me back with ERR state and I see 10K++ slow ops, 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down due to compactions problem. At the first check-up I found some drives were replaced by others and they used LVM backend for the replaced drives without using wal+db on nvme. In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM drives without wal+db. We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 workload. RGW index stored on SSD pool. 10 nodes: with - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD Total HDD raw size 2.8PiB, SSD size 26TiB I started fixing all the problems one by one and I'm gonna recreate these LVM drives without wal+db and I wonder 2 questions: 1- Are there any speed or latency differences on XFS and LVM backend OSD's for 16TB 7200rpm NL-SAS drives. 2- Mixing XFS and LVM backend on the same cluster does have any negative effect or problems? Best regards: Extra note: If you wonder please check the LSBLK output for 1/10 server: NODE-01# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 14.6T 0 disk ├─sda1 8:1 0 100M 0 part /var/lib/ceph/osd/ceph-180 └─sda2 8:2 0 14.6T 0 part sdb 8:16 0 14.6T 0 disk ├─sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-181 └─sdb2 8:18 0 14.6T 0 part sdc 8:32 0 14.6T 0 disk ├─sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-182 └─sdc2 8:34 0 14.6T 0 part sdd 8:48 0 14.6T 0 disk ├─sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-183 └─sdd2 8:50 0 14.6T 0 part sde 8:64 0 14.6T 0 disk ├─sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-185 └─sde2 8:66 0 14.6T 0 part sdf 8:80 0 14.6T 0 disk └─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5 253:2 0 14.6T 0 lvm sdg 8:96 0 14.6T 0 disk ├─sdg1 8:97 0 100M 0 part /var/lib/ceph/osd/ceph-186 └─sdg2 8:98 0 14.6T 0 part sdh 8:112 0 14.6T 0 disk ├─sdh1 8:113 0 100M 0 part /var/lib/ceph/osd/ceph-187 └─sdh2 8:114 0 14.6T 0 part sdi 8:128 0 14.6T 0 disk ├─sdi1 8:129 0 100M 0 part /var/lib/ceph/osd/ceph-188 └─sdi2 8:130 0 14.6T 0 part sdj 8:144 0 14.6T 0 disk ├─sdj1 8:145 0 100M 0 part /var/lib/ceph/osd/ceph-189 └─sdj2 8:146 0 14.6T 0 part sdk 8:160 0 14.6T 0 disk ├─sdk1 8:161 0 100M 0 part /var/lib/ceph/osd/ceph-190 └─sdk2 8:162 0 14.6T 0 part sdl 8:176 0 14.6T 0 disk ├─sdl1 8:177 0 100M 0 part /var/lib/ceph/osd/ceph-191 └─sdl2 8:178 0 14.6T 0 part sdm 8:192 0 14.6T 0 disk ├─sdm1 8:193 0 100M 0 part /var/lib/ceph/osd/ceph-192 └─sdm2 8:194 0 14.6T 0 part sdn 8:208 0 14.6T 0 disk ├─sdn1 8:209 0 100M 0 part /var/lib/ceph/osd/ceph-193 └─sdn2 8:210 0 14.6T 0 part sdo 8:224 0 14.6T 0 disk ├─sdo1 8:225 0 100M 0 part /var/lib/ceph/osd/ceph-194 └─sdo2 8:226 0 14.6T 0 part sdp 8:240 0 14.6T 0 disk ├─sdp1 8:241 0 100M 0 part /var/lib/ceph/osd/ceph-195 └─sdp2 8:242 0 14.6T 0 part sdq 65:0 0 14.6T 0 disk ├─sdq1 65:1 0 100M 0 part /var/lib/ceph/osd/ceph-196 └─sdq2 65:2 0 14.6T 0 part sdr 65:16 0 14.6T 0 disk ├─sdr1 65:17 0 100M 0 part /var/lib/ceph/osd/ceph-197 └─sdr2 65:18 0 14.6T 0 part sds 65:32 0 14.6T 0 disk ├─sds1 65:33 0 100M 0 part /var/lib/ceph/osd/ceph-198 └─sds2 65:34 0 14.6T 0 part sdt 65:48 0 894.3G 0 disk ├─sdt1 65:49 0 100M 0 part /var/lib/ceph/osd/ceph-199 └─sdt2 65:50 0 894.2G 0 part sdu 65:64 0 894.3G 0 disk └─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607 253:3 0 894.3G 0 lvm sdv 65:80 0 894.3G 0 disk └─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4 253:1 0 894.3G 0 lvm sdw 65:96 0 14.6T 0 disk └─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845 253:0 0 14.6T 0 lvm sdx 65:112 0 223.5G 0 disk ├─sdx1 65:113 0 1M 0 part └─sdx2 65:114 0 223.5G 0 part rbd0 252:0 0 15G 0 disk rbd1 252:16 0 6T 0 disk nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md nvme1n1 259:3 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md

2 2

Optimizations on "high" latency Ceph clusters
by Victor Rodriguez 02 Oct '24

02 Oct '24

Hello, I'm trying to get the most from a Ceph Reef 3 node clusters, with 6 NVMe OSD each. Each node is between 0'6 and 1 millisecond RTT. Obviously performance isn't as good as with local clusters, usually around ~0'2ms. 4k in Q1D1 I/O write request are the most affected by this, as expected, with a performance loss of around 70% from an all local cluster with similar hardware. In general, measurements match the increased latency. Are there any guidelines about what to tune or adjust to get the most of this kind of setups with "high" latencies? Thanks in advance! Victor

3 4

Is there a way to throttle faster osds due to slow ops?
by Szabo, Istvan (Agoda) 02 Oct '24

02 Oct '24

Hi, We have extended our clusters with some new nodes and currently it is impossible to remove from any old node the nvme drive which holding the index pool in the cluster without generating slow ops and cluster performance degradation. Currently how I want to remove is in quincy non cephadm cluster is to crush reweight to 0 and remove. This data movement makes slow ops all the way during the nvme osd out. In my opinion it might be generated from the faster drives harder push on the old servers nvme which makes high iowait on the old nvmes so I want to somehow throttle the new nvmes. Not sure with mclock or with any wait is it possible? (max backfill, osd recover ops and recovery ops priotity is already 1 and balancer max misplaced ratio 0.01). This some of the slow osd says during remove: https://gist.github.com/Badb0yBadb0y/15b51e524a47dfbd2728bbabc18238fc#file-… 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.583707809s, txc = 0x55af2bd2e300 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.541916847s, txc = 0x55af1a035b00 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.533919334s, txc = 0x55af19fafb00 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.904534340s, txc = 0x55af49814c00 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.911001205s, txc = 0x55af24b19800 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.597061634s, txc = 0x55af4fe0fb00 2024-10-01T11:46:30.889+0700 7f29becf7640 4 rocksdb: [db/db_impl/db_impl_write.cc:1736] [default] New memtable created with log file: #280327. Immutable memtables: 0. 2024-10-01T11:46:30.889+0700 7f29becf7640 4 rocksdb: [db/column_family.cc:983] [default] Increasing compaction threads because we have 18 level-0 files 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: (Original Log Time 2024/10/01-11:46:30.893378) [db/db_impl/db_impl_compaction_flush.cc:2394] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 2, flush slots scheduled 1, compaction slots scheduled 2 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: [db/flush_job.cc:335] [default] [JOB 5604] Flushing memtable with next log file: 280327 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1727757990893428, "job": 5604, "event": "flush_started", "num_memtables": 1, "num_entries": 2437269, "num_deletes": 2384624, "total_data_size": 233695787, "memory_usage": 278437952, "flush_reason": "Write Buffer Full"} 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: [db/flush_job.cc:364] [default] [JOB 5604] Level-0 flush table #280328: started Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Ceph Steering Committee (a.k.a. CLT) Meeting Minutes 2024-09-30
by Patrick Donnelly 30 Sep '24

30 Sep '24

Hello, Today we only discussed the upcoming election [5]. - A test election was conducted using Helios: [3] - A date for the election was discussed. The eventual consensus was that we should hold the election offline and meet during our normal call for any points of discussion. - The executive council's practical responsibilities have been expanded, thanks to Neha, Josh, and Dan. It is available at [4]. - The remaining time was used to discuss proposals [5] for membership [6] changes on the Ceph Steering Committee in advance of the vote on other agenda items (including the Ceph Executive Council membership). + The focus was on removing members who are no longer active on the project. We did not finish. + Many discussions were had on the mechanisms/criteria for membership addition/removal. These included: lack of attendance as a reason for removal; whether a vote should be had at all for removal; or, the number of representatives for each Ceph component. No consensus was reached. + Mark Nelson will be following up with individuals on the CSC who've been proposed to be removed from the membership roll. (Mainly, the criteria used seemed to be lack of attendance.) In general, I don't think anyone wants to vote for removal of someone who wishes to participate in the CSC. + Next week we will focus on proposals for adding members to the CSC. I will be sending an email shortly concerning that. Minutes: https://pad.ceph.com/p/clt-weekly-minutes There is a Ceph Developer Monthly meeting on Wednesday: [2] [1] https://pad.ceph.com/p/clt-weekly-minutes [2] https://www.meetup.com/ceph-user-group/events/303725485/ [3] https://vote.heliosvoting.org/helios/e/CLT-Test-Election/vote [4] https://pad.ceph.com/p/ceph-exec-council-responsibilities [5] https://pad.ceph.com/p/css-vote-2024q4 [6] https://docs.ceph.com/en/latest/governance/#id4 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Dashboard: frequent queries for balancer status
by Eugen Block 30 Sep '24

30 Sep '24

Hi, I just noticed across different Ceph versions that when browsing the dashboard, the MGR is logging lots of prometheus queries for the balancer status: Sep 30 11:15:55 host2 ceph-mgr[3993215]: log_channel(cluster) log [DBG] : pgmap v25341: 381 pgs: 381 active+clean; 3.9 GiB data, 69 GiB used, 311 GiB / 380 GiB avail; 954 B/s rd, 1 op/s Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:57 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:59 host2 conmon[3993190]: 192.168.168.62 - - [30/Sep/2024:09:15:59] "GET /metrics HTTP/1.1" 200 71320 "" "Prometheus/2.33.4" Sep 30 11:16:02 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch At least I assume that this is prometheus if I interpret [0] correctly. In this case, the balancer is even disabled. Why is it necessary to query the balancer status that often? I couldn't find any useful config related to this. This was noticed on Pacific (16.2.15) and Squid (19.1.1), both with Prometheus version 2.33.4. I haven't tried other Prometheus versions yet. I will check it with a newer Prometheus version as well. Thanks, Eugen [0] https://github.com/ceph/ceph/blob/2d93ad5a87a720c9756ea1eac9d95e1067b80976/…

1 0

device_health_metrics pool automatically recreated
by Eugen Block 30 Sep '24

30 Sep '24

Hi, I just looked into one customer cluster that we upgraded some time ago from Octopus to Quincy (17.2.6) and I'm wondering why there are still both pools, "device_health_metrics" and ".mgr". According to the docs [0], it's supposed to be renamed: > Prior to Quincy, the devicehealth module created a > device_health_metrics pool to store device SMART statistics. With > Quincy, this pool is automatically renamed to be the common manager > module pool. Now only .mgr has data while device_health_metrics is empty, but it has a newer ID: ses01:~ # ceph df | grep -E "device_health|.mgr" .mgr 1 1 68 MiB 18 204 MiB 0 254 TiB device_health_metrics 15 1 0 B 0 0 B 0 254 TiB On a test cluster (meanwhile upgraded to latest Reef) I see the same: ceph01:~ # ceph df | grep -E "device_health_metrics|.mgr" .mgr 38 1 577 KiB 2 1.7 MiB 0 71 GiB device_health_metrics 45 1 0 B 0 0 B 0 71 GiB Since there are still many users who haven't upgraded to >= Quincy yet, this should be clarified/fixed. I briefly checked tracker.ceph.com, but didn't find anything related to this. I'm currently trying to reproduce it on a one-node test cluster which I upgraded from Pacific to Quincy, but no results yet, only that the renaming was successful. But for the other clusters I don't have enough logs to find out how/why the device_health_metrics pool had been recreated. Thanks, Eugen [0] https://docs.ceph.com/en/latest/mgr/administrator/#module-pool

2 3

RGW returning HTTP 500 during resharding
by Floris Bos 30 Sep '24

30 Sep '24

Hi, I am doing an attempt at loading a large number (billions) of tiny objects from other existing system into Ceph (18.2.4 reef) RadosGW/S3. I wrote a little program for that using the AWS S3 Java SDK (using CRT/S3AsyncClient), and am using 250 simultanous connections to get the data in. However I am experiencing problems when it starts to dynamically reshard. The first problem I was experiencing is that the SDK then complains connection throughput is too low, and errors out: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Http connectio n channel shut down due to failure to meet throughput minimum == While I configured the SDK to retry several times on errors, that does not seem to be happening. Or perhaps it does retry, but using one of the other connections, which obviously will fail as well for the same reason. Anyway, I shoved this issue under the carpet by configuring a very high timeout (creating a S3CrtConnectionHealthConfiguration and setting minimumThroughputTimeout() ) That seemed to work around it for a while. However I found out not much later that if the resharding takes longer than some minutes, RadosGW starts returning HTTP status code 500 to PUT requests: == Sep 27 18:14:33 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:15:03 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:21:59 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-16f7e. ret=(2 300) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-15b3. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-147ea. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7865710 op status=-2300 http_status=500 latency=450.223876953s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7ff4710 op status=-2300 http_status=500 latency=450.231872559s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfca7c3710 op status=-2300 http_status=500 latency=450.239868164s ====== Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7865710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.851 +0000] "PUT /datastore/some-obj-16f7e HTTP/1.1" 500 905 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.223876953s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7ff4710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.843 +0000] "PUT /datastore/some-obj-15b3 HTTP/1.1" 500 293 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.231872559s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfca7c3710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.835 +0000] "PUT /datastore/some-obj-147ea HTTP/1.1" 500 318 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.239868164s S == AWS SDK considers 500 quite fatal: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates internal server error == No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives) And is it correct behavior that it returns HTTP response code 500, instead of something that could indicate it is a retry'able condition? If I would add my own code that does retry for a very long time, is there any way I can detect the 500 is due to the resharding, instead of some other condition that do is fatal? Also, is there any more efficient way to get a large amount of objects into Ceph than individual PUTs? Yours sincerely, Floris Bos

3 5