- ceph-users - lists.ceph.io

Unexpected ceph pool creation error with Ceph Quincy
by Geert Kloosterman 23 Mar '23

23 Mar '23

Hi all, I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create". When not passing any pg_num and pgp_num values, I get the following error with Quincy: [root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed. I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf: [root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8 At least, this is what appears to be used with Pacific. Is this an intended change of behavior? I could not find anything related in the release notes. Best regards, Geert Kloosterman

2 5

Ceph performance problems
by Dominik Baack 22 Mar '23

22 Mar '23

Hi, we are currently testing out ways to increase Ceph performance because what we experience so far is very close to unusable. For the test cluster we utilizing 4 nodes with the following hardware data: Dual 200GBe Mellanox Ethernet 2x EPYC Rome 7302 16x 32GB 3200MHz ECC 9x 15.36TB Micron 9300 Pro For production this will be extended to all 8 nodes if it shows promising results. - Ceph was installed with cephadm. - MDS and ODS are located on the same nodes. - Mostly using stock config - Network performance tested with iperf3 seems fine, 26Gbits/s with -P4 on single port (details below). Close to 200Gbits with 10 parallel instances and servers. When testing a mounted CephFS on the working nodes in various configurations I only got <50MB/s for fuse mount and <270MB/s for kernel mounts. (dd command and output attached below) In addition ceph dashboard and our graphana monitoring reports packet loss on all relevant interfaces during load. Which does not occur during the normal iperf load tests or rsync/scp file transfer. Rados Bench shows performance around 2000MB/s which is not max performance of the SSDs but fine for us (details below). Why is the filesystem so slow compared to the individual components? Cheers Dominik Test details: ------------------------------------------------------------------------------------------------------ Some tests done on working nodes: Ceph mounted with ceph-fuse root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s Ceph mounted with kernel driver: root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s Storage Node With fuse root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s Kernel mount: dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s _______________________________________________________ Iperf3 iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b 200000000000 Connecting to host ml2ran08s0, port 4701 [ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port 4701 [ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port 4701 [ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port 4701 [ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port 4701 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender [ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec receiver [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender [SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec receiver _________________________________________________________________ Rados Bench on storage node: # rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ml2rsn05_2829244 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 747 731 2923.84 2924 0.0178757 0.0216262 2 16 1506 1490 2979.71 3036 0.0308664 0.0213685 3 16 2267 2251 3000.99 3044 0.0259053 0.0212556 4 16 3058 3042 3041.62 3164 0.0227621 0.0209792 5 16 3850 3834 3066.8 3168 0.0130519 0.0208148 6 16 4625 4609 3072.26 3100 0.151371 0.0207904 7 16 5381 5365 3065.28 3024 0.0300368 0.0208345 8 16 6172 6156 3077.57 3164 0.0197728 0.0207714 9 16 6971 6955 3090.67 3196 0.0142751 0.0206786 10 14 7772 7758 3102.76 3212 0.0181034 0.020605 Total time run: 10.0179 Total writes made: 7772 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3103.23 Stddev Bandwidth: 93.3676 Max bandwidth (MB/sec): 3212 Min bandwidth (MB/sec): 2924 Average IOPS: 775 Stddev IOPS: 23.3419 Max IOPS: 803 Min IOPS: 731 Average Latency(s): 0.020598 Stddev Latency(s): 0.00731743 Max latency(s): 0.151371 Min latency(s): 0.00966991 # rados bench -p testbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 15 657 642 2567.32 2568 0.011104 0.022631 2 15 1244 1229 2456.9 2348 0.0115248 0.019485 3 16 1499 1483 1976.64 1016 0.00722983 0.0177887 4 15 1922 1907 1906.16 1696 0.0142242 0.0327382 5 15 2593 2578 2061.6 2684 0.011758 0.0301774 6 16 3142 3126 2083.23 2192 0.00915926 0.027478 7 16 3276 3260 1862.23 536 0.00824714 0.0267449 8 16 3606 3590 1794.43 1320 0.0118938 0.0350541 9 16 4293 4277 1900.32 2748 0.0301886 0.0330604 10 14 5003 4989 1995.04 2848 0.0389717 0.0314977 Total time run: 10.0227 Total reads made: 5003 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1996.67 Average IOPS: 499 Stddev IOPS: 202.3 Max IOPS: 712 Min IOPS: 134 Average Latency(s): 0.0314843 Max latency(s): 3.04463 Min latency(s): 0.00551523 # rados bench -p testbench 10 rand hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 15 15 0 0 0 - 0 1 15 680 665 2657.61 2660 0.00919807 0.0224833 2 15 1273 1258 2514.26 2372 0.00839656 0.0247125 3 16 1863 1847 2461.4 2356 0.00994467 0.0236565 4 16 2064 2048 2047.14 804 0.00809139 0.0223506 5 16 2064 2048 1637.79 0 - 0.0223506 6 16 2477 2461 1640.12 826 0.0286315 0.0383254 7 16 3102 3086 1762.89 2500 0.0267464 0.0349189 8 16 3513 3497 1748 1644 0.00890952 0.032269 9 16 3617 3601 1600 416 0.00626917 0.0316019 10 15 4014 3999 1599.18 1592 0.0461076 0.0393606 Total time run: 10.0481 Total reads made: 4014 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1597.91 Average IOPS: 399 Stddev IOPS: 239.089 Max IOPS: 665 Min IOPS: 0 Average Latency(s): 0.0394035 Max latency(s): 3.00962 Min latency(s): 0.00449537

3 2

MDS host in OSD blacklist
by Frank Schilder 22 Mar '23

22 Mar '23

Hi all, we have an octopus v15.2.17 cluster and observe that one of our MDS hosts showed up in the OSD blacklist: # ceph osd blacklist ls 192.168.32.87:6801/3841823949 2023-03-22T10:08:02.589698+0100 192.168.32.87:6800/3841823949 2023-03-22T10:08:02.589698+0100 I see an MDS restart that might be related; see log snippets below. There are no clients running on this host, only OSDs and one MDS. What could be the reason for the blacklist entries? Thanks! Log snippets: Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:54 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: *** Caught signal (Aborted) ** Mar 21 10:07:54 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:54 ceph-23 journal: 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:54 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:54 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:54 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:54 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:54 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:54 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:54 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:54 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:54 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:54 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -1> 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:55 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:55 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: 0> 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:55 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:55 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:55 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:55 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:55 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -9999> 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:55 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:55 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -9998> 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:55 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:55 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:55 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:55 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:55 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: reraise_fatal: default handler for signal 6 didn't terminate the process? Mar 21 10:07:58 ceph-23 dockerd-current: time="2023-03-21T10:07:58.119559277+01:00" level=warning msg="040c1e98a0669204e0e98bdbcdde893f8acf63444f3827358e663a13a2869478 cleanup: failed to unmount secrets: invalid argument" Mar 21 10:07:58 ceph-23 kernel: overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior. Mar 21 10:07:58 ceph-23 kernel: overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior. Mar 21 10:07:58 ceph-23 journal: 118 get_config /opt/ceph-container/bin/config.static.sh Mar 21 10:07:58 ceph-23 journal: 5 start_mds /opt/ceph-container/bin/start_mds.sh Mar 21 10:07:58 ceph-23 journal: 120 main /opt/ceph-container/bin/entrypoint.sh Mar 21 10:07:58 ceph-23 journal: 2023-03-21 10:07:58 /opt/ceph-container/bin/entrypoint.sh: static: does not generate config Mar 21 10:07:58 ceph-23 journal: 58 start_mds /opt/ceph-container/bin/start_mds.sh Mar 21 10:07:58 ceph-23 journal: 120 main /opt/ceph-container/bin/entrypoint.sh Mar 21 10:07:58 ceph-23 journal: 2023-03-21 10:07:58 /opt/ceph-container/bin/entrypoint.sh: SUCCESS Mar 21 10:07:58 ceph-23 journal: starting mds.ceph-23 at ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

s3 compatible interface
by Marc 22 Mar '23

22 Mar '23

Anyone know of a s3 compatible interface that I can just run, and reads/writes files from a local file system and not from object storage?

13 18

S3 notification for backup
by Olivier Audry 22 Mar '23

22 Mar '23

Hello I would like to know if using the bucket notification system to http for backuping S3 bucket is a good move or not. Is someone already did it ? I got around 100to of little documents to backup and archive for legal purpose and dr each day. Many thanks for your help. oau

1 0

Moving From BlueJeans to Jitsi for Ceph meetings
by Mike Perez 22 Mar '23

22 Mar '23

Hi everyone, We have been using BlueJeans to meet and record some of our meetings that later get posted to our YouTube channel. Unfortunately, we have to figure out a new meeting platform due to Red Hat discontinuing BlueJeans by the end of this month. Google Meets is an option, but some users in other countries have trouble using Google's services. For some meetings, we have tried out Jitsi, and it works well, meets our requirements, and is free. Does anyone else have suggestions for another free meeting platform that provides recording capabilities? -- Mike Perez

6 6

Cephalocon Amsterdam 2023 Photographer Volunteer Help Needed
by Mike Perez 22 Mar '23

22 Mar '23

Hi everyone, To help with costs for Cephalocon Amsterdam 2023, we wanted to see if anyone would like to volunteer to help with photography for the event. A group of people would be ideal so that we have good coverage in the expo hall and sessions. If you're interested, please reply to me directly for more information. -- Mike Perez Community Manager Ceph Foundation

3 2

Re: Very slow backfilling/remapping of EC pool PGs
by Gauvain Pocentek 21 Mar '23

21 Mar '23

(adding back the list) On Tue, Mar 21, 2023 at 11:25 AM Joachim Kraftmayer < joachim.kraftmayer(a)clyso.com> wrote: > i added the questions and answers below. > > ___________________________________ > Best Regards, > Joachim Kraftmayer > CEO | Clyso GmbH > > Clyso GmbH > p: +49 89 21 55 23 91 2 > a: Loristraße 8 | 80335 München | Germany > w: https://clyso.com | e: joachim.kraftmayer(a)clyso.com > > We are hiring: https://www.clyso.com/jobs/ > --- > CEO: Dipl. Inf. (FH) Joachim Kraftmayer > Unternehmenssitz: Utting am Ammersee > Handelsregister beim Amtsgericht: Augsburg > Handelsregister-Nummer: HRB 25866 > USt. ID-Nr.: DE275430677 > > Am 21.03.23 um 11:14 schrieb Gauvain Pocentek: > > Hi Joachim, > > > On Tue, Mar 21, 2023 at 10:13 AM Joachim Kraftmayer < > joachim.kraftmayer(a)clyso.com> wrote: > >> Which Ceph version are you running, is mclock active? >> >> > We're using Quincy (17.2.5), upgraded step by step from Luminous if I > remember correctly. > > did you recreate the osds? if yes, at which version? > I actually don't remember all the history, but I think we added the HDD nodes while running Pacific. > > mlock seems active, set to high_client_ops profile. HDD OSDs have very > different settings for max capacity iops: > > osd.137 basic osd_mclock_max_capacity_iops_hdd > 929.763899 > osd.161 basic osd_mclock_max_capacity_iops_hdd > 4754.250946 > osd.222 basic osd_mclock_max_capacity_iops_hdd > 540.016984 > osd.281 basic osd_mclock_max_capacity_iops_hdd > 1029.193945 > osd.282 basic osd_mclock_max_capacity_iops_hdd > 1061.762870 > osd.283 basic osd_mclock_max_capacity_iops_hdd > 462.984562 > > We haven't set those explicitly, could they be the reason of the slow > recovery? > > i recommend to disable mclock for now, and yes we have seen slow recovery > caused by mclock. > Stupid question: how do you do that? I've looked through the docs but could only find information about changing the settings. > > > Bonus question: does ceph set that itself? > > yes and if you have a setup with HDD + SSD (db & wal) the discovery works > not in the right way. > Good to know! Gauvain > > Thanks! > > Gauvain > > > > >> Joachim >> >> ___________________________________ >> Clyso GmbH - Ceph Foundation Member >> >> Am 21.03.23 um 06:53 schrieb Gauvain Pocentek: >> > Hello all, >> > >> > We have an EC (4+2) pool for RGW data, with HDDs + SSDs for WAL/DB. This >> > pool has 9 servers with each 12 disks of 16TBs. About 10 days ago we >> lost a >> > server and we've removed its OSDs from the cluster. Ceph has started to >> > remap and backfill as expected, but the process has been getting slower >> and >> > slower. Today the recovery rate is around 12 MiB/s and 10 objects/s. All >> > the remaining unclean PGs are backfilling: >> > >> > data: >> > volumes: 1/1 healthy >> > pools: 14 pools, 14497 pgs >> > objects: 192.38M objects, 380 TiB >> > usage: 764 TiB used, 1.3 PiB / 2.1 PiB avail >> > pgs: 771559/1065561630 objects degraded (0.072%) >> > 1215899/1065561630 objects misplaced (0.114%) >> > 14428 active+clean >> > 50 active+undersized+degraded+remapped+backfilling >> > 18 active+remapped+backfilling >> > 1 active+clean+scrubbing+deep >> > >> > We've checked the health of the remaining servers, and everything looks >> > like (CPU/RAM/network/disks). >> > >> > Any hints on what could be happening? >> > >> > Thank you, >> > Gauvain >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

2 2

Re: Ceph Bluestore tweaks for Bcache
by Matthias Ferdinand 21 Mar '23

21 Mar '23

ceph version: 17.2.0 on Ubuntu 22.04 non-containerized ceph from Ubuntu repos cluster started on luminous I have been using bcache on filestore on rotating disks for many years without problems. Now converting OSDs to bluestore, there are some strange effects. If I create the bcache device, set its rotational flag to '1', then do ceph-volume lvm create ... --crush-device-class=hdd the OSD comes up with the right parameters and much improved latency compared to OSD directly on /dev/sdX. ceph osd metatdata ... shows "bluestore_bdev_type": "hdd", "rotational": "1" But after reboot, bcache rotational flag is set '0' again, and the OSD now comes up with "rotational": "0" Latency immediately starts to increase (and continually increases over the next days, possibly due to accumulating fragmention). These wrong settings stay in place even if I stop the OSD, set the bcache rotational flag to '1' again and restart the OSD. I have found no way to get back to the original settings other than destroying and recreating the OSD. I guess I am just not seeing something obvious, like from where these settings get pulled at OSD startup. I even created udev rules to set bcache rotational=1 at boot time, before any ceph daemon starts, but it did not help. Something running after these rules reset the bcache rotationl flags back to 0. Haven't found the culprit yet, but not sure if it even matters. Are these OSD settings (bluestore_bdev_type, rotational) persisted somewhere and can they be edited and pinned? Alternatively, can I manually set and persist the relevant bluestore tunables (per OSD / per device class) so as to make the bcache rotational flag irrelevant after the OSD is first created? Regards Matthias On Fri, Apr 08, 2022 at 03:05:38PM +0300, Igor Fedotov wrote: > Hi Frank, > > in fact this parameter impacts OSD behavior at both build-time and during > regular operationing. It simply substitutes hdd/ssd auto-detection with > manual specification. And hence relevant config parameters are applied. If > e.g. min_alloc_size is persistent after OSD creation - it wouldn't be > updated. But if specific setting allows at run-time - it would be altered. > > So the proper usage would definitely be manual ssd/hdd mode selection before > the first OSD creation and keeping it in that mode along the whole OSD > lifecycle. But technically one can change the mode at any arbitrary point in > time which would result in run-rime setting being out-of-sync with creation > ones. With some unclear side-effects.. > > Please also note that this setting was orignally intended mostly for > development/testing purposes not regular usage. Hence it's flexible but > rather unsafe if used improperly. > > > Thanks, > > Igor > > On 4/7/2022 2:40 PM, Frank Schilder wrote: > > Hi Richard and Igor, > > > > are these tweaks required at build-time (osd prepare) only or are they required for every restart? > > > > Is this setting "bluestore debug enforce settings=hdd" in the ceph config data base or set somewhere else? How does this work if deploying HDD- and SSD-OSDs at the same time? > > > > Ideally, all these tweaks should be applicable and settable at creation time only without affecting generic settings (that is, at the ceph-volume command line and not via config side effects). Otherwise it becomes really tedious to manage these. > > > > For example, would the following work-flow apply the correct settings *permanently* across restarts: > > > > 1) Prepare OSD on fresh HDD with ceph-volume lvm batch --prepare ... > > 2) Assign dm_cache to logical OSD volume created in step 1 > > 3) Start OSD, restart OSDs, boot server ... > > > > I would assume that the HDD settings are burned into the OSD in step 1 and will be used in all future (re-)starts without the need to do anything despite the device being detected as non-rotational after step 2. Is this assumption correct? > > > > Thanks and best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Richard Bade <hitrich(a)gmail.com> > > Sent: 06 April 2022 00:43:48 > > To: Igor Fedotov > > Cc: Ceph Users > > Subject: [Warning Possible spam] [ceph-users] Re: Ceph Bluestore tweaks for Bcache > > > > Just for completeness for anyone that is following this thread. Igor > > added that setting in Octopus, so unfortunately I am unable to use it > > as I am still on Nautilus. > > > > Thanks, > > Rich > > > > On Wed, 6 Apr 2022 at 10:01, Richard Bade <hitrich(a)gmail.com> wrote: > > > Thanks Igor for the tip. I'll see if I can use this to reduce the > > > number of tweaks I need. > > > > > > Rich > > > > > > On Tue, 5 Apr 2022 at 21:26, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > > > > Hi Richard, > > > > > > > > just FYI: one can use "bluestore debug enforce settings=hdd" config > > > > parameter to manually enforce HDD-related settings for a BlueStore > > > > > > > > > > > > Thanks, > > > > > > > > Igor > > > > > > > > On 4/5/2022 1:07 AM, Richard Bade wrote: > > > > > Hi Everyone, > > > > > I just wanted to share a discovery I made about running bluestore on > > > > > top of Bcache in case anyone else is doing this or considering it. > > > > > We've run Bcache under Filestore for a long time with good results but > > > > > recently rebuilt all the osds on bluestore. This caused some > > > > > degradation in performance that I couldn't quite put my finger on. > > > > > Bluestore osds have some smarts where they detect the disk type. > > > > > Unfortunately in the case of Bcache it detects as SSD, when in fact > > > > > the HDD parameters are better suited. > > > > > I changed the following parameters to match the HDD default values and > > > > > immediately saw my average osd latency during normal workload drop > > > > > from 6ms to 2ms. Peak performance didn't change really, but a test > > > > > machine that I have running a constant iops workload was much more > > > > > stable as was the average latency. > > > > > Performance has returned to Filestore or better levels. > > > > > Here are the parameters. > > > > > > > > > > ; Make sure that we use values appropriate for HDD not SSD - Bcache > > > > > gets detected as SSD > > > > > bluestore_prefer_deferred_size = 32768 > > > > > bluestore_compression_max_blob_size = 524288 > > > > > bluestore_deferred_batch_ops = 64 > > > > > bluestore_max_blob_size = 524288 > > > > > bluestore_min_alloc_size = 65536 > > > > > bluestore_throttle_cost_per_io = 670000 > > > > > > > > > > ; Try to improve responsiveness when some disks are fully utilised > > > > > osd_op_queue = wpq > > > > > osd_op_queue_cut_off = high > > > > > > > > > > Hopefully someone else finds this useful. > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > > > Igor Fedotov > > > > Ceph Lead Developer > > > > > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > > > > > croit GmbH, Freseniusstr. 31h, 81247 Munich > > > > CEO: Martin Verges - VAT-ID: DE310638492 > > > > Com. register: Amtsgericht Munich HRB 231263 > > > > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Igor Fedotov > Ceph Lead Developer > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Ceph Days India 2023 - Call for proposals
by Gaurav Sitlani 21 Mar '23

21 Mar '23

Hello Cephers, We're happy to share that we're organizing Ceph Day India on 5th May 2023 this year. The event is now sold out! If you missed getting a ticket, consider submitting the CFP and we'll provide a ticket if accepted. https://ceph.io/en/community/events/2023/ceph-days-india/ Please reach out to us if you need any help regarding the submissions. Thanks and regards, Gaurav Sitlani Ceph Community Ambassador

1 0