- ceph-users - lists.ceph.io

Quincy osd bench in order to define osd_mclock_max_capacity_iops_[hdd|ssd]
by Rafael Diaz Maurin 30 Jun '23

30 Jun '23

Hello, I've just upgraded a Pacific cluster into Quincy, and all my osd have the low value osd_mclock_max_capacity_iops_hdd : 315.000000. The manuel does not explain how to benchmark the OSD with fio or ceph bench with good options. Can someone have the good ceph bench options or fio options in order to configure osd_mclock_max_capacity_iops_hdd for each osd ? I ran this bench various times on the same OSD (class hdd) and I obtain different results. ceph tell ${osd} cache drop ceph tell ${osd} bench 12288000 4096 4194304 100 example : osd.21 (hdd): osd_mclock_max_capacity_iops_hdd = 315.000000 bench 1 : 3006.2271379745534 bench 2 : 819.503206458996 bench 3 : 946.5406320134085 How can I succeed in getting the good values for the osd_mclock_max_capacity_iops_[hdd|ssd] options ? Thank you for your help, Rafael

2 2

ceph-fuse crash
by hakesudu＠gmail.com 30 Jun '23

30 Jun '23

Hi, I've deployed a ceph-quincy for HPC. Recently, I always encounter the problem of ceph-fuse crash kernel version is 4.18.0-348.el8.0.2.x86_64 here is part of ceph-fuse log: -59> 2023-06-28T09:51:00.452+0800 155546ff7700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -58> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 ll_opendir 0x10003e1408d.head -57> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 may_open 0x1554e79123d0 = 0 -56> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 ll_opendir 0x10003e1408d.head = 0 (0x155328079380) -55> 2023-06-28T09:51:00.453+0800 1555473f9700 3 client.159239 seekdir(0x155328079380, 0) -54> 2023-06-28T09:51:00.452+0800 155546bf5700 5 client.159239 put_cap_ref dropped last FILE_CACHE ref on 0x20004a6e548.head(faked_ino=0 nref=14 ll_ref=2 cap_refs={1024=0,2048=1} open={1=1} mode=100644 size=5626/0 nlink=1 btime=2023-06-05T14:38:36.471178+0800 mtime=2023-06-05T14:38:36.471178+0800 ctime=2023-06-05T14:38:36.471178+0800 change_attr=1 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x20004a6e548 ts 0/0 objects 1 dirty_or_tx 0] 0x1554e7902300) -53> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_read 0x15531806b970 0~8192 = 5626 -52> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 may_lookup 0x155420004840 = 0 -51> 2023-06-28T09:51:00.453+0800 1554a5dee700 3 client.159239 ll_lookup 0x10003e1406f.head html -50> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -49> 2023-06-28T09:51:00.453+0800 1554a5dee700 3 client.159239 may_lookup 0x1554c1a89e40 = 0 -48> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_flush 0x15531806b970 0x20004a6e548 -47> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x100019b7a43.head envs -46> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 may_lookup 0x155420343eb0 = 0 -45> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -44> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_release (fh)0x15531806b970 0x20004a6e548 -43> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 seekdir(0x15544c054010, 1152360438801891331) -42> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 seekdir(0x15544c029ee0, 1152690945930559491) -41> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_releasedir 0x15544c029ee0 -40> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x20004a66459.head tests -39> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100040bd04e.head att5410-w -> 0 (200047cbc02) -38> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -37> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -36> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -> 0 (100040bcf3a) -35> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x155533c970f0 = 0 -34> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x20004a66459.head tests -> 0 (20004a6e3e0) -33> 2023-06-28T09:51:00.453+0800 155545bed700 3 client.159239 ll_releasedir 0x15544c054010 -32> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_getattr 0x200047cbc02.head = 0 -31> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -30> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x155420004840 = 0 -29> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -28> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_lookup 0x100040bcf3a.head lib -27> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 may_lookup 0x1554d7d1d240 = 0 -26> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_lookup 0x100040bcf3a.head lib -> 0 (100040bd018) -25> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x20004a6e3e0.head test_trainer -24> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 may_lookup 0x155531e5a6c0 = 0 -23> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -22> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 may_lookup 0x155420004840 = 0 -21> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -20> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100019b7a43.head envs -19> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 may_lookup 0x155420343eb0 = 0 -18> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -17> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x100019b7a43.head envs -16> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 may_lookup 0x155420343eb0 = 0 -15> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -14> 2023-06-28T09:51:00.453+0800 155545bed700 3 client.159239 ll_lookup 0x100040bd018.head terminfo -13> 2023-06-28T09:51:00.453+0800 1555475fa700 3 client.159239 ll_lookup 0x1000383ca0a.head train.py -> 0 (1000383caf7) -12> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -11> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 may_lookup 0x155420004840 = 0 -10> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -9> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -8> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -7> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -> 0 (100040bcf3a) -6> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x100019b7a43.head envs -5> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_getattr 0x1000383caf7.head = 0 -4> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_lookup 0x200035de5d8.head llmzoo -3> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -2> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_lookup 0x200035de5d8.head llmzoo -> 0 (10003e11a9e) -1> 2023-06-28T09:51:00.453+0800 15554c646700 3 client.159239 ll_lookup 0x10003e11a9e.head lib 0> 2023-06-28T09:51:00.458+0800 1554a77fb700 -1 *** Caught signal (Segmentation fault) ** in thread 1554a77fb700 thread_name:ceph-fuse ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x1555535eece0] 2: (Client::_readdir_cache_cb(dir_result_t*, int (*)(void*, dirent*, ceph_statx*, long, Inode*), void*, int, bool)+0x2f4) [0x555555647d64] 3: (Client::readdir_r_cb(dir_result_t*, int (*)(void*, dirent*, ceph_statx*, long, Inode*), void*, unsigned int, unsigned int, bool)+0xae7) [0x55555564cd37] 4: ceph-fuse(+0xadbf8) [0x555555601bf8] 5: /lib64/libfuse.so.2(+0x16706) [0x1555550fd706] 6: /lib64/libfuse.so.2(+0x17868) [0x1555550fe868] 7: /lib64/libfuse.so.2(+0x14440) [0x1555550fb440] 8: /lib64/libpthread.so.0(+0x81cf) [0x1555535e41cf] 9: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 1/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_cleaner 0/ 5 seastore_lba 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 alienstore 1/ 5 mclock 1/ 5 ceph_exporter -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 1554947e3700 / ceph-fuse 155494fe7700 / ceph-fuse 1554975fa700 / ceph-fuse 1554a57eb700 / ceph-fuse 1554a59ec700 / ceph-fuse 1554a5dee700 / ceph-fuse 1554a5fef700 / ceph-fuse 1554a61f0700 / 1554a63f1700 / ceph-fuse 1554a65f2700 / ceph-fuse 1554a67f3700 / ceph-fuse 1554a6bf5700 / ceph-fuse 1554a6df6700 / ceph-fuse 1554a71f8700 / ceph-fuse 1554a73f9700 / ceph-fuse 1554a75fa700 / 1554a77fb700 / ceph-fuse 1554a7dfe700 / ceph-fuse 1554a7fff700 / ceph-fuse 1555452da700 / ceph-fuse 1555455ea700 / ceph-fuse 1555459ec700 / ceph-fuse 155545bed700 / ceph-fuse 155545fef700 / ceph-fuse 1555465f2700 / ceph-fuse 1555469f4700 / ceph-fuse 155546bf5700 / ceph-fuse 155546df6700 / ceph-fuse 155546ff7700 / ceph-fuse 1555471f8700 / ceph-fuse 1555473f9700 / ceph-fuse 1555475fa700 / ceph-fuse 1555477fb700 / 155547dfe700 / ceph-fuse 155547fff700 / ceph-fuse 15554c244700 / ceph-fuse 15554c646700 / ceph-fuse 15554c847700 / ceph-fuse 15554cc49700 / ceph-fuse 15554ce4a700 / ceph-fuse 15554da50700 / ms_dispatch max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2023-06-28T01:51:00.459615Z_3ddbaa44-d8cd-437b-908b-c3772520c7a6/log --- end dump of recent events --- Has anyone encountered this kind of problem?

2 1

Quincy osd bench in order to define osd_mclock_max_capacity_iops_[hdd|ssd]
by Rafael Diaz Maurin 30 Jun '23

30 Jun '23

Hello, I've just upgraded a Pacific cluster into Quincy, and all my osd have the low value osd_mclock_max_capacity_iops_hdd : 315.000000. The manuel does not explain how to benchmark the OSD with fio or ceph bench with good options. Can someone have the good ceph bench options or fio options in order to configure osd_mclock_max_capacity_iops_hdd for each osd ? I ran this bench various times on the same OSD (class hdd) and I obtain different results. ceph tell ${osd} cache drop ceph tell ${osd} bench 12288000 4096 4194304 100 example : osd.21 (hdd): osd_mclock_max_capacity_iops_hdd = 315.000000 bench 1 : 3006.2271379745534 bench 2 : 819.503206458996 bench 3 : 946.5406320134085 How can I succeed in getting the good values for the osd_mclock_max_capacity_iops_[hdd|ssd] options ? Thank you for your help, Rafael

1 0

warning: CEPHADM_APPLY_SPEC_FAIL
by Adiga, Anantha 29 Jun '23

29 Jun '23

Hi, I am not finding any reference to clear this warning AND stop the service. See below After creating OSD with iops_optimized option, this WARN mesg appear. Ceph 17.2.6 [cid:image001.png@01D9AAA5.8639A1F0] 6/29/23 4:10:45 PM [WRN] Health check failed: Failed to apply 1 service(s): osd.iops_optimized (CEPHADM_APPLY_SPEC_FAIL) 6/29/23 4:10:45 PM [ERR] Failed to apply osd.iops_optimized spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: iops_optimized service_name: osd.iops_optimized placement: host_pattern: '*' spec: data_devices: rotational: 0 filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/mon.fl31ca104ja0203/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e -e NODE_NAME=fl31ca104ja0203 -e #:/var/log/ceph# grep optimi cephadm.log cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized', '--image', 'quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e', 'ceph-volume', '--fsid', 'd0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/nvme10n1', '/dev/nvme11n1', '/dev/nvme12n1', '/dev/nvme13n1', '/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1', '/dev/nvme5n1', '/dev/nvme6n1', '/dev/nvme7n1', '/dev/nvme8n1', '/dev/nvme9n1', '--yes', '--no-systemd'] 2023-06-29 23:06:28,340 7fc2668a7740 INFO Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e -e NODE_NAME=fl31ca104ja0202 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/run/ceph:z -v /var/log/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/log/ceph:z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1v09i0jx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphy3pnh46:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e lvm batch --no-auto /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 --yes --no-systemd #:/var/log/ceph# grep optimi cephadm.log cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized', '--image', 'quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e', 'ceph-volume', '--fsid', 'd0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/nvme10n1', '/dev/nvme11n1', '/dev/nvme12n1', '/dev/nvme13n1', '/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1', '/dev/nvme5n1', '/dev/nvme6n1', '/dev/nvme7n1', '/dev/nvme8n1', '/dev/nvme9n1', '--yes', '--no-systemd'] 2023-06-29 23:06:28,340 7fc2668a7740 INFO Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> -e NODE_NAME=fl31ca104ja0202 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/run/ceph:z -v /var/log/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/log/ceph:z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1v09i0jx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphy3pnh46:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> lvm batch --no-auto /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 --yes --no-systemd The warning message clears on its on for a few seconds..ceph health status foes green, but comes back. I am not finding any reference to clear this msg AND not able to stop the service. /#ceph orch ls node-exporter ?:9100 4/4 8m ago 10w * osd 33 8m ago - <unmanaged> osd.iops_optimized 0 - 2d * /# ceph orch stop --service_name osd.iops_optimized Error EINVAL: No daemons exist under service name "osd.iops_optimized". View currently running services using "ceph orch ls" /# Thank you, Anantha

1 1

1 pg inconsistent and does not recover
by Niklas Hambüchen 29 Jun '23

29 Jun '23

Hi, I have a 3x-replicated pool with Ceph 12.2.7. One HDD broke, its OSD "2" was automatically marked as "out", the disk was physically replaced by a new one, and that added back in. Now `ceph health detail` continues to permanently show: [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.87 is active+clean+inconsistent, acting [33,2,20] What exactly is wrong here? Why can Ceph not fix the issue? With BlueStore I have checksums, on two unbroken disks, so what remaining inconsistency can there be? The suggested command in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-d… does not work: # rados list-inconsistent-obj 2.87 No scrub information available for pg 2.87 error 2: (2) No such file or directory Further, I find the documentation in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informati… extremely unclear. It says > In the case of replicated pools, recovery is beyond the scope of pg repair. while many people on the Internet suggest that `ceph pg repair` might fix the issue. Yet again others claim that Ceph will fix the issue itself. I am hesitant to run "ceph pg repair" without understanding what the problem is and what exactly this will do. I have already reported the "error 2" and the documentation in issue https://tracker.ceph.com/issues/61739 but not received a reply yet, and my cluster stays "inconsistent". How can this be fixed? I would appreciate any help!

5 15

device class for nvme disk is ssd
by Boris Behrens 29 Jun '23

29 Jun '23

Hi, is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally? I've added them via ceph-volume lvm create --bluestore --data /dev/nvme2n1 and they only show up as ssd root@a0423f621aaa:~# ceph osd metadata osd.0 { "id": 0, "arch": "x86_64", ... "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-2", "bluestore_bdev_devices": "nvme0n1", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_partition_path": "/dev/dm-2", "bluestore_bdev_rotational": "0", "bluestore_bdev_size": "1920378863616", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "ssd", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "ceph_version_short": "16.2.13", "ceph_version_when_created": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "cpu": "Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz", "created_at": "2023-06-20T14:03:35.167741Z", "default_device_class": "ssd", "device_ids": "nvme0n1=SAMSUNG_MZQLB1T9HAJR-00007_S439NF0M506164", "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "devices": "nvme0n1", "distro": "ubuntu", "distro_description": "Ubuntu 20.04.6 LTS", "distro_version": "20.04", ... "journal_rotational": "0", "kernel_description": "#169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023", "kernel_version": "5.4.0-152-generic", "mem_swap_kb": "0", "mem_total_kb": "196668116", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_node": "0", "objectstore_numa_nodes": "0", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "osdspec_affinity": "", "rotational": "0" } Cheers Boris

5 6

CLT Meeting Notes June 28th, 2023
by Adam King 28 Jun '23

28 Jun '23

Reef RC linking failure on Alpine Linux. Do we worry about that? 1. https://tracker.ceph.com/issues/61718 2. Nice to fix, but not a requirement 3. If there are patches available, we should accept them, but probably don't put too much work into it currently debian bullseye build failure on reef rc: 1. https://tracker.ceph.com/issues/61845 2. want to fix before final release clean-up in AuthMonitor – CephFS and core are fine. Any other component interested in? 1. https://github.com/ceph/ceph/pull/52008#issuecomment-1606581139 2. Already tested for rados and cephfs 3. No other components requesting testing Reef rc v18.1.2 in progress 1. Next rc build in progress 2. build issue to be looked at 1. Failure on a jammy arm build, not a platform we test on meaningfully 2. Think this is an infrastructure issue 3. Generally, this shouldn't be a release blocker 4. Priority of arm builds might rise in the future though 5. investigate today, if we can't figure it out quickly, publish rc with a known issue 6. NOTE: Rook expects arm builds to be present 3. Would like to release next rc later this week if things work out 4. Would also like to upgrade lrc CDS agenda https://pad.ceph.com/p/cds-squid 1. leads should add topics 2. plan is for this to happen week of July 17th mempool monitoring in teuthology tests https://github.com/ceph/ceph/pull/51853 1. Just an FYI 2. ceph task will now have ability to dump memtools 3. might add a bit of delay to how long tests take 4. expected to be merged soon 5. will follow up in performance meeting iSCSI packages old/not signed -- want to fix before final release 1. https://tracker.ceph.com/issues/57388 2. tcmu-runner, since containerization of ceph, is being pulled from our build system 3. This tcmu-runner package is not signed 4. ceph-iscsi package is signed, but outdated (seems to be because this is the newest one that is signed and pushed to download.ceph.com) 5. someone with access to tools to sign the packages would have to help fix this 6. been like this for a long time and nobody noticed 7. only ceph-iscsi package, not tcmu-runner, is distributed through download.ceph.com 8. getting updated ceph-iscsi package on download.ceph.com should be done before reef release 9. tcmu-runner inside the container being unsigned is not as big of a deal (was this way in quincy/pacific as well)

1 0

Re: [multisite] period update and zonegroup
by Yixin Jin 28 Jun '23

28 Jun '23

Mystery solved. I loaded the zonegroup up with 100K hostnanes. That caused the period to fail to decode when other zones try to pick up the new period. Apparently, there is a size limit on how big zonegroup info can be but implicitly. Regards, Yixin Sent from Yahoo Mail on Android

1 0

[multisite] period update and zonegroup
by Yixin Jin 28 Jun '23

28 Jun '23

Hi folks, Has anyone experienced the situation where "radosgw-admin period get" shows the latest zonegroup information but "radosgw-admin zonegroup get" shows the old one? I am now seeing this with v17.2.6. I used to play with v17.2.5 but never saw this situation. I tried to do various "period update --commit" and "period pull" to see if I could get the update happen. No avail. Any ideas? Thanks,Yixin

1 0

cephadm, new OSD
by Shashi Dahal 28 Jun '23

28 Jun '23

Hi, I added new OSD on ceph servers. ( orch is cephadm) Its recognized as osd.12 and osd.13 ceph pg dump shows no pg are there in osd 12 and 13 .. they are all empty .. ceph osd tree shows them that they are up. ceph osd df shows them to be all 0 in reweight and size etc. ceph orch device ls shows the status as "Insufficient space (<10 extents) on vgs, LVM detected, locked" However, this is the same status shown in all the existing osds ( working ones also ) I tried ceph osd crush reweight osd.12 1 reweighted item id 12 name 'osd.12' to 1 in crush map but the reweight is still 0 . and these disks are not in the cluster yet. How to add these disks so that they are part of the cluster ? Thanks

2 1