January 2023 - ceph-users

deploying Ceph using FQDN for MON / MDS Services

by Lokendra Rathour

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

1 year

7
20
0 0

ceph/daemon stable tag

by Jonas Nemeikšis

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

1 year

2
6
0 0

cephadm cluster move /var/lib/docker to separate device fails

by Karsten Nielsen

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

1 year

2
1
0 0

Stuck OSD service specification - can't remove

by David Orman

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

1 year, 1 month

3
4
0 0

OSD upgrade problem nautilus->octopus - snap_mapper upgrade stuck

by Jan Pekař - Imatic

Hi all, I have problem upgrading nautilus to octopus on my OSD. Upgrade mon and mgr was OK and first OSD stuck on 2023-01-12T09:25:54.122+0100 7f49ff3eae00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) and there were no activity after that for more than 48 hours. No disk activity. I restarted OSD many times and nothing changed. It is old, filestore OSD based on XFS filesystem. Is upgrade to snap mapper 2 reliable? What is OSD waiting for? Can I start OSD without upgrade and get cluster healthy with old snap structure? Or should I skip octopus upgrade and go to pacific directly (some bug backport is missing?). Thank you for help, I'm sending some logs below.. Log shows 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 ceph version 15.2.17 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable), process ceph-osd, pid 2566563 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 pidfile_write: ignore empty --pid-file 2023-01-09T19:12:49.499+0100 7f41f60f1e00 -1 missing 'type' file, inferring filestore from current/ dir 2023-01-09T19:12:49.531+0100 7f41f60f1e00 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2023-01-09T19:12:49.531+0100 7f41f60f1e00 -1 Falling back to public interface 2023-01-09T19:12:49.871+0100 7f41f60f1e00 0 load: jerasure load: lrc load: isa 2023-01-09T19:12:49.875+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice() is disabled via 'filestore splice' config option 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf 2023-01-09T19:12:50.015+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) start omap initiation 2023-01-09T19:12:50.079+0100 7f41f60f1e00 1 leveldb: Recovering log #165531 2023-01-09T19:12:50.083+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: started 2023-01-09T19:12:50.235+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: 1598 bytes OK 2023-01-09T19:12:50.583+0100 7f41f60f1e00 1 leveldb: Delete type=0 #165531 2023-01-09T19:12:50.615+0100 7f41f60f1e00 1 leveldb: Delete type=3 #165529 2023-01-09T19:12:51.339+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) mount(1861): enabling WRITEAHEAD journal mode: checkpoint is not enabled 2023-01-09T19:12:51.379+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.931+0100 7f41f60f1e00 -1 journal do_read_entry(243675136): bad header magic 2023-01-09T19:12:51.939+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.943+0100 7f41f60f1e00 1 filestore(/var/lib/ceph/osd/ceph-0) upgrade(1466) 2023-01-09T19:12:52.015+0100 7f41f60f1e00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) lsof shows COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ceph-osd 225860 ceph cwd DIR 9,127 4096 2 / ceph-osd 225860 ceph rtd DIR 9,127 4096 2 / ceph-osd 225860 ceph txt REG 9,127 31762544 5021 /usr/bin/ceph-osd ceph-osd 225860 ceph mem REG 8,70 2147237 68104224 /var/lib/ceph/osd/ceph-0/current/omap/165546.ldb ceph-osd 225860 ceph mem REG 8,70 2147792 68104190 /var/lib/ceph/osd/ceph-0/current/omap/165545.ldb ceph-osd 225860 ceph mem REG 8,70 2147689 68104240 /var/lib/ceph/osd/ceph-0/current/omap/165466.ldb ceph-osd 225860 ceph mem REG 8,70 2142721 68102679 /var/lib/ceph/osd/ceph-0/current/omap/165544.ldb ceph-osd 225860 ceph mem REG 8,70 2142677 68104239 /var/lib/ceph/osd/ceph-0/current/omap/165465.ldb ceph-osd 225860 ceph mem REG 8,70 2144979 68078254 /var/lib/ceph/osd/ceph-0/current/omap/165543.ldb ceph-osd 225860 ceph mem REG 8,70 2143705 68163491 /var/lib/ceph/osd/ceph-0/current/omap/165526.ldb ceph-osd 225860 ceph mem REG 8,70 2141468 68163492 /var/lib/ceph/osd/ceph-0/current/omap/165527.ldb ceph-osd 225860 ceph mem REG 8,70 145986 68018644 /var/lib/ceph/osd/ceph-0/current/omap/165541.ldb ceph-osd 225860 ceph mem REG 8,70 2143434 68163490 /var/lib/ceph/osd/ceph-0/current/omap/165525.ldb ceph-osd 225860 ceph mem REG 8,70 2136002 68122351 /var/lib/ceph/osd/ceph-0/current/omap/165472.ldb ceph-osd 225860 ceph mem REG 8,70 1965262 68119647 /var/lib/ceph/osd/ceph-0/current/omap/165467.ldb ceph-osd 225860 ceph mem REG 8,70 2145206 68104229 /var/lib/ceph/osd/ceph-0/current/omap/165464.ldb ceph-osd 225860 ceph mem REG 8,70 61600 68002130 /var/lib/ceph/osd/ceph-0/current/omap/165536.ldb ceph-osd 225860 ceph mem REG 8,70 352689 67945734 /var/lib/ceph/osd/ceph-0/current/omap/165530.ldb ceph-osd 225860 ceph mem REG 8,70 82500 67480519 /var/lib/ceph/osd/ceph-0/current/omap/165539.ldb ceph-osd 225860 ceph mem REG 9,127 189200 2878 /usr/lib/ceph/erasure-code/libec_isa.so ceph-osd 225860 ceph mem REG 9,127 2021136 3033 /usr/lib/ceph/erasure-code/libec_lrc.so ceph-osd 225860 ceph mem REG 9,127 360544 2942 /usr/lib/ceph/erasure-code/libec_jerasure.so ceph-osd 225860 ceph mem REG 9,127 55792 15227 /lib/x86_64-linux-gnu/libnss_files-2.28.so ceph-osd 225860 ceph mem REG 9,127 22368 133997 /usr/lib/x86_64-linux-gnu/liburcu-common.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 42976 133996 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 51128 149821 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0 ceph-osd 225860 ceph mem REG 8,70 33977 68116674 /var/lib/ceph/osd/ceph-0/current/omap/165547.ldb ceph-osd 225860 ceph DEL REG 0,19 64261192 /[aio] ceph-osd 225860 ceph DEL REG 0,19 64261191 /[aio] ceph-osd 225860 ceph mem REG 9,127 158400 2523 /lib/x86_64-linux-gnu/liblzma.so.5.2.4 ceph-osd 225860 ceph mem REG 9,127 135064 12635 /lib/x86_64-linux-gnu/libnl-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 488272 133248 /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 35808 15263 /lib/x86_64-linux-gnu/librt-2.28.so ceph-osd 225860 ceph mem REG 9,127 55512 133042 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1 ceph-osd 225860 ceph mem REG 9,127 30776 16463 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 ceph-osd 225860 ceph mem REG 9,127 1820400 15070 /lib/x86_64-linux-gnu/libc-2.28.so ceph-osd 225860 ceph mem REG 9,127 100712 7687 /lib/x86_64-linux-gnu/libgcc_s.so.1 ceph-osd 225860 ceph mem REG 9,127 1579448 15202 /lib/x86_64-linux-gnu/libm-2.28.so ceph-osd 225860 ceph mem REG 9,127 1570256 133576 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 ceph-osd 225860 ceph mem REG 9,127 112800 133289 /usr/lib/x86_64-linux-gnu/librdmacm.so.1.1.22.1 ceph-osd 225860 ceph mem REG 9,127 104496 133287 /usr/lib/x86_64-linux-gnu/libibverbs.so.1.5.22.1 ceph-osd 225860 ceph mem REG 9,127 149704 5527 /lib/x86_64-linux-gnu/libudev.so.1.6.13 ceph-osd 225860 ceph mem REG 9,127 146968 15251 /lib/x86_64-linux-gnu/libpthread-2.28.so ceph-osd 225860 ceph mem REG 9,127 3044224 135371 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 ceph-osd 225860 ceph mem REG 9,127 14592 15128 /lib/x86_64-linux-gnu/libdl-2.28.so ceph-osd 225860 ceph mem REG 9,127 93000 15255 /lib/x86_64-linux-gnu/libresolv-2.28.so ceph-osd 225860 ceph mem REG 9,127 301288 134118 /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3 ceph-osd 225860 ceph mem REG 9,127 14184 134682 /usr/lib/x86_64-linux-gnu/libaio.so.1.0.1 ceph-osd 225860 ceph mem REG 9,127 117184 2738 /lib/x86_64-linux-gnu/libz.so.1.2.11 ceph-osd 225860 ceph mem REG 9,127 121184 136027 /usr/lib/x86_64-linux-gnu/liblz4.so.1.8.3 ceph-osd 225860 ceph mem REG 9,127 31104 133334 /usr/lib/x86_64-linux-gnu/libsnappy.so.1.1.7 ceph-osd 225860 ceph mem REG 9,127 378160 133086 /usr/lib/x86_64-linux-gnu/libleveldb.so.1d.20 ceph-osd 225860 ceph mem REG 9,127 256120 7650 /lib/x86_64-linux-gnu/libfuse.so.2.9.9 ceph-osd 225860 ceph mem REG 9,127 343008 16890 /lib/x86_64-linux-gnu/libblkid.so.1.1.0 ceph-osd 225860 ceph mem REG 8,70 1546 68022625 /var/lib/ceph/osd/ceph-0/current/omap/165542.ldb ceph-osd 225860 ceph mem REG 8,70 1598 67109142 /var/lib/ceph/osd/ceph-0/current/omap/165533.ldb ceph-osd 225860 ceph mem REG 9,127 35112 133884 /usr/lib/x86_64-linux-gnu/liburcu-bp.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 165632 15000 /lib/x86_64-linux-gnu/ld-2.28.so ceph-osd 225860 ceph 0r CHR 1,3 0t0 6 /dev/null ceph-osd 225860 ceph 1u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 2u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 3u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 4r FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 5w FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 6u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 7r FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 8w FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 9u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 10r FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 11w FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 12w REG 9,127 8765 3314 /var/log/ceph/ceph-osd.0.log ceph-osd 225860 ceph 13r FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 14w FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 15u unix 0x0000000097fdc46a 0t0 64259951 /var/run/ceph/ceph-osd.0.asok type=STREAM ceph-osd 225860 ceph 24r FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 25w FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 26r FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 27w FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 28uW REG 8,70 37 21 /var/lib/ceph/osd/ceph-0/fsid ceph-osd 225860 ceph 29r DIR 8,70 262 16 /var/lib/ceph/osd/ceph-0 ceph-osd 225860 ceph 30r DIR 8,70 24576 27 /var/lib/ceph/osd/ceph-0/current ceph-osd 225860 ceph 31u REG 8,70 10 28 /var/lib/ceph/osd/ceph-0/current/commit_op_seq ceph-osd 225860 ceph 32uW REG 8,70 0 67108882 /var/lib/ceph/osd/ceph-0/current/omap/LOCK ceph-osd 225860 ceph 33w REG 8,70 813568 68035819 /var/lib/ceph/osd/ceph-0/current/omap/165540.log ceph-osd 225860 ceph 34w REG 8,70 32068 68072533 /var/lib/ceph/osd/ceph-0/current/omap/MANIFEST-165538 ceph-osd 225860 ceph 36u BLK 8,65 0t0 422 /dev/sde1 ceph-osd 225860 ceph 37u REG 8,70 538 134217745 /var/lib/ceph/osd/ceph-0/current/meta/DIR_E/DIR_D/DIR_C/osd\\usuperblock__0_23C2FCDE__none ceph-osd 225860 ceph 38u REG 8,70 22453 1075330317 /var/lib/ceph/osd/ceph-0/current/meta/DIR_4/DIR_B/DIR_4/osdmap.126556__0_DF0524B4__none strace shows strace: Process 225860 attached futex(0x555f3e1c05c8, FUTEX_WAIT_PRIVATE, 0, NULL -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 https://www.imatic.cz | +420326555326 ============ --

1 year, 1 month

3
8
0 0

MDS stuck in "up:replay"

by Thomas Widhalm

Hi, I'm really lost with my Ceph system. I built a small cluster for home usage which has two uses for me: I want to replace an old NAS and I want to learn about Ceph so that I have hands-on experience. We're using it in our company but I need some real-life experience without risking any company or customers data. That's my preferred way of learning. The cluster consists of 3 Raspberry Pis plus a few VMs running on Proxmox. I'm not using Proxmox' built in Ceph because I want to focus on Ceph and not just use it as a preconfigured tool. All hosts are running Fedora (x86_64 and arm64) and during an Upgrade from F36 to F37 my cluster suddenly showed all PGs as unavailable. I worked nearly a week to get it back online and I learned a lot about Ceph management and recovery. The cluster is back but I still can't access my data. Maybe you can help me? Here are my versions: [ceph: root@ceph04 /]# ceph versions { "mon": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 5 }, "mds": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 4 }, "overall": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 15 } } Here's MDS status output of one MDS: [ceph: root@ceph04 /]# ceph tell mds.mds01.ceph05.pqxmvt status 2023-01-14T15:30:28.607+0000 7fb9e17fa700 0 client.60986454 ms_handle_reset on v2:192.168.23.65:6800/2680651694 2023-01-14T15:30:28.640+0000 7fb9e17fa700 0 client.60986460 ms_handle_reset on v2:192.168.23.65:6800/2680651694 { "cluster_fsid": "ff6e50de-ed72-11ec-881c-dca6325c2cc4", "whoami": 0, "id": 60984167, "want_state": "up:replay", "state": "up:replay", "fs_name": "cephfs", "replay_status": { "journal_read_pos": 0, "journal_write_pos": 0, "journal_expire_pos": 0, "num_events": 0, "num_segments": 0 }, "rank_uptime": 1127.54018615, "mdsmap_epoch": 98056, "osdmap_epoch": 12362, "osdmap_epoch_barrier": 0, "uptime": 1127.957307273 } It's staying like that for days now. If there was a counter moving, I just would wait but it doesn't change anything and alle stats says, the MDS aren't working at all. The symptom I have is that Dashboard and all other tools I use say, it's more or less ok. (Some old messages about failed daemons and scrubbing aside). But I can't mount anything. When I try to start a VM that's on RDS I just get a timeout. And when I try to mount a CephFS, mount just hangs forever. Whatever command I give MDS or journal, it just hangs. The only thing I could do, was take all CephFS offline, kill the MDS's and do a "ceph fs reset <fs name> --yes-i-really-mean-it". After that I rebooted all nodes, just to be sure but I still have no access to data. Could you please help me? I'm kinda desperate. If you need any more information, just let me know. Cheers, Thomas -- Thomas Widhalm Lead Systems Engineer NETWAYS Professional Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB34510 https://www.netways.de | thomas.widhalm(a)netways.de ** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** New at NWS: Managed Database - https://nws.netways.de/managed-database ** ** NETWAYS Web Services - https://nws.netways.de **

1 year, 1 month

7
23
0 0

kernel client osdc ops stuck and mds slow reqs

by Dan van der Ster

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

1 year, 1 month

4
12
0 0

Problem with IO after renaming File System .data pool

by murilo＠evocorp.com.br

Good morning everyone. On this Thursday night we went through an accident, where they accidentally renamed the .data pool of a File System making it instantly inaccessible, when renaming it again to the correct name it was possible to mount and list the files, but could not read or write. When trying to write, the FS returned as Read Only, when trying to read it returned Operation not allowed. After a period of breaking my head I tried to mount with the ADMIN user and everything worked correctly. I tried to remove the authentication of the current user through `ceph auth rm`, I created a new user through `ceph fs authorize <fs_name> client.<user> / rw` and it continued the same way, I also tried to recreate it through `ceph auth get-or-create` and nothing different happened, it stayed exactly the same. After setting `allow *` in mon, mds and osd I was able to mount, read and write again with the new user. I can understand why the File System stopped after renaming the pool, what I don't understand is why users are unable to perform operations on FS even with RW or any other user created. What could have happened behind the scenes to not be able to perform IO even with the correct permissions? Or did I apply incorrect permissions that caused this problem? Right now everything is working, I would really like to understand what happened, because I didn't find anything documented about this type of incident.

1 year, 1 month

3
2
0 0

17.2.5 ceph fs status: AssertionError

by Robert Sander

Hi, I have a healthy (test) cluster running 17.2.5: root@cephtest20:~# ceph status cluster: id: ba37db20-2b13-11eb-b8a9-871ba11409f6 health: HEALTH_OK services: mon: 3 daemons, quorum cephtest31,cephtest41,cephtest21 (age 2d) mgr: cephtest22.lqzdnk(active, since 4d), standbys: cephtest32.ybltym, cephtest42.hnnfaf mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 48 osds: 48 up (since 4d), 48 in (since 4M) rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 6 portals active (3 hosts) data: volumes: 1/1 healthy pools: 17 pools, 513 pgs objects: 28.25k objects, 4.7 GiB usage: 26 GiB used, 4.7 TiB / 4.7 TiB avail pgs: 513 active+clean io: client: 4.3 KiB/s rd, 170 B/s wr, 5 op/s rd, 0 op/s wr CephFS is mounted and can be used without any issue. But I get an error when I when querying its status: root@cephtest20:~# ceph fs status Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1757, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 159, in handle_fs_status assert metadata AssertionError The dashboard's filesystem page shows no error and displays all information about cephfs. Where does this AssertionError come from? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 year, 1 month

3
3
0 0

Very slow snaptrim operations blocking client I/O

by Victor Rodriguez

Hello, Asking for help with an issue. Maybe someone has a clue about what's going on. Using ceph 15.2.17 on Proxmox 7.3. A big VM had a snapshot and I removed it. A bit later, nearly half of the PGs of the pool entered snaptrim and snaptrim_wait state, as expected. The problem is that such operations ran extremely slow and client I/O was nearly nothing, so all VMs in the cluster got stuck as they could not I/O to the storage. Taking and removing big snapshots is a normal operation that we do often and this is the first time I see this issue in any of my clusters. Disks are all Samsung PM1733 and network is 25G. It gives us plenty of performance for the use case and never had an issue with the hardware. Both disk I/O and network I/O was very low. Still, client I/O seemed to get queued forever. Disabling snaptrim (ceph osd set nosnaptrim) stops any active snaptrim operation and client I/O resumes back to normal. Enabling snaptrim again makes client I/O to almost halt again. I've been playing with some settings: ceph tell 'osd.*' injectargs '--osd-max-trimming-pgs 1' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep 30' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep-ssd 30' ceph tell 'osd.*' injectargs '--osd-pg-max-concurrent-snap-trims 1' None really seemed to help. Also tried restarting OSD services. This cluster was upgraded from 14.2.x to 15.2.17 a couple of months. Is there any setting that must be changed which may cause this problem? I have scheduled a maintenance window, what should I look for to diagnose this problem? Any help is very appreciated. Thanks in advance. Victor

1 year, 1 month

8
18
0 0

2024

2023

2022

2021

2020

2019

ceph-users January 2023