- ceph-users - lists.ceph.io

CephFS space usage
by Thorne Lawler 27 Mar '24

27 Mar '24

Hi everyone! My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going? How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space? Thank you. -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

9 27

Ceph versus Zabbix: failure: no data sent
by John Jasen 26 Mar '24

26 Mar '24

If the documentation is to be believed, it's just install the zabbix sender, then; ceph mgr module enable zabbix ceph zabbix config-set zabbix_host my-zabbix-server (Optional) Set the identifier to the fsid. And poof. I should now have a discovered entity on my zabbix server to add templates to. However, this has not worked yet on either of my ceph clusters (one RHEL, one proxmox). Reference: https://docs.ceph.com/en/latest/mgr/zabbix/ On Reddit advice, I installed the Ceph templates for Zabbix. https://raw.githubusercontent.com/ceph/ceph/master/src/pybind/mgr/zabbix/za… Still no dice. No traffic at all seems to be generated, that I've seen from packet traces, ... OK. I su'ed to the ceph user on both clusters, and ran zabbix_send: zabbix_sender -v -z 10.0.0.1 -s "$my_fsid" -k ceph.osd_avg_pgs -o 1 Response from "10.0.0.1:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000042" sent: 1; skipped: 0; total: 1 As the ceph user, ceph zabbix send/discovery still fail. I am officially stumped. Any ideas as to which tree I should be barking up? Thanks in advance!

2 2

stretch mode item not defined
by ronny.lippold 26 Mar '24

26 Mar '24

hi there, need some help please. we are planning to replace our rbd-mirror setup and go to stretch mode. the goal is, to have the cluster in 2 fire compartment server rooms. start was a default proxmox/ceph setup. now, i followed the howto from: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ with the crushmap i ended in an error: in rule 'stretch_rule' item 'dc1' not defined (dc1,2,3 is like site1,2,3 in the docu). my osd tree looks like pve-test02-01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 3.80951 root default -15 0.06349 host pve-test01-01 0 ssd 0.06349 osd.0 up 1.00000 1.00000 -19 0.06349 host pve-test01-02 1 ssd 0.06349 osd.1 up 1.00000 1.00000 -17 0.06349 host pve-test01-03 2 ssd 0.06349 osd.2 up 1.00000 1.00000 ... i think there is something missing. should i need adding the buckets manually? ... for disaster failover, any ideas are welcome. man thanks for help and kind regards, ronny

2 2

CephFS filesystem mount tanks on some nodes?
by Erich Weiler 26 Mar '24

26 Mar '24

Hi All, We have a CephFS filesystem where we are running Reef on the servers (OSD/MDS/MGR/MON) and Quincy on the clients. Every once in a while, one of the clients will stop allowing access to my CephFS filesystem, the error being "permission denied" while try to access the filesystem on that node. The fix is to force unmount the filesystem and remount it, then it's fine again. Any idea how I can prevent this? I see this in the client node logs: Mar 25 11:34:46 phoenix-07 kernel: [50508.354036] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:34:46 phoenix-07 kernel: [50508.359650] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:34:46 phoenix-07 kernel: [50508.367657] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:36:46 phoenix-07 kernel: [50629.189000] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:36:46 phoenix-07 kernel: [50629.192579] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:36:46 phoenix-07 kernel: [50629.196103] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:38:47 phoenix-07 kernel: [50750.024268] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:38:47 phoenix-07 kernel: [50750.031520] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:38:47 phoenix-07 kernel: [50750.038594] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:40:48 phoenix-07 kernel: [50870.853281] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 22:55:38 phoenix-07 kernel: [91360.583032] libceph: mds0 (1)10.50.1.75:6801 socket closed (con state OPEN) Mar 25 22:55:38 phoenix-07 kernel: [91360.667914] libceph: mds0 (1)10.50.1.75:6801 session reset Mar 25 22:55:38 phoenix-07 kernel: [91360.667923] ceph: mds0 closed our session Mar 25 22:55:38 phoenix-07 kernel: [91360.667925] ceph: mds0 reconnect start Mar 25 22:55:52 phoenix-07 kernel: [91374.541614] ceph: mds0 reconnect denied Mar 25 22:55:52 phoenix-07 kernel: [91374.541726] ceph: dropping dirty+flushing Fw state for 00000000ea96c18f 1099683115069 Mar 25 22:55:52 phoenix-07 kernel: [91374.541732] ceph: dropping dirty+flushing Fw state for 00000000ce495f00 1099687100635 Mar 25 22:55:52 phoenix-07 kernel: [91374.541737] ceph: dropping dirty+flushing Fw state for 0000000073ebb190 1099687100636 Mar 25 22:55:52 phoenix-07 kernel: [91374.541744] ceph: dropping dirty+flushing Fw state for 0000000091337e6a 1099687100637 Mar 25 22:55:52 phoenix-07 kernel: [91374.541746] ceph: dropping dirty+flushing Fw state for 000000009075ecd8 1099687100634 Mar 25 22:55:52 phoenix-07 kernel: [91374.541751] ceph: dropping dirty+flushing Fw state for 00000000d1d4c51f 1099687100633 Mar 25 22:55:52 phoenix-07 kernel: [91374.541781] ceph: dropping dirty+flushing Fw state for 0000000063dec1e4 1099687100632 Mar 25 22:55:52 phoenix-07 kernel: [91374.541793] ceph: dropping dirty+flushing Fw state for 000000008b3124db 1099687100638 Mar 25 22:55:52 phoenix-07 kernel: [91374.541796] ceph: dropping dirty+flushing Fw state for 00000000d9e76d8b 1099687100471 Mar 25 22:55:52 phoenix-07 kernel: [91374.541798] ceph: dropping dirty+flushing Fw state for 00000000b57da610 1099685041085 Mar 25 22:55:52 phoenix-07 kernel: [91374.542235] libceph: mds0 (1)10.50.1.75:6801 socket closed (con state V1_CONNECT_MSG) Mar 25 22:55:52 phoenix-07 kernel: [91374.791652] ceph: mds0 rejected session Mar 25 23:01:51 phoenix-07 kernel: [91733.308806] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 25 23:01:56 phoenix-07 kernel: [91738.182127] ceph: check_quota_exceeded: ino (1000a1cb4a8.fffffffffffffffe) null i_snap_realm Mar 25 23:01:56 phoenix-07 kernel: [91738.188225] ceph: check_quota_exceeded: ino (1000a1cb4a8.fffffffffffffffe) null i_snap_realm Mar 25 23:01:56 phoenix-07 kernel: [91738.233658] ceph: check_quota_exceeded: ino (1000a1cb4aa.fffffffffffffffe) null i_snap_realm Mar 25 23:25:52 phoenix-07 kernel: [93174.787630] libceph: mds0 (1)10.50.1.75:6801 socket closed (con state OPEN) Mar 25 23:39:45 phoenix-07 kernel: [94007.751879] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 00:03:28 phoenix-07 kernel: [95430.158646] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 00:39:45 phoenix-07 kernel: [97607.685421] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 00:43:34 phoenix-07 kernel: [97836.681145] ceph: check_quota_exceeded: ino (1000a306503.fffffffffffffffe) null i_snap_realm Mar 26 00:43:34 phoenix-07 kernel: [97836.686797] ceph: check_quota_exceeded: ino (1000a306503.fffffffffffffffe) null i_snap_realm Mar 26 00:43:34 phoenix-07 kernel: [97836.729046] ceph: check_quota_exceeded: ino (1000a306505.fffffffffffffffe) null i_snap_realm Mar 26 00:49:39 phoenix-07 kernel: [98201.302564] ceph: check_quota_exceeded: ino (1000a75677d.fffffffffffffffe) null i_snap_realm Mar 26 00:49:39 phoenix-07 kernel: [98201.305676] ceph: check_quota_exceeded: ino (1000a75677d.fffffffffffffffe) null i_snap_realm Mar 26 00:49:39 phoenix-07 kernel: [98201.347267] ceph: check_quota_exceeded: ino (1000a755fe3.fffffffffffffffe) null i_snap_realm Mar 26 01:04:49 phoenix-07 kernel: [99111.892854] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 01:39:45 phoenix-07 kernel: [101207.645602] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 02:05:35 phoenix-07 kernel: [102757.494073] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 02:39:45 phoenix-07 kernel: [104807.617467] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 03:03:14 phoenix-07 kernel: [106216.519979] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 03:39:45 phoenix-07 kernel: [108407.731139] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 04:04:05 phoenix-07 kernel: [109867.406047] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 04:39:45 phoenix-07 kernel: [112007.672100] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 05:04:39 phoenix-07 kernel: [113501.101302] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 05:39:45 phoenix-07 kernel: [115607.696806] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 06:05:08 phoenix-07 kernel: [117130.484942] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 06:39:45 phoenix-07 kernel: [119207.706740] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 07:03:33 phoenix-07 kernel: [120635.419910] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 07:39:45 phoenix-07 kernel: [122807.700416] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 08:02:44 phoenix-07 kernel: [124186.804150] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 08:39:45 phoenix-07 kernel: [126407.696256] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 09:02:15 phoenix-07 kernel: [127757.573231] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 09:39:45 phoenix-07 kernel: [130007.718852] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm And after I unmount/remount the filesystem I see this: Mar 26 10:02:57 phoenix-07 kernel: [131399.351230] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 10:04:28 phoenix-07 kernel: [131490.722855] ceph: No path or : separator in source Mar 26 10:04:28 phoenix-07 kernel: [131490.727023] libceph: mon0 (1)10.50.1.74:6789 session established Mar 26 10:04:28 phoenix-07 kernel: [131490.730211] libceph: client102330 fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 Is it possibly losing connection to the MDS servers or something? These lines specifically concern me: Mar 25 22:55:38 phoenix-07 kernel: [91360.667923] ceph: mds0 closed our session Mar 25 22:55:38 phoenix-07 kernel: [91360.667925] ceph: mds0 reconnect start Mar 25 22:55:52 phoenix-07 kernel: [91374.541614] ceph: mds0 reconnect denied It looks like it lost its connection, tried to reconnect and was denied? Does that ring a bell for anyone? The filesystem *is* extremely busy... Thanks again! -erich

1 0

cephfs client not released caps when running rsync
by Nikita Borisenkov 26 Mar '24

26 Mar '24

We transfer data (300 million small files) using rsync between cephfs from version 12.2.13 to 18.2.1. After about the same time (about 7 hours in this case), copying stops for a minute ``` health:HEALTH_WARN 1 clients failing to advance oldest client/flush tid 1 MDSs report slow metadata IOs 1 MDSs behind on trimming ``` The only interesting messages in the logs are: ``` ceph-mds[640738]: mds.beacon.cephfs.X missed beacon ack from the monitors ``` I watched debugging on the client (destination) via watch -n1 /sys/kernel/debug/ceph/451eea44-d7a0-11ee-9117-b496914b4c02.client32497 ``` item total ------------------------------------------ opened files / total inodes 1 / 866757 pinned i_caps / total inodes 866757 / 866757 opened inodes / total inodes 1 / 866757 item total avg_lat(us) min_lat(us) max_lat(us) stdev(us) -------------------------------------------------- --------------------------------- read 0 0 0 0 0 write 5129252 194689 9080 54693415 1023 metadata 29045143 670 161 87794124 369 item total miss hit ------------------------------------------------- d_lease 5361 3026 315488442 caps 866757 11 483177478 ``` During the copy, "pinned i_caps / total inodes" are gradually increased until it reaches the value "mds_max_caps_per_client" (default: 1Mi). Then "pinned i_caps / total inodes" begins to decrease to almost 0, at which time HEALTH_WARN appears and transfer stops. "op/s wr" increases from 200 to 1.5k. Then total inodes begin to increase again along with the resumption of copying and the cluster goes into the HEALTHY state. Mount options: /mnt/cephfs-old 10.77.12.90:6789,10.77.12.91:6789,10.77.12.92:6789:/ ceph rw,noatime,nodiratime,name=admin,secret=<hidden>,acl /mnt/cephfs-new 10.77.12.139:6789,10.77.12.140:6789,10.77.12.141:6789:/ ceph rw,noatime,nodiratime,name=admin,secret=<hidden>,acl,caps_max=10000 Client properties on the MDS server (removed unnecessary): ceph daemon mds.cephfs.X client ls 32497 [ { ... "id": 32497, "state": "open", "num_leases": 0, "num_caps": 980679, "request_load_avg": 7913, "requests_in_flight": 466, "num_completed_flushes": 464, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 1732.2552002208533, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 1.35000000000000001 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 42186.620275326415, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 30 }, ... } ] ceph daemonperf mds.cephfs.X ``` -------------------------------------------mds------ ------------------------------------- --mds_cache--- ------mds_log ------------ -mds_mem- -------mds_server------- mds_ -----objecter------ purg req rlat slr fwd inos caps exi imi hifc crev cgra ctru cfsa cfa hcc hccd hccr prcr|stry recy recd|subm evts segs repl|ino dn |hcr hcs hsr cre cat |sess|actv rd wr rdwr|purg| 114 0 0 0 1.9M 438k 0 0 0 0 253 0 0 0 0 0 0 59 | 0 0 0 |128 123k 129 0 |1.3M 1.9M|114 0 0 0 0 | 3 | 0 0 440 0 | 0 101 0 0 0 1.9M 438k 0 0 0 0 0 0 0 0 0 0 0 53 | 0 0 0 |106 123k 129 0 |1.3M 1.9M|101 0 0 0 0 | 3 | 0 0 0 0 | 0 ... ``` - from this output it is clear that the client does not send cap release at all (column hccr, decoding of the columns "ceph daemon mds.X perf schema") Then I was able to find the right query to google similar problems: https://www.spinics.net/lists/ceph-users/msg50573.html https://ceph-users.ceph.narkive.com/mcyPtEyz/rsync-kernel-client-cepfs-mkst… https://www.spinics.net/lists/ceph-users/msg50158.html https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B7K6B5VXM3I… It turns out that the problem is in rsync, in the way it works? The only "solution" is to do it on the client according to a schedule (or upon reaching a certain number of open caps) “echo 2 > /proc/sys/vm/drop_caches”. After this command, the cephfs client releases the cached caps. And if there were a lot of them, then MDS becomes slow again. We also tried to mount cephfs with the option "caps_max=10000" so that the client would do a forced release when the specified value is reached, but this did not help. We can limit mds_max_caps_per_client (not tested), but this also affects all clients at once. The command "ceph daemon mds.cephfs.X cache drop" (with or without an additional parameter) does not help Tested on Linux kernels (client side): 5.10 and 6.1 Did I understand everything correctly? is this the expected behavior when running rsync? And one more problem (I don’t know if it’s related or not), when rsync finishes copying, all caps are freed except the last two (pinned i_caps / total inodes 2 / 2) At this moment a warning appears (or remains after releasing a large number of caps): 1 clients failing to advance oldest client/flush tid But then it doesn't disappear. I waited 12 hours. Warning disappears only after executing the "sync" command on the client. and in the client metrics you can see "pinned i_caps / total inodes 1 / 1" Note: running "echo 2 > /proc/sys/vm/drop_caches" does not help in this case.

2 1

Cephadm on mixed architecture hosts
by Iain Stott 26 Mar '24

26 Mar '24

Hi, We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs. During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo. [global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment. Is there a config setting we are missing or a workaround for this? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

3 3

Clients failing to advance oldest client?
by Erich Weiler 26 Mar '24

26 Mar '24

Hi Y'all, I'm seeing this warning via 'ceph -s' (this is on Reef): # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 3 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 3d) mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz mds: 1/1 daemons up, 1 standby osd: 46 osds: 46 up (since 3d), 46 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 258.13M objects, 454 TiB usage: 688 TiB used, 441 TiB / 1.1 PiB avail pgs: 1303 active+clean 8 active+clean+scrubbing 2 active+clean+scrubbing+deep io: client: 131 MiB/s rd, 111 MiB/s wr, 41 op/s rd, 613 op/s wr I googled around and looked at the docs and it seems like this isn't a critical problem, but I couldn't find a clear path to resolution. Does anyone have any advice on what I can do to resolve the health issues up top? My CephFS filesystem is incredibly busy so I have a feeling that has some impact here, but not 100% sure... Thanks as always for the help! cheers, erich

3 5

How can I set osd fast shutdown = true
by Suyash Dongre 26 Mar '24

26 Mar '24

Hello, I want to set osd fast shutdown = true, how should I achieve this? Regards, Suyash

2 1

Lot log message from one server
by Albert Shih 26 Mar '24

26 Mar '24

Hi, On my active mgr (only one) on my cluster I got repeatedly something like Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists I check the dashboard (first hit with google on that message) and that seem OK, no config about it listen address. Those messages come each ~ 1 hour. I notice those message appear each time just after Mar 26 08:50:04 cthulhu2 ceph-mgr[2843]: [volumes INFO volumes.module] Starting _cmd_fs_subvolume_ls(prefix:fs subvolume ls, target:['mon-mgr', ''], vol_name:cephfs) < "" but I don't known if that's related. Any clue ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 26 mars 2024 10:52:53 CET

1 0

Mounting A RBD Via Kernal Modules
by duluxoz 26 Mar '24

26 Mar '24

Hi All, I'm trying to mount a Ceph Reef (v18.2.2 - latest version) RBD Image as a 2nd HDD on a Rocky Linux v9.3 (latest version) host. The EC pool has been created and initialised and the image has been created. The ceph-common package has been installed on the host. The correct keyring has been added to the host (with a chmod of 600) and the host has been configure with an rbdmap file as follows: `my_pool.meta/my_image id=ceph_user,keyring=/etc/ceph/ceph.client.ceph_user.keyring`. When running the rbdmap.service the image appears as both `/dev/rbd0` and `/dev/rbd/my_pool.meta/my_image`, exactly as the Ceph Doco says it should. So everything *appears* AOK up to this point. My question now is: Should I run `mkfs xfs` on `/dev/rbd0` *before* or *after* I try to mount the image (via fstab: `/dev/rbd/my_pool.meta/my_image /mnt/my_image xfs noauto 0 0` - as per the Ceph doco)? The reason I ask is that I've tried this *both* ways and all I get is an error message (sorry, can't remember the exact messages and I'm not currently in front of the host to confirm it :-) - but from memory it was something about not being able to recognise the 1st block - or something like that). So, I'm obviously doing something wrong, but I can't work out what exactly (and the logs don't show any useful info). Do I, for instance, have the process wrong / don't understand the exact process, or is there something else wrong? All comments/suggestions/etc greatly appreciated - thanks in advance Cheers Dulux-Oz

6 16