- ceph-users - lists.ceph.io

[grafana] ceph-cluster-advanced: wrong title for object count
by Eugen Block 01 Apr '25

01 Apr '25

Hi Ankush, I found another (minor) Grafana issue. The "Ceph Cluster - Advanced" panel contains a graph for "object count", but its title is "OSD Type Count". I see that in Squid 19.2.0 (Grafana version 9.4.12) and 19.2.1 (Grafana 10.4.0), and the latest ceph-cluster-advanced.json [0] also still contains that mistake. Regards, Eugen [0] https://github.com/ceph/ceph/blob/89f21db321ddd404654b0064a61b0aa5428d6f7e/…

1 0

Purpose of the cephadm account
by dhivagar selvam 01 Apr '25

01 Apr '25

Hi, We have set up a ceph cluster via ceph-ansible. When we add a new osd or mds to the ceph cluster, the "cephadm" account is automatically created with "/bin/bash". 1. Is this account necessary? 2. Can I delete this account or do I need to change something in the ceph-ansible configuration? Thanks,

1 0

How/when the osd_mclock_max_capacity_iops is updated?
by Michel Jouvin 31 Mar '25

31 Mar '25

Hi, Looking at our configuration during a recovery operation (misplaced objects after adding PG in a pool) that I found a bit slow, I saw that ~30 OSD over 500 had an entry for osd_mclock_max_capacity_iops_hdd in the configuration DB (and almost all the SSD). Despite all HDD and SSD being the same models (and the server HW being the same). As this value can have a significant impact on recovery time/speed with mclock, I was wondering if this value is periodically refreshed, in case one measurement was wrong (perturbated by external factors). The documentation is not very clear about this and seems to say that the bench used to estimate the IOPS capacity is run only when the OSD is activated (which may make sense as it is before there is any other activity on it)? I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers? Currently, I tend to remove all the osd_mclock_max_capacity_iops_xxx entries from the DB periodically but it's probably not the right approach... Cheers, Michel

3 7

Ceph MDS stuck in reconnect -> rejoin -> failover loop
by Kasper Rasmussen 31 Mar '25

31 Mar '25

Hi Ceph pacific 16.2.15 I have 5 MDS hosts, 4 active (4 FS), and 1 standby. One MDS was restarted today (as part of OS Patching), resulting in a failover. This is usually not an issue but today,it got stuck in a reconnect -> rejoin -> failover loop for the specific FS. A ceph fs status shows that during the time the FS is in state "rejoin" the INOS rise to +50M (usually it is around 10-12M ) The memory on the MDS host is eaten, (MDS cache size is 36GB, but it rises to +140 GB.) Finaly it fails over, and the cycle starts over. We are currently restarting all clients, in an effort to rule out buggy clients. Any help on this issue will be very much appreciated. Thank you

2 2

Lifecycle question
by Szabo, Istvan (Agoda) 31 Mar '25

31 Mar '25

Hi, Is there a way to have hourly or minute based lifecycle rather than a 1 day set on a bucket? When I checked multiple xml rules all of them day based and can't set hour or minute value: Multipart cleaning: <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Incomplete Multipart Uploads</ID> <Prefix/> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> Basic retention: <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>5 days retention</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>5</Days> </Expiration> </Rule> </LifecycleConfiguration> Tag based retention: <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>1 day object expiry</ID> <Filter> <Tag> <Key>expiration</Key> <Value>1d</Value> </Tag> </Filter> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> </Rule> </LifecycleConfiguration> Prefix based retention <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>hermes-1-day-retention</ID> <Filter> <Prefix>her-</Prefix> </Filter> <Status>Enabled</Status> <Expiration> <Days>1</Days> //retention </Expiration> </Rule> </LifecycleConfiguration> ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

Deleting a pool with data
by Richard Bade 30 Mar '25

30 Mar '25

Hi Everyone, We're reducing back down from multisite to a single rgw zone. This will mean that some pools will be unused so I'd like to delete them. However there are some objects and data remaining in the pool even though the buckets are all deleted. It's just shadow objects. All the actual data has been deleted. So my question is around performance impact of deleting a pool with data in it. Will ceph handle things nicely or will it try remove all that data at once? I'm on Nautilus 14.2.22 with all bluestore osds on spinning disks with nvme db. Pools are erasure coded k=4 m=2. I'm thinking it might be best to do rados ls and loop through rados rm the objects to control the speed. There is one mailing list thread from 7 years ago basically saying the same but I was wondering if anyone else had any input around this? Thanks, Rich

4 5

Device missing from "ceph device ls"
by Torkil Svensgaard 29 Mar '25

29 Mar '25

Hi 19.2.1 " [root@franky ~]# ceph device ls | grep franky ATA_HGST_HDN726060ALE614_K1GV9P4B franky:sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P franky:sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ franky:sdg osd.580 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA franky:sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA franky:sdf osd.527 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA franky:sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A franky:sdx osd.583 ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN franky:sdp osd.532 5% now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN franky:sdt osd.531 7% now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH franky:sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH franky:sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH franky:sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH franky:sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH franky:sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 franky:sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 franky:sdr osd.530 ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 franky:sdl osd.528 ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 franky:sdaa osd.529 ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ franky:sdd osd.537 HGST_HUS724040ALS640_PBG458BX franky:sdb osd.568 HGST_HUS724040ALS640_PCJ2VXRX franky:sdj osd.578 HGST_HUS724040ALS640_PCJN3LWX franky:sdi osd.539 HGST_HUS724040ALS640_PCKX3UWX franky:sdo osd.564 HGST_HUS724040ALS640_PCKX4KSX franky:sds osd.575 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 franky:nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 0% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 franky:nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 4% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 franky:nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 4% [root@franky ~]# ceph orch device ls | grep franky franky /dev/md127 ssd 221G No 12m ago Has a FileSystem franky /dev/nvme0n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme1n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme2n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme3n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/sda hdd ATA_HGST_HDN726060ALE614_K1GV9P4B 5589G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdaa hdd ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdb hdd HGST_HUS724040ALS640_PBG458BX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdc hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdd hdd ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sde hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdf hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdg hdd ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdh hdd ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdi hdd HGST_HUS724040ALS640_PCJN3LWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdj hdd HGST_HUS724040ALS640_PCJ2VXRX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdk hdd ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdl hdd ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdm hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdn hdd ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdo hdd HGST_HUS724040ALS640_PCKX3UWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdp ssd ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN 186G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdq hdd ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdr hdd ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sds hdd HGST_HUS724040ALS640_PCKX4KSX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdt ssd ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN 447G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdu hdd ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdv hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdw hdd ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2273861 2794G Yes 12m ago franky /dev/sdx hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected " Any suggestions as to why a device (nvme2n1) might be missing from "ceph device ls"? It is being used as db device for 4 OSDs. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 3

OSD creation from service spec fails to check all db_devices for available space
by Torkil Svensgaard 29 Mar '25

29 Mar '25

Hi I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL. The orchaetrator now fails to create any more OSDs due to: " /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled " It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed. Suggestions? " 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd " " [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd--block--76e673e9--d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block--b7330837--b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd--block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd--block--8a0c4a74--fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block--c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd--block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd--block--618f7582--fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd--block--8e9b21b6--cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd--block--14f68253--b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd--block--2b475461--a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block--f08c7e71--ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd--block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block--c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block--c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd--block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block--a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd--block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block--ff12d109--dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block--b3e38c3d--e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# " Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 4

space size issue
by Mihai Ciubancan 29 Mar '25

29 Mar '25

Hello, My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB): sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000 sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B While on the client side I have this: $ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data I don't know where it's gone all the space that was at the beginning. Someone has any hint? Best regards, Mihai

6 16

Ceph Commvault Experience
by DAN ALBRIGHT 28 Mar '25

28 Mar '25

Hi Ceph Community, looking to understand Ceph implementation as an S3 storage pool in support of the Commvault. Please share your experiences and any important tips to consider. Many Thanks! Dan Albright Americas Sales Leader/Specialist - IBM Object Storage (COS & Ceph)- Systems Sales M: (303) 908-5203

2 1