external multipath disk not mounted after power off/on the server
Dear cephers, We have a series of devices that mount several SATA disks via an external cabinet. These disks have 4 paths managed by multipath. (local disks about 20 work perfectly) cephosd23:~ # multipath -ll mpathe (35000c500d88657e3) dm-30 LENOVO-X,ST14000NM004J size=13T features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 2:0:9:0 sdae 65:224 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:21:0 sdap 66:144 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:33:0 sdba 67:64 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 2:0:45:0 sdbl 67:240 active ready running mpathd (35000c500d886ac87) dm-23 LENOVO-X,ST14000NM004J size=13T features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 2:0:2:0 sdx 65:112 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:14:0 sdai 66:32 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:26:0 sdat 66:208 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 2:0:38:0 sdbe 67:128 active ready running mpathc (35000c500d87a5c5f) dm-20 LENOVO-X,ST14000NM004J The disks are accessible from the server, but Ceph has not been able to mount them and they are not accessible from Ceph. ... tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-110 tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-113 tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-112 the directory is empty and of course: /usr/bin/ceph-osd -f --cluster ceph --id 110 --setuser ceph --setgroup ceph 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-110/keyring: (2) No such file or directory 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 AuthRegistry(0x55841ec96940) no keyring found at /var/lib/ceph/osd/ceph-110/keyring, disabling cephx 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-110/keyring: (2) No such file or directory 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 AuthRegistry(0x7ffd668863a0) no keyring found at /var/lib/ceph/osd/ceph-110/keyring, disabling cephx Is there any way to fix this error and mount them again? The data should be intact. Best regards -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
Hi more info, The ceph-volume lvm list, show the wrong osds for example: ====== osd.82 ====== [block] /dev/ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd-block-f96c826d-3570-4c78-9ef6-bea191589102 block device /dev/ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd-block-f96c826d-3570-4c78-9ef6-bea191589102 block uuid avfhQj-uiRt-ZwjG-4jkD-G4tg-a3BM-bcbUfc cephx lockbox secret cluster fsid 6f5a65a7-316c-4825-afcb-428608941dd1 cluster name ceph crush device class encrypted 0 osd fsid f96c826d-3570-4c78-9ef6-bea191589102 osd id 82 osdspec affinity type block vdo 0 devices /dev/mapper/mpatha the /dev/mapper/mpatha is linked to dm-29. The path ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd-block-f96c826d-3570-4c78-9ef6-bea191589102 does not exits I just try to create the path and link to dm-29: ln -s ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd-block-f96c826d-3570-4c78-9ef6-bea191589102 ../dm-29 and reboot, but the path has disappear again Any advice is welcomed. Regards, I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
Hi, I am Still debbuging, ceph-volume lvm activate -all work in 2 servers, but in the other always the same error: failed to read label: cephosd23:~ # ceph-volume lvm activate --all --> Activating OSD ID 112 FSID d8fc1a6f-3a29-41f3-aebb-3c6be84047e5 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-112 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-03c9064e-51af-4a08-afe0-a9515ee513af/osd-block-d8fc1a6f-3a29-41f3-aebb-3c6be84047e5 --path /var/lib/ceph/osd/ceph-112 --no-mon-config stderr: failed to read label for /dev/ceph-03c9064e-51af-4a08-afe0-a9515ee513af/osd-block-d8fc1a6f-3a29-41f3-aebb-3c6be84047e5: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1 [2025-02-27 17:11:31,865][ceph_volume.devices.lvm.activate][DEBUG ] Found block device (osd-block-d8fc1a6f-3a29-41f3-aebb-3c6be84047e5) with encryption: False [2025-02-27 17:11:31,865][ceph_volume.devices.lvm.activate][DEBUG ] Found block device (osd-block-d8fc1a6f-3a29-41f3-aebb-3c6be84047e5) with encryption: False [2025-02-27 17:11:31,866][ceph_volume.process][INFO ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-112 [2025-02-27 17:11:31,870][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-03c9064e-51af-4a08-afe0-a9515ee513af/osd-block-d8fc1a6f-3a29-41f3-aebb-3c6be84047e5 --path /var/lib/ceph/osd/ceph-112 --no-mon-config [2025-02-27 17:11:31,904][ceph_volume.process][INFO ] stderr failed to read label for /dev/ceph-03c9064e-51af-4a08-afe0-a9515ee513af/osd-block-d8fc1a6f-3a29-41f3-aebb-3c6be84047e5: (2) No such file or directory -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
Hello, I recently installed a new cluster. After the first node was working, I started transferring the files I needed. As I was in some urgency to do rsync, I enabled size=1 for the CephFS data pool. After a few days, when I managed to place a new node, I put size = 2 for that pool. Replicas of existing objects are already being recorded, but the available space has not yet been updated. The available space should automatically increase as I add more disks. Right? Could you help me identify where I'm going wrong? Rafael.
Please send the outputs of ceph osd df ceph df ceph osd crush dump ceph osd tree At first I will ask if you intend for size=2 to be final, as that’s a recipe for data loss.
On Feb 27, 2025, at 12:40 PM, quaglio@bol.com.br wrote:
Hello, I recently installed a new cluster.
After the first node was working, I started transferring the files I needed. As I was in some urgency to do rsync, I enabled size=1 for the CephFS data pool. After a few days, when I managed to place a new node, I put size = 2 for that pool.
Replicas of existing objects are already being recorded, but the available space has not yet been updated.
The available space should automatically increase as I add more disks. Right?
Could you help me identify where I'm going wrong?
Rafael. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
No, your available space hasn't increased. If you start with 1 host at let's say 100TB capacity and size 1, your available space is at most 100TB (ceph will actually not allow you to fill drives to 100% by default) - Available capacity = raw capacity If you now add a second host also with 100TB capacity and increase pool size to 2, your available space is still 100TB because all your data is now stored twice - Available capacity = raw capacity / 2 If you go to 3 nodes and size 3 (which you should if you value your data) then available capacity will still only be 100TB because all your data is stored 3x - Available capacity = raw capacity / 3 Only if you add more nodes and keep the replication factor at 3 your available space will increase. A bit simplified but that's what it boils down to. On Fri, 28 Feb 2025 at 06:48, quaglio@bol.com.br <quaglio@bol.com.br> wrote:
Hello, I recently installed a new cluster.
After the first node was working, I started transferring the files I needed. As I was in some urgency to do rsync, I enabled size=1 for the CephFS data pool. After a few days, when I managed to place a new node, I put size = 2 for that pool.
Replicas of existing objects are already being recorded, but the available space has not yet been updated.
The available space should automatically increase as I add more disks. Right?
Could you help me identify where I'm going wrong?
Rafael. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, 2025-02-28 at 08:49 +1300, Christian Wuerdig wrote:
Only if you add more nodes and keep the replication factor at 3 your available space will increase.
I would imagine adding more drives to the existing nodes (once you hit the three node number) would also increase space?
Den tors 27 feb. 2025 kl 18:48 skrev quaglio@bol.com.br <quaglio@bol.com.br>:
Hello, I recently installed a new cluster. After the first node was working, I started transferring the files I needed. As I was in some urgency to do rsync, I enabled size=1 for the CephFS data pool. After a few days, when I managed to place a new node, I put size = 2 for that pool. Replicas of existing objects are already being recorded, but the available space has not yet been updated. The available space should automatically increase as I add more disks. Right?
Could you help me identify where I'm going wrong?
The free space listed in "ceph df" and so on is based on "number of OSDs, times the free-size of the OSD with smallest amount of free space currently". Since you can't control where individual objects end up, "ceph df" will assume worst-case placement and hence give you info about the least possible space it can promise to give you, so if your objects dont get placed on the worst possible place, the "ceph df" output will not be lower by adding data until it makes a new OSD the smallest-free one. In the end, you need to wait until rebalancing has moved the data into the new OSDs so that one of the previous OSDs get more free space. This can take a while. -- May the most significant bit of your life be positive.
On Thu, 2025-02-27 at 20:50 +0100, Janne Johansson wrote:
In the end, you need to wait until rebalancing has moved the data into the new OSDs so that one of the previous OSDs get more free space. This can take a while.
I can attest to this. PG Autoscaler had an issue, so I had an extremely small number of PGs that placed data on only 3 OSDs (out of a total of 20). By the time I noticed the OSDs were about 70% full (16TB HDDs), and when I finally sorted out the PG Autoscaler issue and a better number of PGs were created, it took about three weeks to balance everything out.
I'm coming in late so I don't know the whole story here, but that name's indicative of a Managed (containerized) resource. You can't manually construct, delete or change the systemd services for such items. I learned that the hard way. The service declaration/control files are dynamically created by Ceph under /var/run or something like that and they don't survive reboots. I expect that something similar applies to /dev items, Certainly I've created my share of /dev USB devices dynamically via the udev hot-swap subsystem. What might help is to look at what's under /var/lib/ceph-3b5662ac-854a- 4954-aa44-8951feaa1840. The resources defined there seem to be the prime sources used to generate dynamica services. Tim On Thu, 2025-02-27 at 15:29 +0000, Iban Cabrillo wrote:
Hi more info,
The ceph-volume lvm list, show the wrong osds for example:
====== osd.82 ======
[block] /dev/ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd- block-f96c826d-3570-4c78-9ef6-bea191589102
block device /dev/ceph-3b5662ac-854a-4954-aa44- 8951feaa1840/osd-block-f96c826d-3570-4c78-9ef6-bea191589102 block uuid avfhQj-uiRt-ZwjG-4jkD-G4tg-a3BM- bcbUfc cephx lockbox secret cluster fsid 6f5a65a7-316c-4825-afcb-428608941dd1 cluster name ceph crush device class encrypted 0 osd fsid f96c826d-3570-4c78-9ef6-bea191589102 osd id 82 osdspec affinity type block vdo 0 devices /dev/mapper/mpatha
the /dev/mapper/mpatha is linked to dm-29. The path ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd-block- f96c826d-3570-4c78-9ef6-bea191589102 does not exits
I just try to create the path and link to dm-29: ln -s ceph-3b5662ac-854a-4954-aa44-8951feaa1840/osd-block- f96c826d-3570-4c78-9ef6-bea191589102 ../dm-29
and reboot, but the path has disappear again
Any advice is welcomed. Regards, I
-- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ===================================================================== ==================== ===================================================================== ==================== All our suppliers must know and accept IFCA policy available at:
https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ===================================================================== ===================== _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Alan Murrell
-
Anthony D'Atri
-
Christian Wuerdig
-
Iban Cabrillo
-
Janne Johansson
-
quaglio＠bol.com.br
-
Tim Holloway