- ceph-users - lists.ceph.io

Sometimes PGs inconsistent (although there is no load on them)
by Marianne Spiller 11 Mar '25

11 Mar '25

Dear list, I'm currently maintaining several Ceph (prod) installations. One of them consists of 3 MON hosts and 6 OSD hosts hosting 40 OSDs in total. And there are 5 separate Proxmox-Hosts - they only host the VMs and use the storage provided by Ceph, but they are not part of Ceph. The worst case happened: due to an outage, all these hosts crashed hardly the same time. Last week, I began to restart (only the Ceph hosts; Proxmox servers are still down). Ceph was very unhappy with the situation as a whole - one OSD host (and its 6 OSDs) is completely gone, some hardware issues (33 OSDs left, networking, PSU, I'm working on it) and 73 out of 129 PGs inconsistent. Meanwhile, the overall status of the cluster is "HEALTHY" again. But nearly every day, one or two PGs get damaged. Never on the same OSDs. And there is no traffic on the storage as the virtualization hosts are not running. I see no further reason in the logs: everything is fine, scrub starts and leaves one or more PGs damaged. Repairing them is successful, but maybe next night, another PG is stuck. Do you have hints to investigate this any further? I would love to understand more before starting the Proxmox cluster again. Using Ceph 18.2.4 (Proxmox packages). Thanks a lot, Marianne

4 3

Move block.db to new ssd
by Roland Giesler 11 Mar '25

11 Mar '25

I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland

6 13

Submitting proposals for Ceph Day London 2025
by Ivan Clayson 10 Mar '25

10 Mar '25

Hi, I was interested in applying to give a talk at the upcoming Ceph day in London this June (04/06/25) but I can't find an email or form to submit my proposal to. Is there an email I can send my proposal to? Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N269 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

2 2

Ceph cluster unable to read/write data properly and cannot recover normally.
by 小小 10 Mar '25

10 Mar '25

Hi, I'm facing a critical issue with my Ceph cluster. It has become unable to read/write data properly and cannot recover normally. What steps should I take to resolve this? [root@ceph-node1 ~]# ceph -s cluster: id: 76956086-25f5-445d-a49e-b7824393c17b health: HEALTH_WARN 1 pools have many more objects per pg than average 102131233/124848552 objects misplaced (81.804%) Reduced data availability: 40 pgs inactive Degraded data redundancy: 6402821/124848552 objects degraded (5.128%), 11 pgs degraded, 31 pgs undersized services: mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 mgr: ceph-node2(active), standbys: ceph-node3, ceph-node1 osd: 6 osds: 6 up, 6 in; 169 remapped pgs data: pools: 7 pools, 216 pgs objects: 41.62 M objects, 2.6 TiB usage: 18 TiB used, 12 TiB / 30 TiB avail pgs: 12.500% pgs unknown 6.019% pgs not active 6402821/124848552 objects degraded (5.128%) 102131233/124848552 objects misplaced (81.804%) 143 active+clean+remapped 27 unknown 15 active+undersized+remapped 13 active+clean 9 undersized+degraded+peered 4 undersized+peered 2 active+undersized+degraded 1 active+undersized 1 active+remapped+backfilling 1 active+clean+remapped+scrubbing+deep io: recovery: 1.3 MiB/s, 20 objects/s， [root@ceph-node1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 30.00000 - 30 TiB 18 TiB 17 TiB 125 GiB 302 GiB 12 TiB 58.88 1.00 - root default -3 10.00000 - 10 TiB 5.9 TiB 5.8 TiB 38 GiB 101 GiB 4.1 TiB 59.11 1.00 - host ceph-node1 0 hdd 5.00000 0.21257 5.0 TiB 2.3 TiB 2.2 TiB 25 GiB 41 GiB 2.7 TiB 46.21 0.78 61 osd.0 3 hdd 5.00000 0.13638 5.0 TiB 3.6 TiB 3.5 TiB 13 GiB 60 GiB 1.4 TiB 72.02 1.22 84 osd.3 -5 10.00000 - 10 TiB 6.2 TiB 6.1 TiB 42 GiB 105 GiB 3.8 TiB 62.40 1.06 - host ceph-node2 1 hdd 5.00000 0.13644 5.0 TiB 3.6 TiB 3.5 TiB 11 GiB 61 GiB 1.4 TiB 71.99 1.22 95 osd.1 4 hdd 5.00000 0.16666 5.0 TiB 2.6 TiB 2.6 TiB 31 GiB 44 GiB 2.4 TiB 52.81 0.90 104 osd.4 -7 10.00000 - 10 TiB 5.5 TiB 5.4 TiB 45 GiB 96 GiB 4.5 TiB 55.11 0.94 - host ceph-node3 2 hdd 5.00000 0.16666 5.0 TiB 3.5 TiB 3.4 TiB 22 GiB 62 GiB 1.5 TiB 70.19 1.19 79 osd.2 5 hdd 5.00000 0.21664 5.0 TiB 2.0 TiB 1.9 TiB 23 GiB 34 GiB 3.0 TiB 40.04 0.68 100 osd.5 TOTAL 30 TiB 18 TiB 17 TiB 125 GiB 302 GiB 12 TiB 58.88 MIN/MAX VAR: 0.68/1.22 STDDEV: 13.38 Thanks~

4 4

Created no osd(s) on host, already created?
by Gustavo Garcia Rondina 10 Mar '25

10 Mar '25

Hello list, We have a Ceph cluster with two management nodes and six data nodes. Each data node has 28 HDD disks. One disk recently failed in one of the nodes, corresponding to osd.2. To replace the disk, we took the osd.2 out, stopped it, and after a few days removed it, basically: ceph osd out osd.2 ceph osd ok-to-stop osd.2 Once Ok to stop, then: ceph orch daemon stop osd.2 Once stopped: ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2 Then checked `ceph osd tree`, `ceph orch ps`, `ceph -s`, and all confirmed that the OSD was gone. We then proceeded to physically replace the failed drive on the node. It was /dev/sdac before, and after replacing it (hot swap) the system identified it as /dev/sdah. I zapped it on the node with `sgdisk --zap-all /dev/sdah` and, back on the management node, I could see that the disk was now showing up and marked as available with `ceph orch device ls`. Then I proceeded to add a new OSD to that disk with: `ceph orch daemon add osd node-osd1:/dev/sdah`, which then failed with: Created no osd(s) on host server-osd1; already created? Which was rather strange, because total OSDs was still showing as 1 less, no new OSD was showing up in `ceph osd tree`. What was even odder is that the disk that was showing as available in `ceph orch device ls` now shows as *not* available, and looking at lsblk's output in the node it seems that it was populated by Ceph: # lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm Any hints on how to proceed to get the OSD added back with this disk? Thank you for any suggestions! - Gustavo

4 10

DC 4 EC 4+5 with 4 servers make sense?
by Torkil Svensgaard 10 Mar '25

10 Mar '25

Hi We currently have a 3 DC setup with 6 HDD servers in each of 3 DCs, running replica 3 and EC 4+5 pools. This works fine for mostly dormant data but of course not so well for other stuff requiring low latency etc. Mostly big RBDs shared by kernel NFS with a bit of CephFS. We want to start migrating some of these big RBD backed NFS shares to NVMe. We currently have 3 E3.S servers, one in each DC, with an EC 4+5 pool with DC->OSD crush selection. We are wondering how to cheapest increase redundancy short term since we do not have enterprise money. With what we have we can lose 1 DC (aka host in this case) and 1 OSD and still be online, but there's nowhere to backfill and some hardware problems, like a faulty mainboard or CPU can take down an entire "DC". Ideally we would have several more hosts in each DC so we had DC->Host->OSD but that's not an option. I am thinking it could make sense to add just 1 more server in a 4th DC and keep the 4+5 rule as is, simply giving us a buffer of 1 more failed host before we hit problems. Thoughts? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

cephadm bootstrap failed with docker
by farhad kh 08 Mar '25

08 Mar '25

hi , for trying deploy cluster with cephadm version 19.2.1 and using docker version 28.0.1 i get this error : ------- # cephadm --image opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 bootstrap --mon-ip 10.248.35.143 --registry-json /root/reg.json --allow-fqdn-hostname --initial-dashboard-user admin --initial-dashboard-password P@ssw0rd1404 --dashboard-password-noupdate --ssh-user cephadmin Verifying ssh connectivity using standard pubkey authentication ... Adding key to cephadmin@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... docker (/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 15d9eaee-fbe0-11ef-ad63-005056a83619 Verifying IP 10.248.35.143 port 3300 ... Verifying IP 10.248.35.143 port 6789 ... Mon IP `10.248.35.143` is in CIDR network `10.248.35.0/24` Mon IP `10.248.35.143` is in CIDR network `10.248.35.0/24` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Logging into custom registry. Pulling custom registry login info from /root/reg.json. Pulling container image opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1... Non-zero exit code 125 from /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 -e NODE_NAME=opcpmfpsksa0403 opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 --version ceph: stderr docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown ceph: stderr ceph: stderr Run 'docker run --help' for more information RuntimeError: Failed command: /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 -e NODE_NAME=opcpmfpsksa0403 opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 --version *************** Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically. To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous broken installation, users must use the following command to completely delete the broken cluster: > cephadm rm-cluster --force --zap-osds --fsid <fsid> for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster *************** Deleting cluster with fsid: 15d9eaee-fbe0-11ef-ad63-005056a83619 Traceback (most recent call last): File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 2628, in _rollback File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 446, in _default_image File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 2763, in command_bootstrap File "/tmp/tmpfe1bt8s9.cephadm.build/app/cephadmlib/container_types.py", line 429, in run File "/tmp/tmpfe1bt8s9.cephadm.build/app/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 -e NODE_NAME=opcpmfpsksa0403 opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 --version During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 5581, in <module> File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 5569, in main File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 2657, in _rollback File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 4391, in _rm_cluster File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 4317, in get_ceph_cluster_count FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ceph' ----- for that i checked everything lik docker version also containerd and runc and just find this log in journalctl -u docker : ------- Mar 02 06:23:00 opcpmfpsksa0403 systemd[1]: Starting Docker Application Container Engine... Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.533468075Z" level=info msg="Starting up" Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.535018237Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider" Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.566901059Z" level=info msg="[graphdriver] using prior storage driver: overlay2" Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.567653709Z" level=info msg="Loading containers: start." Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.325500958Z" level=info msg="Loading containers: done." Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.344752401Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2 Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.344892780Z" level=info msg="Docker daemon" commit=bbd0a17 containerd-snapshotter=false storage-driver=overlay2 version=28.0.1 Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.344968676Z" level=info msg="Initializing buildkit" Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.393298555Z" level=info msg="Completed buildkit initialization" Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.401141531Z" level=info msg="Daemon has completed initialization" Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.401227515Z" level=info msg="API listen on /run/docker.sock" Mar 02 06:23:01 opcpmfpsksa0403 systemd[1]: Started Docker Application Container Engine. Mar 08 05:42:50 opcpmfpsksa0403 dockerd[18363]: time="2025-03-08T05:42:50.468402014Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stderr Mar 08 05:42:50 opcpmfpsksa0403 dockerd[18363]: time="2025-03-08T05:42:50.468433232Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stdout Mar 08 05:42:51 opcpmfpsksa0403 dockerd[18363]: time="2025-03-08T05:42:51.103965419Z" level=error msg="Handler for POST /v1.48/containers/40b1295b9067eb570d01b1509c59593f29e7ad61fb61e8ed4a82166441d52d53/start returned error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown" -------- we use linux oracle 9.5 with kernel 5.15.0-305.176.4.el9uek.x86_64, we searched anywhere but we can't understand that what happen, anybody know how to can resolve that ? or what's happening?

2 1

BlueStore and BlueFS warnings after upgrade to 19.2.1
by Nicola Mori 07 Mar '25

07 Mar '25

Dear Ceph users, after upgrading from 19.2.0 to 19.2.1 (via cephadm) my cluster started showing some warnings never seen before: 29 OSD(s) experiencing slow operations in BlueStore 13 OSD(s) experiencing stalled read in db device of BlueFS I searched for these messages but didn't find much. I also noticed that when browsing the CephFS folders (using the kernel module for the client) sometimes the client gets stuck for a long time before showing the folder content; however I don't know if this can be related to the above warnings. I'd need help to understand what's going on and eventually how to troubleshoot it. Thanks, Nicola

4 8

ceph orch is not working
by xadhoom76＠gmail.com 06 Mar '25

06 Mar '25

Hi , we have issue using ceph orch , it doesn't send any command for real. We try to restart or redeploy but nothing works This happened downgrading from 17.2.8 to 17.2.6 Thanks

3 3

RGW Cloud-Sync Configuration Help
by Mark Selby 06 Mar '25

06 Mar '25

I am trying to use the RGW Cloud-Sync module to replicate data from an on-prem RGW cluster to AWS. This is a multi-tenant cluster with many tenants each who have multiple buckets. I only want to replicate a single bucket from a single tenant and am having problems creating a tier config that does this. It seems that the tier config requires a default connection and will try and replicate all buckets that do not have a specific profile. I have used sync groups, flows, and pipes with “standard” multisite replication but it does not seem to be applicable to cloud sync. Does anyone know how to configure cloud-sync to only replicate a single bucket from a single tenant. Here are some details on my current tier-config. I want to replicate the bucket roadrunner from the tenant wile to the beepbeep bucket in aws. This the only bucket I want cloud-sync to replicate. Any and all help is greatly appreciated. "tier_config": { "connection_id": "aws-default", "connections": [ { "id": "aws-default", "endpoint": https://s3.amazonaws.com, "access_key": "XXXXXXXXXXXXXXXXXXXX", "secret": "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" }, { "id": "aws-cloud-sync", "endpoint": https://s3.amazonaws.com, "access_key": "XXXXXXXXXXXXXXXXXXXX", "secret": "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" }, ], "profiles": [ { "connection_id": "aws-cloud-sync", "source_bucket": "wile:roadrunner", "target_path": "beepbeep" } ] } -- Mark Selby Sr Linux Administrator, The Voleon Group mselby(a)voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/.

1 0