Hi, I'm still evaluating ceph 15.2.5 in a lab so the problem is not really hurting me, but I want to understand it and hopefully fix it. It is a good practice. To test the resilience of the cluster I try to break it by doing all kinds of things. Today I powered off (clean shutdown) one osd node and powered it back on. Last time I tried this there was no problem getting it back online. After a few minutes the cluster health was back to ok. This time it stayed degraded forever. I checked and noticed that the service osd.0 on the osd node was failing. So i used google and there people recommended to simply delete the osd and re-create it. I tried it and still can't get the osd back in service. First I removed the osd: [root@gedasvl02 ~]# ceph osd out 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 osd.0 is already out. [root@gedasvl02 ~]# ceph auth del 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 Error EINVAL: bad entity name [root@gedasvl02 ~]# ceph auth del osd.0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 updated [root@gedasvl02 ~]# ceph osd rm 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 removed osd.0 [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 Looks ok it's gone... Then i zapped it: [root@gedasvl02 ~]# ceph orch device zap gedaopl02 /dev/sdb --force INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 INFO:cephadm:/usr/bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices. INFO:cephadm:/usr/bin/podman:stderr --> Zapping: /dev/sdb INFO:cephadm:/usr/bin/podman:stderr --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323 bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0314447 s, 333 MB/s INFO:cephadm:/usr/bin/podman:stderr stderr: INFO:cephadm:/usr/bin/podman:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-3bf1bb28-0858-4464-a848-d7f56319b40a INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/vgremove -v -f ceph-3bf1bb28-0858-4464-a848-d7f56319b40a INFO:cephadm:/usr/bin/podman:stderr stderr: Removing ceph--3bf1bb28--0858--4464--a848--d7f56319b40a-osd--block--3a79800d--2a19--45d8--a850--82c6a8113323 (253:0) INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" metadata (seqno 5). INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume "osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" (seqno 6). INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume "osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" successfully removed INFO:cephadm:/usr/bin/podman:stderr stderr: Removing physical volume "/dev/sdb" from volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" INFO:cephadm:/usr/bin/podman:stderr stdout: Volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" successfully removed INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0355641 s, 295 MB/s INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <Raw Device: /dev/sdb> And re-added it: [root@gedasvl02 ~]# ceph orch daemon add osd gedaopl02:/dev/sdb INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 Created osd(s) 0 on host 'gedaopl02' But the osd is still out... [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 Looking at the cluster log in the webui i see the following error: Failed to apply osd.dashboard-admin-1606745745154 spec DriveGroupSpec(name=dashboard-admin-1606745745154->placement=PlacementSpec(host_pattern='*'), service_id='dashboard-admin-1606745745154', service_type='osd', data_devices=DeviceSelection(size='223.6GB', rotational=False, all=False), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): No filters applied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 2108, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/module.py", line 2005, in _apply_service self.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 43, in create_from_spec ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 127, in prepare_drivegroup drive_selection = DriveSelection(drive_group, inventory_for_host) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 32, in __init__ self._data = self.assign_devices(self.spec.data_devices) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 138, in assign_devices if not all(m.compare(disk) for m in FilterGenerator(device_filter)): File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 138, in <genexpr> if not all(m.compare(disk) for m in FilterGenerator(device_filter)): File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/matchers.py", line 410, in compare raise Exception("No filters applied") Exception: No filters applied I have another error "pgs undersized", maybe this is also causing trouble? [root@gedasvl02 ~]# ceph -s INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: d0920c36-2368-11eb-a5de-005056b703af health: HEALTH_WARN Degraded data redundancy: 13142/39426 objects degraded (33.333%), 176 pgs degraded, 225 pgs undersized services: mon: 1 daemons, quorum gedasvl02 (age 2w) mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby osd: 3 osds: 2 up (since 4d), 2 in (since 94m) task status: scrub status: mds.cephfs.gedaopl01.zjuhem: idle data: pools: 7 pools, 225 pgs objects: 13.14k objects, 77 GiB usage: 148 GiB used, 299 GiB / 447 GiB avail pgs: 13142/39426 objects degraded (33.333%) 176 active+undersized+degraded 49 active+undersized io: client: 0 B/s rd, 6.1 KiB/s wr, 0 op/s rd, 0 op/s wr Best Regards, Oliver
On 2020-11-30 15:55, Oliver Weinmann wrote:
I have another error "pgs undersized", maybe this is also causing trouble?
This is a result of the loss of one OSD, and the PGs located on it. As you only have 1 OSDs left, the cluster cannot recover on a third OSD (assuming defaults here). The cluster will heal itself as soon as the third OSD will be back online. Can you start the OSD? If not, can you provide logs of the failing OSD? Gr. Stefan
Hi Stefan, unfortunately It doesn't start. The failed osd (osd.0) is located on gedaopl02 [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 [root@gedaopl02 ~]# systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● ceph-d0920c36-2368-11eb-a5de-005056b703af@mgr.gedaopl02.pijxbm.service loaded failed failed Ceph mgr.gedaopl02.pijxbm for d0920c36-2368-11eb-a5de-005056b703af ● ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service loaded failed failed Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af ● ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.1.service loaded failed failed Ceph osd.1 for d0920c36-2368-11eb-a5de-005056b703af LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 3 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. I can start the service but then after a minute or so it fails. Maybe I'm looking at the wrong log file, but it's empty: [root@gedaopl02 ~]# tail -f /var/log/ceph/d0920c36-2368-11eb-a5de-005056b703af/ceph-osd.0.log Yesterday when I deleted the failed osd and recreated it there were lots of message in the log file: https://pastebin.com/5hH27pdR Cheers, Oliver Am 01.12.2020 um 09:22 schrieb Stefan Kooman:
On 2020-11-30 15:55, Oliver Weinmann wrote:
I have another error "pgs undersized", maybe this is also causing trouble? This is a result of the loss of one OSD, and the PGs located on it. As you only have 1 OSDs left, the cluster cannot recover on a third OSD (assuming defaults here). The cluster will heal itself as soon as the third OSD will be back online.
Can you start the OSD? If not, can you provide logs of the failing OSD?
Gr. Stefan
On 2020-12-01 10:21, Oliver Weinmann wrote:
Hi Stefan,
unfortunately It doesn't start.
The failed osd (osd.0) is located on gedaopl02
I can start the service but then after a minute or so it fails. Maybe I'm looking at the wrong log file, but it's empty:
Maybe it hits a timeout.
[root@gedaopl02 ~]# tail -f /var/log/ceph/d0920c36-2368-11eb-a5de-005056b703af/ceph-osd.0.log
Yesterday when I deleted the failed osd and recreated it there were lots of message in the log file:
Mostly housekeeping logs. Are your containers running in docker? A docker logs $container-id should give you the right logs in that case. Gr. Stefan
Yes, I deployed via cephadm on CentOS 7, it is using podman. The container doesn't even start up so I don't get a container id. But i checked journalctl -xe, and it seems that its trying to use a container name that still exists. -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has begun starting up. Dec 01 11:39:29 gedaopl02 podman[9976]: Error: no container with name or ID ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0 found: no such container Dec 01 11:39:29 gedaopl02 systemd[1]: Started Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af. -- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished starting up. -- -- The start-up result is done. Dec 01 11:39:29 gedaopl02 bash[9993]: WARNING: The same type, major and minor should not be used for multiple devices. Dec 01 11:39:29 gedaopl02 bash[9993]: Error: error creating container storage: the container name "ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-activate" is already in use by "e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6". You have to remove that container to be able to reuse that name.: that name is already in use Dec 01 11:39:29 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service: main process exited, code=exited, status=125/n/a Dec 01 11:39:29 gedaopl02 bash[10033]: WARNING: The same type, major and minor should not be used for multiple devices. Dec 01 11:39:29 gedaopl02 bash[10033]: Error: error creating container storage: the container name "ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-deactivate" is already in use by "ef696c5a92ea891cbd7651cdab66abe6c4ba49b70ef06e44b51c9be1cdfc36d9". You have to remove that container to be able to reuse that name.: that name is already in use Dec 01 11:39:29 gedaopl02 systemd[1]: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service entered failed state. Dec 01 11:39:29 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service failed. Dec 01 11:39:39 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service holdoff time over, scheduling restart. Dec 01 11:39:39 gedaopl02 systemd[1]: Stopped Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af. -- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished shutting down. Dec 01 11:39:39 gedaopl02 systemd[1]: Starting Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af... -- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has begun starting up. Dec 01 11:39:39 gedaopl02 podman[10134]: Error: no container with name or ID ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0 found: no such container Dec 01 11:39:39 gedaopl02 systemd[1]: Started Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af. -- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished starting up. -- -- The start-up result is done. Dec 01 11:39:40 gedaopl02 bash[10150]: WARNING: The same type, major and minor should not be used for multiple devices. Dec 01 11:39:40 gedaopl02 bash[10150]: Error: error creating container storage: the container name "ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-activate" is already in use by "e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6". You have to remove that container to be able to reuse that name.: that name is already in use Dec 01 11:39:40 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service: main process exited, code=exited, status=125/n/a Dec 01 11:39:40 gedaopl02 bash[10175]: WARNING: The same type, major and minor should not be used for multiple devices. Dec 01 11:39:40 gedaopl02 bash[10175]: Error: error creating container storage: the container name "ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-deactivate" is already in use by "ef696c5a92ea891cbd7651cdab66abe6c4ba49b70ef06e44b51c9be1cdfc36d9". You have to remove that container to be able to reuse that name.: that name is already in use Dec 01 11:39:40 gedaopl02 systemd[1]: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service entered failed state. Dec 01 11:39:40 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service failed. Dec 01 11:39:50 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service holdoff time over, scheduling restart. Dec 01 11:39:50 gedaopl02 systemd[1]: Stopped Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af. -- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has finished shutting down. Dec 01 11:39:50 gedaopl02 systemd[1]: start request repeated too quickly for ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service Dec 01 11:39:50 gedaopl02 systemd[1]: Failed to start Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af. -- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service has failed. -- -- The result is failed. Dec 01 11:39:50 gedaopl02 systemd[1]: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service entered failed state. Dec 01 11:39:50 gedaopl02 systemd[1]: ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service failed. Dec 01 11:39:59 gedaopl02 postfix/smtpd[10257]: connect from localhost[127.0.0.1] Dec 01 11:39:59 gedaopl02 postfix/smtpd[10257]: disconnect from localhost[127.0.0.1] Dec 01 11:40:00 gedaopl02 sshd[10264]: rexec line 32: Deprecated option ServerKeyBits Dec 01 11:40:00 gedaopl02 sshd[10264]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key Dec 01 11:40:00 gedaopl02 sshd[10264]: Connection closed by 127.0.0.1 port 52624 [preauth] podman ps -a didn't show that container. So I googled and stumbled over this post: https://github.com/containers/podman/issues/2553 I was able to fix it by running: podman rm --storage e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6 After that I reset the failure of the service and started it again. systemctl reset-failed ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service systemctl start ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service Now ceph is doing its magic :) [root@gedasvl02 ~]# ceph -s INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: d0920c36-2368-11eb-a5de-005056b703af health: HEALTH_WARN Degraded data redundancy: 1941/39432 objects degraded (4.922%), 19 pgs degraded, 19 pgs undersized 8 pgs not deep-scrubbed in time services: mon: 1 daemons, quorum gedasvl02 (age 2w) mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby osd: 3 osds: 3 up (since 9m), 3 in (since 9m); 18 remapped pgs task status: scrub status: mds.cephfs.gedaopl01.zjuhem: idle data: pools: 7 pools, 225 pgs objects: 13.14k objects, 77 GiB usage: 214 GiB used, 457 GiB / 671 GiB avail pgs: 1941/39432 objects degraded (4.922%) 206 active+clean 18 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+degraded+remapped+backfilling io: recovery: 105 MiB/s, 25 objects/s Many thanks for your help. This was an excellent "Recovery training" :) Am 01.12.2020 um 11:50 schrieb Stefan Kooman:
On 2020-12-01 10:21, Oliver Weinmann wrote:
Hi Stefan,
unfortunately It doesn't start.
The failed osd (osd.0) is located on gedaopl02 I can start the service but then after a minute or so it fails. Maybe I'm looking at the wrong log file, but it's empty: Maybe it hits a timeout. [root@gedaopl02 ~]# tail -f /var/log/ceph/d0920c36-2368-11eb-a5de-005056b703af/ceph-osd.0.log
Yesterday when I deleted the failed osd and recreated it there were lots of message in the log file:
https://pastebin.com/5hH27pdR Mostly housekeeping logs. Are your containers running in docker? A docker logs $container-id should give you the right logs in that case.
Gr. Stefan
On 2020-12-01 13:19, Oliver Weinmann wrote:
podman ps -a didn't show that container. So I googled and stumbled over this post:
https://github.com/containers/podman/issues/2553
I was able to fix it by running:
podman rm --storage e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6
After that I reset the failure of the service and started it again.
systemctl reset-failed ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service systemctl start ceph-d0920c36-2368-11eb-a5de-005056b703af@osd.0.service
Ah, I had this issue once with my first venture into Ceph and docker. There might be a "kill container" option in cephadm, just to make sure it cleans up the container bit. I haven't touched cephadm yet, but IIRC there is such an option.
Now ceph is doing its magic :)
Great!
[root@gedasvl02 ~]# ceph -s INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: d0920c36-2368-11eb-a5de-005056b703af health: HEALTH_WARN Degraded data redundancy: 1941/39432 objects degraded (4.922%), 19 pgs degraded, 19 pgs undersized 8 pgs not deep-scrubbed in time
services: mon: 1 daemons, quorum gedasvl02 (age 2w) mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby osd: 3 osds: 3 up (since 9m), 3 in (since 9m); 18 remapped pgs
task status: scrub status: mds.cephfs.gedaopl01.zjuhem: idle
data: pools: 7 pools, 225 pgs objects: 13.14k objects, 77 GiB usage: 214 GiB used, 457 GiB / 671 GiB avail pgs: 1941/39432 objects degraded (4.922%) 206 active+clean 18 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+degraded+remapped+backfilling
io: recovery: 105 MiB/s, 25 objects/s
Many thanks for your help. This was an excellent "Recovery training" :)
Yes, certainly the best way (and moment) to break Ceph and gain experience ;-). Gr. Stefan
participants (2)
-
Oliver Weinmann
-
Stefan Kooman