30 Nov
2020
30 Nov
'20
10:55 a.m.
Hi,
I'm still evaluating ceph 15.2.5 in a lab so the problem is not really hurting me, but
I want to understand it and hopefully fix it. It is a good practice. To test the
resilience of the cluster I try to break it by doing all kinds of things. Today I powered
off (clean shutdown) one osd node and powered it back on. Last time I tried this there was
no problem getting it back online. After a few minutes the cluster health was back to ok.
This time it stayed degraded forever. I checked and noticed that the service osd.0 on the
osd node was failing. So i used google and there people recommended to simply delete the
osd and re-create it. I tried it and still can't get the osd back in service.
First I removed the osd:
[root@gedasvl02 ~]# ceph osd out 0
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
osd.0 is already out.
[root@gedasvl02 ~]# ceph auth del 0
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
Error EINVAL: bad entity name
[root@gedasvl02 ~]# ceph auth del osd.0
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
updated
[root@gedasvl02 ~]# ceph osd rm 0
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
removed osd.0
[root@gedasvl02 ~]# ceph osd tree
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43658 root default
-7 0.21829 host gedaopl01
2 ssd 0.21829 osd.2 up 1.00000 1.00000
-3 0 host gedaopl02
-5 0.21829 host gedaopl03
3 ssd 0.21829 osd.3 up 1.00000 1.00000
Looks ok it's gone...
Then i zapped it:
[root@gedasvl02 ~]# ceph orch device zap gedaopl02 /dev/sdb --force
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
INFO:cephadm:/usr/bin/podman:stderr WARNING: The same type, major and minor should not be
used for multiple devices.
INFO:cephadm:/usr/bin/podman:stderr --> Zapping: /dev/sdb
INFO:cephadm:/usr/bin/podman:stderr --> Zapping lvm member /dev/sdb. lv_path is
/dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323
INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero
of=/dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323
bs=1M count=10 conv=fsync
INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in
INFO:cephadm:/usr/bin/podman:stderr 10+0 records out
INFO:cephadm:/usr/bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0314447 s,
333 MB/s
INFO:cephadm:/usr/bin/podman:stderr stderr:
INFO:cephadm:/usr/bin/podman:stderr --> Only 1 LV left in VG, will proceed to destroy
volume group ceph-3bf1bb28-0858-4464-a848-d7f56319b40a
INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/vgremove -v -f
ceph-3bf1bb28-0858-4464-a848-d7f56319b40a
INFO:cephadm:/usr/bin/podman:stderr stderr: Removing
ceph--3bf1bb28--0858--4464--a848--d7f56319b40a-osd--block--3a79800d--2a19--45d8--a850--82c6a8113323
(253:0)
INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group
"ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" metadata (seqno 5).
INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume
"osd-block-3a79800d-2a19-45d8-a850-82c6a8113323"
INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group backup
"/etc/lvm/backup/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" (seqno 6).
INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume
"osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" successfully removed
INFO:cephadm:/usr/bin/podman:stderr stderr: Removing physical volume "/dev/sdb"
from volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a"
INFO:cephadm:/usr/bin/podman:stderr stdout: Volume group
"ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" successfully removed
INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb
bs=1M count=10 conv=fsync
INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in
INFO:cephadm:/usr/bin/podman:stderr 10+0 records out
INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied,
0.0355641 s, 295 MB/s
INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <Raw Device:
/dev/sdb>
And re-added it:
[root@gedasvl02 ~]# ceph orch daemon add osd gedaopl02:/dev/sdb
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
Created osd(s) 0 on host 'gedaopl02'
But the osd is still out...
[root@gedasvl02 ~]# ceph osd tree
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43658 root default
-7 0.21829 host gedaopl01
2 ssd 0.21829 osd.2 up 1.00000 1.00000
-3 0 host gedaopl02
-5 0.21829 host gedaopl03
3 ssd 0.21829 osd.3 up 1.00000 1.00000
0 0 osd.0 down 0 1.00000
Looking at the cluster log in the webui i see the following error:
Failed to apply osd.dashboard-admin-1606745745154 spec
DriveGroupSpec(name=dashboard-admin-1606745745154->placement=PlacementSpec(host_pattern='*'),
service_id='dashboard-admin-1606745745154', service_type='osd',
data_devices=DeviceSelection(size='223.6GB', rotational=False, all=False),
osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): No
filters applied Traceback (most recent call last): File
"/usr/share/ceph/mgr/cephadm/module.py", line 2108, in _apply_all_services if
self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/module.py", line
2005, in _apply_service self.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File
"/usr/share/ceph/mgr/cephadm/services/osd.py", line 43, in create_from_spec ret
= create_from_spec_one(self.prepare_drivegroup(drive_group)) File
"/usr/share/ceph/mgr/cephadm/services/osd.py", line 127, in prepare_drivegroup
drive_selection = DriveSelection(drive_group, inventory_for_host) File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line
32, in __init__ self._data = self.assign_devices(self.spec.data_devices) File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line
138, in assign_devices if not all(m.compare(disk) for m in
FilterGenerator(device_filter)): File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line
138, in <genexpr> if not all(m.compare(disk) for m in
FilterGenerator(device_filter)): File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/matchers.py", line
410, in compare raise Exception("No filters applied") Exception: No filters
applied
I have another error "pgs undersized", maybe this is also causing trouble?
[root@gedasvl02 ~]# ceph -s
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
cluster:
id: d0920c36-2368-11eb-a5de-005056b703af
health: HEALTH_WARN
Degraded data redundancy: 13142/39426 objects degraded (33.333%), 176 pgs
degraded, 225 pgs undersized
services:
mon: 1 daemons, quorum gedasvl02 (age 2w)
mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh
mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby
osd: 3 osds: 2 up (since 4d), 2 in (since 94m)
task status:
scrub status:
mds.cephfs.gedaopl01.zjuhem: idle
data:
pools: 7 pools, 225 pgs
objects: 13.14k objects, 77 GiB
usage: 148 GiB used, 299 GiB / 447 GiB avail
pgs: 13142/39426 objects degraded (33.333%)
176 active+undersized+degraded
49 active+undersized
io:
client: 0 B/s rd, 6.1 KiB/s wr, 0 op/s rd, 0 op/s wr
Best Regards,
Oliver
1 Dec
1 Dec
4:22 a.m.
On 2020-11-30 15:55, Oliver Weinmann wrote:
I have another error "pgs undersized", maybe
this is also causing trouble?
This is a result of the loss of one OSD, and the PGs located on it. As
you only have 1 OSDs left, the cluster cannot recover on a third OSD
(assuming defaults here). The cluster will heal itself as soon as the
third OSD will be back online.
Can you start the OSD? If not, can you provide logs of the failing OSD?
Gr. Stefan
5:21 a.m.
Hi Stefan,
unfortunately It doesn't start.
The failed osd (osd.0) is located on gedaopl02
[root@gedasvl02 ~]# ceph osd tree
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43658 root default
-7 0.21829 host gedaopl01
2 ssd 0.21829 osd.2 up 1.00000 1.00000
-3 0 host gedaopl02
-5 0.21829 host gedaopl03
3 ssd 0.21829 osd.3 up 1.00000 1.00000
0 0 osd.0 down 0 1.00000
[root@gedaopl02 ~]# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● ceph-d0920c36-2368-11eb-a5de-005056b703af(a)mgr.gedaopl02.pijxbm.service
loaded failed failed Ceph mgr.gedaopl02.pijxbm for
d0920c36-2368-11eb-a5de-005056b703af
● ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service loaded failed
failed Ceph osd.0 for d0920c36-2368-11eb-a5de-005056b703af
● ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.1.service loaded failed
failed Ceph osd.1 for d0920c36-2368-11eb-a5de-005056b703af
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
3 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
I can start the service but then after a minute or so it fails. Maybe
I'm looking at the wrong log file, but it's empty:
[root@gedaopl02 ~]# tail -f
/var/log/ceph/d0920c36-2368-11eb-a5de-005056b703af/ceph-osd.0.log
Yesterday when I deleted the failed osd and recreated it there were lots
of message in the log file:
https://pastebin.com/5hH27pdR
Cheers,
Oliver
Am 01.12.2020 um 09:22 schrieb Stefan Kooman:
> On 2020-11-30 15:55, Oliver Weinmann wrote:
>
>> I have another error "pgs undersized", maybe this is also causing
trouble?
> This is a result of the loss of one OSD, and the PGs located on it. As
> you only have 1 OSDs left, the cluster cannot recover on a third OSD
> (assuming defaults here). The cluster will heal itself as soon as the
> third OSD will be back online.
>
> Can you start the OSD? If not, can you provide logs of the failing OSD?
>
> Gr. Stefan
6:50 a.m.
On 2020-12-01 10:21, Oliver Weinmann wrote:
Hi Stefan,
unfortunately It doesn't start.
The failed osd (osd.0) is located on gedaopl02
I can start the service but then after a minute or so it fails. Maybe
I'm looking at the wrong log file, but it's empty:
Maybe it hits a timeout.
[root@gedaopl02 ~]# tail -f
/var/log/ceph/d0920c36-2368-11eb-a5de-005056b703af/ceph-osd.0.log
Yesterday when I deleted the failed osd and recreated it there were lots
of message in the log file:
https://pastebin.com/5hH27pdR
Mostly housekeeping logs. Are your containers running in docker? A
docker logs $container-id should give you the right logs in that case.
Gr. Stefan
8:19 a.m.
Yes, I deployed via cephadm on CentOS 7, it is using podman. The
container doesn't even start up so I don't get a container id. But i
checked journalctl -xe, and it seems that its trying to use a container
name that still exists.
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has
begun starting up.
Dec 01 11:39:29 gedaopl02 podman[9976]: Error: no container with name or
ID ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0 found: no such container
Dec 01 11:39:29 gedaopl02 systemd[1]: Started Ceph osd.0 for
d0920c36-2368-11eb-a5de-005056b703af.
-- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has
finished starting up.
--
-- The start-up result is done.
Dec 01 11:39:29 gedaopl02 bash[9993]: WARNING: The same type, major and
minor should not be used for multiple devices.
Dec 01 11:39:29 gedaopl02 bash[9993]: Error: error creating container
storage: the container name
"ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-activate" is already in
use by
"e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6". You
have to remove that container to be able to reuse that name.: that name
is already in use
Dec 01 11:39:29 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service: main process
exited, code=exited, status=125/n/a
Dec 01 11:39:29 gedaopl02 bash[10033]: WARNING: The same type, major and
minor should not be used for multiple devices.
Dec 01 11:39:29 gedaopl02 bash[10033]: Error: error creating container
storage: the container name
"ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-deactivate" is already
in use by
"ef696c5a92ea891cbd7651cdab66abe6c4ba49b70ef06e44b51c9be1cdfc36d9". You
have to remove that container to be able to reuse that name.: that name
is already in use
Dec 01 11:39:29 gedaopl02 systemd[1]: Unit
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service entered failed
state.
Dec 01 11:39:29 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service failed.
Dec 01 11:39:39 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service holdoff time
over, scheduling restart.
Dec 01 11:39:39 gedaopl02 systemd[1]: Stopped Ceph osd.0 for
d0920c36-2368-11eb-a5de-005056b703af.
-- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has
finished shutting down.
Dec 01 11:39:39 gedaopl02 systemd[1]: Starting Ceph osd.0 for
d0920c36-2368-11eb-a5de-005056b703af...
-- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has
begun starting up.
Dec 01 11:39:39 gedaopl02 podman[10134]: Error: no container with name
or ID ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0 found: no such
container
Dec 01 11:39:39 gedaopl02 systemd[1]: Started Ceph osd.0 for
d0920c36-2368-11eb-a5de-005056b703af.
-- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has
finished starting up.
--
-- The start-up result is done.
Dec 01 11:39:40 gedaopl02 bash[10150]: WARNING: The same type, major and
minor should not be used for multiple devices.
Dec 01 11:39:40 gedaopl02 bash[10150]: Error: error creating container
storage: the container name
"ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-activate" is already in
use by
"e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6". You
have to remove that container to be able to reuse that name.: that name
is already in use
Dec 01 11:39:40 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service: main process
exited, code=exited, status=125/n/a
Dec 01 11:39:40 gedaopl02 bash[10175]: WARNING: The same type, major and
minor should not be used for multiple devices.
Dec 01 11:39:40 gedaopl02 bash[10175]: Error: error creating container
storage: the container name
"ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0-deactivate" is already
in use by
"ef696c5a92ea891cbd7651cdab66abe6c4ba49b70ef06e44b51c9be1cdfc36d9". You
have to remove that container to be able to reuse that name.: that name
is already in use
Dec 01 11:39:40 gedaopl02 systemd[1]: Unit
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service entered failed
state.
Dec 01 11:39:40 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service failed.
Dec 01 11:39:50 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service holdoff time
over, scheduling restart.
Dec 01 11:39:50 gedaopl02 systemd[1]: Stopped Ceph osd.0 for
d0920c36-2368-11eb-a5de-005056b703af.
-- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has
finished shutting down.
Dec 01 11:39:50 gedaopl02 systemd[1]: start request repeated too quickly
for ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
Dec 01 11:39:50 gedaopl02 systemd[1]: Failed to start Ceph osd.0 for
d0920c36-2368-11eb-a5de-005056b703af.
-- Subject: Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service has failed.
--
-- The result is failed.
Dec 01 11:39:50 gedaopl02 systemd[1]: Unit
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service entered failed
state.
Dec 01 11:39:50 gedaopl02 systemd[1]:
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service failed.
Dec 01 11:39:59 gedaopl02 postfix/smtpd[10257]: connect from
localhost[127.0.0.1]
Dec 01 11:39:59 gedaopl02 postfix/smtpd[10257]: disconnect from
localhost[127.0.0.1]
Dec 01 11:40:00 gedaopl02 sshd[10264]: rexec line 32: Deprecated option
ServerKeyBits
Dec 01 11:40:00 gedaopl02 sshd[10264]: error: Could not load host key:
/etc/ssh/ssh_host_dsa_key
Dec 01 11:40:00 gedaopl02 sshd[10264]: Connection closed by 127.0.0.1
port 52624 [preauth]
podman ps -a didn't show that container. So I googled and stumbled over
this post:
https://github.com/containers/podman/issues/2553
I was able to fix it by running:
podman rm --storage
e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6
After that I reset the failure of the service and started it again.
systemctl reset-failed
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
systemctl start ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
Now ceph is doing its magic :)
[root@gedasvl02 ~]# ceph -s
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
cluster:
id: d0920c36-2368-11eb-a5de-005056b703af
health: HEALTH_WARN
Degraded data redundancy: 1941/39432 objects degraded
(4.922%), 19 pgs degraded, 19 pgs undersized
8 pgs not deep-scrubbed in time
services:
mon: 1 daemons, quorum gedasvl02 (age 2w)
mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh
mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby
osd: 3 osds: 3 up (since 9m), 3 in (since 9m); 18 remapped pgs
task status:
scrub status:
mds.cephfs.gedaopl01.zjuhem: idle
data:
pools: 7 pools, 225 pgs
objects: 13.14k objects, 77 GiB
usage: 214 GiB used, 457 GiB / 671 GiB avail
pgs: 1941/39432 objects degraded (4.922%)
206 active+clean
18 active+undersized+degraded+remapped+backfill_wait
1 active+undersized+degraded+remapped+backfilling
io:
recovery: 105 MiB/s, 25 objects/s
Many thanks for your help. This was an excellent "Recovery training" :)
Am 01.12.2020 um 11:50 schrieb Stefan Kooman:
> On 2020-12-01 10:21, Oliver Weinmann wrote:
>> Hi Stefan,
>>
>> unfortunately It doesn't start.
>>
>> The failed osd (osd.0) is located on gedaopl02
>> I can start the service but then after a minute or so it fails. Maybe
>> I'm looking at the wrong log file, but it's empty:
> Maybe it hits a timeout.
>> [root@gedaopl02 ~]# tail -f
>> /var/log/ceph/d0920c36-2368-11eb-a5de-005056b703af/ceph-osd.0.log
>>
>> Yesterday when I deleted the failed osd and recreated it there were lots
>> of message in the log file:
>>
>> https://pastebin.com/5hH27pdR
> Mostly housekeeping logs. Are your containers running in docker? A
> docker logs $container-id should give you the right logs in that case.
>
> Gr. Stefan
8:51 a.m.
On 2020-12-01 13:19, Oliver Weinmann wrote:
podman ps -a didn't show that container. So I googled and stumbled over
this post:
https://github.com/containers/podman/issues/2553
I was able to fix it by running:
podman rm --storage
e43f8533d6418267d7e6f3a408a566b4221df4fb51b13d71c634ee697914bad6
After that I reset the failure of the service and started it again.
systemctl reset-failed
ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
systemctl start ceph-d0920c36-2368-11eb-a5de-005056b703af(a)osd.0.service
Ah, I had this issue once with my first venture into Ceph and docker.
There might be a "kill container" option in cephadm, just to make sure
it cleans up the container bit. I haven't touched cephadm yet, but IIRC
there is such an option.
Now ceph is doing its magic :)
Great!
[root@gedasvl02 ~]# ceph -s
INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af
INFO:cephadm:Inferring config
/var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
cluster:
id: d0920c36-2368-11eb-a5de-005056b703af
health: HEALTH_WARN
Degraded data redundancy: 1941/39432 objects degraded
(4.922%), 19 pgs degraded, 19 pgs undersized
8 pgs not deep-scrubbed in time
services:
mon: 1 daemons, quorum gedasvl02 (age 2w)
mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh
mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby
osd: 3 osds: 3 up (since 9m), 3 in (since 9m); 18 remapped pgs
task status:
scrub status:
mds.cephfs.gedaopl01.zjuhem: idle
data:
pools: 7 pools, 225 pgs
objects: 13.14k objects, 77 GiB
usage: 214 GiB used, 457 GiB / 671 GiB avail
pgs: 1941/39432 objects degraded (4.922%)
206 active+clean
18 active+undersized+degraded+remapped+backfill_wait
1 active+undersized+degraded+remapped+backfilling
io:
recovery: 105 MiB/s, 25 objects/s
Many thanks for your help. This was an excellent "Recovery training" :)
Yes, certainly the best way (and moment) to break Ceph and gain
experience ;-).
Gr. Stefan
1248
days inactive
1249
days old
5 comments
2 participants
participants (2)
-
Oliver Weinmann -
Stefan Kooman