desaster recovery Ceph Storage , urgent help needed
Hi I have a worst case, osd's in a 3 node cluster each 4 nvme's won't start we had a ip config change in public network, and mon's died so we managed mon's to come back with new ip's. corosync on 2 rings is fine, all 3 mon's are up osd's won't start how to get back to the pool, already 3vm's are configured and valuable data would be lost... this is like a scenario when all systemdisks on each 3 nodes failed, but osd disks are healthy ... any help to reconstruct this storage is highly appreciated! Gerhard |root@pve01:/var/log# systemctl status ceph-osd@0.service.service ● ceph-osd@0.service.service - Ceph object storage daemon osd.0.service Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: enabled) Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d └─ceph-after-pve-cluster.conf Active: failed (Result: exit-code) since Thu 2020-10-22 00:30:09 CEST; 37min ago Process: 31402 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 0.service (code=exited, status=1/FAILURE) Oct 22 00:30:09 pve01 systemd[1]: ceph-osd@0.service.service: Service RestartSec=100ms expired, scheduling restart. Oct 22 00:30:09 pve01 systemd[1]: ceph-osd@0.service.service: Scheduled restart job, restart counter is at 3. Oct 22 00:30:09 pve01 systemd[1]: Stopped Ceph object storage daemon osd.0.service. Oct 22 00:30:09 pve01 systemd[1]: ceph-osd@0.service.service: Start request repeated too quickly. Oct 22 00:30:09 pve01 systemd[1]: ceph-osd@0.service.service: Failed with result 'exit-code'. Oct 22 00:30:09 pve01 systemd[1]: Failed to start Ceph object storage daemon osd.0.service. | ||ceph mon dump dumped monmap epoch 3 epoch 3 fsid 92d063d7-647c-44b8-95d7-86057ee0ab22 last_changed 2020-10-21 23:31:50.584796 created 2020-10-21 21:00:54.077449 min_mon_release 14 (nautilus) 0: [v2:10.100.200.141:3300/0,v1:10.100.200.141:6789/0] mon.pve01 1: [v2:10.100.200.142:3300/0,v1:10.100.200.142:6789/0] mon.pve02 2: [v2:10.100.200.143:3300/0,v1:10.100.200.143:6789/0] mon.pve03 || |||Networks: auto lo iface lo inet loopback auto eno1np0 iface eno1np0 inet static address 10.110.200.131/24 mtu 9000 #corosync1 10GB auto eno2np1 iface eno2np1 inet static address 10.111.200.131/24 mtu 9000 #Corosync2 10GB iface enp69s0f0 inet manual mtu 9000 auto enp69s0f1 iface enp69s0f1 inet static address 10.112.200.131/24 mtu 9000 #Cluster private 100GB auto vmbr0 iface vmbr0 inet static address 10.100.200.141/24 gateway 10.100.200.1 bridge-ports enp69s0f0 bridge-stp off bridge-fd 0 mtu 9000 #Cluster public 100GB =================================================================================================== ||| ||||ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.112.200.0/24 fsid = 92d063d7-647c-44b8-95d7-86057ee0ab22 mon_allow_pool_delete = true mon_host = 10.100.200.141 10.100.200.142 10.100.200.143 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.100.200.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mon.pve01] public_addr = 10.100.200.141 [mon.pve02] public_addr = 10.100.200.142 [mon.pve03] public_addr = 10.100.200.143 |||| |||||ceph -s cluster: id: 92d063d7-647c-44b8-95d7-86057ee0ab22 health: HEALTH_WARN 1 daemons have recently crashed OSD count 0 < osd_pool_default_size 3 services: mon: 3 daemons, quorum pve01,pve02,pve03 (age 63m) mgr: pve01(active, since 64m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: df -h Filesystem Size Used Avail Use% Mounted on udev 252G 0 252G 0% /dev tmpfs 51G 11M 51G 1% /run rpool/ROOT/pve-1 229G 16G 214G 7% / tmpfs 252G 63M 252G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 252G 0 252G 0% /sys/fs/cgroup rpool 214G 128K 214G 1% /rpool rpool/data 214G 128K 214G 1% /rpool/data rpool/ROOT 214G 128K 214G 1% /rpool/ROOT tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-3 tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-2 tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-0 tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-1 /dev/fuse 30M 32K 30M 1% /etc/pve tmpfs 51G 0 51G 0% /run/user/0 lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme4n1 259:0 0 238.5G 0 disk ├─nvme4n1p1 259:5 0 1007K 0 part ├─nvme4n1p2 259:6 0 512M 0 part └─nvme4n1p3 259:7 0 238G 0 part nvme5n1 259:1 0 238.5G 0 disk ├─nvme5n1p1 259:2 0 1007K 0 part ├─nvme5n1p2 259:3 0 512M 0 part └─nvme5n1p3 259:4 0 238G 0 part nvme0n1 259:12 0 2.9T 0 disk └─ceph--cc77fe1b--c8d4--48be--a7c4--36109439c85c-osd--block--80e0127e--836e--44b8--882d--ac49bfc85866 253:3 0 2.9T 0 lvm nvme1n1 259:13 0 2.9T 0 disk └─ceph--eb8b2fc7--775e--4b94--8070--784e7bbf861e-osd--block--4d433222--e1e8--43ac--8dc7--2e6e998ff122 253:2 0 2.9T 0 lvm nvme3n1 259:14 0 2.9T 0 disk └─ceph--5724bdf7--5124--4244--91d6--e254210c2174-osd--block--2d6fe149--f330--415a--a762--44d037c900b1 253:1 0 2.9T 0 lvm nvme2n1 259:15 0 2.9T 0 disk └─ceph--cb5762e9--40fa--4148--98f4--5b5ddef4c1de-osd--block--793d52e6--4cc7--42aa--8326--df25b21c1237 253:0 0 2.9T 0 lvm |||||
Hi, your mail is formatted in a way that makes it impossible to get all information, so a number of questions first: - are the mons up, or are the mon up and in a quorum? you cannot change mon IP addresses without also adjusting them in the mon map. use the daemon socket on the systems to qeury the current state of the mons - the osd systemd output is useless for debugging. it only states that the osd is not running and not able to start The real log files are located in /var/log/ceph/. If the mon are in quorum, you should find more information here. Keep in mind that you also need to change ceph.conf on the OSD hosts if you change the mon IP addresses, otherwise the OSDs won't be able to find the mon and the processes will die. And I do not understand how corosync should affect your ceph cluster. Ceph does not use corosync... If you need fast help I can recommend the ceph irc channel ;-) Regards, Burkhard
This is a proxmox cluster ... sorry for formating problems of my post :( short plot, we messed with ip addr. change of public network, so monitors went down. we changed monitor information in ceph.conf and with ceph-mon -i pve01 --extract-monmap /tmp/monmap monmaptool --rm pve01 --rm pve02 --rm pve03 /tmp/monmap monmaptool --add pve01 10.100.200.141 --add pve02 10.100.200.142 --add pve03 10.100.200.143 /tmp/monmap monmaptool --print /tmp/monmap ceph-mon -i pve01 --inject-monmap /tmp/monmap restart of all three nodes, but osd's dont't come up so howto recover from this disaster ? # ceph -s cluster: id: 92d063d7-647c-44b8-95d7-86057ee0ab22 health: HEALTH_WARN 1 daemons have recently crashed OSD count 0 < osd_pool_default_size 3 services: mon: 3 daemons, quorum pve01,pve02,pve03 (age 19h) mgr: pve01(active, since 19h) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: cat /etc/pve/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.112.200.0/24 fsid = 92d063d7-647c-44b8-95d7-86057ee0ab22 mon_allow_pool_delete = true mon_host = 10.100.200.141 10.100.200.142 10.100.200.143 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.100.200.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mon.pve01] public_addr = 10.100.200.141 [mon.pve02] public_addr = 10.100.200.142 [mon.pve03] public_addr = 10.100.200.143 Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing +49 8191 4283888 +49 171 4802507 Am 23.10.2020 um 13:50 schrieb Burkhard Linke:
Hi,
your mail is formatted in a way that makes it impossible to get all information, so a number of questions first:
- are the mons up, or are the mon up and in a quorum? you cannot change mon IP addresses without also adjusting them in the mon map. use the daemon socket on the systems to qeury the current state of the mons
- the osd systemd output is useless for debugging. it only states that the osd is not running and not able to start
The real log files are located in /var/log/ceph/. If the mon are in quorum, you should find more information here. Keep in mind that you also need to change ceph.conf on the OSD hosts if you change the mon IP addresses, otherwise the OSDs won't be able to find the mon and the processes will die.
And I do not understand how corosync should affect your ceph cluster. Ceph does not use corosync...
If you need fast help I can recommend the ceph irc channel ;-)
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hace you tried to recover old IPs ? El 23/10/20 a las 14:22, Gerhard W. Recher escribió:
This is a proxmox cluster ... sorry for formating problems of my post :(
short plot, we messed with ip addr. change of public network, so monitors went down.
we changed monitor information in ceph.conf and with ceph-mon -i pve01 --extract-monmap /tmp/monmap monmaptool --rm pve01 --rm pve02 --rm pve03 /tmp/monmap monmaptool --add pve01 10.100.200.141 --add pve02 10.100.200.142 --add pve03 10.100.200.143 /tmp/monmap monmaptool --print /tmp/monmap ceph-mon -i pve01 --inject-monmap /tmp/monmap
restart of all three nodes, but osd's dont't come up
so howto recover from this disaster ?
# ceph -s cluster: id: 92d063d7-647c-44b8-95d7-86057ee0ab22 health: HEALTH_WARN 1 daemons have recently crashed OSD count 0 < osd_pool_default_size 3
services: mon: 3 daemons, quorum pve01,pve02,pve03 (age 19h) mgr: pve01(active, since 19h) osd: 0 osds: 0 up, 0 in
data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:
cat /etc/pve/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.112.200.0/24 fsid = 92d063d7-647c-44b8-95d7-86057ee0ab22 mon_allow_pool_delete = true mon_host = 10.100.200.141 10.100.200.142 10.100.200.143 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.100.200.0/24
[client] keyring = /etc/pve/priv/$cluster.$name.keyring
[mon.pve01] public_addr = 10.100.200.141
[mon.pve02] public_addr = 10.100.200.142
[mon.pve03] public_addr = 10.100.200.143
Gerhard W. Recher
net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing
+49 8191 4283888 +49 171 4802507 Am 23.10.2020 um 13:50 schrieb Burkhard Linke:
Hi,
your mail is formatted in a way that makes it impossible to get all information, so a number of questions first:
- are the mons up, or are the mon up and in a quorum? you cannot change mon IP addresses without also adjusting them in the mon map. use the daemon socket on the systems to qeury the current state of the mons
- the osd systemd output is useless for debugging. it only states that the osd is not running and not able to start
The real log files are located in /var/log/ceph/. If the mon are in quorum, you should find more information here. Keep in mind that you also need to change ceph.conf on the OSD hosts if you change the mon IP addresses, otherwise the OSDs won't be able to find the mon and the processes will die.
And I do not understand how corosync should affect your ceph cluster. Ceph does not use corosync...
If you need fast help I can recommend the ceph irc channel ;-)
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
yep, I now reverted ip changes osd's still do not come up and i see no error in ceph.log , osd logs are empty ... Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing +49 8191 4283888 +49 171 4802507 Am 23.10.2020 um 14:28 schrieb Eneko Lacunza:
Hace you tried to recover old IPs ?
El 23/10/20 a las 14:22, Gerhard W. Recher escribió:
This is a proxmox cluster ... sorry for formating problems of my post :(
short plot, we messed with ip addr. change of public network, so monitors went down.
we changed monitor information in ceph.conf and with ceph-mon -i pve01 --extract-monmap /tmp/monmap monmaptool --rm pve01 --rm pve02 --rm pve03 /tmp/monmap monmaptool --add pve01 10.100.200.141 --add pve02 10.100.200.142 --add pve03 10.100.200.143 /tmp/monmap monmaptool --print /tmp/monmap ceph-mon -i pve01 --inject-monmap /tmp/monmap
restart of all three nodes, but osd's dont't come up
so howto recover from this disaster ?
# ceph -s cluster: id: 92d063d7-647c-44b8-95d7-86057ee0ab22 health: HEALTH_WARN 1 daemons have recently crashed OSD count 0 < osd_pool_default_size 3
services: mon: 3 daemons, quorum pve01,pve02,pve03 (age 19h) mgr: pve01(active, since 19h) osd: 0 osds: 0 up, 0 in
data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:
cat /etc/pve/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.112.200.0/24 fsid = 92d063d7-647c-44b8-95d7-86057ee0ab22 mon_allow_pool_delete = true mon_host = 10.100.200.141 10.100.200.142 10.100.200.143 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.100.200.0/24
[client] keyring = /etc/pve/priv/$cluster.$name.keyring
[mon.pve01] public_addr = 10.100.200.141
[mon.pve02] public_addr = 10.100.200.142
[mon.pve03] public_addr = 10.100.200.143
Gerhard W. Recher
net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing
+49 8191 4283888 +49 171 4802507 Am 23.10.2020 um 13:50 schrieb Burkhard Linke:
Hi,
your mail is formatted in a way that makes it impossible to get all information, so a number of questions first:
- are the mons up, or are the mon up and in a quorum? you cannot change mon IP addresses without also adjusting them in the mon map. use the daemon socket on the systems to qeury the current state of the mons
- the osd systemd output is useless for debugging. it only states that the osd is not running and not able to start
The real log files are located in /var/log/ceph/. If the mon are in quorum, you should find more information here. Keep in mind that you also need to change ceph.conf on the OSD hosts if you change the mon IP addresses, otherwise the OSDs won't be able to find the mon and the processes will die.
And I do not understand how corosync should affect your ceph cluster. Ceph does not use corosync...
If you need fast help I can recommend the ceph irc channel ;-)
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 10/23/20 2:22 PM, Gerhard W. Recher wrote:
This is a proxmox cluster ... sorry for formating problems of my post :(
short plot, we messed with ip addr. change of public network, so monitors went down.
*snipsnap*
so howto recover from this disaster ?
# ceph -s cluster: id: 92d063d7-647c-44b8-95d7-86057ee0ab22 health: HEALTH_WARN 1 daemons have recently crashed OSD count 0 < osd_pool_default_size 3
services: mon: 3 daemons, quorum pve01,pve02,pve03 (age 19h) mgr: pve01(active, since 19h) osd: 0 osds: 0 up, 0 in
data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:
Are you sure that the existing mons have been restarted? If the mon database is still present, the status output should contain at least the pool and osd information. But those numbers are zero... Please check the local osd logs for the actual reason of the failed restart. Regards, Burkhard
participants (3)
-
Burkhard Linke
-
Eneko Lacunza
-
Gerhard W. Recher