Sorry, dont know where to find mds logs. I just found some logs in
/var/log/ceph/ceph-volume.log from mds pod and here it is(just a piece):
...
[2023-02-15 12:09:07,460][ceph_volume.main][INFO ] Running command: ceph-volume
inventory --format json /dev/sda3
[2023-02-15 12:09:07,461][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk
-plno KNAME,NAME,TYPE
[2023-02-15 12:09:07,471][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda
disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda2 /dev/sda2
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda3 /dev/sda3
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda4 /dev/sda4
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sdb /dev/sdb
disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sr0 /dev/sr0
rom
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/dm-0
/dev/mapper/ceph--893e62b0--dc66--4db6--9487--404cad5907d6-osd--block--19ef4373--d828--4259--9159--86f2e89f4397
lvm
[2023-02-15 12:09:07,479][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs
--noheadings --readonly --separator=";" -a --units=b --nosuffix -S
lv_path=/dev/sda3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
[2023-02-15 12:09:07,525][ceph_volume.process][INFO ] stderr Udev is running and
DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical
volume symlinks in device directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] stderr Udev is running and
DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list
by scanning device directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk
--nodeps -P -o
NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
/dev/sda3
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] stdout NAME="sda3"
KNAME="sda3" MAJ:MIN="8:3" FSTYPE="" MOUNTPOINT=""
LABEL="" UUID="" RO="0" RM="0" MODEL=""
SIZE="1M" STATE="" OWNER="root" GROUP="disk"
MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512"
LOG-SEC="512" ROTA="1" SCHED="bfq" TYPE="part"
DISC-ALN="0" DISC-GRAN="4K" DISC-MAX="1G"
DISC-ZERO="0" PKNAME="sda" PARTLABEL="BIOS-BOOT"
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c
/dev/null -p /dev/sda3
[2023-02-15 12:09:07,553][ceph_volume.process][INFO ] stdout /dev/sda3:
PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="BIOS-BOOT"
PART_ENTRY_UUID="860433f4-8158-432c-88aa-e42901086299"
PART_ENTRY_TYPE="21686148-6449-6e6f-744e-656564454649"
PART_ENTRY_NUMBER="3" PART_ENTRY_OFFSET="1048576"
PART_ENTRY_SIZE="2048" PART_ENTRY_DISK="8:0"
[2023-02-15 12:09:07,554][ceph_volume.process][INFO ] Running command: /usr/sbin/pvs
--noheadings --readonly --units=b --nosuffix --separator=";" -o
vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size /dev/sda3
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev is running and
DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical
volume symlinks in device directory.
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev is running and
DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list
by scanning device directory.
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] stderr Failed to find device for
physical volume "/dev/sda3".
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] Running command:
/usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] stderr unable to read label for
/dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] Running command:
/usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,672][ceph_volume.process][INFO ] stderr unable to read label for
/dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,673][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm
info --query=property /dev/sda3
[2023-02-15 12:09:07,687][ceph_volume.process][INFO ] stdout
DEVLINKS=/dev/disk/by-partuuid/860433f4-8158-432c-88aa-e42901086299
/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3
/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3
/dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:0-part3 /dev/disk/by-partlabel/BIOS-BOOT
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVNAME=/dev/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
DEVPATH=/devices/pci0000:00/0000:00:06.0/virtio2/host2/target2:0:0/2:0:0:0/block/sda/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVTYPE=partition
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_BUS=scsi
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_MODEL=QEMU_HARDDISK
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_PART_ENTRY_DISK=8:0
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_PART_ENTRY_NAME=BIOS-BOOT
....
Also after some manipulation our filesystem now fine(few steps later we create new
metadata pool and restore current from new). But mds still read only. Our ceph shows only
this issue :
cluster:
id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119
health: HEALTH_WARN
1 MDSs are read only
21 daemons have recently crashed
services:
mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 8h)
mgr: a(active, since 102m)
mds: 1/1 daemons up, 1 hot standby
osd: 9 osds: 9 up (since 8h), 9 in (since 7d)
rgw: 1 daemon active (1 hosts, 1 zones)
Thank you