Hello everybody! I am fascinated by Ceph, but now I live in moments of terror and despair. I'm using ceph 18.2.2 (reef) and at the moment We need to import 4 OSDs from an old cluster (which was removed by accident). In short, we suspect that the cause and solution for this case lies in the information in the OSD log below: (all OSD daemons repeat the same information) After importing I receive the following information in the "journalctl -u ceph-osd@2" log: --- Jul 17 20:09:55 pxm3 ceph-osd[647313]: in thread 702342c006c0 thread_name:ms_dispatch Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) Jul 17 20:09:55 pxm3 ceph-osd[647313]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x70235785b050] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x7023578a9e2c] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 3: gsignal() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 4: abort() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18a) [0x62edaeab47> Jul 17 20:09:55 pxm3 ceph-osd[647313]: 6: (OSD::handle_osd_map(MOSDMap*)+0x384a) [0x62edaec0aeca] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 7: (OSD::ms_dispatch(Message*)+0x62) [0x62edaec0b332] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc1) [0x62edaf5eea51] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 9: (DispatchQueue::entry()+0x6cf) [0x62edaf5ed53f] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x62edaf40fd5d] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7023578a8134] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x7023579287dc] Jul 17 20:09:55 pxm3 ceph-osd[647313]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 17 20:09:55 pxm3 ceph-osd[647313]: -142> 2024-07-17T20:09:54.845-0300 702356d616c0 -1 osd.2 71 log_to_monitors true Jul 17 20:09:55 pxm3 ceph-osd[647313]: -2> 2024-07-17T20:09:55.362-0300 702342c006c0 -1 osd.2 71 ERROR: bad fsid? i have 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 and inc has f4466e33-b57d-4d68-9909-346> Jul 17 20:09:55 pxm3 ceph-osd[647313]: -1> 2024-07-17T20:09:55.366-0300 702342c006c0 -1 ./src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 702342c006c0 time 2024-07-17T20:09:5> Jul 17 20:09:55 pxm3 ceph-osd[647313]: ./src/osd/OSD.cc: 8098: ceph_abort_msg("bad fsid") Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) --- Above in the OSDs daemon log, we see the "bad fsid" error which indicates that the OSDs are trying to connect to a different Ceph cluster than the one they were initially configured for. Each Ceph cluster has a unique identifier, the FSID, and if the OSDs detect a different FSID, they cannot connect correctly. Note: I believe this is the root cause of the problem and the solution. Problem: OSD has fsid from the old cluster. This would be the solution, change the ceph_fsid (fsid) of the OSDs to the fsid of the new cluster, or recreate a new cluster using the fsid that is recorded in the OSD metadata, that is, recreate the cluster with the fsid of the old cluster? would it be possible?) I also opened a topic for this case on the proxmox forum and inserted a lot of data about this scenario there: https://forum.proxmox.com/threads/ceph-cluster-rebuild-import-bluestore-osds... ----- ~# cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 192.168.0.2/24 fsid = f4466e33-b57d-4d68-9909-3468afd9e5c2 mon_allow_pool_delete = true mon_host = 192.168.0.2 192.168.0.3 192.168.0.1 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.0.0/24 [client] # keyring = /etc/pve/priv/$cluster.$name.keyring keyring = /etc/pve/priv/ceph.client.admin.keyring #[client.crash] # keyring = /etc/pve/ceph/$cluster.$name.keyring [client.crash] key = AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== [client.admin] key = AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mon.pxm1] public_addr = 192.168.0.1 [mon.pxm2] public_addr = 192.168.0.2 [mon.pxm3] public_addr = 192.168.0.3 --- root@pxm3:~# ceph fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 --- ~# ceph -s cluster: id: f4466e33-b57d-4d68-9909-3468afd9e5c2 health: HEALTH_WARN mon pxm1 is low on available space 4 osds down 2 hosts (4 osds) down 1 root (4 osds) down 170 daemons have recently crashed services: mon: 3 daemons, quorum pxm2,pxm3,pxm1 (age 3h) mgr: pxm2(active, since 3h), standbys: pxm1, pxm3 osd: 4 osds: 0 up, 4 in (since 2h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: --- ~# cephadm ls [ { "style": "legacy", "name": "osd.0", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@0", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.1", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@1", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" } ] root@pxm3:~# cephadm ls [ { "style": "legacy", "name": "osd.3", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@3", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.2", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@2", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" } ] --- To import the OSDs, we first increased the epoch of the new cluster, repeating the commands below until the cluster's epoch count number was greater than the OSDs' epoch number: Bash: ceph osd set noin ceph osd set noout ceph osd set noup ceph osd set nodown ceph osd set norebalance ceph osd set nobackfill ceph osd unset noin ceph osd unset noout ceph osd unset noup ceph osd unset nodown ceph osd unset norebalance ceph osd unset nobackfill --- OSD volumes: Code: pxm2: -> OSD.0: /dev/ceph-1740d41a-2ae7-4c4d-820f-ec3702e3ba90/osd-block-39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa -> OSD.1: /dev/ceph-ad425d70-4aa3-419a-997f-f3a4082c9904/osd-block-bb4df480-2b9b-4604-a44d-6151d5c0cb33 pxm3: -> OSD.2: /dev/ceph-94682b88-d09c-4eab-9170-c6d31eac79e6/osd-block-3f6756d6-e64b-4c60-9ac2-305c0e71cc51 -> OSD.3: /dev/ceph-d5ffd027-8289-4a1c-9378-6687d9f950ad/osd-block-eece9fc9-44d6-460b-aced-572c79a98be8 --- ceph-bluestore-tool show-label: -> osb.0: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-0/block": { "osd_uuid": "39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa", "size": 2000397795328, "btime": "2024-07-07T21:32:30.861509-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:32.459102Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg==", "ready": "ready", "require_osd_release": "18", "whoami": "0" } } -> osd.1: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "bb4df480-2b9b-4604-a44d-6151d5c0cb33", "size": 2000397795328, "btime": "2024-07-07T21:32:43.729638-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:45.577456Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA==", "ready": "ready", "require_osd_release": "18", "whoami": "1" } } -> osd.2 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-2 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-2/block": { "osd_uuid": "3f6756d6-e64b-4c60-9ac2-305c0e71cc51", "size": 2000397795328, "btime": "2024-07-07T21:33:07.812888-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:09.404317Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg==", "ready": "ready", "require_osd_release": "18", "whoami": "2" } } -> osd.3 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-3 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-3/block": { "osd_uuid": "eece9fc9-44d6-460b-aced-572c79a98be8", "size": 2000397795328, "btime": "2024-07-07T21:33:25.725294-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:27.323085Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw==", "ready": "ready", "require_osd_release": "18", "whoami": "3" } } --- root@pxm2:~# cat /var/lib/ceph/osd/ceph-0/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 --- ~# ceph daemon osd.0 status no valid command found; 10 closest matches: 0 1 2 abort assert bluefs debug_inject_read_zeros bluefs files list bluefs stats bluestore allocator dump block bluestore allocator fragmentation block admin_socket: invalid command --- ~# ceph osd info osd.0 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa --- ~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 0 0 0 0 0 0 exists 1 0 0 0 0 0 0 exists 2 0 0 0 0 0 0 exists 3 0 0 0 0 0 0 exists root@pxm3:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.27759 root default -2 3.63879 host pxm2 0 1.81940 osd.0 down 1.00000 1.00000 1 1.81940 osd.1 down 1.00000 1.00000 -5 3.63879 host pxm3 3 1.81940 osd.3 down 1.00000 1.00000 2 ssd 1.81940 osd.2 down 1.00000 1.00000 --- ~# ceph osd dump epoch 164 fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 created 2024-07-14T13:04:53.354056-0300 modified 2024-07-17T20:24:17.550123-0300 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 63 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release reef stretch_mode_enabled false max_osd 4 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa osd.1 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists bb4df480-2b9b-4604-a44d-6151d5c0cb33 osd.2 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 3f6756d6-e64b-4c60-9ac2-305c0e71cc51 osd.3 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists eece9fc9-44d6-460b-aced-572c79a98be8 blocklist 192.168.0.1:0/1377959162 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6800/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:0/3150227222 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1759809784 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.1:0/3747372149 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6801/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1472036822 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:0/571633622 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6801/10543 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6800/10543 expires 2024-07-18T00:56:24.333969-0300 --- ~# ceph osd metadata [ { "id": 0 }, { "id": 1 }, { "id": 2 }, { "id": 3 } ] --- ~# cat /var/lib/ceph/osd/ceph-0/keyring [osd.0] key = AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg== root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/keyring [osd.1] key = AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA== root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/keyring [osd.2] key = AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/keyring [osd.3] key = AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw== --- ~# ceph auth list osd.0 key: AQCKQJhmMYWeEhAAosrE8Ff+1kZbKcroi22TvQ== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQCTQJhm2uLvAxAAC3uIcRk9d0sxLJgIxcivtw== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== caps: [mon] allow profile osd caps: [osd] allow * osd.3 key: AQBYMZhmmlaFDRAAtoy4//XweaI94OvrPV1aiQ== caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQAl95NmWuofFRAAhV7/f1PykW/KlcB8Rede9w== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQAl95NmmPQfFRAAG2cQNiWQgdx0Rvr7ZFmuhg== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQAl95Nmr/0fFRAA4TahG5PCbZIsltFgkRNEgA== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQAl95Nm0gYgFRAAWBvqfphKk62InqY9x0ijHg== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rbd-mirror key: AQAl95Nmxg8gFRAAomFkp299Ca04NwGpfbSZRg== caps: [mon] allow profile bootstrap-rbd-mirror client.bootstrap-rgw key: AQAl95Nm3RwgFRAA/QoIemXPFv5Gs1/PWfJuYw== caps: [mon] allow profile bootstrap-rgw client.crash key: AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== caps: [mgr] profile crash caps: [mon] profile crash mgr.pxm1 key: AQDm+JNmtJ3CCRAAl/wZaT6z12LCfAghvm6s4w== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm2 key: AQAm95Nm6+dmABAA6405D8ROtNpJf6iVhMegQA== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm3 key: AQDp95Nm3ZbLDxAAWc1zLaVOhE0wMxLJCL5IGg== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow *