- ceph-users - lists.ceph.io

Please Reply my Mail Sent to you ON (24th July 2024)
by Mrs. Hanana Shrawi 02 Aug '24

02 Aug '24

Dear Greetings I have the pleasure to communicate with you via this medium. My intent is to discuss investment business matters with you so as to be able to learn of the available investment opportunities in your region. I have attech my Personal Documents for you to see, Do you remember me, Looking forward to hearing from you. Best regards, Mrs. Hanan Ashrawi

1 0

Difficulty importing bluestore OSDs from the old cluster (bad fsid) - OSD does not start
by Vinícius Barreto 02 Aug '24

02 Aug '24

Hello everybody! I am fascinated by Ceph, but now I live in moments of terror and despair. I'm using ceph 18.2.2 (reef) and at the moment We need to import 4 OSDs from an old cluster (which was removed by accident). In short, we suspect that the cause and solution for this case lies in the information in the OSD log below: (all OSD daemons repeat the same information) After importing I receive the following information in the "journalctl -u ceph-osd@2" log: --- Jul 17 20:09:55 pxm3 ceph-osd[647313]: in thread 702342c006c0 thread_name:ms_dispatch Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) Jul 17 20:09:55 pxm3 ceph-osd[647313]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x70235785b050] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x7023578a9e2c] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 3: gsignal() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 4: abort() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18a) [0x62edaeab47> Jul 17 20:09:55 pxm3 ceph-osd[647313]: 6: (OSD::handle_osd_map(MOSDMap*)+0x384a) [0x62edaec0aeca] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 7: (OSD::ms_dispatch(Message*)+0x62) [0x62edaec0b332] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc1) [0x62edaf5eea51] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 9: (DispatchQueue::entry()+0x6cf) [0x62edaf5ed53f] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x62edaf40fd5d] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7023578a8134] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x7023579287dc] Jul 17 20:09:55 pxm3 ceph-osd[647313]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 17 20:09:55 pxm3 ceph-osd[647313]: -142> 2024-07-17T20:09:54.845-0300 702356d616c0 -1 osd.2 71 log_to_monitors true Jul 17 20:09:55 pxm3 ceph-osd[647313]: -2> 2024-07-17T20:09:55.362-0300 702342c006c0 -1 osd.2 71 ERROR: bad fsid? i have 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 and inc has f4466e33-b57d-4d68-9909-346> Jul 17 20:09:55 pxm3 ceph-osd[647313]: -1> 2024-07-17T20:09:55.366-0300 702342c006c0 -1 ./src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 702342c006c0 time 2024-07-17T20:09:5> Jul 17 20:09:55 pxm3 ceph-osd[647313]: ./src/osd/OSD.cc: 8098: ceph_abort_msg("bad fsid") Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) --- Above in the OSDs daemon log, we see the "bad fsid" error which indicates that the OSDs are trying to connect to a different Ceph cluster than the one they were initially configured for. Each Ceph cluster has a unique identifier, the FSID, and if the OSDs detect a different FSID, they cannot connect correctly. Note: I believe this is the root cause of the problem and the solution. Problem: OSD has fsid from the old cluster. This would be the solution, change the ceph_fsid (fsid) of the OSDs to the fsid of the new cluster, or recreate a new cluster using the fsid that is recorded in the OSD metadata, that is, recreate the cluster with the fsid of the old cluster? would it be possible?) I also opened a topic for this case on the proxmox forum and inserted a lot of data about this scenario there: https://forum.proxmox.com/threads/ceph-cluster-rebuild-import-bluestore-osd… ----- ~# cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 192.168.0.2/24 fsid = f4466e33-b57d-4d68-9909-3468afd9e5c2 mon_allow_pool_delete = true mon_host = 192.168.0.2 192.168.0.3 192.168.0.1 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.0.0/24 [client] # keyring = /etc/pve/priv/$cluster.$name.keyring keyring = /etc/pve/priv/ceph.client.admin.keyring #[client.crash] # keyring = /etc/pve/ceph/$cluster.$name.keyring [client.crash] key = AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== [client.admin] key = AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mon.pxm1] public_addr = 192.168.0.1 [mon.pxm2] public_addr = 192.168.0.2 [mon.pxm3] public_addr = 192.168.0.3 --- root@pxm3:~# ceph fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 --- ~# ceph -s cluster: id: f4466e33-b57d-4d68-9909-3468afd9e5c2 health: HEALTH_WARN mon pxm1 is low on available space 4 osds down 2 hosts (4 osds) down 1 root (4 osds) down 170 daemons have recently crashed services: mon: 3 daemons, quorum pxm2,pxm3,pxm1 (age 3h) mgr: pxm2(active, since 3h), standbys: pxm1, pxm3 osd: 4 osds: 0 up, 4 in (since 2h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: --- ~# cephadm ls [ { "style": "legacy", "name": "osd.0", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@0", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.1", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@1", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" } ] root@pxm3:~# cephadm ls [ { "style": "legacy", "name": "osd.3", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@3", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.2", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@2", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" } ] --- To import the OSDs, we first increased the epoch of the new cluster, repeating the commands below until the cluster's epoch count number was greater than the OSDs' epoch number: Bash: ceph osd set noin ceph osd set noout ceph osd set noup ceph osd set nodown ceph osd set norebalance ceph osd set nobackfill ceph osd unset noin ceph osd unset noout ceph osd unset noup ceph osd unset nodown ceph osd unset norebalance ceph osd unset nobackfill --- OSD volumes: Code: pxm2: -> OSD.0: /dev/ceph-1740d41a-2ae7-4c4d-820f-ec3702e3ba90/osd-block-39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa -> OSD.1: /dev/ceph-ad425d70-4aa3-419a-997f-f3a4082c9904/osd-block-bb4df480-2b9b-4604-a44d-6151d5c0cb33 pxm3: -> OSD.2: /dev/ceph-94682b88-d09c-4eab-9170-c6d31eac79e6/osd-block-3f6756d6-e64b-4c60-9ac2-305c0e71cc51 -> OSD.3: /dev/ceph-d5ffd027-8289-4a1c-9378-6687d9f950ad/osd-block-eece9fc9-44d6-460b-aced-572c79a98be8 --- ceph-bluestore-tool show-label: -> osb.0: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-0/block": { "osd_uuid": "39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa", "size": 2000397795328, "btime": "2024-07-07T21:32:30.861509-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:32.459102Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg==", "ready": "ready", "require_osd_release": "18", "whoami": "0" } } -> osd.1: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "bb4df480-2b9b-4604-a44d-6151d5c0cb33", "size": 2000397795328, "btime": "2024-07-07T21:32:43.729638-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:45.577456Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA==", "ready": "ready", "require_osd_release": "18", "whoami": "1" } } -> osd.2 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-2 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-2/block": { "osd_uuid": "3f6756d6-e64b-4c60-9ac2-305c0e71cc51", "size": 2000397795328, "btime": "2024-07-07T21:33:07.812888-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:09.404317Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg==", "ready": "ready", "require_osd_release": "18", "whoami": "2" } } -> osd.3 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-3 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-3/block": { "osd_uuid": "eece9fc9-44d6-460b-aced-572c79a98be8", "size": 2000397795328, "btime": "2024-07-07T21:33:25.725294-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:27.323085Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw==", "ready": "ready", "require_osd_release": "18", "whoami": "3" } } --- root@pxm2:~# cat /var/lib/ceph/osd/ceph-0/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 --- ~# ceph daemon osd.0 status no valid command found; 10 closest matches: 0 1 2 abort assert bluefs debug_inject_read_zeros bluefs files list bluefs stats bluestore allocator dump block bluestore allocator fragmentation block admin_socket: invalid command --- ~# ceph osd info osd.0 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa --- ~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 0 0 0 0 0 0 exists 1 0 0 0 0 0 0 exists 2 0 0 0 0 0 0 exists 3 0 0 0 0 0 0 exists root@pxm3:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.27759 root default -2 3.63879 host pxm2 0 1.81940 osd.0 down 1.00000 1.00000 1 1.81940 osd.1 down 1.00000 1.00000 -5 3.63879 host pxm3 3 1.81940 osd.3 down 1.00000 1.00000 2 ssd 1.81940 osd.2 down 1.00000 1.00000 --- ~# ceph osd dump epoch 164 fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 created 2024-07-14T13:04:53.354056-0300 modified 2024-07-17T20:24:17.550123-0300 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 63 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release reef stretch_mode_enabled false max_osd 4 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa osd.1 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists bb4df480-2b9b-4604-a44d-6151d5c0cb33 osd.2 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 3f6756d6-e64b-4c60-9ac2-305c0e71cc51 osd.3 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists eece9fc9-44d6-460b-aced-572c79a98be8 blocklist 192.168.0.1:0/1377959162 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6800/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:0/3150227222 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1759809784 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.1:0/3747372149 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6801/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1472036822 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:0/571633622 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6801/10543 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6800/10543 expires 2024-07-18T00:56:24.333969-0300 --- ~# ceph osd metadata [ { "id": 0 }, { "id": 1 }, { "id": 2 }, { "id": 3 } ] --- ~# cat /var/lib/ceph/osd/ceph-0/keyring [osd.0] key = AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg== root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/keyring [osd.1] key = AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA== root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/keyring [osd.2] key = AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/keyring [osd.3] key = AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw== --- ~# ceph auth list osd.0 key: AQCKQJhmMYWeEhAAosrE8Ff+1kZbKcroi22TvQ== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQCTQJhm2uLvAxAAC3uIcRk9d0sxLJgIxcivtw== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== caps: [mon] allow profile osd caps: [osd] allow * osd.3 key: AQBYMZhmmlaFDRAAtoy4//XweaI94OvrPV1aiQ== caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQAl95NmWuofFRAAhV7/f1PykW/KlcB8Rede9w== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQAl95NmmPQfFRAAG2cQNiWQgdx0Rvr7ZFmuhg== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQAl95Nmr/0fFRAA4TahG5PCbZIsltFgkRNEgA== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQAl95Nm0gYgFRAAWBvqfphKk62InqY9x0ijHg== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rbd-mirror key: AQAl95Nmxg8gFRAAomFkp299Ca04NwGpfbSZRg== caps: [mon] allow profile bootstrap-rbd-mirror client.bootstrap-rgw key: AQAl95Nm3RwgFRAA/QoIemXPFv5Gs1/PWfJuYw== caps: [mon] allow profile bootstrap-rgw client.crash key: AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== caps: [mgr] profile crash caps: [mon] profile crash mgr.pxm1 key: AQDm+JNmtJ3CCRAAl/wZaT6z12LCfAghvm6s4w== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm2 key: AQAm95Nm6+dmABAA6405D8ROtNpJf6iVhMegQA== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm3 key: AQDp95Nm3ZbLDxAAWc1zLaVOhE0wMxLJCL5IGg== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow *

2 1

rbd-mirror keeps crashing
by armsby＠armsby.dk 02 Aug '24

02 Aug '24

Hi everyone, I've been running rbd-mirror between my old Ceph system (16.2.10) and my new system (18.2.2). I'm using journaling mode on a pool that contains 7,500 images. Everything was running perfectly until it processed about 5,608 images. Now, it keeps crashing with the following message: 2024-07-19T05:49:32.425+0000 7f582b3fd6c0 0 set uid:gid to 167:167 (ceph:ceph) 2024-07-19T05:49:32.425+0000 7f582b3fd6c0 0 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), process rbd-mirror, pid 7 2024-07-19T05:49:32.429+0000 7f582b3fd6c0 1 mgrc service_daemon_register rbd-mirror.3606956688 metadata {arch=x86_64, ceph_release=pacific, ceph_version=ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), ceph_version_short=16.2.10, container_hostname=mon-001, container_image=quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd, cpu=Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz, distro=centos, distro_description=CentOS Stream 8, distro_version=8, hostname=mon-001, id=mon-001.lcqrti, instance_id=3606956688, kernel_description=#1 SMP Mon Jul 18 17:42:52 UTC 2022, kernel_version=4.18.0-408.el8.x86_64, mem_swap_kb=4194300, mem_total_kb=131393360, os=Linux} 2024-07-19T05:50:28.305+0000 7f5812582700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f5812582700 time 2024-07-19T05:50:28.303536+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/Thread.cc: 165: FAILED ceph_assert(ret == 0) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58218b6de8] 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f58218b7002] 3: /usr/lib64/ceph/libceph-common.so.2(+0x362fd7) [0x7f58219a2fd7] 4: (CommonSafeTimer<std::mutex>::init()+0x1fe) [0x7f58219a963e] 5: (journal::Journaler::Threads::Threads(ceph::common::CephContext*)+0x2fc) [0x55c9b33c6ddc] 6: (journal::Journaler::Journaler(librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, journal::Settings const&, journal::CacheManagerHandler*)+0x50) [0x55c9b33c6f10] 7: (librbd::Journal<librbd::ImageCtx>::get_tag_owner(librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, librbd::asio::ContextWQ*, Context*)+0x19f) [0x55c9b2fa65af] 8: (librbd::mirror::GetInfoRequest<librbd::ImageCtx>::get_journal_tag_owner()+0x210) [0x55c9b31869f0] 9: (librbd::mirror::GetInfoRequest<librbd::ImageCtx>::handle_get_mirror_image(int)+0x8c8) [0x55c9b3189d78] 10: /lib64/librados.so.2(+0xa8546) [0x7f582aedb546] 11: /lib64/librados.so.2(+0xc17e5) [0x7f582aef47e5] 12: /lib64/librados.so.2(+0xc3742) [0x7f582aef6742] 13: /lib64/librados.so.2(+0xc914a) [0x7f582aefc14a] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f581fb03ba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5820cec1ca] 16: clone() Has anyone encountered a similar problem or have any insight into what might be causing this crash? Thanks in advance for your help.

2 1

Help with osd spec needed
by Kristaps Cudars 02 Aug '24

02 Aug '24

3 nodes each: 3 hdd – 21G 1 ssd – 80G Create osd containing block_data with block_db size 15G located on ssd- This par works Create block_data osd on remaining space 35G in ssd- This part is not working ceph orch apply osd -i /path/to/osd_spec.yml service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' spec: block_db_size: 15G db_slots: 3 data_devices: rotational: 1 db_devices: rotational: 0 --- service_type: osd service_id: odd_spec_ssd placement: host_pattern: '*' spec: data_devices: rotational: 0

2 1

Ceph Quarterly #5 - April 2024 to June 2024
by Zac Dover 02 Aug '24

02 Aug '24

The fifth issue of "Ceph Quarterly" is linked below in this email. Ceph Quarterly (or "CQ") is an overview of the past three months of upstream Ceph development. CQ#5 covers the period from April 2024 to June 2024. https://ceph.io/en/community/cq/Ceph_Quarterly_005_A4.pdf https://ceph.io/en/community/cq/Ceph_Quarterly_005_Letter.pdf Zac Dover Head of Documentation Ceph Foundation

1 0

Debian package for 18.2.4 broken
by Burkhard Linke 01 Aug '24

01 Aug '24

Hi, the Debian bookworm packages for 18.2.4 are completely broken and unusable: 1. missing dependencies, e.g. python3-packaging: root@cc-r3-ceph-2:~# ceph-volume lvm list Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 33, in <module> sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/sbin/ceph-volume", line 25, in importlib_load_entry_point return next(matches).load() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1206, in _gcd_import File "<frozen importlib._bootstrap>", line 1178, in _find_and_load File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 690, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 940, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 9, in <module> from ceph_volume import log, devices, configuration, conf, exceptions, terminal, inventory, drive_group, activate File "/usr/lib/python3/dist-packages/ceph_volume/devices/__init__.py", line 1, in <module> from . import lvm, simple, raw # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/__init__.py", line 1, in <module> from .main import LVM # noqa ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 4, in <module> from . import activate File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/activate.py", line 9, in <module> from ceph_volume.util import encryption as encryption_utils File "/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py", line 9, in <module> from packaging import version ModuleNotFoundError: No module named 'packaging' 2. After installing missing dependencies activating encrypted OSDs fails: root@cc-r3-ceph-2:~# ceph-volume lvm activate --all --> Activating OSD ID 875 FSID 06296c01-4c59-4fde-afe4-0e1e4f078ba9 --> InvalidVersion: Invalid version: 'cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI ' Downgrade to 18.2.2 resolves the problem. Did anyone test the packages before release? Regards, Burkhard

2 1

Problems with crash and k8sevents modules
by Massimo Sgaravatto 01 Aug '24

01 Aug '24

Dear all Today I realized that the syslog on the mon/mgr modules was complaining about missing credentials for the crash client So. following what is reported in: https://docs.ceph.com/en/quincy/mgr/crash/ I did: root@ceph-mon-01 ~]# ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash' [client.crash] key = AQAqNKtmny3yNRAALoVRCgPju2Epc+BPfqOSdw== and on the 3 mon/mgr nodes I created the file /etc/ceph/ceph.client.crash.keyring root@ceph-mon-01 ~]# ll /etc/ceph/ceph.client.crash.keyring -rw-r--r-- 1 ceph ceph 64 Aug 1 09:21 /etc/ceph/ceph.client.crash.keyring [root@ceph-mon-01 ~]# cat /etc/ceph/ceph.client.crash.keyring [client.crash] key = AQAqNKtmny3yNRAALoVRCgPju2Epc+BPfqOSdw== But it is still complaining in the syslog [*] Do I need to also create a client.crash.$hostname ? I have to admit that the doc is not fully clear to me .... At any rate now ceph shows: [root@ceph-mon-01 ~]# ceph health detail HEALTH_WARN 8 mgr modules have recently crashed [WRN] RECENT_MGR_MODULE_CRASH: 8 mgr modules have recently crashed mgr module k8sevents crashed in daemon mgr.ceph-mon-01 on host ceph-mon-01.cloud.pd.infn.it at 2024-07-23T08:53:52.543451Z mgr module k8sevents crashed in daemon mgr.ceph-mon-01 on host ceph-mon-01.cloud.pd.infn.it at 2024-07-23T09:09:17.703402Z mgr module k8sevents crashed in daemon mgr.ceph-mon-02 on host ceph-mon-02.cloud.pd.infn.it at 2024-07-22T13:44:18.779525Z mgr module k8sevents crashed in daemon mgr.ceph-mon-02 on host ceph-mon-02.cloud.pd.infn.it at 2024-07-22T13:57:24.207309Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-07-22T10:42:42.002880Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-07-22T11:09:21.639576Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-08-01T06:02:01.139099Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-08-01T07:45:16.249664Z but that module is disabled [**]. So I am a bit confused Thanks, Massimo [**] [root@ceph-mon-01 ~]# ceph mgr module ls MODULE balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) dashboard on restful on alerts - cephadm - influx - insights - iostat - k8sevents - localpool - mds_autoscaler - mirroring - nfs - osd_perf_query - osd_support - prometheus - rgw - rook - selftest - snap_schedule - stats - telegraf - test_orchestrator - zabbix - [root@ceph-mon-01 ~]# ceph mgr module disable k8sevents module 'k8sevents' is already disabled [root@ceph-mon-01 ~]# [*] Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-07-23T08:53:52.543451Z_da7be91d-5025-4bd0-b6a9-8fe2a1d3b71d as client.crash.ceph-mon-01.cloud.pd.infn.it failed: 2024-08-01T09:26:39.599+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.599+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44b4060ac0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44b408c3d0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44ba8ad0c0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: [errno 2] RADOS object not found (error connecting to the cluster) Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-07-23T09:09:17.703402Z_911fe89d-8f28-4f0e-8280-09b2e8a2e342 as client.crash.ceph-mon-01.cloud.pd.infn.it failed: 2024-08-01T09:26:40.277+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.277+0200 7fdfa4314640 -1 AuthRegistry(0x7fdf9c060ac0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.282+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.282+0200 7fdfa4314640 -1 AuthRegistry(0x7fdf9c08c3d0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.283+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.283+0200 7fdfa4314640 -1 AuthRegistry(0x7fdfa43130c0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: [errno 2] RADOS object not found (error connecting to the cluster)

1 0

Re: 18.2.4 regression: 'diskprediction_local' has failed: No module named 'sklearn'
by Devender Singh 01 Aug '24

01 Aug '24

What is the fix for above error. My OS is : ubuntu 20.04 and ceph version is 18.2.4 Currently I disabled the prediction. : ceph mgr module disable diskprediction_local Regards Dev

1 0

ceph-mgr memory problems 16.2.15
by timureh＠gmail.com 01 Aug '24

01 Aug '24

I have faced a similar problem that is described here https://tracker.ceph.com/issues/59580#change-270387 upon update to 16.2.15. I have a question for developers. Are you planning to fix this problem? Thank you in advance.

1 0

Osds going down/flapping after Luminous to Nautilus upgrade part 2
by Mark Kirkwood 01 Aug '24

01 Aug '24

This 2nd post is about the next type of flapping osds we encountered after upgrading. We started to see osds going down with this in 'ceph -w': 2024-08-01 12:02:57.437135 mon.cat-hlz-stor001 [INF] osd.479 marked down after no beacon for 902.637005 seconds 2024-08-01 12:02:57.468372 mon.cat-hlz-stor001 [WRN] Health check failed: 1 osds down (OSD_DOWN) We have the beacon interval set to 300. To fix this we tried: - restarting osds - restarting mons - ntp tidyup - restarting mgrs However it is still happening. Poking around in the osd and mon logs we did see some lines that hinted that the mon might be listening for beacons using v1 - which could be broken (see part 1). Hence restarting them again. This did not have any effect. Apart from enabling the v2 msgr we have not altered our Luminous config for Nautilus, are we missing something? Regards Mark

1 1