Is this the same cluster as the one your reported down OSDs for? Can you share the logs from before the "probing" status? You may have to increase the log level to something like debug_mon = 20. But be cautious and monitor the used disk space, it can increase quite a lot. Did you have any changes in the network infrastructure (links going up and down, usually visible in dmesg)? Are the nodes correctly configured ntp wise? Zitat von Mosharaf Hossain <mosharaf.hossain@bol-online.com>:
Dear Concern I am observing a mon is out of quorum. The current running version of Ceph is octopus.
Total Node in the cluster: 13 Mon: 3/3 Network: 20G(10Gx 2) bonded link Each node capacity: 512GB RAM + 72 Core CPU
root@ceph6:/var/run/ceph# systemctl status ceph-mon@ceph6.service ◠ceph-mon@ceph6.service - Ceph cluster monitor daemon Loaded: loaded (/lib/systemd/system/ceph-mon@.service; indirect; vendor preset: enabled) Active: active (running) since Mon 2023-11-13 14:18:28 +06; 55min ago Main PID: 165299 (ceph-mon) Tasks: 26 CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@ceph6.service └─165299 /usr/bin/ceph-mon -f --cluster ceph --id ceph6 --setuser ceph --setgroup ceph
Nov 13 15:13:09 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:09.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:14 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:14.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:34 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:34.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:39 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:39.191+0600 7
root@ceph6:/var/run/ceph# ceph -s cluster: id: f8096ec7-51db-4557-85e6-57d7fdfe9423 health: HEALTH_WARN 1/3 mons down, quorum ceph2,mon1 nodeep-scrub flag(s) set 614 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph2,mon1 (age 8h), out of quorum: ceph6 mgr: ceph4(active, since 2w), standbys: mon1, ceph3, ceph1, ceph6 mds: cephfs:1 {0=ceph8=up:active} 1 up:standby osd: 107 osds: 102 up (since 3h), 102 in (since 3h); 24 remapped pgs flags nodeep-scrub rgw: 4 daemons active (ceph10.rgw0, ceph7.rgw0, ceph9.rgw0, mon1.rgw0) rgw-nfs: 2 daemons active (ceph7, ceph9)
root@ceph6:/var/run/ceph# ceph daemon mon.ceph6 mon_status { "name": "ceph6", "rank": 2, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958747315978244", "required_mon": [ "kraken", "luminous",
Total 3 mon in the cluster and all addresses are able to ping and telnet between them.
1. mon1: 10.10.10.71 2. ceph2: 10.10.10.52 3. ceph6: 10.10.10.56
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io