Stephane, If you are using cephadm, the OS (distrib and version) you use should not matter. When using cephadm with several servers (the general case!), it is important to setup properly the SSH key used by cephadm for the communication between nodes (cephadm is sort of a SSH-based management cluster) and to check that you can log in from one node to the other using SSH. Can you confirm that it is the case? Also cephadm has a specific log file. I don't use much the dashboard, not sure how you display it (it may be part of the logs displayed by the dashboard) but you can access it with the command: ceph log last debug cephadm Michel Le 21/07/2025 à 09:19, Stéphane Barthes a écrit :

Hi,

Yes, I did use cephadm, to bootstrap the 1st node in the cluster, installed cephadm on the other nodes, and used the dashboard to add the nodes to the cluster.

Regards,

S. Barthes

Le 21/07/2025 à 09:12, Michel Jouvin a écrit :

...

Hi Stephane,

How did you configure your cluster? Have you been using cephadm? If not, I really advise you to recreate your cluster with cephadm, that includes a script to bootstrap the cluster. In particular if you don't have a detail knowledge about Ceph architecture and management, it will ensure that your cluster is properly configured and let you progressively learn about Ceph details...

Best regards.

Michel

Le 21/07/2025 à 09:02, Stéphane Barthes a écrit :

...

Hello,

I am very new to ceph and have started a small cluster to get started with ceph.

But so far my experience, is not very impressive, probably by lack of knowledge and good practices.

I started with Ubuntu 24, installed 3 VM for a ceph cluster, and some how could not get it running. Adding nodes would fail adding OSDs with some weird error(I found it on the web but could not solve the problem).

I then made a new cluster with 3 ubuntu 22 VM. Install ok, start ok, I created 1 pool to test storing stuff there and work my way across crash testing. However the cluster dies during the weekly vm snapshot. It may not a good idea to run vm backups on a ceph host, but I find this a little surprising. (crash testing started earlier than expected)

Bottom line is that, after the backup the cluster is in warning state with missing mons, or logrotate and  sometimes crashed machines. systemctl restart service or Rebooting node usually fixes it.

I am now stuck in a situation I cannot fix :

    - 1 Machine is ceph rbd client cannot auth : auth method 'x' error -13. I have tried quite a few things, and none unlocked the situation. I am currently trying to reboot the machine, but the busy/stuck rbd device seems to block it. I am not looking forward to hard reset it.

    - Node with the mgr service will not restart mon, or logrotate. I did reboot it again today, but I guess this is not how a node is expected to behave.

So my questions :

    - How can I unlock my stuck ceph client, when this kind of error occurs?

    - Is this expected behavior that client looses access to cluster, which kind of kills the machine?

    - Where should I look in the ceph nodes logs to figure what is going wrong, and how to fix it, so that is run in a stable manner?

Regards,

-- S. Barthes

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Stéphane, On ceph-02, I am not sure why the ceph command is not installed as on the other nodes, if you installed it the same way. One way to get access to the ceph command on this server should be to execute: cephadm shell This will start a container where you have the ceph environment installed and configured for your cluster. The situation is not as bad as I thought reading your first message. You have the mon quorum so at least ceph command should be usable. The first thing to do is probably to log on your ceph-01 node and try to understand why the mon daemon is crashing. You may want to run on this node: cephadm ls  ---> Look for the exact daemon name corresponding to the mon cephadm logs --daemon $daemon_name Apart from this, it is strange that ceph-03 report a RADOS error with 'ceph log last...', this probably hides another issue. Could you tell what the same command says on ceph-02 (when run in cephadm shell)? Michel Le 21/07/2025 à 09:44, Stéphane Barthes a écrit :

Michel,

I ran "ceph log last debug cephadm" on my 3 nodes, and "mileage varies"

ceph-01 :

some errors, and it ends with

2025-07-20T03:24:18.887889+0000 mgr.srvr-ceph-03.dhzbpe (mgr.134360) 1368 : cephadm [INF] Deploying daemon mon.srvr-ceph-03 on srvr-ceph-03

when I had to remove the mon daemon and redeploy on ceph-03.

ceph-02 :

root@srvr-ceph-02:~# ceph log last debug cephadm Command 'ceph' not found, but can be installed with: snap install microceph    # version 18.2.4+snapc9f2b08f92, or apt  install ceph-common  # version 17.2.7-0ubuntu0.22.04.2 See 'snap info microceph' for additional versions.

??? should I install ceph-common ???

ceph-03 :

root@srvr-ceph-03:~# ceph log last debug cephadm Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') root@srvr-ceph-03:~#

FWIW : ceph health is :

root@srvr-ceph-01:~# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum srvr-ceph-03,srvr-ceph-02; 10 daemons have recently crashed [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)     daemon mon.srvr-ceph-01 on srvr-ceph-01 is in error state [WRN] MON_DOWN: 1/3 mons down, quorum srvr-ceph-03,srvr-ceph-02     mon.srvr-ceph-01 (rank 0) addr [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] is down (out of quorum) [WRN] RECENT_CRASH: 10 daemons have recently crashed     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:50:10.202091Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:49:47.712267Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:50:21.464475Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:49:36.609442Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:49:58.966663Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:51:36.947240Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:52:21.751711Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:51:48.490875Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:51:59.651129Z     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at 2025-07-20T17:52:10.552756Z

S. Barthes Le 21/07/2025 à 09:31, Michel Jouvin a écrit :

...

Stephane,

If you are using cephadm, the OS (distrib and version) you use should not matter. When using cephadm with several servers (the general case!), it is important to setup properly the SSH key used by cephadm for the communication between nodes (cephadm is sort of a SSH-based management cluster) and to check that you can log in from one node to the other using SSH. Can you confirm that it is the case?

Also cephadm has a specific log file. I don't use much the dashboard, not sure how you display it (it may be part of the logs displayed by the dashboard) but you can access it with the command:

ceph log last debug cephadm

Michel

Le 21/07/2025 à 09:19, Stéphane Barthes a écrit :

...

Hi,

Yes, I did use cephadm, to bootstrap the 1st node in the cluster, installed cephadm on the other nodes, and used the dashboard to add the nodes to the cluster.

Regards,

S. Barthes

Le 21/07/2025 à 09:12, Michel Jouvin a écrit :

...

Hi Stephane,

How did you configure your cluster? Have you been using cephadm? If not, I really advise you to recreate your cluster with cephadm, that includes a script to bootstrap the cluster. In particular if you don't have a detail knowledge about Ceph architecture and management, it will ensure that your cluster is properly configured and let you progressively learn about Ceph details...

Best regards.

Michel

Le 21/07/2025 à 09:02, Stéphane Barthes a écrit :

...

Hello,

I am very new to ceph and have started a small cluster to get started with ceph.

But so far my experience, is not very impressive, probably by lack of knowledge and good practices.

I started with Ubuntu 24, installed 3 VM for a ceph cluster, and some how could not get it running. Adding nodes would fail adding OSDs with some weird error(I found it on the web but could not solve the problem).

I then made a new cluster with 3 ubuntu 22 VM. Install ok, start ok, I created 1 pool to test storing stuff there and work my way across crash testing. However the cluster dies during the weekly vm snapshot. It may not a good idea to run vm backups on a ceph host, but I find this a little surprising. (crash testing started earlier than expected)

Bottom line is that, after the backup the cluster is in warning state with missing mons, or logrotate and sometimes crashed machines. systemctl restart service or Rebooting node usually fixes it.

I am now stuck in a situation I cannot fix :

    - 1 Machine is ceph rbd client cannot auth : auth method 'x' error -13. I have tried quite a few things, and none unlocked the situation. I am currently trying to reboot the machine, but the busy/stuck rbd device seems to block it. I am not looking forward to hard reset it.

    - Node with the mgr service will not restart mon, or logrotate. I did reboot it again today, but I guess this is not how a node is expected to behave.

So my questions :

    - How can I unlock my stuck ceph client, when this kind of error occurs?

    - Is this expected behavior that client looses access to cluster, which kind of kills the machine?

    - Where should I look in the ceph nodes logs to figure what is going wrong, and how to fix it, so that is run in a stable manner?

Regards,

-- S. Barthes

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hello Stéphane, I think, you're mixing and mismatching up a lot! You always have to show us the output of: ceph -s And more! Logs and stuff, e. g.: ceph health detail It is clear you missed something here and there. It is repairable but since it is a test cluster, just delete it and start again. And follow the documentation for cephadm. And do not use Ubuntu 24.04. Best, Malte On 22.07.25 09:02, Stéphane Barthes wrote:

Hello,

Today, things have degraded a bit more. ceph-03 mon has failed and will not restart. It shows the same kind of checksum error in rocksdb compact operation during startup. As a consequence, I lost quorum, and ceph commands hang.

Would it be wise to disable rocksbd compact, to restart and find quorum back? If yes, what is the exactt syntax of the setting in ceph.conf, I have seen one for OSD, but not sure if it would apply:

[osd]

osd_compact_on_start = true

If I can restart, I will try to out the OSDs, and recreate them. Last time I saw the OSD seemed fine in the dashboard. Since I hove no dashboard, is there a command I can use to check their status?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny

Le 21/07/2025 à 14:27, Stéphane Barthes a écrit :

...

Michel,

cephadm shell starts on all 3 nodes without error, and each host as the same ceph public key entry in the .ssh/autorized_key file of the root user.

ceph-01 also has ceph.pub in /etc/ceph with the same key (this is the node I started the install from)

ceph-2 has no/etc/ceph folder

ceph-3 has a /etc/ceph folder, but no ceph.pub file there

S. Barthes Le 21/07/2025 à 12:36, Michel Jouvin a écrit :

...

Hi Stéphane,

Sorry I was busy and did not look at your previous answers... It is a bit difficult for me to understand how you ended up in this situation but for me it is strange that ceph-02 complains about a missing keyring and the corruped rocks.db on a freshly created cluster is also a bit strange for me. I don't think it makes sense to destroy and recreate the OSD, I am running several clusters with hundreds of OSDs and I never saw a mis-initialized one. The problem is hiding something else I'm afraid. Because of some misconfiguration, may be one OSD is in a bad state and may need to be reinitialized but first we should get the 3 mons running properly and `cephadm shell` working properly on the 3 hosts. And the rocks.db compaction issue for me is related to your mon, not to an OSD.

Have you checked that SSH configuration for cephadm is working well from any host to any other one in your cluster (with 3 hosts, it should be really straighforward to check). The ceph-02 problem may be the sign of SSH misconfiguration as cephadm will use SSH connection to push the keyring, if I am right.

Michel

Le 21/07/2025 à 12:17, Stéphane Barthes a écrit :

...

Hi,

Should I just wipe the OSD and let ceph rebuild it (as suggested there : https://ceph-users.ceph.narkive.com/LO6ebu9r/how-to-recover- from-corrupted-rocksdb) ?

Which would the suggested way be  :

cephadm rm-daemon osd.ceph-01

then

cephadm deploy ?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 21/07/2025 à 10:33, Stéphane Barthes a écrit :

...

Michel,

ceph-02 logs :

root@srvr-ceph-02:/# ceph log last debug cephadm 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory

2025-07-21T08:16:54.814+0000 7efe1a884640 -1 AuthRegistry(0x7efe14064de0) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ keyring,/etc/ceph/keyring.bin, disabling cephx

2025-07-21T08:16:54.818+0000 7efe1a884640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory

2025-07-21T08:16:54.818+0000 7efe1a884640 -1 AuthRegistry(0x7efe1a883000) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ keyring,/etc/ceph/keyring.bin, disabling cephx

2025-07-21T08:16:54.818+0000 7efe137fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:16:54.818+0000 7efe18e21640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:16:57.818+0000 7efe137fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:16:57.818+0000 7efe13fff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:17:00.818+0000 7efe137fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:17:00.818+0000 7efe18e21640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

^CCluster connection aborted root@srvr-ceph-02:/#

Regarding the ceph-01 log, there is a LOT. looking from the end, I see this :

Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -19> 2025-07-20T17:52:21.137+0000 7f359f42d8c0  2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-srvr-ceph-01/keyring Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -18> 2025-07-20T17:52:21.137+0000 7f359f42d8c0  2 mon.srvr- ceph-01@-1(???) e5 init Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -17> 2025-07-20T17:52:21.137+0000 7f359f42d8c0  4 mgrc handle_mgr_map Got map version 73 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -16> 2025-07-20T17:52:21.137+0000 7f359f42d8c0  4 mgrc handle_mgr_map Active mgr is now [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -15> 2025-07-20T17:52:21.137+0000 7f359f42d8c0  4 mgrc reconnect Starting new session with [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -14> 2025-07-20T17:52:21.137+0000 7f359c206640 -1 mon.srvr- ceph-01@-1(???) e5 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -13> 2025-07-20T17:52:21.137+0000 7f359f42d8c0  0 mon.srvr- ceph-01@-1(probing) e5  my rank is now 0 (was -1) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -12> 2025-07-20T17:52:21.161+0000 7f359d208640  3 rocksdb: [db/db_impl/ db_impl_compaction_flush.cc:3026] Compaction error: Corruption: block checksum mismatch: stored = 3368055299, computed = 2100551158  in /var/lib/ceph/mon/ceph-srvr-ceph-01/ store.db/061999.sst offset 10379525 size 91317 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -11> 2025-07-20T17:52:21.161+0000 7f359d208640  4 rocksdb: (Original Log Time 2025/07/20-17:52:21.164193) [db/compaction/ compaction_job.cc:812] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 1] max score 0.00, MB/sec: 514.9 rd, 272.6 wr, level 6, files in(4, 1) out(0) MB in(4.0, 14.8) out(9.9), read-write-amplify(7.2) write- amplify(2.5) Corruption: block checksum mismatch: stored = 3368055299, computed = 2100551158 in /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst offset 10379525 size 91317, records in: 25191, records dropped: 3 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug    -10> 2025-07-20T17:52:21.161+0000 7f359d208640  4 rocksdb: (Original Log Time 2025/07/20-17:52:21.164212) EVENT_LOG_v1 {"time_micros": 1753033941164205, "job": 3, "event": "compaction_finished", "compaction_time_micros": 38166, "compaction_time_cpu_micros": 25133, "output_level": 6, "num_output_files": 0, "total_output_size": 10404253, "num_input_records": 25191, "num_output_records": 21216, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 0, 0, 0, 0, 0, 1]} Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -9> 2025-07-20T17:52:21.161+0000 7f359d208640  2 rocksdb: [db/db_impl/ db_impl_compaction_flush.cc:2545] Waiting after background compaction error: Corruption: block checksum mismatch: stored = 3368055299, computed = 2100551158  in /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst offset 10379525 size 91317, Accumulated background error counts: 1 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -8> 2025-07-20T17:52:21.341+0000 7f359c206640 -1 mon.srvr- ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -7> 2025-07-20T17:52:21.741+0000 7f359c206640 -1 mon.srvr- ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -6> 2025-07-20T17:52:21.741+0000 7f359c206640  1 mon.srvr- ceph-01@0(probing) e5 handle_auth_request failed to assign global_id Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -5> 2025-07-20T17:52:21.749+0000 7f35981fe640  5 mon.srvr- ceph-01@0(probing) e5 _ms_dispatch setting monitor caps on this connection Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -4> 2025-07-20T17:52:21.749+0000 7f35981fe640  1 mon.srvr- ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -3> 2025-07-20T17:52:21.749+0000 7f35981fe640  1 mon.srvr- ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap obtained monmap e5 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: [285B blob data] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = mon_sync key = 'latest_monmap' value size = 508) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = mon_sync key = 'in_sync' value size = 8) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = mon_sync key = 'last_committed_floor' value size = 8) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug     -1> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 /home/jenkins-build/ build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f35981fe640 time 2025-07-20T17:52:21.750611+0000 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: /home/jenkins-build/build/ workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: 355: ceph_abort_msg("failed to write to db") Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7f35a03a5469] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /usr/bin/ceph- mon(+0x1e968e) [0x55e079c5768e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) [0x55e079c8a145] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) [0x55e079c90baf] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) [0x55e079c925dc] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) [0x55e079cad20d] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: /usr/bin/ceph- mon(+0x1f6d3e) [0x55e079c64d3e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: /usr/lib64/ceph/ libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: /lib64/ libc.so.6(+0x89e92) [0x7f359fb6ce92] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /lib64/ libc.so.6(+0x10ef20) [0x7f359fbf1f20] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug      0> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 *** Caught signal (Aborted) ** Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  in thread 7f35981fe640 thread_name:ms_dispatch Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: /lib64/ libc.so.6(+0x3e730) [0x7f359fb21730] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /lib64/ libc.so.6(+0x8bbdc) [0x7f359fb6ebdc] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: raise() Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: abort() Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x190) [0x7f35a03a5526] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: /usr/bin/ceph- mon(+0x1e968e) [0x55e079c5768e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) [0x55e079c8a145] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) [0x55e079c90baf] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) [0x55e079c925dc] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) [0x55e079cad20d] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /usr/bin/ceph- mon(+0x1f6d3e) [0x55e079c64d3e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  13: (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  14: /usr/lib64/ceph/ libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  15: /lib64/ libc.so.6(+0x89e92) [0x7f359fb6ce92] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  16: /lib64/ libc.so.6(+0x10ef20) [0x7f359fbf1f20] Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

I do not know if the logs are purged from sensitive data that would prevent emailing them. looking for "checksum mismatch", in the logs, there are many of them (138).

How can I fix this checksum issue?

Regards,

S. Barthes

Le 21/07/2025 à 09:59, Michel Jouvin a écrit :

...

Stéphane,

On ceph-02, I am not sure why the ceph command is not installed as on the other nodes, if you installed it the same way. One way to get access to the ceph command on this server should be to execute:

cephadm shell

This will start a container where you have the ceph environment installed and configured for your cluster.

The situation is not as bad as I thought reading your first message. You have the mon quorum so at least ceph command should be usable. The first thing to do is probably to log on your ceph-01 node and try to understand why the mon daemon is crashing. You may want to run on this node:

cephadm ls  ---> Look for the exact daemon name corresponding to the mon

cephadm logs --daemon $daemon_name

Apart from this, it is strange that ceph-03 report a RADOS error with 'ceph log last...', this probably hides another issue. Could you tell what the same command says on ceph-02 (when run in cephadm shell)?

Michel

Le 21/07/2025 à 09:44, Stéphane Barthes a écrit : > > Michel, > > > I ran "ceph log last debug cephadm" on my 3 nodes, and "mileage > varies" > > ceph-01 : > > some errors, and it ends with > > 2025-07-20T03:24:18.887889+0000 mgr.srvr-ceph-03.dhzbpe > (mgr.134360) 1368 : cephadm [INF] Deploying daemon mon.srvr- > ceph-03 on srvr-ceph-03 > > when I had to remove the mon daemon and redeploy on ceph-03. > > ceph-02 : > > root@srvr-ceph-02:~# ceph log last debug cephadm > Command 'ceph' not found, but can be installed with: > snap install microceph    # version 18.2.4+snapc9f2b08f92, or > apt  install ceph-common  # version 17.2.7-0ubuntu0.22.04.2 > See 'snap info microceph' for additional versions. > > ??? should I install ceph-common ??? > > ceph-03 : > > root@srvr-ceph-03:~# ceph log last debug cephadm > Error initializing cluster client: ObjectNotFound('RADOS object > not found (error calling conf_read_file)') > root@srvr-ceph-03:~# > > FWIW : ceph health is : > > root@srvr-ceph-01:~# ceph health detail > HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum > srvr-ceph-03,srvr-ceph-02; 10 daemons have recently crashed > [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) >     daemon mon.srvr-ceph-01 on srvr-ceph-01 is in error state > [WRN] MON_DOWN: 1/3 mons down, quorum srvr-ceph-03,srvr-ceph-02 >     mon.srvr-ceph-01 (rank 0) addr > [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] is down (out of > quorum) > [WRN] RECENT_CRASH: 10 daemons have recently crashed >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:50:10.202091Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:49:47.712267Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:50:21.464475Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:49:36.609442Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:49:58.966663Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:51:36.947240Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:52:21.751711Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:51:48.490875Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:51:59.651129Z >     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at > 2025-07-20T17:52:10.552756Z > > S. Barthes > Le 21/07/2025 à 09:31, Michel Jouvin a écrit : >> Stephane, >> >> If you are using cephadm, the OS (distrib and version) you use >> should not matter. When using cephadm with several servers (the >> general case!), it is important to setup properly the SSH key >> used by cephadm for the communication between nodes (cephadm is >> sort of a SSH-based management cluster) and to check that you >> can log in from one node to the other using SSH. Can you confirm >> that it is the case? >> >> Also cephadm has a specific log file. I don't use much the >> dashboard, not sure how you display it (it may be part of the >> logs displayed by the dashboard) but you can access it with the >> command: >> >> ceph log last debug cephadm >> >> Michel >> >> Le 21/07/2025 à 09:19, Stéphane Barthes a écrit : >>> >>> Hi, >>> >>> Yes, I did use cephadm, to bootstrap the 1st node in the >>> cluster, installed cephadm on the other nodes, and used the >>> dashboard to add the nodes to the cluster. >>> >>> >>> Regards, >>> >>> S. Barthes >>> >>> Le 21/07/2025 à 09:12, Michel Jouvin a écrit : >>>> Hi Stephane, >>>> >>>> How did you configure your cluster? Have you been using >>>> cephadm? If not, I really advise you to recreate your cluster >>>> with cephadm, that includes a script to bootstrap the cluster. >>>> In particular if you don't have a detail knowledge about Ceph >>>> architecture and management, it will ensure that your cluster >>>> is properly configured and let you progressively learn about >>>> Ceph details... >>>> >>>> Best regards. >>>> >>>> Michel >>>> >>>> Le 21/07/2025 à 09:02, Stéphane Barthes a écrit : >>>>> >>>>> Hello, >>>>> >>>>> >>>>> I am very new to ceph and have started a small cluster to get >>>>> started with ceph. >>>>> >>>>> But so far my experience, is not very impressive, probably by >>>>> lack of knowledge and good practices. >>>>> >>>>> >>>>> I started with Ubuntu 24, installed 3 VM for a ceph cluster, >>>>> and some how could not get it running. Adding nodes would >>>>> fail adding OSDs with some weird error(I found it on the web >>>>> but could not solve the problem). >>>>> >>>>> I then made a new cluster with 3 ubuntu 22 VM. Install ok, >>>>> start ok, I created 1 pool to test storing stuff there and >>>>> work my way across crash testing. However the cluster dies >>>>> during the weekly vm snapshot. It may not a good idea to run >>>>> vm backups on a ceph host, but I find this a little >>>>> surprising. (crash testing started earlier than expected) >>>>> >>>>> Bottom line is that, after the backup the cluster is in >>>>> warning state with missing mons, or logrotate and sometimes >>>>> crashed machines. systemctl restart service or Rebooting node >>>>> usually fixes it. >>>>> >>>>> I am now stuck in a situation I cannot fix : >>>>> >>>>>     - 1 Machine is ceph rbd client cannot auth : auth method >>>>> 'x' error -13. I have tried quite a few things, and none >>>>> unlocked the situation. I am currently trying to reboot the >>>>> machine, but the busy/stuck rbd device seems to block it. I >>>>> am not looking forward to hard reset it. >>>>> >>>>>     - Node with the mgr service will not restart mon, or >>>>> logrotate. I did reboot it again today, but I guess this is >>>>> not how a node is expected to behave. >>>>> >>>>> So my questions : >>>>> >>>>>     - How can I unlock my stuck ceph client, when this kind >>>>> of error occurs? >>>>> >>>>>     - Is this expected behavior that client looses access to >>>>> cluster, which kind of kills the machine? >>>>> >>>>>     - Where should I look in the ceph nodes logs to figure >>>>> what is going wrong, and how to fix it, so that is run in a >>>>> stable manner? >>>>> >>>>> >>>>> Regards, >>>>> >>>>> -- >>>>> S. Barthes >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Stéphane, 'ceph -s' requires the mon quorum to be reached, else the Ceph cluster hangs. cephadm is not using the Ceph cluster internal communication but is building a management cluster on top of it so it can manage the cluster even if the quorum is lost but it cannot provide any information requires the quorum to be reached. Michel Le 22/07/2025 à 10:33, Stéphane Barthes a écrit :

Hi Malte,

Thank for  your reply.  Here are a some info :

ceph -s hangs and times out monhunting after 300s

But I can run cephadm shell. Is there a similar command under cephadm shell?

ceph health detail : same as above.

I would like to repair it, instead of wipe & restart, as it is (from my point of view) a good way to learn (and there are a few data I'd like to recover).

What is the problem with ubuntu 24? I did not see warnings regarding this specific version in https://docs.ceph.com/en/latest/cephadm/install/#cephadm-install-distros

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 22/07/2025 à 10:02, Malte Stroem a écrit :

...

Hello Stéphane,

I think, you're mixing and mismatching up a lot!

You always have to show us the output of:

ceph -s

And more! Logs and stuff, e. g.:

ceph health detail

It is clear you missed something here and there.

It is repairable but since it is a test cluster, just delete it and start again.

And follow the documentation for cephadm. And do not use Ubuntu 24.04.

Best, Malte

On 22.07.25 09:02, Stéphane Barthes wrote:

...

Hello,

Today, things have degraded a bit more. ceph-03 mon has failed and will not restart. It shows the same kind of checksum error in rocksdb compact operation during startup. As a consequence, I lost quorum, and ceph commands hang.

Would it be wise to disable rocksbd compact, to restart and find quorum back? If yes, what is the exactt syntax of the setting in ceph.conf, I have seen one for OSD, but not sure if it would apply:

[osd]

osd_compact_on_start = true

If I can restart, I will try to out the OSDs, and recreate them. Last time I saw the OSD seemed fine in the dashboard. Since I hove no dashboard, is there a command I can use to check their status?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny

Le 21/07/2025 à 14:27, Stéphane Barthes a écrit :

...

Michel,

cephadm shell starts on all 3 nodes without error, and each host as the same ceph public key entry in the .ssh/autorized_key file of the root user.

ceph-01 also has ceph.pub in /etc/ceph with the same key (this is the node I started the install from)

ceph-2 has no/etc/ceph folder

ceph-3 has a /etc/ceph folder, but no ceph.pub file there

S. Barthes Le 21/07/2025 à 12:36, Michel Jouvin a écrit :

...

Hi Stéphane,

Sorry I was busy and did not look at your previous answers... It is a bit difficult for me to understand how you ended up in this situation but for me it is strange that ceph-02 complains about a missing keyring and the corruped rocks.db on a freshly created cluster is also a bit strange for me. I don't think it makes sense to destroy and recreate the OSD, I am running several clusters with hundreds of OSDs and I never saw a mis-initialized one. The problem is hiding something else I'm afraid. Because of some misconfiguration, may be one OSD is in a bad state and may need to be reinitialized but first we should get the 3 mons running properly and `cephadm shell` working properly on the 3 hosts. And the rocks.db compaction issue for me is related to your mon, not to an OSD.

Have you checked that SSH configuration for cephadm is working well from any host to any other one in your cluster (with 3 hosts, it should be really straighforward to check). The ceph-02 problem may be the sign of SSH misconfiguration as cephadm will use SSH connection to push the keyring, if I am right.

Michel

Le 21/07/2025 à 12:17, Stéphane Barthes a écrit :

...

Hi,

Should I just wipe the OSD and let ceph rebuild it (as suggested there : https://ceph-users.ceph.narkive.com/LO6ebu9r/how-to-recover- from-corrupted-rocksdb) ?

Which would the suggested way be  :

cephadm rm-daemon osd.ceph-01

then

cephadm deploy ?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 21/07/2025 à 10:33, Stéphane Barthes a écrit : > > Michel, > > ceph-02 logs : > > root@srvr-ceph-02:/# ceph log last debug cephadm > 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 auth: unable to > find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ > ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No > such file or directory > > 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 > AuthRegistry(0x7efe14064de0) no keyring found at /etc/ceph/ > ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ > keyring,/etc/ceph/keyring.bin, disabling cephx > > 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 auth: unable to > find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ > ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No > such file or directory > > 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 > AuthRegistry(0x7efe1a883000) no keyring found at /etc/ceph/ > ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ > keyring,/etc/ceph/keyring.bin, disabling cephx > > 2025-07-21T08:16:54.818+0000 7efe137fe640 -1 monclient(hunting): > handle_auth_bad_method server allowed_methods [2] but i only > support [1] > > 2025-07-21T08:16:54.818+0000 7efe18e21640 -1 monclient(hunting): > handle_auth_bad_method server allowed_methods [2] but i only > support [1] > > 2025-07-21T08:16:57.818+0000 7efe137fe640 -1 monclient(hunting): > handle_auth_bad_method server allowed_methods [2] but i only > support [1] > > 2025-07-21T08:16:57.818+0000 7efe13fff640 -1 monclient(hunting): > handle_auth_bad_method server allowed_methods [2] but i only > support [1] > > 2025-07-21T08:17:00.818+0000 7efe137fe640 -1 monclient(hunting): > handle_auth_bad_method server allowed_methods [2] but i only > support [1] > > 2025-07-21T08:17:00.818+0000 7efe18e21640 -1 monclient(hunting): > handle_auth_bad_method server allowed_methods [2] but i only > support [1] > > ^CCluster connection aborted > root@srvr-ceph-02:/# > > > Regarding the ceph-01 log, there is a LOT. looking from the end, > I see this : > > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -19> > 2025-07-20T17:52:21.137+0000 7f359f42d8c0  2 auth: > KeyRing::load: loaded key file > /var/lib/ceph/mon/ceph-srvr-ceph-01/keyring > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -18> > 2025-07-20T17:52:21.137+0000 7f359f42d8c0  2 mon.srvr- > ceph-01@-1(???) e5 init > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -17> > 2025-07-20T17:52:21.137+0000 7f359f42d8c0  4 mgrc handle_mgr_map > Got map version 73 > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -16> > 2025-07-20T17:52:21.137+0000 7f359f42d8c0  4 mgrc handle_mgr_map > Active mgr is now > [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -15> > 2025-07-20T17:52:21.137+0000 7f359f42d8c0  4 mgrc reconnect > Starting new session with > [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -14> > 2025-07-20T17:52:21.137+0000 7f359c206640 -1 mon.srvr- > ceph-01@-1(???) e5 handle_auth_bad_method hmm, they didn't like > 2 result (13) Permission denied > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -13> > 2025-07-20T17:52:21.137+0000 7f359f42d8c0  0 mon.srvr- > ceph-01@-1(probing) e5  my rank is now 0 (was -1) > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -12> > 2025-07-20T17:52:21.161+0000 7f359d208640  3 rocksdb: > [db/db_impl/ db_impl_compaction_flush.cc:3026] Compaction error: > Corruption: block checksum mismatch: stored = 3368055299, > computed = 2100551158  in /var/lib/ceph/mon/ceph-srvr-ceph-01/ > store.db/061999.sst offset 10379525 size 91317 > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -11> > 2025-07-20T17:52:21.161+0000 7f359d208640  4 rocksdb: (Original > Log Time 2025/07/20-17:52:21.164193) [db/compaction/ > compaction_job.cc:812] [default] compacted to: base level 6 > level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 > 0 1] max score 0.00, MB/sec: 514.9 rd, 272.6 wr, level 6, files > in(4, 1) out(0) MB in(4.0, 14.8) out(9.9), > read-write-amplify(7.2) write- amplify(2.5) Corruption: block > checksum mismatch: stored = 3368055299, computed = 2100551158 in > /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst offset > 10379525 size 91317, records in: 25191, records dropped: 3 > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -10> > 2025-07-20T17:52:21.161+0000 7f359d208640  4 rocksdb: (Original > Log Time 2025/07/20-17:52:21.164212) EVENT_LOG_v1 > {"time_micros": 1753033941164205, "job": 3, "event": > "compaction_finished", "compaction_time_micros": 38166, > "compaction_time_cpu_micros": 25133, "output_level": 6, > "num_output_files": 0, "total_output_size": 10404253, > "num_input_records": 25191, "num_output_records": 21216, > "num_subcompactions": 1, "output_compression": "NoCompression", > "num_single_delete_mismatches": 0, > "num_single_delete_fallthrough": 0, "lsm_state": [4, 0, 0, 0, 0, > 0, 1]} > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -9> > 2025-07-20T17:52:21.161+0000 7f359d208640  2 rocksdb: > [db/db_impl/ db_impl_compaction_flush.cc:2545] Waiting after > background compaction error: Corruption: block checksum > mismatch: stored = 3368055299, computed = 2100551158  in > /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst offset > 10379525 size 91317, Accumulated background error counts: 1 > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -8> > 2025-07-20T17:52:21.341+0000 7f359c206640 -1 mon.srvr- > ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't > like 2 result (13) Permission denied > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -7> > 2025-07-20T17:52:21.741+0000 7f359c206640 -1 mon.srvr- > ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't > like 2 result (13) Permission denied > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -6> > 2025-07-20T17:52:21.741+0000 7f359c206640  1 mon.srvr- > ceph-01@0(probing) e5 handle_auth_request failed to assign > global_id > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -5> > 2025-07-20T17:52:21.749+0000 7f35981fe640  5 mon.srvr- > ceph-01@0(probing) e5 _ms_dispatch setting monitor caps on this > connection > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -4> > 2025-07-20T17:52:21.749+0000 7f35981fe640  1 mon.srvr- > ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -3> > 2025-07-20T17:52:21.749+0000 7f35981fe640  1 mon.srvr- > ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap obtained > monmap e5 > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: [285B blob data] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = > mon_sync key = 'latest_monmap' value size = 508) > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = > mon_sync key = 'in_sync' value size = 8) > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = > mon_sync key = 'last_committed_floor' value size = 8) > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -1> > 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 > /home/jenkins-build/ > build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ > AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ > release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: > In function 'int > MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' > thread 7f35981fe640 time 2025-07-20T17:52:21.750611+0000 > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: > /home/jenkins-build/build/ > workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ > AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ > release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: > 355: ceph_abort_msg("failed to write to db") > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version 17.2.8 > (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: > (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0xd3) [0x7f35a03a5469] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /usr/bin/ceph- > mon(+0x1e968e) [0x55e079c5768e] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: > (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) > [0x55e079c8a145] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: > (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) > [0x55e079c90baf] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: > (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) > [0x55e079c925dc] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: > (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) > [0x55e079cad20d] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: > (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: /usr/bin/ceph- > mon(+0x1f6d3e) [0x55e079c64d3e] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: > (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: /usr/lib64/ceph/ > libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: /lib64/ > libc.so.6(+0x89e92) [0x7f359fb6ce92] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /lib64/ > libc.so.6(+0x10ef20) [0x7f359fbf1f20] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug 0> > 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 *** Caught signal > (Aborted) ** > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  in thread 7f35981fe640 > thread_name:ms_dispatch > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version 17.2.8 > (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: /lib64/ > libc.so.6(+0x3e730) [0x7f359fb21730] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /lib64/ > libc.so.6(+0x8bbdc) [0x7f359fb6ebdc] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: raise() > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: abort() > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: > (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0x190) [0x7f35a03a5526] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: /usr/bin/ceph- > mon(+0x1e968e) [0x55e079c5768e] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: > (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) > [0x55e079c8a145] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: > (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) > [0x55e079c90baf] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: > (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) > [0x55e079c925dc] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: > (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) > [0x55e079cad20d] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: > (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /usr/bin/ceph- > mon(+0x1f6d3e) [0x55e079c64d3e] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  13: > (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  14: /usr/lib64/ceph/ > libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  15: /lib64/ > libc.so.6(+0x89e92) [0x7f359fb6ce92] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  16: /lib64/ > libc.so.6(+0x10ef20) [0x7f359fbf1f20] > Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  NOTE: a copy of the > executable, or `objdump -rdS <executable>` is needed to > interpret this. > > I do not know if the logs are purged from sensitive data that > would prevent emailing them. looking for "checksum mismatch", in > the logs, there are many of them (138). > > How can I fix this checksum issue? > > > Regards, > > > S. Barthes > > Le 21/07/2025 à 09:59, Michel Jouvin a écrit : >> Stéphane, >> >> On ceph-02, I am not sure why the ceph command is not installed >> as on the other nodes, if you installed it the same way. One >> way to get access to the ceph command on this server should be >> to execute: >> >> cephadm shell >> >> This will start a container where you have the ceph environment >> installed and configured for your cluster. >> >> The situation is not as bad as I thought reading your first >> message. You have the mon quorum so at least ceph command >> should be usable. The first thing to do is probably to log on >> your ceph-01 node and try to understand why the mon daemon is >> crashing. You may want to run on this node: >> >> cephadm ls  ---> Look for the exact daemon name corresponding >> to the mon >> >> cephadm logs --daemon $daemon_name >> >> Apart from this, it is strange that ceph-03 report a RADOS >> error with 'ceph log last...', this probably hides another >> issue. Could you tell what the same command says on ceph-02 >> (when run in cephadm shell)? >> >> Michel >> >> Le 21/07/2025 à 09:44, Stéphane Barthes a écrit : >>> >>> Michel, >>> >>> >>> I ran "ceph log last debug cephadm" on my 3 nodes, and >>> "mileage varies" >>> >>> ceph-01 : >>> >>> some errors, and it ends with >>> >>> 2025-07-20T03:24:18.887889+0000 mgr.srvr-ceph-03.dhzbpe >>> (mgr.134360) 1368 : cephadm [INF] Deploying daemon mon.srvr- >>> ceph-03 on srvr-ceph-03 >>> >>> when I had to remove the mon daemon and redeploy on ceph-03. >>> >>> ceph-02 : >>> >>> root@srvr-ceph-02:~# ceph log last debug cephadm >>> Command 'ceph' not found, but can be installed with: >>> snap install microceph    # version 18.2.4+snapc9f2b08f92, or >>> apt  install ceph-common  # version 17.2.7-0ubuntu0.22.04.2 >>> See 'snap info microceph' for additional versions. >>> >>> ??? should I install ceph-common ??? >>> >>> ceph-03 : >>> >>> root@srvr-ceph-03:~# ceph log last debug cephadm >>> Error initializing cluster client: ObjectNotFound('RADOS >>> object not found (error calling conf_read_file)') >>> root@srvr-ceph-03:~# >>> >>> FWIW : ceph health is : >>> >>> root@srvr-ceph-01:~# ceph health detail >>> HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum >>> srvr-ceph-03,srvr-ceph-02; 10 daemons have recently crashed >>> [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) >>>     daemon mon.srvr-ceph-01 on srvr-ceph-01 is in error state >>> [WRN] MON_DOWN: 1/3 mons down, quorum srvr-ceph-03,srvr-ceph-02 >>>     mon.srvr-ceph-01 (rank 0) addr >>> [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] is down (out >>> of quorum) >>> [WRN] RECENT_CRASH: 10 daemons have recently crashed >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:50:10.202091Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:49:47.712267Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:50:21.464475Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:49:36.609442Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:49:58.966663Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:51:36.947240Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:52:21.751711Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:51:48.490875Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:51:59.651129Z >>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>> 2025-07-20T17:52:10.552756Z >>> >>> S. Barthes >>> Le 21/07/2025 à 09:31, Michel Jouvin a écrit : >>>> Stephane, >>>> >>>> If you are using cephadm, the OS (distrib and version) you >>>> use should not matter. When using cephadm with several >>>> servers (the general case!), it is important to setup >>>> properly the SSH key used by cephadm for the communication >>>> between nodes (cephadm is sort of a SSH-based management >>>> cluster) and to check that you can log in from one node to >>>> the other using SSH. Can you confirm that it is the case? >>>> >>>> Also cephadm has a specific log file. I don't use much the >>>> dashboard, not sure how you display it (it may be part of the >>>> logs displayed by the dashboard) but you can access it with >>>> the command: >>>> >>>> ceph log last debug cephadm >>>> >>>> Michel >>>> >>>> Le 21/07/2025 à 09:19, Stéphane Barthes a écrit : >>>>> >>>>> Hi, >>>>> >>>>> Yes, I did use cephadm, to bootstrap the 1st node in the >>>>> cluster, installed cephadm on the other nodes, and used the >>>>> dashboard to add the nodes to the cluster. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> S. Barthes >>>>> >>>>> Le 21/07/2025 à 09:12, Michel Jouvin a écrit : >>>>>> Hi Stephane, >>>>>> >>>>>> How did you configure your cluster? Have you been using >>>>>> cephadm? If not, I really advise you to recreate your >>>>>> cluster with cephadm, that includes a script to bootstrap >>>>>> the cluster. In particular if you don't have a detail >>>>>> knowledge about Ceph architecture and management, it will >>>>>> ensure that your cluster is properly configured and let you >>>>>> progressively learn about Ceph details... >>>>>> >>>>>> Best regards. >>>>>> >>>>>> Michel >>>>>> >>>>>> Le 21/07/2025 à 09:02, Stéphane Barthes a écrit : >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> >>>>>>> I am very new to ceph and have started a small cluster to >>>>>>> get started with ceph. >>>>>>> >>>>>>> But so far my experience, is not very impressive, probably >>>>>>> by lack of knowledge and good practices. >>>>>>> >>>>>>> >>>>>>> I started with Ubuntu 24, installed 3 VM for a ceph >>>>>>> cluster, and some how could not get it running. Adding >>>>>>> nodes would fail adding OSDs with some weird error(I found >>>>>>> it on the web but could not solve the problem). >>>>>>> >>>>>>> I then made a new cluster with 3 ubuntu 22 VM. Install ok, >>>>>>> start ok, I created 1 pool to test storing stuff there and >>>>>>> work my way across crash testing. However the cluster dies >>>>>>> during the weekly vm snapshot. It may not a good idea to >>>>>>> run vm backups on a ceph host, but I find this a little >>>>>>> surprising. (crash testing started earlier than expected) >>>>>>> >>>>>>> Bottom line is that, after the backup the cluster is in >>>>>>> warning state with missing mons, or logrotate and >>>>>>> sometimes crashed machines. systemctl restart service or >>>>>>> Rebooting node usually fixes it. >>>>>>> >>>>>>> I am now stuck in a situation I cannot fix : >>>>>>> >>>>>>>     - 1 Machine is ceph rbd client cannot auth : auth >>>>>>> method 'x' error -13. I have tried quite a few things, and >>>>>>> none unlocked the situation. I am currently trying to >>>>>>> reboot the machine, but the busy/stuck rbd device seems to >>>>>>> block it. I am not looking forward to hard reset it. >>>>>>> >>>>>>>     - Node with the mgr service will not restart mon, or >>>>>>> logrotate. I did reboot it again today, but I guess this >>>>>>> is not how a node is expected to behave. >>>>>>> >>>>>>> So my questions : >>>>>>> >>>>>>>     - How can I unlock my stuck ceph client, when this >>>>>>> kind of error occurs? >>>>>>> >>>>>>>     - Is this expected behavior that client looses access >>>>>>> to cluster, which kind of kills the machine? >>>>>>> >>>>>>>     - Where should I look in the ceph nodes logs to figure >>>>>>> what is going wrong, and how to fix it, so that is run in >>>>>>> a stable manner? >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> -- >>>>>>> S. Barthes >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Stéphane, Basically you cannot do anything in your cluster until you reach the quorum. Except managing it with cephadm to restore a functionning cluster. If 'ceph -s' doesn't return, it means you lost the quorum, it is the only reason I'm aware for this. As your cluster is quite simple, it should be easy to see the state of the monitor daemon on each host where one should run using `cephadm ls` and/or `podman/docker ps`. And you should be able to get access to the daemon logs of the monitor daemons. In one of your message yesterday you reported a log saying the rocks.db of one of the mon was corrupted. I personnally never saw that but the first thing to do is to fix this as it will prevent the mon to start. Follow the doc mentioned by Eugen to reduce your quorum to 1 mon (deleting the 2 broken ones from the monmap) if necessary (if you don't find a way to start at least 2 mon). And as said in another message, ensure you added the label _admin to the hosts where you want to be able to use the ceph command else the required information to connect to the cluster will be missing. It is done with 'ceph orch host label add' command which requires that you fixed the quorum issue. One possibility if you have one mon healthy and you manage to reduce the quorum to 1 is to delete the 2 other mon and readd them as new mon so that they are reinitialized. This way you will not loose anything. Look at cephadm documentation to learn how to remove and add daemons. One thing not fully clear for me is how you installed your different hosts. It seems they are not configured exactly the same way as on one host the ceph command is not available where it is on the other ones. Ceph doesn't need a lot of things from the OS when using cephadm but it is pretty important to ensure that all your Ceph hosts are deployed the same way/with the same config else you just add to the entropy... I fully agree with you and Eugen that trying to fix things is a way to learn a lot but at the same time it is not very easy to help you with the very limited information we have on what you did to be in such a strange situation... So if you don't manage to converge, may be it is better to restart from scratch following carefully the instructions: you will have plenty of other occasions to learn anyway! Michel Le 22/07/2025 à 11:04, Stéphane Barthes a écrit :

Hi Michel,

Does this mean I need to recover quorum, before some fixing happens?

Should I kick a new VM, and add a mon to the cluster, via cephaadm? This would allow to have 2 running mons?

S. Barthes Le 22/07/2025 à 10:39, Michel Jouvin a écrit :

...

Hi Stéphane,

'ceph -s' requires the mon quorum to be reached, else the Ceph cluster hangs. cephadm is not using the Ceph cluster internal communication but is building a management cluster on top of it so it can manage the cluster even if the quorum is lost but it cannot provide any information requires the quorum to be reached.

Michel

Le 22/07/2025 à 10:33, Stéphane Barthes a écrit :

...

Hi Malte,

Thank for  your reply.  Here are a some info :

ceph -s hangs and times out monhunting after 300s

But I can run cephadm shell. Is there a similar command under cephadm shell?

ceph health detail : same as above.

I would like to repair it, instead of wipe & restart, as it is (from my point of view) a good way to learn (and there are a few data I'd like to recover).

What is the problem with ubuntu 24? I did not see warnings regarding this specific version in https://docs.ceph.com/en/latest/cephadm/install/#cephadm-install-distros

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 22/07/2025 à 10:02, Malte Stroem a écrit :

...

Hello Stéphane,

I think, you're mixing and mismatching up a lot!

You always have to show us the output of:

ceph -s

And more! Logs and stuff, e. g.:

ceph health detail

It is clear you missed something here and there.

It is repairable but since it is a test cluster, just delete it and start again.

And follow the documentation for cephadm. And do not use Ubuntu 24.04.

Best, Malte

On 22.07.25 09:02, Stéphane Barthes wrote:

...

Hello,

Today, things have degraded a bit more. ceph-03 mon has failed and will not restart. It shows the same kind of checksum error in rocksdb compact operation during startup. As a consequence, I lost quorum, and ceph commands hang.

Would it be wise to disable rocksbd compact, to restart and find quorum back? If yes, what is the exactt syntax of the setting in ceph.conf, I have seen one for OSD, but not sure if it would apply:

[osd]

osd_compact_on_start = true

If I can restart, I will try to out the OSDs, and recreate them. Last time I saw the OSD seemed fine in the dashboard. Since I hove no dashboard, is there a command I can use to check their status?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny

Le 21/07/2025 à 14:27, Stéphane Barthes a écrit :

...

Michel,

cephadm shell starts on all 3 nodes without error, and each host as the same ceph public key entry in the .ssh/autorized_key file of the root user.

ceph-01 also has ceph.pub in /etc/ceph with the same key (this is the node I started the install from)

ceph-2 has no/etc/ceph folder

ceph-3 has a /etc/ceph folder, but no ceph.pub file there

S. Barthes Le 21/07/2025 à 12:36, Michel Jouvin a écrit : > Hi Stéphane, > > Sorry I was busy and did not look at your previous answers... It > is a bit difficult for me to understand how you ended up in this > situation but for me it is strange that ceph-02 complains about > a missing keyring and the corruped rocks.db on a freshly created > cluster is also a bit strange for me. I don't think it makes > sense to destroy and recreate the OSD, I am running several > clusters with hundreds of OSDs and I never saw a mis-initialized > one. The problem is hiding something else I'm afraid. Because of > some misconfiguration, may be one OSD is in a bad state and may > need to be reinitialized but first we should get the 3 mons > running properly and `cephadm shell` working properly on the 3 > hosts. And the rocks.db compaction issue for me is related to > your mon, not to an OSD. > > Have you checked that SSH configuration for cephadm is working > well from any host to any other one in your cluster (with 3 > hosts, it should be really straighforward to check). The ceph-02 > problem may be the sign of SSH misconfiguration as cephadm will > use SSH connection to push the keyring, if I am right. > > Michel > > Le 21/07/2025 à 12:17, Stéphane Barthes a écrit : >> >> Hi, >> >> >> Should I just wipe the OSD and let ceph rebuild it (as >> suggested there : >> https://ceph-users.ceph.narkive.com/LO6ebu9r/how-to-recover- >> from-corrupted-rocksdb) ? >> >> Which would the suggested way be  : >> >> cephadm rm-daemon osd.ceph-01 >> >> then >> >> cephadm deploy ? >> >> >> Regards, >> >> >> S. Barthes >> T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 >> InTest S.A. >> 4 Allée du Levant >> 69890 La Tour de Salvagny >> Le 21/07/2025 à 10:33, Stéphane Barthes a écrit : >>> >>> Michel, >>> >>> ceph-02 logs : >>> >>> root@srvr-ceph-02:/# ceph log last debug cephadm >>> 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 auth: unable to >>> find a keyring on >>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ >>> ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No >>> such file or directory >>> >>> 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 >>> AuthRegistry(0x7efe14064de0) no keyring found at /etc/ceph/ >>> ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ >>> keyring,/etc/ceph/keyring.bin, disabling cephx >>> >>> 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 auth: unable to >>> find a keyring on >>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ >>> ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No >>> such file or directory >>> >>> 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 >>> AuthRegistry(0x7efe1a883000) no keyring found at /etc/ceph/ >>> ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ >>> keyring,/etc/ceph/keyring.bin, disabling cephx >>> >>> 2025-07-21T08:16:54.818+0000 7efe137fe640 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>> >>> 2025-07-21T08:16:54.818+0000 7efe18e21640 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>> >>> 2025-07-21T08:16:57.818+0000 7efe137fe640 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>> >>> 2025-07-21T08:16:57.818+0000 7efe13fff640 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>> >>> 2025-07-21T08:17:00.818+0000 7efe137fe640 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>> >>> 2025-07-21T08:17:00.818+0000 7efe18e21640 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>> >>> ^CCluster connection aborted >>> root@srvr-ceph-02:/# >>> >>> >>> Regarding the ceph-01 log, there is a LOT. looking from the >>> end, I see this : >>> >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -19> >>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 2 auth: >>> KeyRing::load: loaded key file >>> /var/lib/ceph/mon/ceph-srvr-ceph-01/keyring >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -18> >>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 2 mon.srvr- >>> ceph-01@-1(???) e5 init >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -17> >>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc >>> handle_mgr_map Got map version 73 >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -16> >>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc >>> handle_mgr_map Active mgr is now >>> [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -15> >>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc reconnect >>> Starting new session with >>> [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -14> >>> 2025-07-20T17:52:21.137+0000 7f359c206640 -1 mon.srvr- >>> ceph-01@-1(???) e5 handle_auth_bad_method hmm, they didn't >>> like 2 result (13) Permission denied >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -13> >>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 0 mon.srvr- >>> ceph-01@-1(probing) e5  my rank is now 0 (was -1) >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -12> >>> 2025-07-20T17:52:21.161+0000 7f359d208640 3 rocksdb: >>> [db/db_impl/ db_impl_compaction_flush.cc:3026] Compaction >>> error: Corruption: block checksum mismatch: stored = >>> 3368055299, computed = 2100551158  in >>> /var/lib/ceph/mon/ceph-srvr-ceph-01/ store.db/061999.sst >>> offset 10379525 size 91317 >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -11> >>> 2025-07-20T17:52:21.161+0000 7f359d208640 4 rocksdb: (Original >>> Log Time 2025/07/20-17:52:21.164193) [db/compaction/ >>> compaction_job.cc:812] [default] compacted to: base level 6 >>> level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 >>> 0 0 1] max score 0.00, MB/sec: 514.9 rd, 272.6 wr, level 6, >>> files in(4, 1) out(0) MB in(4.0, 14.8) out(9.9), >>> read-write-amplify(7.2) write- amplify(2.5) Corruption: block >>> checksum mismatch: stored = 3368055299, computed = 2100551158 >>> in /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst >>> offset 10379525 size 91317, records in: 25191, records dropped: 3 >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -10> >>> 2025-07-20T17:52:21.161+0000 7f359d208640 4 rocksdb: (Original >>> Log Time 2025/07/20-17:52:21.164212) EVENT_LOG_v1 >>> {"time_micros": 1753033941164205, "job": 3, "event": >>> "compaction_finished", "compaction_time_micros": 38166, >>> "compaction_time_cpu_micros": 25133, "output_level": 6, >>> "num_output_files": 0, "total_output_size": 10404253, >>> "num_input_records": 25191, "num_output_records": 21216, >>> "num_subcompactions": 1, "output_compression": >>> "NoCompression", "num_single_delete_mismatches": 0, >>> "num_single_delete_fallthrough": 0, "lsm_state": [4, 0, 0, 0, >>> 0, 0, 1]} >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -9> >>> 2025-07-20T17:52:21.161+0000 7f359d208640 2 rocksdb: >>> [db/db_impl/ db_impl_compaction_flush.cc:2545] Waiting after >>> background compaction error: Corruption: block checksum >>> mismatch: stored = 3368055299, computed = 2100551158  in >>> /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst >>> offset 10379525 size 91317, Accumulated background error >>> counts: 1 >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -8> >>> 2025-07-20T17:52:21.341+0000 7f359c206640 -1 mon.srvr- >>> ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't >>> like 2 result (13) Permission denied >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -7> >>> 2025-07-20T17:52:21.741+0000 7f359c206640 -1 mon.srvr- >>> ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't >>> like 2 result (13) Permission denied >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -6> >>> 2025-07-20T17:52:21.741+0000 7f359c206640 1 mon.srvr- >>> ceph-01@0(probing) e5 handle_auth_request failed to assign >>> global_id >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -5> >>> 2025-07-20T17:52:21.749+0000 7f35981fe640 5 mon.srvr- >>> ceph-01@0(probing) e5 _ms_dispatch setting monitor caps on >>> this connection >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -4> >>> 2025-07-20T17:52:21.749+0000 7f35981fe640 1 mon.srvr- >>> ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -3> >>> 2025-07-20T17:52:21.749+0000 7f35981fe640 1 mon.srvr- >>> ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap obtained >>> monmap e5 >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: [285B blob data] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = >>> mon_sync key = 'latest_monmap' value size = 508) >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = >>> mon_sync key = 'in_sync' value size = 8) >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = >>> mon_sync key = 'last_committed_floor' value size = 8) >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -1> >>> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 >>> /home/jenkins-build/ >>> build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ >>> AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ >>> release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: >>> In function 'int >>> MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' >>> thread 7f35981fe640 time 2025-07-20T17:52:21.750611+0000 >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>> /home/jenkins-build/build/ >>> workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ >>> AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ >>> release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: >>> 355: ceph_abort_msg("failed to write to db") >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version 17.2.8 >>> (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: >>> (ceph::__ceph_abort(char const*, int, char const*, >>> std::__cxx11::basic_string<char, std::char_traits<char>, >>> std::allocator<char> > const&)+0xd3) [0x7f35a03a5469] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /usr/bin/ceph- >>> mon(+0x1e968e) [0x55e079c5768e] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: >>> (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) >>> [0x55e079c8a145] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: >>> (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) >>> [0x55e079c90baf] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: >>> (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) >>> [0x55e079c925dc] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: >>> (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) >>> [0x55e079cad20d] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: >>> (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: /usr/bin/ceph- >>> mon(+0x1f6d3e) [0x55e079c64d3e] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: >>> (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: /usr/lib64/ceph/ >>> libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: /lib64/ >>> libc.so.6(+0x89e92) [0x7f359fb6ce92] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /lib64/ >>> libc.so.6(+0x10ef20) [0x7f359fbf1f20] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug 0> >>> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 *** Caught signal >>> (Aborted) ** >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  in thread >>> 7f35981fe640 thread_name:ms_dispatch >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version 17.2.8 >>> (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: /lib64/ >>> libc.so.6(+0x3e730) [0x7f359fb21730] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /lib64/ >>> libc.so.6(+0x8bbdc) [0x7f359fb6ebdc] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: raise() >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: abort() >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: >>> (ceph::__ceph_abort(char const*, int, char const*, >>> std::__cxx11::basic_string<char, std::char_traits<char>, >>> std::allocator<char> > const&)+0x190) [0x7f35a03a5526] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: /usr/bin/ceph- >>> mon(+0x1e968e) [0x55e079c5768e] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: >>> (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) >>> [0x55e079c8a145] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: >>> (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) >>> [0x55e079c90baf] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: >>> (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) >>> [0x55e079c925dc] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: >>> (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) >>> [0x55e079cad20d] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: >>> (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /usr/bin/ceph- >>> mon(+0x1f6d3e) [0x55e079c64d3e] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  13: >>> (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  14: /usr/lib64/ceph/ >>> libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  15: /lib64/ >>> libc.so.6(+0x89e92) [0x7f359fb6ce92] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  16: /lib64/ >>> libc.so.6(+0x10ef20) [0x7f359fbf1f20] >>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  NOTE: a copy of the >>> executable, or `objdump -rdS <executable>` is needed to >>> interpret this. >>> >>> I do not know if the logs are purged from sensitive data that >>> would prevent emailing them. looking for "checksum mismatch", >>> in the logs, there are many of them (138). >>> >>> How can I fix this checksum issue? >>> >>> >>> Regards, >>> >>> >>> S. Barthes >>> >>> Le 21/07/2025 à 09:59, Michel Jouvin a écrit : >>>> Stéphane, >>>> >>>> On ceph-02, I am not sure why the ceph command is not >>>> installed as on the other nodes, if you installed it the same >>>> way. One way to get access to the ceph command on this server >>>> should be to execute: >>>> >>>> cephadm shell >>>> >>>> This will start a container where you have the ceph >>>> environment installed and configured for your cluster. >>>> >>>> The situation is not as bad as I thought reading your first >>>> message. You have the mon quorum so at least ceph command >>>> should be usable. The first thing to do is probably to log on >>>> your ceph-01 node and try to understand why the mon daemon is >>>> crashing. You may want to run on this node: >>>> >>>> cephadm ls  ---> Look for the exact daemon name corresponding >>>> to the mon >>>> >>>> cephadm logs --daemon $daemon_name >>>> >>>> Apart from this, it is strange that ceph-03 report a RADOS >>>> error with 'ceph log last...', this probably hides another >>>> issue. Could you tell what the same command says on ceph-02 >>>> (when run in cephadm shell)? >>>> >>>> Michel >>>> >>>> Le 21/07/2025 à 09:44, Stéphane Barthes a écrit : >>>>> >>>>> Michel, >>>>> >>>>> >>>>> I ran "ceph log last debug cephadm" on my 3 nodes, and >>>>> "mileage varies" >>>>> >>>>> ceph-01 : >>>>> >>>>> some errors, and it ends with >>>>> >>>>> 2025-07-20T03:24:18.887889+0000 mgr.srvr-ceph-03.dhzbpe >>>>> (mgr.134360) 1368 : cephadm [INF] Deploying daemon mon.srvr- >>>>> ceph-03 on srvr-ceph-03 >>>>> >>>>> when I had to remove the mon daemon and redeploy on ceph-03. >>>>> >>>>> ceph-02 : >>>>> >>>>> root@srvr-ceph-02:~# ceph log last debug cephadm >>>>> Command 'ceph' not found, but can be installed with: >>>>> snap install microceph    # version 18.2.4+snapc9f2b08f92, or >>>>> apt  install ceph-common  # version 17.2.7-0ubuntu0.22.04.2 >>>>> See 'snap info microceph' for additional versions. >>>>> >>>>> ??? should I install ceph-common ??? >>>>> >>>>> ceph-03 : >>>>> >>>>> root@srvr-ceph-03:~# ceph log last debug cephadm >>>>> Error initializing cluster client: ObjectNotFound('RADOS >>>>> object not found (error calling conf_read_file)') >>>>> root@srvr-ceph-03:~# >>>>> >>>>> FWIW : ceph health is : >>>>> >>>>> root@srvr-ceph-01:~# ceph health detail >>>>> HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, >>>>> quorum srvr-ceph-03,srvr-ceph-02; 10 daemons have recently >>>>> crashed >>>>> [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) >>>>>     daemon mon.srvr-ceph-01 on srvr-ceph-01 is in error state >>>>> [WRN] MON_DOWN: 1/3 mons down, quorum srvr-ceph-03,srvr-ceph-02 >>>>>     mon.srvr-ceph-01 (rank 0) addr >>>>> [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] is down (out >>>>> of quorum) >>>>> [WRN] RECENT_CRASH: 10 daemons have recently crashed >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:50:10.202091Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:49:47.712267Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:50:21.464475Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:49:36.609442Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:49:58.966663Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:51:36.947240Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:52:21.751711Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:51:48.490875Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:51:59.651129Z >>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>> 2025-07-20T17:52:10.552756Z >>>>> >>>>> S. Barthes >>>>> Le 21/07/2025 à 09:31, Michel Jouvin a écrit : >>>>>> Stephane, >>>>>> >>>>>> If you are using cephadm, the OS (distrib and version) you >>>>>> use should not matter. When using cephadm with several >>>>>> servers (the general case!), it is important to setup >>>>>> properly the SSH key used by cephadm for the communication >>>>>> between nodes (cephadm is sort of a SSH-based management >>>>>> cluster) and to check that you can log in from one node to >>>>>> the other using SSH. Can you confirm that it is the case? >>>>>> >>>>>> Also cephadm has a specific log file. I don't use much the >>>>>> dashboard, not sure how you display it (it may be part of >>>>>> the logs displayed by the dashboard) but you can access it >>>>>> with the command: >>>>>> >>>>>> ceph log last debug cephadm >>>>>> >>>>>> Michel >>>>>> >>>>>> Le 21/07/2025 à 09:19, Stéphane Barthes a écrit : >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Yes, I did use cephadm, to bootstrap the 1st node in the >>>>>>> cluster, installed cephadm on the other nodes, and used >>>>>>> the dashboard to add the nodes to the cluster. >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> S. Barthes >>>>>>> >>>>>>> Le 21/07/2025 à 09:12, Michel Jouvin a écrit : >>>>>>>> Hi Stephane, >>>>>>>> >>>>>>>> How did you configure your cluster? Have you been using >>>>>>>> cephadm? If not, I really advise you to recreate your >>>>>>>> cluster with cephadm, that includes a script to bootstrap >>>>>>>> the cluster. In particular if you don't have a detail >>>>>>>> knowledge about Ceph architecture and management, it will >>>>>>>> ensure that your cluster is properly configured and let >>>>>>>> you progressively learn about Ceph details... >>>>>>>> >>>>>>>> Best regards. >>>>>>>> >>>>>>>> Michel >>>>>>>> >>>>>>>> Le 21/07/2025 à 09:02, Stéphane Barthes a écrit : >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> >>>>>>>>> I am very new to ceph and have started a small cluster >>>>>>>>> to get started with ceph. >>>>>>>>> >>>>>>>>> But so far my experience, is not very impressive, >>>>>>>>> probably by lack of knowledge and good practices. >>>>>>>>> >>>>>>>>> >>>>>>>>> I started with Ubuntu 24, installed 3 VM for a ceph >>>>>>>>> cluster, and some how could not get it running. Adding >>>>>>>>> nodes would fail adding OSDs with some weird error(I >>>>>>>>> found it on the web but could not solve the problem). >>>>>>>>> >>>>>>>>> I then made a new cluster with 3 ubuntu 22 VM. Install >>>>>>>>> ok, start ok, I created 1 pool to test storing stuff >>>>>>>>> there and work my way across crash testing. However the >>>>>>>>> cluster dies during the weekly vm snapshot. It may not a >>>>>>>>> good idea to run vm backups on a ceph host, but I find >>>>>>>>> this a little surprising. (crash testing started earlier >>>>>>>>> than expected) >>>>>>>>> >>>>>>>>> Bottom line is that, after the backup the cluster is in >>>>>>>>> warning state with missing mons, or logrotate and >>>>>>>>> sometimes crashed machines. systemctl restart service or >>>>>>>>> Rebooting node usually fixes it. >>>>>>>>> >>>>>>>>> I am now stuck in a situation I cannot fix : >>>>>>>>> >>>>>>>>>     - 1 Machine is ceph rbd client cannot auth : auth >>>>>>>>> method 'x' error -13. I have tried quite a few things, >>>>>>>>> and none unlocked the situation. I am currently trying >>>>>>>>> to reboot the machine, but the busy/stuck rbd device >>>>>>>>> seems to block it. I am not looking forward to hard >>>>>>>>> reset it. >>>>>>>>> >>>>>>>>>     - Node with the mgr service will not restart mon, or >>>>>>>>> logrotate. I did reboot it again today, but I guess this >>>>>>>>> is not how a node is expected to behave. >>>>>>>>> >>>>>>>>> So my questions : >>>>>>>>> >>>>>>>>>     - How can I unlock my stuck ceph client, when this >>>>>>>>> kind of error occurs? >>>>>>>>> >>>>>>>>>     - Is this expected behavior that client looses >>>>>>>>> access to cluster, which kind of kills the machine? >>>>>>>>> >>>>>>>>>     - Where should I look in the ceph nodes logs to >>>>>>>>> figure what is going wrong, and how to fix it, so that >>>>>>>>> is run in a stable manner? >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> -- >>>>>>>>> S. Barthes >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Stéphane, As you use VMs for your deployment, would it make sense to stop and keep them and restart with a new set of VMs, following https://docs.ceph.com/en/latest/cephadm/install/. I personnally don't have experience to expand the cluster through the dashboard (it should work! just that I am an old guy so used to command line tools!). Probably using the command line makes easier to identify when there is a problem, without always digging in the logs. I remember following this doc a couple of years ago, the last time I created a cluster and it was working as expected. If you restart something fresh, I'd start with the Squid version rather than Reef. It should not make any difference for the installation but will bring you the latest-and-greatest Ceph! Once you have something running properly, you may compare what is different in the other configuration. I don't have any opinion on Debian 24 as we are using RedHat (AlmaLinux in fact) but for me the OS version should not make a big difference when using cephadm, as Ceph in fact runs in containers. But the devil is in the details, Maite may have a reason for her warning... One thing we didn't mention/check with you in the previous exchanges is the Ceph network configuration you used. Any discrepancy in the network configuration, in particular if you have a separate cluster (network used only by OSD) and public (network used by monitors and Ceph clients) networks, you may have a situation where one of them is not working as expected and may lead to daemons not being able to see eachothers. But I think your problem is much more basic: the daemons were not able to restart after some corruptions of their DB, it is pretty unusual and may also be the result of something weird that happened in your VM infrastructure leading to storage corruption... What are you using to manage to your VMs? Michel Le 22/07/2025 à 13:48, Stéphane Barthes a écrit :

Michel,

Thank you very much for the help.

I will look into the documentation provided by Eugen to try reduce to 1 mon, remove and re-add the 2 mon on the node ceph01 and ceph02.

Regarding the installation, I have created a vm template with ubuntu installed ready for ceph.

Installed cephadm on ceph01, bootstrapped the cluster, and added the other vms from the dashboard. I guess this is all I did to setup the cluster.

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 22/07/2025 à 12:12, Michel Jouvin a écrit :

...

Stéphane,

Basically you cannot do anything in your cluster until you reach the quorum. Except managing it with cephadm to restore a functionning cluster. If 'ceph -s' doesn't return, it means you lost the quorum, it is the only reason I'm aware for this. As your cluster is quite simple, it should be easy to see the state of the monitor daemon on each host where one should run using `cephadm ls` and/or `podman/docker ps`. And you should be able to get access to the daemon logs of the monitor daemons.

In one of your message yesterday you reported a log saying the rocks.db of one of the mon was corrupted. I personnally never saw that but the first thing to do is to fix this as it will prevent the mon to start. Follow the doc mentioned by Eugen to reduce your quorum to 1 mon (deleting the 2 broken ones from the monmap) if necessary (if you don't find a way to start at least 2 mon). And as said in another message, ensure you added the label _admin to the hosts where you want to be able to use the ceph command else the required information to connect to the cluster will be missing. It is done with 'ceph orch host label add' command which requires that you fixed the quorum issue. One possibility if you have one mon healthy and you manage to reduce the quorum to 1 is to delete the 2 other mon and readd them as new mon so that they are reinitialized. This way you will not loose anything. Look at cephadm documentation to learn how to remove and add daemons.

One thing not fully clear for me is how you installed your different hosts. It seems they are not configured exactly the same way as on one host the ceph command is not available where it is on the other ones. Ceph doesn't need a lot of things from the OS when using cephadm but it is pretty important to ensure that all your Ceph hosts are deployed the same way/with the same config else you just add to the entropy...

I fully agree with you and Eugen that trying to fix things is a way to learn a lot but at the same time it is not very easy to help you with the very limited information we have on what you did to be in such a strange situation... So if you don't manage to converge, may be it is better to restart from scratch following carefully the instructions: you will have plenty of other occasions to learn anyway!

Michel

Le 22/07/2025 à 11:04, Stéphane Barthes a écrit :

...

Hi Michel,

Does this mean I need to recover quorum, before some fixing happens?

Should I kick a new VM, and add a mon to the cluster, via cephaadm? This would allow to have 2 running mons?

S. Barthes Le 22/07/2025 à 10:39, Michel Jouvin a écrit :

...

Hi Stéphane,

'ceph -s' requires the mon quorum to be reached, else the Ceph cluster hangs. cephadm is not using the Ceph cluster internal communication but is building a management cluster on top of it so it can manage the cluster even if the quorum is lost but it cannot provide any information requires the quorum to be reached.

Michel

Le 22/07/2025 à 10:33, Stéphane Barthes a écrit :

...

Hi Malte,

Thank for  your reply.  Here are a some info :

ceph -s hangs and times out monhunting after 300s

But I can run cephadm shell. Is there a similar command under cephadm shell?

ceph health detail : same as above.

I would like to repair it, instead of wipe & restart, as it is (from my point of view) a good way to learn (and there are a few data I'd like to recover).

What is the problem with ubuntu 24? I did not see warnings regarding this specific version in https://docs.ceph.com/en/latest/cephadm/install/#cephadm-install-distros

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 22/07/2025 à 10:02, Malte Stroem a écrit :

...

Hello Stéphane,

I think, you're mixing and mismatching up a lot!

You always have to show us the output of:

ceph -s

And more! Logs and stuff, e. g.:

ceph health detail

It is clear you missed something here and there.

It is repairable but since it is a test cluster, just delete it and start again.

And follow the documentation for cephadm. And do not use Ubuntu 24.04.

Best, Malte

On 22.07.25 09:02, Stéphane Barthes wrote: > Hello, > > > Today, things have degraded a bit more. ceph-03 mon has failed > and will not restart. It shows the same kind of checksum error > in rocksdb compact operation during startup. As a consequence, I > lost quorum, and ceph commands hang. > > > Would it be wise to disable rocksbd compact, to restart and find > quorum back? If yes, what is the exactt syntax of the setting in > ceph.conf, I have seen one for OSD, but not sure if it would apply: > > [osd] > > osd_compact_on_start = true > > > If I can restart, I will try to out the OSDs, and recreate them. > Last time I saw the OSD seemed fine in the dashboard. Since I > hove no dashboard, is there a command I can use to check their > status? > > > Regards, > > S. Barthes > T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 > InTest S.A. > 4 Allée du Levant > 69890 La Tour de Salvagny > > Le 21/07/2025 à 14:27, Stéphane Barthes a écrit : >> >> Michel, >> >> >> cephadm shell starts on all 3 nodes without error, and each >> host as the same ceph public key entry in the >> .ssh/autorized_key file of the root user. >> >> ceph-01 also has ceph.pub in /etc/ceph with the same key (this >> is the node I started the install from) >> >> ceph-2 has no/etc/ceph folder >> >> ceph-3 has a /etc/ceph folder, but no ceph.pub file there >> >> >> S. Barthes >> Le 21/07/2025 à 12:36, Michel Jouvin a écrit : >>> Hi Stéphane, >>> >>> Sorry I was busy and did not look at your previous answers... >>> It is a bit difficult for me to understand how you ended up in >>> this situation but for me it is strange that ceph-02 complains >>> about a missing keyring and the corruped rocks.db on a freshly >>> created cluster is also a bit strange for me. I don't think it >>> makes sense to destroy and recreate the OSD, I am running >>> several clusters with hundreds of OSDs and I never saw a >>> mis-initialized one. The problem is hiding something else I'm >>> afraid. Because of some misconfiguration, may be one OSD is in >>> a bad state and may need to be reinitialized but first we >>> should get the 3 mons running properly and `cephadm shell` >>> working properly on the 3 hosts. And the rocks.db compaction >>> issue for me is related to your mon, not to an OSD. >>> >>> Have you checked that SSH configuration for cephadm is working >>> well from any host to any other one in your cluster (with 3 >>> hosts, it should be really straighforward to check). The >>> ceph-02 problem may be the sign of SSH misconfiguration as >>> cephadm will use SSH connection to push the keyring, if I am >>> right. >>> >>> Michel >>> >>> Le 21/07/2025 à 12:17, Stéphane Barthes a écrit : >>>> >>>> Hi, >>>> >>>> >>>> Should I just wipe the OSD and let ceph rebuild it (as >>>> suggested there : >>>> https://ceph-users.ceph.narkive.com/LO6ebu9r/how-to-recover- >>>> from-corrupted-rocksdb) ? >>>> >>>> Which would the suggested way be  : >>>> >>>> cephadm rm-daemon osd.ceph-01 >>>> >>>> then >>>> >>>> cephadm deploy ? >>>> >>>> >>>> Regards, >>>> >>>> >>>> S. Barthes >>>> T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 >>>> InTest S.A. >>>> 4 Allée du Levant >>>> 69890 La Tour de Salvagny >>>> Le 21/07/2025 à 10:33, Stéphane Barthes a écrit : >>>>> >>>>> Michel, >>>>> >>>>> ceph-02 logs : >>>>> >>>>> root@srvr-ceph-02:/# ceph log last debug cephadm >>>>> 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 auth: unable to >>>>> find a keyring on >>>>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ >>>>> ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No >>>>> such file or directory >>>>> >>>>> 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 >>>>> AuthRegistry(0x7efe14064de0) no keyring found at /etc/ceph/ >>>>> ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ >>>>> keyring,/etc/ceph/keyring.bin, disabling cephx >>>>> >>>>> 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 auth: unable to >>>>> find a keyring on >>>>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ >>>>> ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No >>>>> such file or directory >>>>> >>>>> 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 >>>>> AuthRegistry(0x7efe1a883000) no keyring found at /etc/ceph/ >>>>> ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ >>>>> keyring,/etc/ceph/keyring.bin, disabling cephx >>>>> >>>>> 2025-07-21T08:16:54.818+0000 7efe137fe640 -1 >>>>> monclient(hunting): handle_auth_bad_method server >>>>> allowed_methods [2] but i only support [1] >>>>> >>>>> 2025-07-21T08:16:54.818+0000 7efe18e21640 -1 >>>>> monclient(hunting): handle_auth_bad_method server >>>>> allowed_methods [2] but i only support [1] >>>>> >>>>> 2025-07-21T08:16:57.818+0000 7efe137fe640 -1 >>>>> monclient(hunting): handle_auth_bad_method server >>>>> allowed_methods [2] but i only support [1] >>>>> >>>>> 2025-07-21T08:16:57.818+0000 7efe13fff640 -1 >>>>> monclient(hunting): handle_auth_bad_method server >>>>> allowed_methods [2] but i only support [1] >>>>> >>>>> 2025-07-21T08:17:00.818+0000 7efe137fe640 -1 >>>>> monclient(hunting): handle_auth_bad_method server >>>>> allowed_methods [2] but i only support [1] >>>>> >>>>> 2025-07-21T08:17:00.818+0000 7efe18e21640 -1 >>>>> monclient(hunting): handle_auth_bad_method server >>>>> allowed_methods [2] but i only support [1] >>>>> >>>>> ^CCluster connection aborted >>>>> root@srvr-ceph-02:/# >>>>> >>>>> >>>>> Regarding the ceph-01 log, there is a LOT. looking from the >>>>> end, I see this : >>>>> >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -19> >>>>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 2 auth: >>>>> KeyRing::load: loaded key file >>>>> /var/lib/ceph/mon/ceph-srvr-ceph-01/keyring >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -18> >>>>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 2 mon.srvr- >>>>> ceph-01@-1(???) e5 init >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -17> >>>>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc >>>>> handle_mgr_map Got map version 73 >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -16> >>>>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc >>>>> handle_mgr_map Active mgr is now >>>>> [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] >>>>> >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -15> >>>>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc reconnect >>>>> Starting new session with >>>>> [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] >>>>> >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -14> >>>>> 2025-07-20T17:52:21.137+0000 7f359c206640 -1 mon.srvr- >>>>> ceph-01@-1(???) e5 handle_auth_bad_method hmm, they didn't >>>>> like 2 result (13) Permission denied >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -13> >>>>> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 0 mon.srvr- >>>>> ceph-01@-1(probing) e5  my rank is now 0 (was -1) >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -12> >>>>> 2025-07-20T17:52:21.161+0000 7f359d208640 3 rocksdb: >>>>> [db/db_impl/ db_impl_compaction_flush.cc:3026] Compaction >>>>> error: Corruption: block checksum mismatch: stored = >>>>> 3368055299, computed = 2100551158  in >>>>> /var/lib/ceph/mon/ceph-srvr-ceph-01/ store.db/061999.sst >>>>> offset 10379525 size 91317 >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -11> >>>>> 2025-07-20T17:52:21.161+0000 7f359d208640 4 rocksdb: >>>>> (Original Log Time 2025/07/20-17:52:21.164193) >>>>> [db/compaction/ compaction_job.cc:812] [default] compacted >>>>> to: base level 6 level multiplier 10.00 max bytes base >>>>> 268435456 files[4 0 0 0 0 0 1] max score 0.00, MB/sec: 514.9 >>>>> rd, 272.6 wr, level 6, files in(4, 1) out(0) MB in(4.0, >>>>> 14.8) out(9.9), read-write-amplify(7.2) write- amplify(2.5) >>>>> Corruption: block checksum mismatch: stored = 3368055299, >>>>> computed = 2100551158 in /var/lib/ceph/mon/ceph-srvr- >>>>> ceph-01/store.db/061999.sst offset 10379525 size 91317, >>>>> records in: 25191, records dropped: 3 >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -10> >>>>> 2025-07-20T17:52:21.161+0000 7f359d208640 4 rocksdb: >>>>> (Original Log Time 2025/07/20-17:52:21.164212) EVENT_LOG_v1 >>>>> {"time_micros": 1753033941164205, "job": 3, "event": >>>>> "compaction_finished", "compaction_time_micros": 38166, >>>>> "compaction_time_cpu_micros": 25133, "output_level": 6, >>>>> "num_output_files": 0, "total_output_size": 10404253, >>>>> "num_input_records": 25191, "num_output_records": 21216, >>>>> "num_subcompactions": 1, "output_compression": >>>>> "NoCompression", "num_single_delete_mismatches": 0, >>>>> "num_single_delete_fallthrough": 0, "lsm_state": [4, 0, 0, >>>>> 0, 0, 0, 1]} >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -9> >>>>> 2025-07-20T17:52:21.161+0000 7f359d208640 2 rocksdb: >>>>> [db/db_impl/ db_impl_compaction_flush.cc:2545] Waiting after >>>>> background compaction error: Corruption: block checksum >>>>> mismatch: stored = 3368055299, computed = 2100551158  in >>>>> /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst >>>>> offset 10379525 size 91317, Accumulated background error >>>>> counts: 1 >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -8> >>>>> 2025-07-20T17:52:21.341+0000 7f359c206640 -1 mon.srvr- >>>>> ceph-01@0(probing) e5 handle_auth_bad_method hmm, they >>>>> didn't like 2 result (13) Permission denied >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -7> >>>>> 2025-07-20T17:52:21.741+0000 7f359c206640 -1 mon.srvr- >>>>> ceph-01@0(probing) e5 handle_auth_bad_method hmm, they >>>>> didn't like 2 result (13) Permission denied >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -6> >>>>> 2025-07-20T17:52:21.741+0000 7f359c206640 1 mon.srvr- >>>>> ceph-01@0(probing) e5 handle_auth_request failed to assign >>>>> global_id >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -5> >>>>> 2025-07-20T17:52:21.749+0000 7f35981fe640 5 mon.srvr- >>>>> ceph-01@0(probing) e5 _ms_dispatch setting monitor caps on >>>>> this connection >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -4> >>>>> 2025-07-20T17:52:21.749+0000 7f35981fe640 1 mon.srvr- >>>>> ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -3> >>>>> 2025-07-20T17:52:21.749+0000 7f35981fe640 1 mon.srvr- >>>>> ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap >>>>> obtained monmap e5 >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: [285B blob data] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = >>>>> mon_sync key = 'latest_monmap' value size = 508) >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = >>>>> mon_sync key = 'in_sync' value size = 8) >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = >>>>> mon_sync key = 'last_committed_floor' value size = 8) >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug  -1> >>>>> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 >>>>> /home/jenkins-build/ >>>>> build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ >>>>> AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ >>>>> release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: >>>>> In function 'int >>>>> MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' >>>>> thread 7f35981fe640 time 2025-07-20T17:52:21.750611+0000 >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>> /home/jenkins-build/build/ >>>>> workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ >>>>> AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ >>>>> release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: >>>>> 355: ceph_abort_msg("failed to write to db") >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version >>>>> 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy >>>>> (stable) >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: >>>>> (ceph::__ceph_abort(char const*, int, char const*, >>>>> std::__cxx11::basic_string<char, std::char_traits<char>, >>>>> std::allocator<char> > const&)+0xd3) [0x7f35a03a5469] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /usr/bin/ceph- >>>>> mon(+0x1e968e) [0x55e079c5768e] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: >>>>> (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) >>>>> [0x55e079c8a145] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: >>>>> (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) >>>>> [0x55e079c90baf] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: >>>>> (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) >>>>> [0x55e079c925dc] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: >>>>> (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) >>>>> [0x55e079cad20d] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: >>>>> (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: /usr/bin/ceph- >>>>> mon(+0x1f6d3e) [0x55e079c64d3e] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: >>>>> (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: >>>>> /usr/lib64/ceph/ libceph-common.so.2(+0x3bdea1) >>>>> [0x7f35a061eea1] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: /lib64/ >>>>> libc.so.6(+0x89e92) [0x7f359fb6ce92] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /lib64/ >>>>> libc.so.6(+0x10ef20) [0x7f359fbf1f20] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug 0> >>>>> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 *** Caught >>>>> signal (Aborted) ** >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  in thread >>>>> 7f35981fe640 thread_name:ms_dispatch >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  ceph version >>>>> 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy >>>>> (stable) >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: /lib64/ >>>>> libc.so.6(+0x3e730) [0x7f359fb21730] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: /lib64/ >>>>> libc.so.6(+0x8bbdc) [0x7f359fb6ebdc] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: raise() >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: abort() >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: >>>>> (ceph::__ceph_abort(char const*, int, char const*, >>>>> std::__cxx11::basic_string<char, std::char_traits<char>, >>>>> std::allocator<char> > const&)+0x190) [0x7f35a03a5526] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: /usr/bin/ceph- >>>>> mon(+0x1e968e) [0x55e079c5768e] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: >>>>> (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) >>>>> [0x55e079c8a145] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: >>>>> (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) >>>>> [0x55e079c90baf] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: >>>>> (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) >>>>> [0x55e079c925dc] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  10: >>>>> (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) >>>>> [0x55e079cad20d] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  11: >>>>> (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  12: /usr/bin/ceph- >>>>> mon(+0x1f6d3e) [0x55e079c64d3e] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  13: >>>>> (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  14: >>>>> /usr/lib64/ceph/ libceph-common.so.2(+0x3bdea1) >>>>> [0x7f35a061eea1] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  15: /lib64/ >>>>> libc.so.6(+0x89e92) [0x7f359fb6ce92] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  16: /lib64/ >>>>> libc.so.6(+0x10ef20) [0x7f359fbf1f20] >>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: NOTE: a copy of the >>>>> executable, or `objdump -rdS <executable>` is needed to >>>>> interpret this. >>>>> >>>>> I do not know if the logs are purged from sensitive data >>>>> that would prevent emailing them. looking for "checksum >>>>> mismatch", in the logs, there are many of them (138). >>>>> >>>>> How can I fix this checksum issue? >>>>> >>>>> >>>>> Regards, >>>>> >>>>> >>>>> S. Barthes >>>>> >>>>> Le 21/07/2025 à 09:59, Michel Jouvin a écrit : >>>>>> Stéphane, >>>>>> >>>>>> On ceph-02, I am not sure why the ceph command is not >>>>>> installed as on the other nodes, if you installed it the >>>>>> same way. One way to get access to the ceph command on this >>>>>> server should be to execute: >>>>>> >>>>>> cephadm shell >>>>>> >>>>>> This will start a container where you have the ceph >>>>>> environment installed and configured for your cluster. >>>>>> >>>>>> The situation is not as bad as I thought reading your first >>>>>> message. You have the mon quorum so at least ceph command >>>>>> should be usable. The first thing to do is probably to log >>>>>> on your ceph-01 node and try to understand why the mon >>>>>> daemon is crashing. You may want to run on this node: >>>>>> >>>>>> cephadm ls  ---> Look for the exact daemon name >>>>>> corresponding to the mon >>>>>> >>>>>> cephadm logs --daemon $daemon_name >>>>>> >>>>>> Apart from this, it is strange that ceph-03 report a RADOS >>>>>> error with 'ceph log last...', this probably hides another >>>>>> issue. Could you tell what the same command says on ceph-02 >>>>>> (when run in cephadm shell)? >>>>>> >>>>>> Michel >>>>>> >>>>>> Le 21/07/2025 à 09:44, Stéphane Barthes a écrit : >>>>>>> >>>>>>> Michel, >>>>>>> >>>>>>> >>>>>>> I ran "ceph log last debug cephadm" on my 3 nodes, and >>>>>>> "mileage varies" >>>>>>> >>>>>>> ceph-01 : >>>>>>> >>>>>>> some errors, and it ends with >>>>>>> >>>>>>> 2025-07-20T03:24:18.887889+0000 mgr.srvr-ceph-03.dhzbpe >>>>>>> (mgr.134360) 1368 : cephadm [INF] Deploying daemon >>>>>>> mon.srvr- ceph-03 on srvr-ceph-03 >>>>>>> >>>>>>> when I had to remove the mon daemon and redeploy on ceph-03. >>>>>>> >>>>>>> ceph-02 : >>>>>>> >>>>>>> root@srvr-ceph-02:~# ceph log last debug cephadm >>>>>>> Command 'ceph' not found, but can be installed with: >>>>>>> snap install microceph    # version 18.2.4+snapc9f2b08f92, or >>>>>>> apt  install ceph-common  # version 17.2.7-0ubuntu0.22.04.2 >>>>>>> See 'snap info microceph' for additional versions. >>>>>>> >>>>>>> ??? should I install ceph-common ??? >>>>>>> >>>>>>> ceph-03 : >>>>>>> >>>>>>> root@srvr-ceph-03:~# ceph log last debug cephadm >>>>>>> Error initializing cluster client: ObjectNotFound('RADOS >>>>>>> object not found (error calling conf_read_file)') >>>>>>> root@srvr-ceph-03:~# >>>>>>> >>>>>>> FWIW : ceph health is : >>>>>>> >>>>>>> root@srvr-ceph-01:~# ceph health detail >>>>>>> HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, >>>>>>> quorum srvr-ceph-03,srvr-ceph-02; 10 daemons have recently >>>>>>> crashed >>>>>>> [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) >>>>>>>     daemon mon.srvr-ceph-01 on srvr-ceph-01 is in error state >>>>>>> [WRN] MON_DOWN: 1/3 mons down, quorum >>>>>>> srvr-ceph-03,srvr-ceph-02 >>>>>>>     mon.srvr-ceph-01 (rank 0) addr >>>>>>> [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] is down >>>>>>> (out of quorum) >>>>>>> [WRN] RECENT_CRASH: 10 daemons have recently crashed >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:50:10.202091Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:49:47.712267Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:50:21.464475Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:49:36.609442Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:49:58.966663Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:51:36.947240Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:52:21.751711Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:51:48.490875Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:51:59.651129Z >>>>>>>     mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >>>>>>> 2025-07-20T17:52:10.552756Z >>>>>>> >>>>>>> S. Barthes >>>>>>> Le 21/07/2025 à 09:31, Michel Jouvin a écrit : >>>>>>>> Stephane, >>>>>>>> >>>>>>>> If you are using cephadm, the OS (distrib and version) >>>>>>>> you use should not matter. When using cephadm with >>>>>>>> several servers (the general case!), it is important to >>>>>>>> setup properly the SSH key used by cephadm for the >>>>>>>> communication between nodes (cephadm is sort of a >>>>>>>> SSH-based management cluster) and to check that you can >>>>>>>> log in from one node to the other using SSH. Can you >>>>>>>> confirm that it is the case? >>>>>>>> >>>>>>>> Also cephadm has a specific log file. I don't use much >>>>>>>> the dashboard, not sure how you display it (it may be >>>>>>>> part of the logs displayed by the dashboard) but you can >>>>>>>> access it with the command: >>>>>>>> >>>>>>>> ceph log last debug cephadm >>>>>>>> >>>>>>>> Michel >>>>>>>> >>>>>>>> Le 21/07/2025 à 09:19, Stéphane Barthes a écrit : >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Yes, I did use cephadm, to bootstrap the 1st node in the >>>>>>>>> cluster, installed cephadm on the other nodes, and used >>>>>>>>> the dashboard to add the nodes to the cluster. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> S. Barthes >>>>>>>>> >>>>>>>>> Le 21/07/2025 à 09:12, Michel Jouvin a écrit : >>>>>>>>>> Hi Stephane, >>>>>>>>>> >>>>>>>>>> How did you configure your cluster? Have you been using >>>>>>>>>> cephadm? If not, I really advise you to recreate your >>>>>>>>>> cluster with cephadm, that includes a script to >>>>>>>>>> bootstrap the cluster. In particular if you don't have >>>>>>>>>> a detail knowledge about Ceph architecture and >>>>>>>>>> management, it will ensure that your cluster is >>>>>>>>>> properly configured and let you progressively learn >>>>>>>>>> about Ceph details... >>>>>>>>>> >>>>>>>>>> Best regards. >>>>>>>>>> >>>>>>>>>> Michel >>>>>>>>>> >>>>>>>>>> Le 21/07/2025 à 09:02, Stéphane Barthes a écrit : >>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I am very new to ceph and have started a small cluster >>>>>>>>>>> to get started with ceph. >>>>>>>>>>> >>>>>>>>>>> But so far my experience, is not very impressive, >>>>>>>>>>> probably by lack of knowledge and good practices. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I started with Ubuntu 24, installed 3 VM for a ceph >>>>>>>>>>> cluster, and some how could not get it running. Adding >>>>>>>>>>> nodes would fail adding OSDs with some weird error(I >>>>>>>>>>> found it on the web but could not solve the problem). >>>>>>>>>>> >>>>>>>>>>> I then made a new cluster with 3 ubuntu 22 VM. Install >>>>>>>>>>> ok, start ok, I created 1 pool to test storing stuff >>>>>>>>>>> there and work my way across crash testing. However >>>>>>>>>>> the cluster dies during the weekly vm snapshot. It may >>>>>>>>>>> not a good idea to run vm backups on a ceph host, but >>>>>>>>>>> I find this a little surprising. (crash testing >>>>>>>>>>> started earlier than expected) >>>>>>>>>>> >>>>>>>>>>> Bottom line is that, after the backup the cluster is >>>>>>>>>>> in warning state with missing mons, or logrotate and >>>>>>>>>>> sometimes crashed machines. systemctl restart service >>>>>>>>>>> or Rebooting node usually fixes it. >>>>>>>>>>> >>>>>>>>>>> I am now stuck in a situation I cannot fix : >>>>>>>>>>> >>>>>>>>>>>     - 1 Machine is ceph rbd client cannot auth : auth >>>>>>>>>>> method 'x' error -13. I have tried quite a few things, >>>>>>>>>>> and none unlocked the situation. I am currently trying >>>>>>>>>>> to reboot the machine, but the busy/stuck rbd device >>>>>>>>>>> seems to block it. I am not looking forward to hard >>>>>>>>>>> reset it. >>>>>>>>>>> >>>>>>>>>>>     - Node with the mgr service will not restart mon, >>>>>>>>>>> or logrotate. I did reboot it again today, but I guess >>>>>>>>>>> this is not how a node is expected to behave. >>>>>>>>>>> >>>>>>>>>>> So my questions : >>>>>>>>>>> >>>>>>>>>>>     - How can I unlock my stuck ceph client, when this >>>>>>>>>>> kind of error occurs? >>>>>>>>>>> >>>>>>>>>>>     - Is this expected behavior that client looses >>>>>>>>>>> access to cluster, which kind of kills the machine? >>>>>>>>>>> >>>>>>>>>>>     - Where should I look in the ceph nodes logs to >>>>>>>>>>> figure what is going wrong, and how to fix it, so that >>>>>>>>>>> is run in a stable manner? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> S. Barthes >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, Zitat von Stéphane Barthes <stephane.barthes@intest.info>:

Hello,

Thank you very much to every one for helping and giving advice, my cluster is backup online with HEALTH_OK, and it looks like no data was lost. 

I have not been able to convince the cluster to run on 1 mon, as all ceph/cephadm commands would either hang or fail.

That's an offline operation, you need to stop the MON processe(s) and then manipulate the monmap of one MON offline, then inject it and start up the single MON.

I resorted to restoring a backup of ceph-03 which gave me quorum for long enough to : 

    - Label ceph-02 & ceph-03 as _admin

    - Remove ceph-01 daemon, and redeploy it

During my investigations, I found that ceph-01 was not running ntpd, so I installed it. But I am under the impression that my problem is hardware linked to the VM host. We have had some VMs behaving strangley on this machine recently. I transferred all ceph-vm to another host, and will see if this remains stable over time (even during periodic backup).

To answer Michel's question, we are running a small proxmox v8.4 cluster.

FWIW, the install document indicates you should label mon nodes as _admin when creating the cluster, and gives examples with the label in the CLI. When using the GUI, as I did there is a label option which is empty by default, and I did not select the _admin label. 

Best regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 22/07/2025 à 14:32, Michel Jouvin a écrit :

...

Stéphane,

As you use VMs for your deployment, would it make sense to stop and keep them and restart with a new set of VMs, following https://docs.ceph.com/en/latest/cephadm/install/. I personnally don't have experience to expand the cluster through the dashboard (it should work! just that I am an old guy so used to command line tools!). Probably using the command line makes easier to identify when there is a problem, without always digging in the logs. I remember following this doc a couple of years ago, the last time I created a cluster and it was working as expected.

If you restart something fresh, I'd start with the Squid version rather than Reef. It should not make any difference for the installation but will bring you the latest-and-greatest Ceph! Once you have something running properly, you may compare what is different in the other configuration. I don't have any opinion on Debian 24 as we are using RedHat (AlmaLinux in fact) but for me the OS version should not make a big difference when using cephadm, as Ceph in fact runs in containers. But the devil is in the details, Maite may have a reason for her warning...

One thing we didn't mention/check with you in the previous exchanges is the Ceph network configuration you used. Any discrepancy in the network configuration, in particular if you have a separate cluster (network used only by OSD) and public (network used by monitors and Ceph clients) networks, you may have a situation where one of them is not working as expected and may lead to daemons not being able to see eachothers. But I think your problem is much more basic: the daemons were not able to restart after some corruptions of their DB, it is pretty unusual and may also be the result of something weird that happened in your VM infrastructure leading to storage corruption... What are you using to manage to your VMs?

Michel

Le 22/07/2025 à 13:48, Stéphane Barthes a écrit :

...

Michel,

Thank you very much for the help.

I will look into the documentation provided by Eugen to try reduce to 1 mon, remove and re-add the 2 mon on the node ceph01 and ceph02.

Regarding the installation, I have created a vm template with ubuntu installed ready for ceph.

Installed cephadm on ceph01, bootstrapped the cluster, and added the other vms from the dashboard. I guess this is all I did to setup the cluster.

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 22/07/2025 à 12:12, Michel Jouvin a écrit :

...

Stéphane,

Basically you cannot do anything in your cluster until you reach the quorum. Except managing it with cephadm to restore a functionning cluster. If 'ceph -s' doesn't return, it means you lost the quorum, it is the only reason I'm aware for this. As your cluster is quite simple, it should be easy to see the state of the monitor daemon on each host where one should run using `cephadm ls` and/or `podman/docker ps`. And you should be able to get access to the daemon logs of the monitor daemons.

In one of your message yesterday you reported a log saying the rocks.db of one of the mon was corrupted. I personnally never saw that but the first thing to do is to fix this as it will prevent the mon to start. Follow the doc mentioned by Eugen to reduce your quorum to 1 mon (deleting the 2 broken ones from the monmap) if necessary (if you don't find a way to start at least 2 mon). And as said in another message, ensure you added the label _admin to the hosts where you want to be able to use the ceph command else the required information to connect to the cluster will be missing. It is done with 'ceph orch host label add' command which requires that you fixed the quorum issue. One possibility if you have one mon healthy and you manage to reduce the quorum to 1 is to delete the 2 other mon and readd them as new mon so that they are reinitialized. This way you will not loose anything. Look at cephadm documentation to learn how to remove and add daemons.

One thing not fully clear for me is how you installed your different hosts. It seems they are not configured exactly the same way as on one host the ceph command is not available where it is on the other ones. Ceph doesn't need a lot of things from the OS when using cephadm but it is pretty important to ensure that all your Ceph hosts are deployed the same way/with the same config else you just add to the entropy...

I fully agree with you and Eugen that trying to fix things is a way to learn a lot but at the same time it is not very easy to help you with the very limited information we have on what you did to be in such a strange situation... So if you don't manage to converge, may be it is better to restart from scratch following carefully the instructions: you will have plenty of other occasions to learn anyway!

Michel

Le 22/07/2025 à 11:04, Stéphane Barthes a écrit :

...

Hi Michel,

Does this mean I need to recover quorum, before some fixing happens?

Should I kick a new VM, and add a mon to the cluster, via cephaadm? This would allow to have 2 running mons?

S. Barthes Le 22/07/2025 à 10:39, Michel Jouvin a écrit :

...

Hi Stéphane,

'ceph -s' requires the mon quorum to be reached, else the Ceph cluster hangs. cephadm is not using the Ceph cluster internal communication but is building a management cluster on top of it so it can manage the cluster even if the quorum is lost but it cannot provide any information requires the quorum to be reached.

Michel

Le 22/07/2025 à 10:33, Stéphane Barthes a écrit :

> Hi Malte, > > Thank for  your reply.  Here are a some info : > > ceph -s hangs and times out monhunting after 300s > > But I can run cephadm shell. Is there a similar command > under cephadm shell? > > ceph health detail : same as above. > > I would like to repair it, instead of wipe & > restart, as it is (from my point of view) a good way to > learn (and there are a few data I'd like to recover). > > What is the problem with ubuntu 24? I did not see > warnings regarding this specific version in > https://docs.ceph.com/en/latest/cephadm/install/#cephadm-install-distros > > Regards, > > S. Barthes > T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 > InTest S.A. > 4 Allée du Levant > 69890 La Tour de Salvagny > Le 22/07/2025 à 10:02, Malte Stroem a écrit : > >> Hello Stéphane, >> >> I think, you're mixing and mismatching up a lot! >> >> You always have to show us the output of: >> >> ceph -s >> >> And more! Logs and stuff, e. g.: >> >> ceph health detail >> >> It is clear you missed something here and there. >> >> It is repairable but since it is a test cluster, just >> delete it and start again. >> >> And follow the documentation for cephadm. And do not >> use Ubuntu 24.04. >> >> Best, >> Malte >> >> On 22.07.25 09:02, Stéphane Barthes wrote: >> >>> Hello, >>> >>> Today, things have degraded a bit more. ceph-03 mon >>> has failed and will not restart. It shows the same >>> kind of checksum error in rocksdb compact operation >>> during startup. As a consequence, I lost quorum, >>> and ceph commands hang. >>> >>> Would it be wise to disable rocksbd compact, to >>> restart and find quorum back? If yes, what is the >>> exactt syntax of the setting in ceph.conf, I have >>> seen one for OSD, but not sure if it would apply: >>> >>> [osd] >>> >>> osd_compact_on_start = true >>> >>> If I can restart, I will try to out the OSDs, and >>> recreate them. Last time I saw the OSD seemed fine >>> in the dashboard. Since I hove no dashboard, is >>> there a command I can use to check their status? >>> >>> Regards, >>> >>> S. Barthes >>> T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 >>> InTest S.A. >>> 4 Allée du Levant >>> 69890 La Tour de Salvagny >>> >>> Le 21/07/2025 à 14:27, Stéphane Barthes a écrit : >>> >>>> Michel, >>>> >>>> cephadm shell starts on all 3 nodes without error, >>>> and each host as the same ceph public key entry in >>>> the .ssh/autorized_key file of the root user. >>>> >>>> ceph-01 also has ceph.pub in /etc/ceph with the >>>> same key (this is the node I started the install >>>> from) >>>> >>>> ceph-2 has no/etc/ceph folder >>>> >>>> ceph-3 has a /etc/ceph folder, but no ceph.pub >>>> file there >>>> >>>> S. Barthes >>>> Le 21/07/2025 à 12:36, Michel Jouvin a écrit : >>>> >>>>> Hi Stéphane, >>>>> >>>>> Sorry I was busy and did not look at your >>>>> previous answers... It is a bit difficult for me >>>>> to understand how you ended up in this situation >>>>> but for me it is strange that ceph-02 >>>>> complains about a missing keyring and >>>>> the corruped rocks.db on a freshly >>>>> created cluster is also a bit strange >>>>> for me. I don't think it makes sense to >>>>> destroy and recreate the OSD, I am running >>>>> several clusters with hundreds of OSDs and I >>>>> never saw a mis-initialized one. The problem is >>>>> hiding something else I'm afraid. Because of >>>>> some misconfiguration, may be one OSD is >>>>> in a bad state and may need to be >>>>> reinitialized but first we should get >>>>> the 3 mons running properly and >>>>> `cephadm shell` working properly on the 3 >>>>> hosts. And the rocks.db compaction issue for me >>>>> is related to your mon, not to an OSD. >>>>> >>>>> Have you checked that SSH configuration for >>>>> cephadm is working well from any host to any >>>>> other one in your cluster (with 3 hosts, it >>>>> should be really straighforward to check). The >>>>> ceph-02 problem may be the sign of SSH >>>>> misconfiguration as cephadm will use SSH >>>>> connection to push the keyring, if I am >>>>> right. >>>>> >>>>> Michel >>>>> >>>>> Le 21/07/2025 à 12:17, Stéphane Barthes a >>>>> écrit : >>>>> >>>>>> Hi, >>>>>> >>>>>> Should I just wipe the OSD and let ceph >>>>>> rebuild it (as suggested there : >>>>>> https://ceph-users.ceph.narkive.com/LO6ebu9r/how-to-recover- >>>>>> from-corrupted-rocksdb) ? >>>>>> >>>>>> Which would the suggested way be  : >>>>>> >>>>>> cephadm rm-daemon osd.ceph-01 >>>>>> >>>>>> then >>>>>> >>>>>> cephadm deploy ? >>>>>> >>>>>> Regards, >>>>>> >>>>>> S. Barthes >>>>>> T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 >>>>>> InTest S.A. >>>>>> 4 Allée du Levant >>>>>> 69890 La Tour de Salvagny >>>>>> Le 21/07/2025 à 10:33, Stéphane Barthes a >>>>>> écrit : >>>>>> >>>>>>> Michel, >>>>>>> >>>>>>> ceph-02 logs : >>>>>>> >>>>>>> root@srvr-ceph-02:/# ceph log last debug >>>>>>> cephadm >>>>>>> 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ >>>>>>> ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No >>>>>>> such file or directory >>>>>>> >>>>>>> 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 >>>>>>> AuthRegistry(0x7efe14064de0) no keyring >>>>>>> found at /etc/ceph/ >>>>>>> >>>>>>> ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ >>>>>>> keyring,/etc/ceph/keyring.bin, >>>>>>> disabling cephx >>>>>>> >>>>>>> 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ >>>>>>> ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No >>>>>>> such file or directory >>>>>>> >>>>>>> 2025-07-21T08:16:54.818+0000 7efe1a884640 -1 >>>>>>> AuthRegistry(0x7efe1a883000) no keyring >>>>>>> found at /etc/ceph/ >>>>>>> >>>>>>> ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ >>>>>>> keyring,/etc/ceph/keyring.bin, >>>>>>> disabling cephx >>>>>>> >>>>>>> 2025-07-21T08:16:54.818+0000 7efe137fe640 -1 >>>>>>> monclient(hunting): handle_auth_bad_method >>>>>>> server allowed_methods [2] but i only >>>>>>> support [1] >>>>>>> >>>>>>> 2025-07-21T08:16:54.818+0000 7efe18e21640 -1 >>>>>>> monclient(hunting): handle_auth_bad_method >>>>>>> server allowed_methods [2] but i only >>>>>>> support [1] >>>>>>> >>>>>>> 2025-07-21T08:16:57.818+0000 7efe137fe640 -1 >>>>>>> monclient(hunting): handle_auth_bad_method >>>>>>> server allowed_methods [2] but i only >>>>>>> support [1] >>>>>>> >>>>>>> 2025-07-21T08:16:57.818+0000 7efe13fff640 -1 >>>>>>> monclient(hunting): handle_auth_bad_method >>>>>>> server allowed_methods [2] but i only >>>>>>> support [1] >>>>>>> >>>>>>> 2025-07-21T08:17:00.818+0000 7efe137fe640 -1 >>>>>>> monclient(hunting): handle_auth_bad_method >>>>>>> server allowed_methods [2] but i only >>>>>>> support [1] >>>>>>> >>>>>>> 2025-07-21T08:17:00.818+0000 7efe18e21640 -1 >>>>>>> monclient(hunting): handle_auth_bad_method >>>>>>> server allowed_methods [2] but i only >>>>>>> support [1] >>>>>>> >>>>>>> ^CCluster connection aborted >>>>>>> root@srvr-ceph-02:/# >>>>>>> >>>>>>> Regarding the ceph-01 log, there is a LOT. >>>>>>> looking from the end, I see this : >>>>>>> >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -19> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359f42d8c0 2 auth: KeyRing::load: loaded >>>>>>> key file >>>>>>> /var/lib/ceph/mon/ceph-srvr-ceph-01/keyring >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -18> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359f42d8c0 2 mon.srvr- ceph-01@-1(???) e5 >>>>>>> init >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -17> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359f42d8c0 4 mgrc handle_mgr_map Got map >>>>>>> version 73 >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -16> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359f42d8c0 4 mgrc handle_mgr_map Active >>>>>>> mgr is now >>>>>>> [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -15> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359f42d8c0 4 mgrc reconnect Starting new >>>>>>> session with >>>>>>> >>>>>>> [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -14> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359c206640 -1 mon.srvr- ceph-01@-1(???) e5 >>>>>>> handle_auth_bad_method hmm, they >>>>>>> didn't like 2 result (13) >>>>>>> Permission denied >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -13> 2025-07-20T17:52:21.137+0000 >>>>>>> 7f359f42d8c0 0 mon.srvr- ceph-01@-1(probing) >>>>>>> e5  my rank is now 0 (was -1) >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -12> 2025-07-20T17:52:21.161+0000 >>>>>>> 7f359d208640 3 rocksdb: [db/db_impl/ >>>>>>> db_impl_compaction_flush.cc:3026] Compaction >>>>>>> error: Corruption: block checksum >>>>>>> mismatch: stored = 3368055299, >>>>>>> computed = 2100551158  in >>>>>>> /var/lib/ceph/mon/ceph-srvr-ceph-01/ >>>>>>> store.db/061999.sst offset 10379525 size >>>>>>> 91317 >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -11> 2025-07-20T17:52:21.161+0000 >>>>>>> 7f359d208640 4 rocksdb: (Original Log Time >>>>>>> 2025/07/20-17:52:21.164193) >>>>>>> [db/compaction/ >>>>>>> compaction_job.cc:812] [default] compacted >>>>>>> to: base level 6 level multiplier 10.00 max >>>>>>> bytes base 268435456 files[4 0 0 0 0 0 1] >>>>>>> max score 0.00, MB/sec: 514.9 rd, >>>>>>> 272.6 wr, level 6, files in(4, 1) >>>>>>> out(0) MB in(4.0, 14.8) out(9.9), >>>>>>> read-write-amplify(7.2) write- >>>>>>> amplify(2.5) Corruption: block >>>>>>> checksum mismatch: stored = 3368055299, >>>>>>> computed = 2100551158 in >>>>>>> /var/lib/ceph/mon/ceph-srvr- >>>>>>> ceph-01/store.db/061999.sst offset 10379525 >>>>>>> size 91317, records in: 25191, records >>>>>>> dropped: 3 >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug -10> 2025-07-20T17:52:21.161+0000 >>>>>>> 7f359d208640 4 rocksdb: (Original Log Time >>>>>>> 2025/07/20-17:52:21.164212) >>>>>>> EVENT_LOG_v1 {"time_micros": >>>>>>> 1753033941164205, "job": 3, >>>>>>> "event": "compaction_finished", >>>>>>> "compaction_time_micros": 38166, >>>>>>> "compaction_time_cpu_micros": 25133, >>>>>>> "output_level": 6, "num_output_files": 0, >>>>>>> "total_output_size": 10404253, >>>>>>> "num_input_records": 25191, >>>>>>> "num_output_records": 21216, >>>>>>> "num_subcompactions": 1, >>>>>>> "output_compression": "NoCompression", >>>>>>> "num_single_delete_mismatches": 0, >>>>>>> "num_single_delete_fallthrough": 0, >>>>>>> "lsm_state": [4, 0, 0, 0, 0, 0, 1]} >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -9> 2025-07-20T17:52:21.161+0000 >>>>>>> 7f359d208640 2 rocksdb: [db/db_impl/ >>>>>>> db_impl_compaction_flush.cc:2545] Waiting >>>>>>> after background compaction error: >>>>>>> Corruption: block checksum mismatch: >>>>>>> stored = 3368055299, computed = >>>>>>> 2100551158  in >>>>>>> /var/lib/ceph/mon/ceph-srvr- >>>>>>> ceph-01/store.db/061999.sst offset 10379525 >>>>>>> size 91317, Accumulated background error >>>>>>> counts: 1 >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -8> 2025-07-20T17:52:21.341+0000 >>>>>>> 7f359c206640 -1 mon.srvr- ceph-01@0(probing) >>>>>>> e5 handle_auth_bad_method hmm, they >>>>>>> didn't like 2 result (13) >>>>>>> Permission denied >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -7> 2025-07-20T17:52:21.741+0000 >>>>>>> 7f359c206640 -1 mon.srvr- ceph-01@0(probing) >>>>>>> e5 handle_auth_bad_method hmm, they >>>>>>> didn't like 2 result (13) >>>>>>> Permission denied >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -6> 2025-07-20T17:52:21.741+0000 >>>>>>> 7f359c206640 1 mon.srvr- ceph-01@0(probing) >>>>>>> e5 handle_auth_request failed to >>>>>>> assign global_id >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -5> 2025-07-20T17:52:21.749+0000 >>>>>>> 7f35981fe640 5 mon.srvr- ceph-01@0(probing) >>>>>>> e5 _ms_dispatch setting monitor caps >>>>>>> on this connection >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -4> 2025-07-20T17:52:21.749+0000 >>>>>>> 7f35981fe640 1 mon.srvr- >>>>>>> ceph-01@0(synchronizing) e5 >>>>>>> sync_obtain_latest_monmap >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -3> 2025-07-20T17:52:21.749+0000 >>>>>>> 7f35981fe640 1 mon.srvr- >>>>>>> ceph-01@0(synchronizing) e5 >>>>>>> sync_obtain_latest_monmap obtained monmap e5 >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> [285B blob data] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> PutCF( prefix = mon_sync key = >>>>>>> 'latest_monmap' value size = 508) >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> PutCF( prefix = mon_sync key = 'in_sync' >>>>>>> value size = 8) >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> PutCF( prefix = mon_sync key = >>>>>>> 'last_committed_floor' value size = 8) >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug  -1> 2025-07-20T17:52:21.749+0000 >>>>>>> 7f35981fe640 -1 /home/jenkins-build/ >>>>>>> >>>>>>> build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ >>>>>>> AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ >>>>>>> >>>>>>> release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f35981fe640 time >>>>>>> 2025-07-20T17:52:21.750611+0000 >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> /home/jenkins-build/build/ >>>>>>> workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ >>>>>>> AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ >>>>>>> >>>>>>> release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: 355: ceph_abort_msg("failed to write to >>>>>>> db") >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> ceph version 17.2.8 >>>>>>> (f817ceb7f187defb1d021d6328fa833eb8e943b3) >>>>>>> quincy (stable) >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: >>>>>>> (ceph::__ceph_abort(char const*, int, char >>>>>>> const*, std::__cxx11::basic_string<char, >>>>>>> std::char_traits<char>, >>>>>>> std::allocator<char> > >>>>>>> const&)+0xd3) [0x7f35a03a5469] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: >>>>>>> /usr/bin/ceph- mon(+0x1e968e) >>>>>>> [0x55e079c5768e] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: >>>>>>> (Monitor::sync_start(entity_addrvec_t&, >>>>>>> bool)+0x3b5) [0x55e079c8a145] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: >>>>>>> (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) >>>>>>> [0x55e079c90baf] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: >>>>>>> (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) >>>>>>> [0x55e079c925dc] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: >>>>>>> (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) >>>>>>> [0x55e079cad20d] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: >>>>>>> (Monitor::_ms_dispatch(Message*)+0x7c9) >>>>>>> [0x55e079cadd89] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: >>>>>>> /usr/bin/ceph- mon(+0x1f6d3e) >>>>>>> [0x55e079c64d3e] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: >>>>>>> (DispatchQueue::entry()+0x53a) >>>>>>> [0x7f35a058d34a] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 10: /usr/lib64/ceph/ >>>>>>> libceph-common.so.2(+0x3bdea1) >>>>>>> [0x7f35a061eea1] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 11: /lib64/ libc.so.6(+0x89e92) >>>>>>> [0x7f359fb6ce92] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 12: /lib64/ libc.so.6(+0x10ef20) >>>>>>> [0x7f359fbf1f20] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> debug 0> 2025-07-20T17:52:21.749+0000 >>>>>>> 7f35981fe640 -1 *** Caught signal (Aborted) >>>>>>> ** >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  in >>>>>>> thread 7f35981fe640 thread_name:ms_dispatch >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> ceph version 17.2.8 >>>>>>> (f817ceb7f187defb1d021d6328fa833eb8e943b3) >>>>>>> quincy (stable) >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  1: >>>>>>> /lib64/ libc.so.6(+0x3e730) [0x7f359fb21730] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  2: >>>>>>> /lib64/ libc.so.6(+0x8bbdc) [0x7f359fb6ebdc] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  3: >>>>>>> raise() >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  4: >>>>>>> abort() >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  5: >>>>>>> (ceph::__ceph_abort(char const*, int, char >>>>>>> const*, std::__cxx11::basic_string<char, >>>>>>> std::char_traits<char>, >>>>>>> std::allocator<char> > >>>>>>> const&)+0x190) [0x7f35a03a5526] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  6: >>>>>>> /usr/bin/ceph- mon(+0x1e968e) >>>>>>> [0x55e079c5768e] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  7: >>>>>>> (Monitor::sync_start(entity_addrvec_t&, >>>>>>> bool)+0x3b5) [0x55e079c8a145] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  8: >>>>>>> (Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) >>>>>>> [0x55e079c90baf] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  9: >>>>>>> (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) >>>>>>> [0x55e079c925dc] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 10: >>>>>>> (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) >>>>>>> [0x55e079cad20d] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 11: (Monitor::_ms_dispatch(Message*)+0x7c9) >>>>>>> [0x55e079cadd89] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 12: /usr/bin/ceph- mon(+0x1f6d3e) >>>>>>> [0x55e079c64d3e] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 13: (DispatchQueue::entry()+0x53a) >>>>>>> [0x7f35a058d34a] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 14: /usr/lib64/ceph/ >>>>>>> libceph-common.so.2(+0x3bdea1) >>>>>>> [0x7f35a061eea1] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 15: /lib64/ libc.so.6(+0x89e92) >>>>>>> [0x7f359fb6ce92] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]:  >>>>>>> 16: /lib64/ libc.so.6(+0x10ef20) >>>>>>> [0x7f359fbf1f20] >>>>>>> Jul 20 17:52:21 srvr-ceph-01 bash[3424]: >>>>>>> NOTE: a copy of the executable, or `objdump >>>>>>> -rdS <executable>` is needed to >>>>>>> interpret this. >>>>>>> >>>>>>> I do not know if the logs are purged from >>>>>>> sensitive data that would prevent emailing >>>>>>> them. looking for "checksum mismatch", in >>>>>>> the logs, there are many of them >>>>>>> (138). >>>>>>> >>>>>>> How can I fix this checksum issue? >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> S. Barthes >>>>>>> >>>>>>> Le 21/07/2025 à 09:59, Michel Jouvin a >>>>>>> écrit : >>>>>>> >>>>>>>> Stéphane, >>>>>>>> >>>>>>>> On ceph-02, I am not sure why the ceph >>>>>>>> command is not installed as on the other >>>>>>>> nodes, if you installed it the same way. >>>>>>>> One way to get access to the >>>>>>>> ceph command on this server >>>>>>>> should be to execute: >>>>>>>> >>>>>>>> cephadm shell >>>>>>>> >>>>>>>> This will start a container where you have >>>>>>>> the ceph environment installed and >>>>>>>> configured for your cluster. >>>>>>>> >>>>>>>> The situation is not as bad as I thought >>>>>>>> reading your first message. You have the >>>>>>>> mon quorum so at least ceph command >>>>>>>> should be usable. The first >>>>>>>> thing to do is probably to log >>>>>>>> on your ceph-01 node and try >>>>>>>> to understand why the mon daemon is >>>>>>>> crashing. You may want to run on this >>>>>>>> node: >>>>>>>> >>>>>>>> cephadm ls  ---> Look for the exact >>>>>>>> daemon name corresponding to the mon >>>>>>>> >>>>>>>> cephadm logs --daemon $daemon_name >>>>>>>> >>>>>>>> Apart from this, it is strange that >>>>>>>> ceph-03 report a RADOS error with 'ceph >>>>>>>> log last...', this probably hides another >>>>>>>> issue. Could you tell what the same >>>>>>>> command says on ceph-02 (when >>>>>>>> run in cephadm shell)? >>>>>>>> >>>>>>>> Michel >>>>>>>> >>>>>>>> Le 21/07/2025 à 09:44, Stéphane Barthes a >>>>>>>> écrit : >>>>>>>> >>>>>>>>> Michel, >>>>>>>>> >>>>>>>>> I ran "ceph log last debug cephadm" on >>>>>>>>> my 3 nodes, and "mileage varies" >>>>>>>>> >>>>>>>>> ceph-01 : >>>>>>>>> >>>>>>>>> some errors, and it ends with >>>>>>>>> >>>>>>>>> 2025-07-20T03:24:18.887889+0000 >>>>>>>>> mgr.srvr-ceph-03.dhzbpe (mgr.134360) >>>>>>>>> 1368 : cephadm [INF] Deploying daemon >>>>>>>>> mon.srvr- ceph-03 on srvr-ceph-03 >>>>>>>>> >>>>>>>>> when I had to remove the mon daemon and >>>>>>>>> redeploy on ceph-03. >>>>>>>>> >>>>>>>>> ceph-02 : >>>>>>>>> >>>>>>>>> root@srvr-ceph-02:~# ceph log last debug >>>>>>>>> cephadm >>>>>>>>> Command 'ceph' not found, but can be >>>>>>>>> installed with: >>>>>>>>> snap install microceph    # version >>>>>>>>> 18.2.4+snapc9f2b08f92, or >>>>>>>>> apt  install ceph-common  # version >>>>>>>>> 17.2.7-0ubuntu0.22.04.2 >>>>>>>>> See 'snap info microceph' for additional >>>>>>>>> versions. >>>>>>>>> >>>>>>>>> ??? should I install ceph-common ??? >>>>>>>>> >>>>>>>>> ceph-03 : >>>>>>>>> >>>>>>>>> root@srvr-ceph-03:~# ceph log last debug >>>>>>>>> cephadm >>>>>>>>> Error initializing cluster client: >>>>>>>>> ObjectNotFound('RADOS object not found >>>>>>>>> (error calling conf_read_file)') >>>>>>>>> root@srvr-ceph-03:~# >>>>>>>>> >>>>>>>>> FWIW : ceph health is : >>>>>>>>> >>>>>>>>> root@srvr-ceph-01:~# ceph health detail >>>>>>>>> HEALTH_WARN 1 failed cephadm daemon(s); >>>>>>>>> 1/3 mons down, quorum >>>>>>>>> srvr-ceph-03,srvr-ceph-02; 10 daemons >>>>>>>>> have recently crashed >>>>>>>>> [WRN] CEPHADM_FAILED_DAEMON: 1 failed >>>>>>>>> cephadm daemon(s) >>>>>>>>>     daemon mon.srvr-ceph-01 on >>>>>>>>> srvr-ceph-01 is in error state >>>>>>>>> [WRN] MON_DOWN: 1/3 mons down, quorum >>>>>>>>> srvr-ceph-03,srvr-ceph-02 >>>>>>>>>     mon.srvr-ceph-01 (rank 0) addr >>>>>>>>> [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] >>>>>>>>> is down (out of quorum) >>>>>>>>> [WRN] RECENT_CRASH: 10 daemons have >>>>>>>>> recently crashed >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:50:10.202091Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:49:47.712267Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:50:21.464475Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:49:36.609442Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:49:58.966663Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:51:36.947240Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:52:21.751711Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:51:48.490875Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:51:59.651129Z >>>>>>>>>     mon.srvr-ceph-01 crashed on host >>>>>>>>> srvr-ceph-01 at >>>>>>>>> 2025-07-20T17:52:10.552756Z >>>>>>>>> >>>>>>>>> S. Barthes >>>>>>>>> Le 21/07/2025 à 09:31, Michel Jouvin a >>>>>>>>> écrit : >>>>>>>>> >>>>>>>>>> Stephane, >>>>>>>>>> >>>>>>>>>> If you are using cephadm, the OS >>>>>>>>>> (distrib and version) you use should >>>>>>>>>> not matter. When using cephadm with >>>>>>>>>> several servers (the >>>>>>>>>> general case!), it is >>>>>>>>>> important to setup properly the >>>>>>>>>> SSH key used by cephadm for the >>>>>>>>>> communication between nodes (cephadm >>>>>>>>>> is sort of a SSH-based >>>>>>>>>> management cluster) and >>>>>>>>>> to check that you can log >>>>>>>>>> in from one node to the other using >>>>>>>>>> SSH. Can you confirm that it is the >>>>>>>>>> case? >>>>>>>>>> >>>>>>>>>> Also cephadm has a specific log file. >>>>>>>>>> I don't use much the dashboard, not >>>>>>>>>> sure how you display it (it may be >>>>>>>>>> part of the logs >>>>>>>>>> displayed by the >>>>>>>>>> dashboard) but you can access it with >>>>>>>>>> the command: >>>>>>>>>> >>>>>>>>>> ceph log last debug cephadm >>>>>>>>>> >>>>>>>>>> Michel >>>>>>>>>> >>>>>>>>>> Le 21/07/2025 à 09:19, Stéphane >>>>>>>>>> Barthes a écrit : >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Yes, I did use cephadm, to bootstrap >>>>>>>>>>> the 1st node in the cluster, >>>>>>>>>>> installed cephadm on the other >>>>>>>>>>> nodes, and used the >>>>>>>>>>> dashboard to add the >>>>>>>>>>> nodes to the cluster. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> S. Barthes >>>>>>>>>>> >>>>>>>>>>> Le 21/07/2025 à 09:12, Michel Jouvin >>>>>>>>>>> a écrit : >>>>>>>>>>> >>>>>>>>>>>> Hi Stephane, >>>>>>>>>>>> >>>>>>>>>>>> How did you configure your >>>>>>>>>>>> cluster? Have you been using >>>>>>>>>>>> cephadm? If not, I really advise >>>>>>>>>>>> you to recreate your >>>>>>>>>>>> cluster with >>>>>>>>>>>> cephadm, that includes a script to >>>>>>>>>>>> bootstrap the cluster. In >>>>>>>>>>>> particular if you don't have a >>>>>>>>>>>> detail knowledge >>>>>>>>>>>> about Ceph >>>>>>>>>>>> architecture and management, it >>>>>>>>>>>> will ensure that your cluster is >>>>>>>>>>>> properly configured and let >>>>>>>>>>>> you progressively >>>>>>>>>>>> learn about Ceph >>>>>>>>>>>> details... >>>>>>>>>>>> >>>>>>>>>>>> Best regards. >>>>>>>>>>>> >>>>>>>>>>>> Michel >>>>>>>>>>>> >>>>>>>>>>>> Le 21/07/2025 à 09:02, Stéphane >>>>>>>>>>>> Barthes a écrit : >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>>> I am very new to ceph and have >>>>>>>>>>>>> started a small cluster to get >>>>>>>>>>>>> started with ceph. >>>>>>>>>>>>> >>>>>>>>>>>>> But so far my experience, is not >>>>>>>>>>>>> very impressive, probably by >>>>>>>>>>>>> lack of knowledge and >>>>>>>>>>>>> good practices. >>>>>>>>>>>>> >>>>>>>>>>>>> I started with Ubuntu 24, >>>>>>>>>>>>> installed 3 VM for a ceph >>>>>>>>>>>>> cluster, and some how could not >>>>>>>>>>>>> get it running. >>>>>>>>>>>>> Adding nodes >>>>>>>>>>>>> would fail adding OSDs with some >>>>>>>>>>>>> weird error(I found it on the >>>>>>>>>>>>> web but could not solve >>>>>>>>>>>>> the problem). >>>>>>>>>>>>> >>>>>>>>>>>>> I then made a new cluster with 3 >>>>>>>>>>>>> ubuntu 22 VM. Install ok, start >>>>>>>>>>>>> ok, I created 1 pool >>>>>>>>>>>>> to test storing >>>>>>>>>>>>> stuff there and work my >>>>>>>>>>>>> way across crash testing. >>>>>>>>>>>>> However the cluster dies during >>>>>>>>>>>>> the weekly vm >>>>>>>>>>>>> snapshot. It may >>>>>>>>>>>>> not a good idea to run vm >>>>>>>>>>>>> backups on a ceph host, but I >>>>>>>>>>>>> find this a little surprising. >>>>>>>>>>>>> (crash testing >>>>>>>>>>>>> started earlier >>>>>>>>>>>>> than expected) >>>>>>>>>>>>> >>>>>>>>>>>>> Bottom line is that, after the >>>>>>>>>>>>> backup the cluster is in warning >>>>>>>>>>>>> state with missing >>>>>>>>>>>>> mons, or >>>>>>>>>>>>> logrotate and sometimes crashed >>>>>>>>>>>>> machines. systemctl restart >>>>>>>>>>>>> service or Rebooting node >>>>>>>>>>>>> usually fixes it. >>>>>>>>>>>>> >>>>>>>>>>>>> I am now stuck in a situation I >>>>>>>>>>>>> cannot fix : >>>>>>>>>>>>> >>>>>>>>>>>>>     - 1 Machine is ceph rbd >>>>>>>>>>>>> client cannot auth : auth method >>>>>>>>>>>>> 'x' error -13. I have >>>>>>>>>>>>> tried quite a >>>>>>>>>>>>> few things, and none >>>>>>>>>>>>> unlocked the situation. I am >>>>>>>>>>>>> currently trying to reboot the >>>>>>>>>>>>> machine, but the >>>>>>>>>>>>> busy/stuck rbd >>>>>>>>>>>>> device seems to block it. I am >>>>>>>>>>>>> not looking forward to hard >>>>>>>>>>>>> reset it. >>>>>>>>>>>>> >>>>>>>>>>>>>     - Node with the mgr service >>>>>>>>>>>>> will not restart mon, or >>>>>>>>>>>>> logrotate. I did reboot it >>>>>>>>>>>>> again today, >>>>>>>>>>>>> but I guess this is not >>>>>>>>>>>>> how a node is expected to >>>>>>>>>>>>> behave. >>>>>>>>>>>>> >>>>>>>>>>>>> So my questions : >>>>>>>>>>>>> >>>>>>>>>>>>>     - How can I unlock my stuck >>>>>>>>>>>>> ceph client, when this kind of >>>>>>>>>>>>> error occurs? >>>>>>>>>>>>> >>>>>>>>>>>>>     - Is this expected behavior >>>>>>>>>>>>> that client looses access to >>>>>>>>>>>>> cluster, which kind of >>>>>>>>>>>>> kills the >>>>>>>>>>>>> machine? >>>>>>>>>>>>> >>>>>>>>>>>>>     - Where should I look in the >>>>>>>>>>>>> ceph nodes logs to figure what >>>>>>>>>>>>> is going wrong, and how >>>>>>>>>>>>> to fix it, so >>>>>>>>>>>>> that is run in a stable >>>>>>>>>>>>> manner? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> --  >>>>>>>>>>>>> S. Barthes >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> ceph-users mailing list -- >>>>>>>>>>>>> ceph-users@ceph.io >>>>>>>>>>>>> To unsubscribe send an email to >>>>>>>>>>>>> ceph-users-leave@ceph.io >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- >>>>>>>>>>>> ceph-users@ceph.io >>>>>>>>>>>> To unsubscribe send an email to >>>>>>>>>>>> ceph-users-leave@ceph.io >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- >>>>>>>>>>> ceph-users@ceph.io >>>>>>>>>>> To unsubscribe send an email to >>>>>>>>>>> ceph-users-leave@ceph.io >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- >>>>>>>>>> ceph-users@ceph.io >>>>>>>>>> To unsubscribe send an email to >>>>>>>>>> ceph-users-leave@ceph.io >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- >>>>>>>>> ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to >>>>>>>>> ceph-users-leave@ceph.io >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- >>>>>>>> ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to >>>>>>>> ceph-users-leave@ceph.io >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to >>>>>> ceph-users-leave@ceph.io >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to >>>>> ceph-users-leave@ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to >>> ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to >> ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Malte why " And do not use Ubuntu 24.04." please ? I just reinstalled my cluster and use 24.04 and 19.2.2. so , if need be, there is still time to redo / reconfigure Steven On Tue, 22 Jul 2025 at 04:05, Malte Stroem <malte.stroem@gmail.com> wrote:

Hello Stéphane,

I think, you're mixing and mismatching up a lot!

You always have to show us the output of:

ceph -s

And more! Logs and stuff, e. g.:

ceph health detail

It is clear you missed something here and there.

It is repairable but since it is a test cluster, just delete it and start again.

And follow the documentation for cephadm. And do not use Ubuntu 24.04.

Best, Malte

On 22.07.25 09:02, Stéphane Barthes wrote:

...

Hello,

Today, things have degraded a bit more. ceph-03 mon has failed and will not restart. It shows the same kind of checksum error in rocksdb compact operation during startup. As a consequence, I lost quorum, and ceph commands hang.

Would it be wise to disable rocksbd compact, to restart and find quorum back? If yes, what is the exactt syntax of the setting in ceph.conf, I have seen one for OSD, but not sure if it would apply:

[osd]

osd_compact_on_start = true

If I can restart, I will try to out the OSDs, and recreate them. Last time I saw the OSD seemed fine in the dashboard. Since I hove no dashboard, is there a command I can use to check their status?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny

Le 21/07/2025 à 14:27, Stéphane Barthes a écrit :

...

Michel,

cephadm shell starts on all 3 nodes without error, and each host as the same ceph public key entry in the .ssh/autorized_key file of the root user.

ceph-01 also has ceph.pub in /etc/ceph with the same key (this is the node I started the install from)

ceph-2 has no/etc/ceph folder

ceph-3 has a /etc/ceph folder, but no ceph.pub file there

S. Barthes Le 21/07/2025 à 12:36, Michel Jouvin a écrit :

...

Hi Stéphane,

Sorry I was busy and did not look at your previous answers... It is a bit difficult for me to understand how you ended up in this situation but for me it is strange that ceph-02 complains about a missing keyring and the corruped rocks.db on a freshly created cluster is also a bit strange for me. I don't think it makes sense to destroy and recreate the OSD, I am running several clusters with hundreds of OSDs and I never saw a mis-initialized one. The problem is hiding something else I'm afraid. Because of some misconfiguration, may be one OSD is in a bad state and may need to be reinitialized but first we should get the 3 mons running properly and `cephadm shell` working properly on the 3 hosts. And the rocks.db compaction issue for me is related to your mon, not to an OSD.

Have you checked that SSH configuration for cephadm is working well from any host to any other one in your cluster (with 3 hosts, it should be really straighforward to check). The ceph-02 problem may be the sign of SSH misconfiguration as cephadm will use SSH connection to push the keyring, if I am right.

Michel

Le 21/07/2025 à 12:17, Stéphane Barthes a écrit :

...

Hi,

Should I just wipe the OSD and let ceph rebuild it (as suggested there : https://ceph-users.ceph.narkive.com/LO6ebu9r/how-to-recover- from-corrupted-rocksdb) ?

Which would the suggested way be :

cephadm rm-daemon osd.ceph-01

then

cephadm deploy ?

Regards,

S. Barthes T: +33 4 72 52 35 40 M: +33 6 14 73 18 34 InTest S.A. 4 Allée du Levant 69890 La Tour de Salvagny Le 21/07/2025 à 10:33, Stéphane Barthes a écrit :

...

Michel,

ceph-02 logs :

root@srvr-ceph-02:/# ceph log last debug cephadm 2025-07-21T08:16:54.814+0000 7efe1a884640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory

2025-07-21T08:16:54.814+0000 7efe1a884640 -1 AuthRegistry(0x7efe14064de0) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ keyring,/etc/ceph/keyring.bin, disabling cephx

2025-07-21T08:16:54.818+0000 7efe1a884640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory

2025-07-21T08:16:54.818+0000 7efe1a884640 -1 AuthRegistry(0x7efe1a883000) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/ keyring,/etc/ceph/keyring.bin, disabling cephx

2025-07-21T08:16:54.818+0000 7efe137fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:16:54.818+0000 7efe18e21640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:16:57.818+0000 7efe137fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:16:57.818+0000 7efe13fff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:17:00.818+0000 7efe137fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

2025-07-21T08:17:00.818+0000 7efe18e21640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]

^CCluster connection aborted root@srvr-ceph-02:/#

Regarding the ceph-01 log, there is a LOT. looking from the end, I see this :

Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -19> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-srvr-ceph-01/keyring Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -18> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 2 mon.srvr- ceph-01@-1(???) e5 init Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -17> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc handle_mgr_map Got map version 73 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -16> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc handle_mgr_map Active mgr is now [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -15> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 4 mgrc reconnect Starting new session with [v2:10.32.100.24:6800/4015208663,v1:10.32.100.24:6801/4015208663] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -14> 2025-07-20T17:52:21.137+0000 7f359c206640 -1 mon.srvr- ceph-01@-1(???) e5 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -13> 2025-07-20T17:52:21.137+0000 7f359f42d8c0 0 mon.srvr- ceph-01@-1(probing) e5 my rank is now 0 (was -1) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -12> 2025-07-20T17:52:21.161+0000 7f359d208640 3 rocksdb: [db/db_impl/ db_impl_compaction_flush.cc:3026] Compaction error: Corruption: block checksum mismatch: stored = 3368055299, computed = 2100551158 in /var/lib/ceph/mon/ceph-srvr-ceph-01/ store.db/061999.sst offset 10379525 size 91317 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -11> 2025-07-20T17:52:21.161+0000 7f359d208640 4 rocksdb: (Original Log Time 2025/07/20-17:52:21.164193) [db/compaction/ compaction_job.cc:812] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 1] max score 0.00, MB/sec: 514.9 rd, 272.6 wr, level 6, files in(4, 1) out(0) MB in(4.0, 14.8) out(9.9), read-write-amplify(7.2) write- amplify(2.5) Corruption: block checksum mismatch: stored = 3368055299, computed = 2100551158 in /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst offset 10379525 size 91317, records in: 25191, records dropped: 3 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -10> 2025-07-20T17:52:21.161+0000 7f359d208640 4 rocksdb: (Original Log Time 2025/07/20-17:52:21.164212) EVENT_LOG_v1 {"time_micros": 1753033941164205, "job": 3, "event": "compaction_finished", "compaction_time_micros": 38166, "compaction_time_cpu_micros": 25133, "output_level": 6, "num_output_files": 0, "total_output_size": 10404253, "num_input_records": 25191, "num_output_records": 21216, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 0, 0, 0, 0, 0, 1]} Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -9> 2025-07-20T17:52:21.161+0000 7f359d208640 2 rocksdb: [db/db_impl/ db_impl_compaction_flush.cc:2545] Waiting after background compaction error: Corruption: block checksum mismatch: stored = 3368055299, computed = 2100551158 in /var/lib/ceph/mon/ceph-srvr- ceph-01/store.db/061999.sst offset 10379525 size 91317, Accumulated background error counts: 1 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -8> 2025-07-20T17:52:21.341+0000 7f359c206640 -1 mon.srvr- ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -7> 2025-07-20T17:52:21.741+0000 7f359c206640 -1 mon.srvr- ceph-01@0(probing) e5 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -6> 2025-07-20T17:52:21.741+0000 7f359c206640 1 mon.srvr- ceph-01@0(probing) e5 handle_auth_request failed to assign global_id Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -5> 2025-07-20T17:52:21.749+0000 7f35981fe640 5 mon.srvr- ceph-01@0(probing) e5 _ms_dispatch setting monitor caps on this connection Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -4> 2025-07-20T17:52:21.749+0000 7f35981fe640 1 mon.srvr- ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -3> 2025-07-20T17:52:21.749+0000 7f35981fe640 1 mon.srvr- ceph-01@0(synchronizing) e5 sync_obtain_latest_monmap obtained monmap e5 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: [285B blob data] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = mon_sync key = 'latest_monmap' value size = 508) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = mon_sync key = 'in_sync' value size = 8) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: PutCF( prefix = mon_sync key = 'last_committed_floor' value size = 8) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug -1> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 /home/jenkins-build/ build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f35981fe640 time 2025-07-20T17:52:21.750611+0000 Jul 20 17:52:21 srvr-ceph-01 bash[3424]: /home/jenkins-build/build/ workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/ AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/ release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/mon/MonitorDBStore.h: 355: ceph_abort_msg("failed to write to db") Jul 20 17:52:21 srvr-ceph-01 bash[3424]: ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7f35a03a5469] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 2: /usr/bin/ceph- mon(+0x1e968e) [0x55e079c5768e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 3: (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) [0x55e079c8a145] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 4:

(Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) [0x55e079c90baf]

...

...

...

...

Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 5: (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) [0x55e079c925dc] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 6: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) [0x55e079cad20d] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 7: (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 8: /usr/bin/ceph- mon(+0x1f6d3e) [0x55e079c64d3e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 9: (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 10: /usr/lib64/ceph/ libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 11: /lib64/ libc.so.6(+0x89e92) [0x7f359fb6ce92] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 12: /lib64/ libc.so.6(+0x10ef20) [0x7f359fbf1f20] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: debug 0> 2025-07-20T17:52:21.749+0000 7f35981fe640 -1 *** Caught signal (Aborted) ** Jul 20 17:52:21 srvr-ceph-01 bash[3424]: in thread 7f35981fe640 thread_name:ms_dispatch Jul 20 17:52:21 srvr-ceph-01 bash[3424]: ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 1: /lib64/ libc.so.6(+0x3e730) [0x7f359fb21730] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 2: /lib64/ libc.so.6(+0x8bbdc) [0x7f359fb6ebdc] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 3: raise() Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 4: abort() Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x190) [0x7f35a03a5526] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 6: /usr/bin/ceph- mon(+0x1e968e) [0x55e079c5768e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 7: (Monitor::sync_start(entity_addrvec_t&, bool)+0x3b5) [0x55e079c8a145] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 8:

(Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0x83f) [0x55e079c90baf]

...

Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 9: (Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x36c) [0x55e079c925dc] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 10: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x112d) [0x55e079cad20d] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 11: (Monitor::_ms_dispatch(Message*)+0x7c9) [0x55e079cadd89] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 12: /usr/bin/ceph- mon(+0x1f6d3e) [0x55e079c64d3e] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 13: (DispatchQueue::entry()+0x53a) [0x7f35a058d34a] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 14: /usr/lib64/ceph/ libceph-common.so.2(+0x3bdea1) [0x7f35a061eea1] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 15: /lib64/ libc.so.6(+0x89e92) [0x7f359fb6ce92] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: 16: /lib64/ libc.so.6(+0x10ef20) [0x7f359fbf1f20] Jul 20 17:52:21 srvr-ceph-01 bash[3424]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

I do not know if the logs are purged from sensitive data that would prevent emailing them. looking for "checksum mismatch", in the logs, there are many of them (138).

How can I fix this checksum issue?

Regards,

S. Barthes

Le 21/07/2025 à 09:59, Michel Jouvin a écrit : > Stéphane, > > On ceph-02, I am not sure why the ceph command is not installed as > on the other nodes, if you installed it the same way. One way to > get access to the ceph command on this server should be to execute: > > cephadm shell > > This will start a container where you have the ceph environment > installed and configured for your cluster. > > The situation is not as bad as I thought reading your first > message. You have the mon quorum so at least ceph command should > be usable. The first thing to do is probably to log on your > ceph-01 node and try to understand why the mon daemon is crashing. > You may want to run on this node: > > cephadm ls ---> Look for the exact daemon name corresponding to > the mon > > cephadm logs --daemon $daemon_name > > Apart from this, it is strange that ceph-03 report a RADOS error > with 'ceph log last...', this probably hides another issue. Could > you tell what the same command says on ceph-02 (when run in > cephadm shell)? > > Michel > > Le 21/07/2025 à 09:44, Stéphane Barthes a écrit : >> >> Michel, >> >> >> I ran "ceph log last debug cephadm" on my 3 nodes, and "mileage >> varies" >> >> ceph-01 : >> >> some errors, and it ends with >> >> 2025-07-20T03:24:18.887889+0000 mgr.srvr-ceph-03.dhzbpe >> (mgr.134360) 1368 : cephadm [INF] Deploying daemon mon.srvr- >> ceph-03 on srvr-ceph-03 >> >> when I had to remove the mon daemon and redeploy on ceph-03. >> >> ceph-02 : >> >> root@srvr-ceph-02:~# ceph log last debug cephadm >> Command 'ceph' not found, but can be installed with: >> snap install microceph # version 18.2.4+snapc9f2b08f92, or >> apt install ceph-common # version 17.2.7-0ubuntu0.22.04.2 >> See 'snap info microceph' for additional versions. >> >> ??? should I install ceph-common ??? >> >> ceph-03 : >> >> root@srvr-ceph-03:~# ceph log last debug cephadm >> Error initializing cluster client: ObjectNotFound('RADOS object >> not found (error calling conf_read_file)') >> root@srvr-ceph-03:~# >> >> FWIW : ceph health is : >> >> root@srvr-ceph-01:~# ceph health detail >> HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum >> srvr-ceph-03,srvr-ceph-02; 10 daemons have recently crashed >> [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) >> daemon mon.srvr-ceph-01 on srvr-ceph-01 is in error state >> [WRN] MON_DOWN: 1/3 mons down, quorum srvr-ceph-03,srvr-ceph-02 >> mon.srvr-ceph-01 (rank 0) addr >> [v2:10.32.100.22:3300/0,v1:10.32.100.22:6789/0] is down (out of >> quorum) >> [WRN] RECENT_CRASH: 10 daemons have recently crashed >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:50:10.202091Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:49:47.712267Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:50:21.464475Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:49:36.609442Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:49:58.966663Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:51:36.947240Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:52:21.751711Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:51:48.490875Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:51:59.651129Z >> mon.srvr-ceph-01 crashed on host srvr-ceph-01 at >> 2025-07-20T17:52:10.552756Z >> >> S. Barthes >> Le 21/07/2025 à 09:31, Michel Jouvin a écrit : >>> Stephane, >>> >>> If you are using cephadm, the OS (distrib and version) you use >>> should not matter. When using cephadm with several servers (the >>> general case!), it is important to setup properly the SSH key >>> used by cephadm for the communication between nodes (cephadm is >>> sort of a SSH-based management cluster) and to check that you >>> can log in from one node to the other using SSH. Can you confirm >>> that it is the case? >>> >>> Also cephadm has a specific log file. I don't use much the >>> dashboard, not sure how you display it (it may be part of the >>> logs displayed by the dashboard) but you can access it with the >>> command: >>> >>> ceph log last debug cephadm >>> >>> Michel >>> >>> Le 21/07/2025 à 09:19, Stéphane Barthes a écrit : >>>> >>>> Hi, >>>> >>>> Yes, I did use cephadm, to bootstrap the 1st node in the >>>> cluster, installed cephadm on the other nodes, and used the >>>> dashboard to add the nodes to the cluster. >>>> >>>> >>>> Regards, >>>> >>>> S. Barthes >>>> >>>> Le 21/07/2025 à 09:12, Michel Jouvin a écrit : >>>>> Hi Stephane, >>>>> >>>>> How did you configure your cluster? Have you been using >>>>> cephadm? If not, I really advise you to recreate your cluster >>>>> with cephadm, that includes a script to bootstrap the cluster. >>>>> In particular if you don't have a detail knowledge about Ceph >>>>> architecture and management, it will ensure that your cluster >>>>> is properly configured and let you progressively learn about >>>>> Ceph details... >>>>> >>>>> Best regards. >>>>> >>>>> Michel >>>>> >>>>> Le 21/07/2025 à 09:02, Stéphane Barthes a écrit : >>>>>> >>>>>> Hello, >>>>>> >>>>>> >>>>>> I am very new to ceph and have started a small cluster to get >>>>>> started with ceph. >>>>>> >>>>>> But so far my experience, is not very impressive, probably by >>>>>> lack of knowledge and good practices. >>>>>> >>>>>> >>>>>> I started with Ubuntu 24, installed 3 VM for a ceph cluster, >>>>>> and some how could not get it running. Adding nodes would >>>>>> fail adding OSDs with some weird error(I found it on the web >>>>>> but could not solve the problem). >>>>>> >>>>>> I then made a new cluster with 3 ubuntu 22 VM. Install ok, >>>>>> start ok, I created 1 pool to test storing stuff there and >>>>>> work my way across crash testing. However the cluster dies >>>>>> during the weekly vm snapshot. It may not a good idea to run >>>>>> vm backups on a ceph host, but I find this a little >>>>>> surprising. (crash testing started earlier than expected) >>>>>> >>>>>> Bottom line is that, after the backup the cluster is in >>>>>> warning state with missing mons, or logrotate and sometimes >>>>>> crashed machines. systemctl restart service or Rebooting node >>>>>> usually fixes it. >>>>>> >>>>>> I am now stuck in a situation I cannot fix : >>>>>> >>>>>> - 1 Machine is ceph rbd client cannot auth : auth method >>>>>> 'x' error -13. I have tried quite a few things, and none >>>>>> unlocked the situation. I am currently trying to reboot the >>>>>> machine, but the busy/stuck rbd device seems to block it. I >>>>>> am not looking forward to hard reset it. >>>>>> >>>>>> - Node with the mgr service will not restart mon, or >>>>>> logrotate. I did reboot it again today, but I guess this is >>>>>> not how a node is expected to behave. >>>>>> >>>>>> So my questions : >>>>>> >>>>>> - How can I unlock my stuck ceph client, when this kind >>>>>> of error occurs? >>>>>> >>>>>> - Is this expected behavior that client looses access to >>>>>> cluster, which kind of kills the machine? >>>>>> >>>>>> - Where should I look in the ceph nodes logs to figure >>>>>> what is going wrong, and how to fix it, so that is run in a >>>>>> stable manner? >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> -- >>>>>> S. Barthes >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io