- ceph-users - lists.ceph.io

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 22 Jul '26

22 Jul '26

Actually, what confused me was this: cephadm bootstrap --mon-ip <my-ip> Among other things, this command: - Generates a new SSH key for the Ceph cluster and adds it to the /root/.ssh/authorized_keys file. - Saves a copy of the public key to the /etc/ceph/ceph.pub file. You then need to add a node. ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> ceph orch host add <newhost> [<ip>] [<label1> ...] I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! -- M Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: > Hi again. > > I'm answering my own question. > > It seems I read the wrong version of the doc > ... sic! > > It should be enough to set the “_admin” flag on all nodes: > > and that we also need to install cephadm on the nodes we > want to use for administration, right!?

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 22 Jul '26

22 Jul '26

Hi again. I'm answering my own question. It seems I read the wrong version of the doc ... sic! It should be enough to set the “_admin” flag on all nodes: and that we also need to install cephadm on the nodes we want to use for administration, right!? -- M Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice.

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Dhairya Parmar 22 Jul '26

22 Jul '26

Hi Andras, This sounds like the client served the lookups from its cached negative? Can you check if you see the lookup calls being made to MDS and maybe set debug_mds=20 to check if the client did send the request but it was MDS which returned ENOENT but since the `ls -l` succeeded I'm wondering if this is actually something from MDS. Maybe try a sync and drop_caches to validate that? *Dhairya Parmar* Software Engineer, CephFS On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < ceph-users(a)ceph.io> wrote: > Hi ceph users/devs, > > We've been encountering a strange issue with snapshots as part of a > project to copy our large ceph file system to a backup in parallel > across nodes. We are taking a snapshot, and then doing a copy of the > snapshot, which we expect to be static (unchanging) - using several > clients in parallel - all kernel mounts. Some clients are unable to > open files and directories that are known to exist (were discovered > traversing the file system on perhaps another node). > After some playing around, I could reproduce the issue on a single node > with just 'ls' as follows: > > # This file doesn't seems to exist - and some number of parent directories > are also "missing": [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # Until we get to this 'Projects' parent > directory, which seems to be there [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > 1961 1961 46 Jun 8 14:02 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > is still not there ... [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # But ... let's see if the "tickle" the Projects > directory a bit ... > [root@cephmds603 ~]# ls -l > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > # Now all subdirectories and the file in question immediately becomes > visible: > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > ceph squid 19.2.3: > [root@cephmds603 ~]# uname -a > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > x86_64 x86_64 GNU/Linux > [root@cephmds603 ~]# ceph --version > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > (stable) > [root@cephmds603 ~]# > > What is most interesting is that in the live tree (non-snapshotted tree) - > the Projects directory exists, but everything under it has been removed! > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > directory > > Am I running into some known issue? This looks to be specific to some > clients - i.e. some clients see the directory and some don't at the same > point in time, i.e. this is perhaps a kclient problem? > Any ideas/suggestions would be appreciated, > > Andras > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Anthony D'Atri 22 Jul '26

22 Jul '26

Exactly. Use the orchestrator, Luke. Additionally I would set count:3 for the mgr service. It’s kind of a hassle to recover from no active mgr so I encourage running one alongside every mon. > On Jul 21, 2026, at 4:30 PM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... > > If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/cephadm/operations/#enabling-management-of-… > > Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > >> Hi all, >> >> I am currently deploying a Ceph cluster with cephadm on 3 nodes with >> MON, MGR, and MDS roles. OSDs will be added later. >> >> My goal is for these 3 nodes to be fully interchangeable, with the >> ability to administer the cluster from any of them without risk if a >> node fails. >> >> I have identified that the solution involves synchronizing >> /var/lib/ceph/ (excluding the bootstrap file) across all nodes. >> >> However: >> >> - Is this the right approach to achieve this goal? >> >> - Which tool should I use for reliable bidirectional synchronization? >> >> - What are the potential risks (conflicts, corruption) and precautions >> to take? >> >> Have you implemented such a configuration in production before? >> >> Thank you for your advice. >> >> -- >> Michel >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Anthony D'Atri 22 Jul '26

22 Jul '26

I suggest gathering the kernel version on all clients, see if there’s a correlation.

1 0

CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Andras Pataki 21 Jul '26

21 Jul '26

Hi ceph users/devs, We've been encountering a strange issue with snapshots as part of a project to copy our large ceph file system to a backup in parallel across nodes. We are taking a snapshot, and then doing a copy of the snapshot, which we expect to be static (unchanging) - using several clients in parallel - all kernel mounts. Some clients are unable to open files and directories that are known to exist (were discovered traversing the file system on perhaps another node). After some playing around, I could reproduce the issue on a single node with just 'ls' as follows: # This file doesn't seems to exist - and some number of parent directories are also "missing": [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': No such file or directory [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # Until we get to this 'Projects' parent directory, which seems to be there [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 1961 1961 46 Jun 8 14:02 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it is still not there ... [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': No such file or directory # But ... let's see if the "tickle" the Projects directory a bit ... [root@cephmds603 ~]# ls -l /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null # Now all subdirectories and the file in question immediately becomes visible: [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK drwxrwsr-x 5 1961 1961 3 May 7 17:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM drwxrwsr-x 12 1961 1961 10 May 11 16:08 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen drwxrwsr-x 5 1961 1961 3 May 11 16:18 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen [root@cephmds603 ~]# ls -ld /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log # This is the kernel client 6.18.32 (relatively recent LTS kernel) with ceph squid 19.2.3: [root@cephmds603 ~]# uname -a Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux [root@cephmds603 ~]# ceph --version ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@cephmds603 ~]# What is most interesting is that in the live tree (non-snapshotted tree) - the Projects directory exists, but everything under it has been removed! [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or directory Am I running into some known issue? This looks to be specific to some clients - i.e. some clients see the directory and some don't at the same point in time, i.e. this is perhaps a kclient problem? Any ideas/suggestions would be appreciated, Andras

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Eugen Block 21 Jul '26

21 Jul '26

Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/operations/#enabling-management-of-… Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice. > > -- > Michel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph User / Dev Meetup - Wednesday, July 22, at 7:00 AM PDT
by Anthony Middleton 21 Jul '26

21 Jul '26

Hey Ceph Friends, Join us tomorrow, Wednesday, July 22, at 7:00 AM PDT / 14:00 UTC, for our July Ceph User + Dev Meetup. This Meetup is an opportunity for Ceph Users and Developers to discuss real-world deployment experiences and upcoming community initiatives. The following topics are on the agenda for this Meetup: - Discussing cloud transition and restore features that aid in integrating tape with Ceph and planning a proof of concept. - Inquire about improvements in the upcoming Umbrella release and ask for a release timeline. - Ongoing issues with SDs - Storage head nodes connected via multipath to a JBOD. RSVP and add your question or topic to the agenda here: https://lnkd.in/gRdjYAvF Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

How to administer Ceph from any node without bootstrap dependency
by Michel Le Cocq 21 Jul '26

21 Jul '26

Hi all, I am currently deploying a Ceph cluster with cephadm on 3 nodes with MON, MGR, and MDS roles. OSDs will be added later. My goal is for these 3 nodes to be fully interchangeable, with the ability to administer the cluster from any of them without risk if a node fails. I have identified that the solution involves synchronizing /var/lib/ceph/ (excluding the bootstrap file) across all nodes. However: - Is this the right approach to achieve this goal? - Which tool should I use for reliable bidirectional synchronization? - What are the potential risks (conflicts, corruption) and precautions to take? Have you implemented such a configuration in production before? Thank you for your advice. -- Michel

1 0

Re: Performance issue after tentacle upgrade
by Moti 21 Jul '26

21 Jul '26

Hello Frédéric Thanks for your response About iops; yes iops drop is good news, but this amount (far more than 50%) was a bit suspicious, and it shouldn't worsen latency. I was worried that slow disk might be doing what fast disk is supposed to do, but that's also not the case, so it looks like io just vanished to near zero. However this very well could be a separate problem/topic. About config defaults; thanks for that link, that's very handy. I checked the diff and tried to point out what might be problematic, these are the configs I tested changing, however did not get any better latency in response: - bluestore_elastic_shared_blob -> false - swapping osd_op_num_shards_hdd and osd_op_num_threads_per_shard that you mentioned - disabling ec partial reads - decreasing rgw thread pool size (I had this set to a higher value than default on reef) - decreasing rgw log trim interval I haven't tried changing envelope mode, I will, and thanks for the hint, but is it straight forward to upgrade wal back to envelope mode using bluestore tool if downgrade didn't seem to solve the issue? Also meanwhile since my original post I've been doing more tests, and the same case happened on other clusters as well (VMs with no separate fast/slow disks), the only time latency peak didn't happen, was when I tested on a single zone cluster, which is confusing since all my multi zone clusters use separate RGWs for running sync threads, and I'm seeing latency peak on client RGWs, not the sync ones. It might as well be another parameter that's different on the single zone one and I'm missing it, but that's my initial observation. Best regards Moti On Thu, 16 Jul 2026, 19:34 Frédéric Nass via ceph-users, <ceph-users(a)ceph.io> wrote: > Hi Moti, > > Anything suspicious on 'ceph osd perf' output? > > > NVMe disks' iops went from something around 5-6K constant io per second > to almost zero (less than a hundred per second, on all host/OSDs, with some > non-frequent high peaks), > > That would be pretty good news, actually. This would likely be the result > of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it > "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) > I don't expect this to explain the performance drop at this time, though. > > I'd recommend that you check the new/old settings and default values > between both versions using our online Config Diff tool here [2]. It might > ring a bell. > > The tool highlights that the default values for 'osd_op_num_shards_hdd' and > 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm > not sure what the impact could be on your cluster in terms of performance, > but if you're using mClock, reverting this change or trying 'wpq' osd > scheduler instead might help. You can run 'ceph config set osd osd_op_queue > wpq' and restart all OSDs to figure out if mClock has anything to do with > what you observe. > > If swapping values or changing the osd scheduler to wpq doesn't help, you > could try disabling 'bluefs_wal_envelope_mode' and running > 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test > this out on a single OSD first). > > Best regards, > > Frédéric > > [1] > > https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/… > [2] > > https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20… > > > Frédéric Nass > > Senior Ceph Engineer > > Ceph Ambassador, France > > +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> > > frederic.nass(a)clyso.com > > www.clyso.com > > Hohenzollernstr. 27, 80801 Muni > <https://www.google.com/maps/search/Hohenzollernstr.+27,+80801+Muni?entry=gm…> > ch > > Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 > > > > Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <ceph-users(a)ceph.io> a > écrit : > > > Hello everyone! > > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing > as > > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET > and > > HEAD requests. > > > > There is no resource cap or significant increase in usage is visible. > > Neither fast ec nor any other new features are enabled yet, just did the > > upgrade and left it as is. For measuring latency I have a services that > > probes the cluster with consistent load so it's not a difference in load > > either. PG status history does not show any pg operation being heavily in > > process. > > > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > block. > > Both ec and replicated pools exist. > > The strange hint was: NVMe disks' iops went from something around 5-6K > > constant io per second to almost zero (less than a hundred per second, on > > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > > not much changed. db blocks on NVMe are still attached to OSDs with no > > issue. > > > > I can't find a clue to explain this. Got the same results on 3 separate > > clusters (all went version 18 --> 20, same hybrid build and configs). > > Nothing in config dump changes (changed defaults) catches my attention to > > have caused this. > > > > Anyone had similar issues? Any tips and ideas on what to check? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0