- ceph-users - lists.ceph.io

Re: v21.1.0 Umbrella RC0 released
by Stefan Möding 22 Jul '26

22 Jul '26

Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> writes: > Are you prevented from deploying in containers? I thought it would be easier to use the provided packages and avoid the additional hassle with setting up and maintaining Docker. Also, if the images are build using Ubuntu 24.04 (I haven’t checked), the images would have the same issues. In addition we normally do not allow SSH logins between two production hosts, which seems to be a requirement for Cephadm. -- Stefan

1 0

Re: v21.1.0 Umbrella RC0 released
by Patrick Donnelly 22 Jul '26

22 Jul '26

On Wed, Jul 22, 2026 at 4:51 AM Stefan Möding via ceph-users <ceph-users(a)ceph.io> wrote: > > Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> writes: > > > This is the first release candidate for Umbrella. > > Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and > > Debian 13 (Trixie) were built for this RC0. > > Are there plans to also provide packages for Ubuntu 26.04 (released in > April)? The planned support is here: https://docs.ceph.com/en/latest/start/os-recommendations/#platforms > We are currently running Tentacle on Ubuntu 24.04. The Ceph code > depends on the Thrift c++ library which itself depends on multiple Qt 5 > modules. In Ubuntu 24.04 the included Qt 5 modules have know security > vulnerabilities and Canonical provides only fixes as part of an Expanded > Security Contract. That makes it difficult to run Ceph in an > environment where security compliance is a concern. Why is Canonical not updating/fixing these packages when the standard EOL for 24.04 is May 2029? In addition to the documentation improvements mentioned above, I am making an effort to have distribution package support predictable for future Ceph releases. I want to avoid situations where we retroactively add support for newer distributions to older Ceph releases. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: v21.1.0 Umbrella RC0 released
by Anthony D'Atri 22 Jul '26

22 Jul '26

Are you prevented from deploying in containers? > On Jul 22, 2026, at 4:51 AM, Stefan Möding via ceph-users <ceph-users(a)ceph.io> wrote: > > Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> writes: > >> This is the first release candidate for Umbrella. >> Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and >> Debian 13 (Trixie) were built for this RC0. > > Are there plans to also provide packages for Ubuntu 26.04 (released in > April)? > > We are currently running Tentacle on Ubuntu 24.04. The Ceph code > depends on the Thrift c++ library which itself depends on multiple Qt 5 > modules. In Ubuntu 24.04 the included Qt 5 modules have know security > vulnerabilities and Canonical provides only fixes as part of an Expanded > Security Contract. That makes it difficult to run Ceph in an > environment where security compliance is a concern. > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Anthony D'Atri 22 Jul '26

22 Jul '26

Excellent summary. In short, the bootstrap process is one time only. > On Jul 22, 2026, at 5:41 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, responses inline... > > > Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > >> Actually, what confused me was this: >> >> cephadm bootstrap --mon-ip <my-ip> >> >> Among other things, this command: >> - Generates a new SSH key for the Ceph cluster and adds it >> to the /root/.ssh/authorized_keys file. >> - Saves a copy of the public key to the /etc/ceph/ceph.pub file. >> >> You then need to add a node. >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> >> >> ceph orch host add <newhost> [<ip>] [<label1> ...] >> >> I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! > > That's stored in the config-key db: > > ceph config-key get mgr/cephadm/ssh_identity_key > >> >> -- >> M >> >> Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: >> >>> Hi again. >>> >>> I'm answering my own question. >>> >>> It seems I read the wrong version of the doc >>> ... sic! >>> >>> It should be enough to set the “_admin” flag on all nodes: >>> >>> and that we also need to install cephadm on the nodes we >>> want to use for administration, right!? > > Not necessarily, but it can be more convenient for debugging. The orchestrator stores a cephadm binary on each node to be able to manage them, it is stored in /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate the cluster/nodes etc. The original bootstrap node requires a cephadm binary either installed as a package or downloaded via curl as described in the docs to be able to bootstrap. After that you'll only need to execute cephadm commands locally on the nodes if you're debugging issues etc., otherwise the orchestrator will only use the file in /var/lib/ceph. > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 22 Jul '26

22 Jul '26

Hi Dhairya, Yeah, makes sense. Since I couldn't reproduce this locally, @Andras Pataki Could you help reproduce it by enabling the debug logs as Dhairya mentioned? And if possible please also try with the fuse client to see could you see too. Thanks Xiubo On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <dev(a)ceph.io> wrote: > > Hi Xiubo, > > I did look at the codepaths you shared when I responded but the thing that made me put my money on kclient is in `Server::handle_client_readdir`, there is this block https://github.com/ceph/ceph/blob/squid/src/mds/Server.cc#L5029-L5041. Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE` state then the readdir should've also served from memory and omit GSK, no? IMO if this is easily reproducible, we should get the MDS logs to verify the exact sequence. WDYT? > > > On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <xiubo.li(a)clyso.com> wrote: >> >> Hi Andras, Dhairya, >> >> I recall hitting a similar issue with this and the workaround involves >> running the 'ls' command, but I can't recall exactly which command or >> the tracker reference. This happened years ago. >> >> This morning I tried to reproduce this locally but failed. And I just >> went through the mds and kclient code just found two suspect places >> could cause this: >> >> MDS side: in https://github.com/ceph/ceph/blob/main/src/mds/MDCache.cc#L8693-L8696 >> >> MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: >> >> if (curdir->is_complete() || // (1) >> (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) >> return -ENOENT; >> >> is_complete() means all live entries are cached in memory. Shortcut >> (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), >> but shortcut (1) applies to all snapids — including snapshots. >> >> When a snapshot exists but the file was later deleted from the live >> tree, the MDS eventually evicts the deleted file's dentry from the >> in-memory cache (a null dentry, which trim_dentry() removes without >> clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits >> is_complete()=true with the dentry absent (ENOENT), without ever >> checking the OSD where the snapshot data actually lives. >> >> READDIR takes a different code path that fetches from the OSD, which >> is why ls -l fixes it. >> >> Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: >> if (snapid == CEPH_NOSNAP && >> (curdir->is_complete() || ...bloom...)) { >> >> @Andras >> >> BTW, to verify the above could you try to reproduce this by using the >> ceph-fuse mount ? Locally I couldn't reproduce it by trying many times >> and I am planing to add some debug code in MDS to ammulated it to >> reproduce it later. >> >> ==== >> >> Kclient side: https://github.com/ceph/ceph-client/blob/for-linus/fs/ceph/dir.c#L1984-L1987 >> Plus a kclient defense: ceph_d_revalidate() unconditionally trusts >> negative dentries in snapshot directories, caching a transient ENOENT >> forever. Change valid = 1 to valid = inode ? 1 : 0. >> >> I just created on trakcer to follow this: https://tracker.ceph.com/issues/78529 >> >> Thanks >> Xiubo >> >> >> >> >> On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users >> <ceph-users(a)ceph.io> wrote: >> > >> > Hi Andras, >> > >> > This sounds like the client served the lookups from its cached negative? >> > Can you check if you see the lookup calls being made to MDS and maybe set >> > debug_mds=20 to check if the client did send the request but it was MDS >> > which returned ENOENT but since the `ls -l` succeeded I'm wondering if this >> > is actually something from MDS. Maybe try a sync and drop_caches to >> > validate that? >> > >> > *Dhairya Parmar* >> > >> > Software Engineer, CephFS >> > >> > >> > On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < >> > ceph-users(a)ceph.io> wrote: >> > >> > > Hi ceph users/devs, >> > > >> > > We've been encountering a strange issue with snapshots as part of a >> > > project to copy our large ceph file system to a backup in parallel >> > > across nodes. We are taking a snapshot, and then doing a copy of the >> > > snapshot, which we expect to be static (unchanging) - using several >> > > clients in parallel - all kernel mounts. Some clients are unable to >> > > open files and directories that are known to exist (were discovered >> > > traversing the file system on perhaps another node). >> > > After some playing around, I could reproduce the issue on a single node >> > > with just 'ls' as follows: >> > > >> > > # This file doesn't seems to exist - and some number of parent directories >> > > are also "missing": [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> > > >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': >> > > >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> > > ls: cannot access >> > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': >> > > No such file or directory [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> > > No such file or directory # Until we get to this 'Projects' parent >> > > directory, which seems to be there [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 >> > > 1961 1961 46 Jun 8 14:02 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it >> > > is still not there ... [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot >> > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': >> > > No such file or directory # But ... let's see if the "tickle" the Projects >> > > directory a bit ... >> > > [root@cephmds603 ~]# ls -l >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null >> > > >> > > # Now all subdirectories and the file in question immediately becomes >> > > visible: >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> > > drwxrwsr-x 5 1961 1961 3 May 7 17:18 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> > > drwxrwsr-x 12 1961 1961 10 May 11 16:08 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> > > drwxrwsr-x 5 1961 1961 3 May 11 16:18 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen >> > > [root@cephmds603 ~]# ls -ld >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> > > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 >> > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log >> > > >> > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with >> > > ceph squid 19.2.3: >> > > [root@cephmds603 ~]# uname -a >> > > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 >> > > x86_64 x86_64 GNU/Linux >> > > [root@cephmds603 ~]# ceph --version >> > > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid >> > > (stable) >> > > [root@cephmds603 ~]# >> > > >> > > What is most interesting is that in the live tree (non-snapshotted tree) - >> > > the Projects directory exists, but everything under it has been removed! >> > > >> > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ >> > > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ >> > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK >> > > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or >> > > directory >> > > >> > > Am I running into some known issue? This looks to be specific to some >> > > clients - i.e. some clients see the directory and some don't at the same >> > > point in time, i.e. this is perhaps a kclient problem? >> > > Any ideas/suggestions would be appreciated, >> > > >> > > Andras >> > > _______________________________________________ >> > > ceph-users mailing list -- ceph-users(a)ceph.io >> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > >> > > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: No container image for 17.2.9 on quay.io?
by Eugen Block 22 Jul '26

22 Jul '26

Hi, correct, there are no container images as announced on the list [0]: > Please be aware that this hotfix was finalized after its end-of-life > (EOL) declaration and subsequent removal from the build system. As a > result, container builds were not created, and the hotfix has only > been subject to limited testing. I concur that upgrading to Reef before adopting is the way to go here. Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YSIXQKUJU4… Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Hi, > > Why is there no container image for 17.2.9 available on quay.io? > > We are currently trying to adopt a Quincy 17.2.9 cluster into the > orchestrator and fail at the step where the image should be pulled. > > Do we need to upgrade this cluster manually to Reef before being > able to adopt it? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

No container image for 17.2.9 on quay.io?
by Robert Sander 22 Jul '26

22 Jul '26

Hi, Why is there no container image for 17.2.9 available on quay.io? We are currently trying to adopt a Quincy 17.2.9 cluster into the orchestrator and fail at the step where the image should be pulled. Do we need to upgrade this cluster manually to Reef before being able to adopt it? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: CephFS+snapshots - unable to open files/directories that are known to exist in snapshots on some clients
by Xiubo Li 22 Jul '26

22 Jul '26

Hi Andras, Dhairya, I recall hitting a similar issue with this and the workaround involves running the 'ls' command, but I can't recall exactly which command or the tracker reference. This happened years ago. This morning I tried to reproduce this locally but failed. And I just went through the mds and kclient code just found two suspect places could cause this: MDS side: in https://github.com/ceph/ceph/blob/main/src/mds/MDCache.cc#L8693-L8696 MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: if (curdir->is_complete() || // (1) (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) return -ENOENT; is_complete() means all live entries are cached in memory. Shortcut (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), but shortcut (1) applies to all snapids — including snapshots. When a snapshot exists but the file was later deleted from the live tree, the MDS eventually evicts the deleted file's dentry from the in-memory cache (a null dentry, which trim_dentry() removes without clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits is_complete()=true with the dentry absent (ENOENT), without ever checking the OSD where the snapshot data actually lives. READDIR takes a different code path that fetches from the OSD, which is why ls -l fixes it. Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: if (snapid == CEPH_NOSNAP && (curdir->is_complete() || ...bloom...)) { @Andras BTW, to verify the above could you try to reproduce this by using the ceph-fuse mount ? Locally I couldn't reproduce it by trying many times and I am planing to add some debug code in MDS to ammulated it to reproduce it later. ==== Kclient side: https://github.com/ceph/ceph-client/blob/for-linus/fs/ceph/dir.c#L1984-L1987 Plus a kclient defense: ceph_d_revalidate() unconditionally trusts negative dentries in snapshot directories, caching a transient ENOENT forever. Change valid = 1 to valid = inode ? 1 : 0. I just created on trakcer to follow this: https://tracker.ceph.com/issues/78529 Thanks Xiubo On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Andras, > > This sounds like the client served the lookups from its cached negative? > Can you check if you see the lookup calls being made to MDS and maybe set > debug_mds=20 to check if the client did send the request but it was MDS > which returned ENOENT but since the `ls -l` succeeded I'm wondering if this > is actually something from MDS. Maybe try a sync and drop_caches to > validate that? > > *Dhairya Parmar* > > Software Engineer, CephFS > > > On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Hi ceph users/devs, > > > > We've been encountering a strange issue with snapshots as part of a > > project to copy our large ceph file system to a backup in parallel > > across nodes. We are taking a snapshot, and then doing a copy of the > > snapshot, which we expect to be static (unchanging) - using several > > clients in parallel - all kernel mounts. Some clients are unable to > > open files and directories that are known to exist (were discovered > > traversing the file system on perhaps another node). > > After some playing around, I could reproduce the issue on a single node > > with just 'ls' as follows: > > > > # This file doesn't seems to exist - and some number of parent directories > > are also "missing": [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > > No such file or directory # Until we get to this 'Projects' parent > > directory, which seems to be there [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > > 1961 1961 46 Jun 8 14:02 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > > is still not there ... [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > > No such file or directory # But ... let's see if the "tickle" the Projects > > directory a bit ... > > [root@cephmds603 ~]# ls -l > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > > > # Now all subdirectories and the file in question immediately becomes > > visible: > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > > ceph squid 19.2.3: > > [root@cephmds603 ~]# uname -a > > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > > x86_64 x86_64 GNU/Linux > > [root@cephmds603 ~]# ceph --version > > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > > (stable) > > [root@cephmds603 ~]# > > > > What is most interesting is that in the live tree (non-snapshotted tree) - > > the Projects directory exists, but everything under it has been removed! > > > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > > directory > > > > Am I running into some known issue? This looks to be specific to some > > clients - i.e. some clients see the directory and some don't at the same > > point in time, i.e. this is perhaps a kclient problem? > > Any ideas/suggestions would be appreciated, > > > > Andras > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: v21.1.0 Umbrella RC0 released
by Stefan Möding 22 Jul '26

22 Jul '26

Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> writes: > This is the first release candidate for Umbrella. > Ubuntu 22.04, 24.04, CentOS 9, Rocky Linux 10, Debian 12 (Bookworm) and > Debian 13 (Trixie) were built for this RC0. Are there plans to also provide packages for Ubuntu 26.04 (released in April)? We are currently running Tentacle on Ubuntu 24.04. The Ceph code depends on the Thrift c++ library which itself depends on multiple Qt 5 modules. In Ubuntu 24.04 the included Qt 5 modules have know security vulnerabilities and Canonical provides only fixes as part of an Expanded Security Contract. That makes it difficult to run Ceph in an environment where security compliance is a concern. -- Stefan

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Eugen Block 22 Jul '26

22 Jul '26

Hi, responses inline... Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Actually, what confused me was this: > > cephadm bootstrap --mon-ip <my-ip> > > Among other things, this command: > - Generates a new SSH key for the Ceph cluster and adds it > to the /root/.ssh/authorized_keys file. > - Saves a copy of the public key to the /etc/ceph/ceph.pub file. > > You then need to add a node. > > ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host> > > ceph orch host add <newhost> [<ip>] [<label1> ...] > > I wondered where the private part of the SSH key /etc/ceph/ceph.pub was ! That's stored in the config-key db: ceph config-key get mgr/cephadm/ssh_identity_key > > -- > M > > Michel Le Cocq via ceph-users <ceph-users(a)ceph.io> writes: > >> Hi again. >> >> I'm answering my own question. >> >> It seems I read the wrong version of the doc >> ... sic! >> >> It should be enough to set the “_admin” flag on all nodes: >> >> and that we also need to install cephadm on the nodes we >> want to use for administration, right!? Not necessarily, but it can be more convenient for debugging. The orchestrator stores a cephadm binary on each node to be able to manage them, it is stored in /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate the cluster/nodes etc. The original bootstrap node requires a cephadm binary either installed as a package or downloaded via curl as described in the docs to be able to bootstrap. After that you'll only need to execute cephadm commands locally on the nodes if you're debugging issues etc., otherwise the orchestrator will only use the file in /var/lib/ceph. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0