- ceph-users - lists.ceph.io

doc: Is there a ceph-users list archive that archives before June 2019?
by Zac Dover 02 Jul '25

02 Jul '25

Here is the archive of the ceph-users mailing list that docs.ceph.com and ceph.io link to: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ This archive reaches back in time only to June of 2019. But I would like to refer to an email that was sent in 2017. Is anyone aware of an archive of ceph-users that reaches that far back in time? Zac

2 1

CephFS with Ldap
by gagan tiwari 01 Jul '25

01 Jul '25

Hi Guys, We have a Ldap server with all users login details. We have to mount data stored in Ceph to several client nodes via CephFS so that users can access that data and start using that data in their processes. But we need to grant permission / ownership to users to enable them to access that data. like chown user:group /dirs ( on Linux ) How will cephfs recognize users , groups that are in Ldap ? Will I need to set-up Ldap authentication on all nodes in Ceph cluster for this purpose ( ceph mgr , ceph mons , ceph mds and all ods nodes ) Please advise. Thanks, Gagan

5 4

orchestrator not behaving strangely
by Holger Naundorf 01 Jul '25

01 Jul '25

Hello, we are running a ceph cluster at version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) and since a few weeks the orchestrator started to misbehave - up to now we could not identify any root cause, so I am fishing in the community to see if there are any hints. Problems: An OSD removal (for disk replacement) gets stuck in the 'purge' step: ceph orch osd rm 406 --replace root@aadm01:~# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 406 acn07 done, waiting for purge 0 True False True 2025-06-25 09:18:07.650734+00:00 (now for more than 24h in this state) At the same time the orchestrator is not restarting OSD daemons - i.e. an 'ceph orch daemon restart osd.xxx' claims its queuing uo the restart, but it never happens. Other services continue to be controlled correctly via 'ceph orch ...' If anyone has an idea where to poke around or can match this to some known problem - I would appreciate any pointers. Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf(a)rz.uni-kiel.de

3 8

RADOS Gateway IAM for Production Environments ( reef to squid upgrade )
by ramodav＠yahoo.com 01 Jul '25

01 Jul '25

Hi Everyone ! I am planning to upgrade ceph from reef to squid to benefit from RGW IAM .Does anybody know is it still a tech-preview ?Would an update be a nice solution to have this feature ?

3 3

Cephfs performance on a small configuration
by Patrick Begou 01 Jul '25

01 Jul '25

Hi everybody, I've setup a small ceph cluster for a cephfs service. - 5 nodes in Almalinux9 and Squid version of Ceph - 4 HDD per node (so 20 HDD in the cluster) - Erasure coding in k=6+m=2 (ceph osd erasure-code-profile set ec-62-profile-isa k=6 m=2 crush-failure-domain=host plugin=isa) - two chunks per node (I add un "virtual" host bucket on each nodes and move 2 HDD to it) - only one network interface (ethernet 25Gb/s) I'm testing I/Os from another cluster (for computation) with a workflow similar to our needs: several mpi processes writing each in a different file. Node is connected to the same switch, 25Gb ethernet, same VLAN (no router). 1 process writing/reading 20GB shows (around) *700MB/s writing and 600MB/s reading* so far from the maximum network bandwidth 4 processes writing/reading each 5GB (so 20GB aggregated) show (around) *900MB/s *(225MB/s per process)*writing and 800MB/s reading* (200MB/s per process) Is this what I can expect from the actual setup or cluster architecture (I do not find any reference) or is something wrong in my first Ceph cluster setup. Of course the goal is to add HDD (for capacitive storage) and servers in the next years, this is the starting point of a Ceph deployment in the laboratory. I'm checking against the performance of my old NFS server (10Gb ethernet, two cluster of 8 HDDS in HW raid6, stripped) and the throughput improvement is small with my test. Thanks Patrick [ceph: root@whitaker01-ceph /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 254.66797 root default -11 25.46680 host whitaker01-ceph 4 hdd 12.73340 osd.4 up 1.00000 1.00000 13 hdd 12.73340 osd.13 up 1.00000 1.00000 -13 25.46680 host whitaker01-virt 7 hdd 12.73340 osd.7 up 1.00000 1.00000 18 hdd 12.73340 osd.18 up 1.00000 1.00000 -7 25.46680 host whitaker02-ceph 0 hdd 12.73340 osd.0 up 1.00000 1.00000 11 hdd 12.73340 osd.11 up 1.00000 1.00000 -15 25.46680 host whitaker02-virt 8 hdd 12.73340 osd.8 up 1.00000 1.00000 17 hdd 12.73340 osd.17 up 1.00000 1.00000 -5 25.46680 host whitaker03-ceph 2 hdd 12.73340 osd.2 up 1.00000 1.00000 12 hdd 12.73340 osd.12 up 1.00000 1.00000 -17 25.46680 host whitaker03-virt 5 hdd 12.73340 osd.5 up 1.00000 1.00000 15 hdd 12.73340 osd.15 up 1.00000 1.00000 -3 25.46680 host whitaker04-ceph 3 hdd 12.73340 osd.3 up 1.00000 1.00000 10 hdd 12.73340 osd.10 up 1.00000 1.00000 -19 25.46680 host whitaker04-virt 6 hdd 12.73340 osd.6 up 1.00000 1.00000 16 hdd 12.73340 osd.16 up 1.00000 1.00000 -9 25.46680 host whitaker05-ceph 1 hdd 12.73340 osd.1 up 1.00000 1.00000 14 hdd 12.73340 osd.14 up 1.00000 1.00000 -21 25.46680 host whitaker05-virt 9 hdd 12.73340 osd.9 up 1.00000 1.00000 19 hdd 12.73340 osd.19 up 1.00000 1.00000

1 0

Loki retention policy missing?
by Stefan Kooman 01 Jul '25

01 Jul '25

Hi, Yesterday we got a disk space warning notification on one of the monitor nodes. Most space consumption was in use by a running Loki container. Looking into online documentation / trackers to find info on Ceph's default Loki retention policy left me empty handed. I did fine some trackers for Prometheus retention policy in the past, so I figured there might be none yet. The documentation states that you can find the jinja2 templates used for the service here: src/pybind/mgr/cephadm/templates So I changed this template to include a retention policy and a compactor like: ... orig template ... compactor: working_directory: /tmp/loki/compactor shared_store: filesystem compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 limits_config: retention_period: 2160h split_queries_by_interval: 24h ceph config-key set mgr/cephadm/services/loki/loki.yml -i $PWD/loki_with_retention_policy.yml.j2 and redeployed loki (ceph orch daemon redeploy loki.mon1). That gained a lot of free space ... as in ... all Loki logs where gone and it had started from scratch. This "/tmp/loki" working directory is ... odd? How is this supposed to work in Ceph? Thanks, Gr. Stefan

1 0

Failure to build on a devuan.
by robin hammond 01 Jul '25

01 Jul '25

I just attempted to build the lasted tarbar on a freshly installed and updated devuan daedalus system. Linux devuan-ceph 6.1.0-37-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.140-1 (2025-05-22) x86_64 GNU/Linux I ran ./install-deps.sh as per https://docs.ceph.com/en/latest/install/build-ceph/ but ./do_cmake.sh resulted in an error. Has anyone any experience with this error ? root@devuan-ceph:/usr/local/src/ceph-19.2.2# cat /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeError.log Determining if the function memset_s exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-7e7NF3 Run Build Command(s):/usr/bin/ninja cmTC_3bf24 && [1/2] Building C object CMakeFiles/cmTC_3bf24.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_3bf24 FAILED: cmTC_3bf24 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=memset_s CMakeFiles/cmTC_3bf24.dir/CheckFunctionExists.c.o -o cmTC_3bf24 && : /usr/bin/ld: CMakeFiles/cmTC_3bf24.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `memset_s' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if the function pthread_set_name_np exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-swTR94 Run Build Command(s):/usr/bin/ninja cmTC_2fec2 && [1/2] Building C object CMakeFiles/cmTC_2fec2.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_2fec2 FAILED: cmTC_2fec2 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=pthread_set_name_np CMakeFiles/cmTC_2fec2.dir/CheckFunctionExists.c.o -o cmTC_2fec2 -lpthread && : /usr/bin/ld: CMakeFiles/cmTC_2fec2.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `pthread_set_name_np' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if the function pthread_get_name_np exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-6fkgqm Run Build Command(s):/usr/bin/ninja cmTC_13767 && [1/2] Building C object CMakeFiles/cmTC_13767.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_13767 FAILED: cmTC_13767 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=pthread_get_name_np CMakeFiles/cmTC_13767.dir/CheckFunctionExists.c.o -o cmTC_13767 -lpthread && : /usr/bin/ld: CMakeFiles/cmTC_13767.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `pthread_get_name_np' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if the function getprogname exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-CZhew8 Run Build Command(s):/usr/bin/ninja cmTC_6c988 && [1/2] Building C object CMakeFiles/cmTC_6c988.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_6c988 FAILED: cmTC_6c988 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=getprogname CMakeFiles/cmTC_6c988.dir/CheckFunctionExists.c.o -o cmTC_6c988 && : /usr/bin/ld: CMakeFiles/cmTC_6c988.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `getprogname' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if files valgrind/helgrind.h exist failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-gqBwr4 Run Build Command(s):/usr/bin/ninja cmTC_c5529 && [1/2] Building C object CMakeFiles/cmTC_c5529.dir/HAVE_VALGRIND_HELGRIND_H.c.o FAILED: CMakeFiles/cmTC_c5529.dir/HAVE_VALGRIND_HELGRIND_H.c.o /usr/bin/gcc-12 -o CMakeFiles/cmTC_c5529.dir/HAVE_VALGRIND_HELGRIND_H.c.o -c /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-gqBwr4/HAVE_VALGRIc /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-gqBwr4/HAVE_VALGRIND_HELGRIND_H.c:2:10: fatal error: valgrind/helgrind.h: No such file or directory 2 | #include <valgrind/helgrind.h> | ^~~~~~~~~~~~~~~~~~~~~ compilation terminated. ninja: build stopped: subcommand failed. Source: /* */ #include <valgrind/helgrind.h> int main(void){return 0;} Performing C SOURCE FILE Test HAVE_STAT_ST_MTIMESPEC_TV_NSEC failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv Run Build Command(s):/usr/bin/ninja cmTC_b96bc && [1/2] Building C object CMakeFiles/cmTC_b96bc.dir/src.c.o FAILED: CMakeFiles/cmTC_b96bc.dir/src.c.o /usr/bin/gcc-12 -DHAVE_STAT_ST_MTIMESPEC_TV_NSEC -o CMakeFiles/cmTC_b96bc.dir/src.c.o -c /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv/c /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv/src.c: In function ‘main’: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv/src.c:6:34: error: ‘struct stat’ has no member named ‘st_mtimespec’ 6 | (void)sizeof(((struct stat *)0)->st_mtimespec.tv_nsec); | ^~ ninja: build stopped: subcommand failed. Source file was: #include <sys/stat.h> int main() { (void)sizeof(((struct stat *)0)->st_mtimespec.tv_nsec); return 0; } Performing C++ SOURCE FILE Test HAVE_CXX11_ATOMIC failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-Qzk4En Run Build Command(s):/usr/bin/ninja cmTC_ba8d6 && [1/2] Building CXX object CMakeFiles/cmTC_ba8d6.dir/src.cxx.o [2/2] Linking CXX executable cmTC_ba8d6 FAILED: cmTC_ba8d6 : && /usr/bin/g++-12 -std=c++11 CMakeFiles/cmTC_ba8d6.dir/src.cxx.o -o cmTC_ba8d6 && : /usr/bin/ld: CMakeFiles/cmTC_ba8d6.dir/src.cxx.o: in function `std::atomic<tagged_ptr>::store(tagged_ptr, std::memory_order)': src.cxx:(.text._ZNSt6atomicI10tagged_ptrE5storeES0_St12memory_order[_ZNSt6atomicI10tagged_ptrE5storeES0_St12memory_order]+0x62): undefined reference to `__atomic_stor' /usr/bin/ld: CMakeFiles/cmTC_ba8d6.dir/src.cxx.o: in function `std::atomic<tagged_ptr>::load(std::memory_order) const': src.cxx:(.text._ZNKSt6atomicI10tagged_ptrE4loadESt12memory_order[_ZNKSt6atomicI10tagged_ptrE4loadESt12memory_order]+0x2d): undefined reference to `__atomic_load_16' /usr/bin/ld: CMakeFiles/cmTC_ba8d6.dir/src.cxx.o: in function `std::atomic<tagged_ptr>::compare_exchange_strong(tagged_ptr&, tagged_ptr, std::memory_order, std::memor: src.cxx:(.text._ZNSt6atomicI10tagged_ptrE23compare_exchange_strongERS0_S0_St12memory_orderS3_[_ZNSt6atomicI10tagged_ptrE23compare_exchange_strongERS0_S0_St12memory_or' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Source file was: #include <atomic> #include <cstdint> #include <cstddef> #if defined(__SIZEOF_INT128__) // Boost needs 16-byte atomics for tagged pointers. // These are implemented via inline instructions on the platform // if 16-byte alignment can be proven, and are delegated to libatomic // library routines otherwise. Whether or not alignment is provably // OK for a std::atomic unfortunately depends on compiler version and // optimization levels, and also on the details of the expression. // We specifically test access via an otherwise unknown pointer here // to ensure we get the most complex case. If this access can be // done without libatomic, then all accesses can be done. struct tagged_ptr { int* ptr; std::size_t tag; }; void atomic16(std::atomic<tagged_ptr> *ptr) __attribute__ ((used)); void atomic16(std::atomic<tagged_ptr> *ptr) { tagged_ptr p{nullptr, 1}; ptr->store(p); tagged_ptr f = ptr->load(); tagged_ptr new_tag{nullptr, 0}; ptr->compare_exchange_strong(f, new_tag); } #endif int main() { #if defined(__SIZEOF_INT128__) std::atomic<tagged_ptr> ptr; atomic16(&ptr); #endif std::atomic<uint8_t> w1; std::atomic<uint16_t> w2; std::atomic<uint32_t> w4; std::atomic<uint64_t> w8; return w1 + w2 + w4 + w8; }

2 1

Re: Suspiciously low PG count for CephFS with many small files
by Niklas Hambüchen 30 Jun '25

30 Jun '25

(Adding back the list) > https://docs.clyso.com/docs/kb/rados/#osd---improved-procedure-for-adding-h… <https://docs.clyso.com/docs/kb/rados/#osd---improved-procedure-for-adding-h…> Is the upstream source for `upmap-remapped.py` the `ceph-scripts` repo? If yes, is it known why there are 2 different places for `ceph-scripts` (Gitlab and Github), with diverged histories, both of which are recently edited, by the same individuals? https://github.com/cernceph/ceph-scripts/commits/master/tools/upmap/upmap-r… https://gitlab.cern.ch/ceph/ceph-scripts/-/commits/master/tools/upmap/upmap… > If I understand what you’re getting at, I don’t think that’s the case. I *think* recent releases give priority to undersized / degraded PGs over remapped PGs. It's not really `degraded` vs `misplaced` I'm getting at. But instead, consider this scneario: We have * osd.1 (very full, 100 GB free) * osd.2 (very full, 100 GB free) * osd.30 (empty) and we have * PGa on osd.1, which Ceph now wants to be on osd.2 * PGb on osd.2, which Ceph now wants to be on osd.30 The simple thing that could theoretically be done is that simultaneously be done is that the PGa (osd.1 -> osd.2) move, and the PGb (osd2. -> osd.30) move, are started immediately and run in parallel. But I suspect that what Ceph does is to make decisions on PG granularity: 1. See that there's no space on osd.2 for PGa, thus not start the move. 2. Move PGb from osd.2 to osd.30. Wait until that's done (takes long if PGs are large). 3. See that now there enough space on osd.2 for PGa, thus only now start the move. Is this really what happens? Thanks!

2 1

Latest stable version of Ceph
by gagan tiwari 30 Jun '25

30 Jun '25

Hi Guys, I am going to set-up a new Ceph Cluster on nodes with OS RHEL8/9. Please let me know which is the most stable version of ceph with all bug fixes that I should go for. Thanks, Gagan

3 4

Suspiciously low PG count for CephFS with many small files
by Niklas Hambüchen 30 Jun '25

30 Jun '25

I have 2 clusters; both have HDDs and SSDs. Reporting only the HDDs which have their own pools: "rep-cluster": hdd-pool 3-replication, 86 OSDs (16 TiB each), 1024 PGs, 78 %RAW USED, 100 M objects "ec-cluster": hdd-pool erasure k=4 m=2, 58 OSDs (16 TiB each), 256 PGs, 60 %RAW USED, 450 M objects Both are Ceph 18.2.1, Bluestore, and have the autoscaler enabled. As you can see, I have many small objects. My PGs-copies-per-OSD seem far off from the recommendation of 100 PGs per OSD (`mon_target_pg_per_osd`): rep-cluster: 35 PGs/OSD (= 1024*3/86) ec-cluster: 26 PGs/OSD (= 256*6/58) So I'm at least 3x-4x off. Why? Should the autoscaler not have increased the PGs here? `ceph osd pool autoscale-status`: rep-cluster: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK data 349.4T 3.0 1343T 0.7802 1.0 1024 on False ec-cluster: data_ec 347.3T 1.5 876.4T 0.5944 1.0 256 on False I believe that because of this I suffer some drawbacks: * On ec-cluster, a PG contains ~2 TiB and ~2 M objects, causing rebalances to happen in coarse, slow steps. Should I take some steps to force the autoscaler to increase PGs, and if yes, which approach would be best here? Thanks for your tips! Niklas

3 9