- ceph-users - lists.ceph.io

set proxy for ceph installation
by Majid Varzideh 27 Sep '23

27 Sep '23

hi friends i have deployed my first node in cluster. we dont have direct internet on my server so i have to set proxy for that.i set it /etc/environment /etc/profile but i get bellow error 2023-09-26 17:09:38,254 7f04058b4b80 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph:v17', 'pull'] 2023-09-26 17:09:38,302 7f04058b4b80 INFO Pulling container image quay.io/ceph/ceph:v17... 2023-09-26 17:09:42,083 7f04058b4b80 INFO Non-zero exit code 125 from /usr/bin/podman pull quay.io/ceph/ceph:v17 2023-09-26 17:09:42,083 7f04058b4b80 INFO /usr/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v17... 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr time="2023-09-26T17:09:38+03:30" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 34.228.154.221:443: connect: connection refused" 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr time="2023-09-26T17:09:39+03:30" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.220.246.53:443: connect: connection refused" 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr time="2023-09-26T17:09:40+03:30" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 18.213.60.205:443: connect: connection refused" 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 34.231.182.47:443: connect: connection refused 2023-09-26 17:09:42,084 7f04058b4b80 ERROR ERROR: Failed command: /usr/bin/podman pull quay.io/ceph/ceph:v17 would you please help. thanks,

3 2

CEPH zero iops after upgrade to Reef and manual read balancer
by Mosharaf Hossain 26 Sep '23

26 Sep '23

Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance. This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage." Kindly guide us to move forward. Regards Mosharaf Hossain

3 8

replacing storage server host (not drives)
by Wyll Ingersoll 26 Sep '23

26 Sep '23

We have a storage node that is failing, but the disks themselves are not. What is the recommended procedure for replacing the host itself without destroying the OSDs or losing data? This cluster is running ceph 16.2.11 using ceph orchestrator with docker containers on Ubuntu 20.04 (focal). thank you, Wyllys Ingersoll

2 1

pgs incossistent every day same osd
by Jorge JP 26 Sep '23

26 Sep '23

Hello, First, sorry for my english... Since a few weeks, I receive every day notifies with HEALTH ERR in my ceph. The notifies are related to inconssistent pgs and ever are on same osd. I ran smartctl test to the disk osd assigned and the result is "passed". Should replace the disk by other new? Regards!

2 2

rbd rados cephfs libs compilation
by Arnaud Morin 26 Sep '23

26 Sep '23

Hey ceph users! I'd like to compile the different lib-xyz from ceph (rados, rbd, cephfs) with ENABLE_SHARED=OFF (I want static libraries). Since few days I am struggling building the whole ceph repo on debian 12. Is there any way to build only the libraries? I dont need ceph, but only the client side libraries. Regards, Arnaud.

1 0

Re: Libcephfs : ceph_readdirplus_r() with ceph_ll_lookup_vino() : ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
by Joseph Fernandes 26 Sep '23

26 Sep '23

CC ceph-user (Apologies, Forgot last time I replied) On Tue, Sep 26, 2023 at 1:11 PM Joseph Fernandes <josephaug26(a)gmail.com> wrote: > Hello Venky, > > Did you get a chance to look into the updated program? Am I > missing something? > I suppose it's something trivial I am missing, As I see these API used in > NFS Ganesha also and would have been tested there. > > https://github.com/nfs-ganesha/nfs-ganesha/blob/2a57b6d53295426247b200cd100… > > Thanks, > ~Joe > > On Fri, Sep 22, 2023 at 10:52 PM Joseph Fernandes <josephaug26(a)gmail.com> > wrote: > >> Hello Venky, >> >> Nice to hear from you :) Hope you are doing well. >> >> I tried as you suggested, >> >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# mkdir >> dir1 dir2 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# echo >> "Hello Worldls!" > file2 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# echo >> "Hello Worldls!" > file1 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ls >> dir1 dir2 file1 file2 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# >> >> Create a new snapshot called "sofs-4-6" >> >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ceph fs >> subvolume snapshot ls cephfs 4 >> [ >> { >> "name": "sofs-4-1" >> }, >> { >> "name": "sofs-4-2" >> }, >> { >> "name": "sofs-4-3" >> }, >> { >> "name": "sofs-4-4" >> }, >> { >> "name": "sofs-4-5" >> }, >> { >> "name": "sofs-4-6" >> } >> ] >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# >> >> And Delete the file and directory from the active filessystem >> >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# rm -rf * >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ls >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# cd .snap >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap# ls >> _sofs-4-6_1099511627778 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap# cd >> _sofs-4-6_1099511627778/ >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778# >> ls >> dir1 dir2 file1 file2 >> root@ss-joe-01 >> (bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778# >> >> Now I modified the program and executed it, I am getting the same result. >> >> root@ss-joe-01(bash):/home/hydrauser# ./snapshot_inode_lookup >> >> =================================/volumes/_nogroup/4/user_root/===================================== >> >> Path/Name :"/volumes/_nogroup/4/user_root/" >> Inode Address : 0x7f7c10008f00 >> Inode Number : 1099511628293 >> Snapshot Number : 18446744073709551614 >> Inode Number : 1099511628293 >> Snapshot Number : 18446744073709551614 >> . Ino: 1099511628293 SnapId: 18446744073709551614 Address: 0x7f7c10008f00 >> .. Ino: 1099511627778 SnapId: 18446744073709551614 Address: 0x7f7c100086f0 >> >> >> =================================1099511628293:7===================================== >> >> Path/Name :"1099511628293:7" >> Inode Address : 0x7f7c10009710 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> .. Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> >> >> =================================/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778===================================== >> >> Path/Name >> :"/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778" >> Inode Address : 0x7f7c10009710 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> .. Ino: 1099511628293 SnapId: 18446744073709551615 Address: 0x55efc15b4640 >> file1 Ino: 1099511628297 SnapId: 7 Address: 0x7f7c1000a030 >> dir1 Ino: 1099511628294 SnapId: 7 Address: 0x7f7c1000a720 >> dir2 Ino: 1099511628295 SnapId: 7 Address: 0x7f7c1000ada0 >> file2 Ino: 1099511628296 SnapId: 7 Address: 0x7f7c1000b420 >> >> >> =================================1099511628293:7===================================== >> >> Path/Name :"1099511628293:7" >> Inode Address : 0x7f7c10009710 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> .. Ino: 1099511628293 SnapId: 18446744073709551615 Address: 0x55efc15b4640 >> file1 Ino: 1099511628297 SnapId: 7 Address: 0x7f7c1000a030 >> dir1 Ino: 1099511628294 SnapId: 7 Address: 0x7f7c1000a720 >> dir2 Ino: 1099511628295 SnapId: 7 Address: 0x7f7c1000ada0 >> file2 Ino: 1099511628296 SnapId: 7 Address: 0x7f7c1000b420 >> >> root@ss-joe-01(bash):/home/hydrauser# >> >> I have attached the modified program and ceph client logs from this run. >> >> Cheers, >> Joe >> >> On Fri, Sep 22, 2023 at 8:54 PM Venky Shankar <vshankar(a)redhat.com> >> wrote: >> >>> Hi Joseph, >>> >>> On Fri, Sep 22, 2023 at 5:27 PM Joseph Fernandes <josephaug26(a)gmail.com> >>> wrote: >>> > >>> > Hello All, >>> > >>> > I found a weird issue with ceph_readdirplus_r() when used along >>> > with ceph_ll_lookup_vino(). >>> > On ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) >>> quincy >>> > (stable) >>> > >>> > Any help is really appreciated. >>> > >>> > Thanks in advance, >>> > -Joe >>> > >>> > Test Scenario : >>> > >>> > A. Create a Ceph Fs Subvolume "4" and created a directory in root of >>> > subvolume "user_root" >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > ceph fs subvolume ls cephfs >>> > [ >>> > { >>> > "name": "4" >>> > } >>> > ] >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > ls -l >>> > total 0 >>> > drwxrwxrwx 2 root root 0 Sep 22 09:16 user_root >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > >>> > >>> > B. In the "user_root" directory create some files and directories >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > mkdir dir1 dir2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > ls >>> > dir1 dir2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > echo >>> > "Hello Worldls!" > file1 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > echo "Hello Worldls!" > file2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > ls >>> > dir1 dir2 file1 file2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > cat file* >>> > Hello Worldls! >>> > Hello Worldls! >>> > >>> > >>> > C. Create a subvolume snapshot "sofs-4-5". Please ignore the older >>> > snapshots. >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > ceph fs subvolume snapshot ls cephfs 4 >>> > [ >>> > { >>> > "name": "sofs-4-1" >>> > }, >>> > { >>> > "name": "sofs-4-2" >>> > }, >>> > { >>> > "name": "sofs-4-3" >>> > }, >>> > { >>> > "name": "sofs-4-4" >>> > }, >>> > { >>> > "name": "sofs-4-5" >>> > } >>> > ] >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > >>> > Here "sofs-4-5" has snapshot id 6. >>> > Got this from libcephfs and have verified at Line >>> > snapshot_inode_lookup.cpp#L212. (Attached to the email) >>> > >>> > >>> > #Content within the snapshot >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > cd .snap/ >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap# >>> > ls >>> > _sofs-4-1_1099511627778 _sofs-4-2_1099511627778 >>> _sofs-4-3_1099511627778 >>> > _sofs-4-4_1099511627778 _sofs-4-5_1099511627778 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap# >>> > cd _sofs-4-5_1099511627778/ >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778# >>> > ls >>> > user_root >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778# >>> > cd user_root/ >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# >>> > ls >>> > dir1 dir2 file1 file2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# >>> > cat file* >>> > Hello Worldls! >>> > Hello Worldls! >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# >>> > >>> > >>> > D. Delete all the files and directories in "user_root" >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > rm -rf * >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > ls >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > >>> > >>> > E. Using Libcephfs in a C++ program do the following,(Attached to this >>> > email) >>> > >>> > 1. Get the Inode of "user_root" using ceph_ll_walk(). >>> > 2. Open the directory using Inode received from ceph_ll_walk() >>> and do >>> > ceph_readdirplus_r() >>> > We don't see any dentries(except "." and "..") as we have deleted >>> all >>> > files and directories in the active filesystem. This is expected and >>> > correct! >>> > >>> > >>> > >>> > >>> =================================/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/===================================== >>> > >>> > Path/Name >>> > >>> :"/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/" >>> > Inode Address : 0x7f5ce0009900 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 18446744073709551614 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 18446744073709551614 >>> > . Ino: 1099511629282 SnapId: 18446744073709551614 Address: >>> 0x7f5ce0009900 >>> > .. Ino: 1099511627779 SnapId: 18446744073709551614 Address: >>> > 0x7f5ce00090f0 >>> > >>> > >>> > 3. Using ceph_ll_lookup_vino() get the Inode * of "user_root" for >>> > snapshot 6, Here "sofs-4-5" has snapshot id 6. >>> >>> Try this for path "/volumes/_nogroup/4/" (snapshots live a level-up in >>> the subvolume path) for the ceph version you use. >>> >>> > Got this from libcephfs and have verified at Line >>> > snapshot_inode_lookup.cpp#L212. (Attached to the email >>> > 4. Open the directory using Inode * received from >>> ceph_ll_lookup_vino() >>> > and do ceph_readdirplus_r() >>> > We don't see any dentries (except "." and "..") This is NOT >>> expected and >>> > NOT correct, as there are files and directories in the snaphot 6. >>> > >>> > >>> > >>> =================================1099511629282:6===================================== >>> > >>> > Path/Name :"1099511629282:6" >>> > Inode Address : 0x7f5ce000a110 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > .. Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > >>> > >>> > 5. Get the Inode of "user_root/ .snap/_sofs-4-5_1099511627778 / " >>> using >>> > ceph_ll_walk(). >>> > 6. Open the directory using Inode received from ceph_ll_walk() >>> and do >>> > ceph_readdirplus_r() >>> > We see ALL dentries of all files and directories in the snapshot. >>> This >>> > is expected and correct! >>> > >>> > >>> > >>> =================================/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/.snap/_sofs-4-5_1099511627778/===================================== >>> > >>> > Path/Name >>> > >>> :"/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/.snap/_sofs-4-5_1099511627778/" >>> > Inode Address : 0x7f5ce000a110 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > .. Ino: 1099511629282 SnapId: 18446744073709551615 Address: >>> > 0x5630ab946340 >>> > file1 Ino: 1099511628291 SnapId: 6 Address: 0x7f5ce000aa90 >>> > dir1 Ino: 1099511628289 SnapId: 6 Address: 0x7f5ce000b180 >>> > dir2 Ino: 1099511628290 SnapId: 6 Address: 0x7f5ce000b800 >>> > file2 Ino: 1099511628292 SnapId: 6 Address: 0x7f5ce000be80 >>> > >>> > 7. Now Again using ceph_ll_lookup_vino() get the Inode * of >>> "user_root" >>> > for snapshot 6, Here "sofs-4-5" has snapshot id 6. >>> > 8. Open the directory using Inode * received from >>> > ceph_ll_lookup_vino() and do ceph_readdirplus_r() >>> > Now! we see all the files and Directories in the snapshot! >>> > >>> > >>> > >>> =================================1099511629282:6===================================== >>> > >>> > Path/Name :"1099511629282:6" >>> > Inode Address : 0x7f5ce000a110 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > .. Ino: 1099511629282 SnapId: 18446744073709551615 Address: >>> > 0x5630ab946340 >>> > file1 Ino: 1099511628291 SnapId: 6 Address: 0x7f5ce000aa90 >>> > dir1 Ino: 1099511628289 SnapId: 6 Address: 0x7f5ce000b180 >>> > dir2 Ino: 1099511628290 SnapId: 6 Address: 0x7f5ce000b800 >>> > file2 Ino: 1099511628292 SnapId: 6 Address: 0x7f5ce000be80 >>> > >>> > >>> > Am I missing something using these APIs? >>> > >>> > File attached to this email >>> > Full out of the program attached to the email. >>> > - snapshot_inode_lookup.cpp_output.txt <Attached> >>> > C++ Program - snapshot_inode_lookup.cpp <Attached> >>> > /etc/ceph/ceph.conf - <attached> >>> > Ceph Client Log during the run of this C++ program - >>> client.log<attached> >>> > >>> > Compile Command: >>> > g++ -o snapshot_inode_lookup ./snapshot_inode_lookup.cpp -g -ldl -ldw >>> > -lcephfs -lboost_filesystem --std=c++17 >>> > >>> > Linux Details, >>> > root@ss-joe-01(bash):/home/hydrauser# uname -a >>> > Linux ss-joe-01 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) >>> > x86_64 GNU/Linux >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > >>> > Ceph Details, >>> > >>> > root@ss-joe-01(bash):/home/hydrauser# ceph -v >>> > ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy >>> > (stable) >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > root@ss-joe-01(bash):/home/hydrauser# ceph -s >>> > cluster: >>> > id: fb43d857-d165-4189-87fc-cf1debce9170 >>> > health: HEALTH_OK >>> > >>> > services: >>> > mon: 3 daemons, quorum ss-joe-01,ss-joe-02,ss-joe-03 (age 4d) >>> > mgr: ss-joe-01(active, since 4d), standbys: ss-joe-03, ss-joe-02 >>> > mds: 1/1 daemons up >>> > osd: 3 osds: 3 up (since 4d), 3 in (since 4d) >>> > >>> > data: >>> > volumes: 1/1 healthy >>> > pools: 3 pools, 49 pgs >>> > objects: 39 objects, 1.0 MiB >>> > usage: 96 MiB used, 30 GiB / 30 GiB avail >>> > pgs: 49 active+clean >>> > >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > root@ss-joe-01(bash):/home/hydrauser# dpkg -l | grep ceph >>> > ii ceph 17.2.5-1~bpo11+1 >>> > amd64 distributed storage and file system >>> > ii ceph-base 17.2.5-1~bpo11+1 >>> > amd64 common ceph daemon libraries and management >>> tools >>> > ii ceph-base-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-base >>> > ii ceph-common 17.2.5-1~bpo11+1 >>> > amd64 common utilities to mount and interact with a >>> ceph >>> > storage cluster >>> > ii ceph-common-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-common >>> > ii ceph-fuse 17.2.5-1~bpo11+1 >>> > amd64 FUSE-based client for the Ceph distributed file >>> > system >>> > ii ceph-fuse-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-fuse >>> > ii ceph-mds 17.2.5-1~bpo11+1 >>> > amd64 metadata server for the ceph distributed file >>> system >>> > ii ceph-mds-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-mds >>> > ii ceph-mgr 17.2.5-1~bpo11+1 >>> > amd64 manager for the ceph distributed storage system >>> > ii ceph-mgr-cephadm 17.2.5-1~bpo11+1 >>> > all cephadm orchestrator module for ceph-mgr >>> > ii ceph-mgr-dashboard 17.2.5-1~bpo11+1 >>> > all dashboard module for ceph-mgr >>> > ii ceph-mgr-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-mgr >>> > ii ceph-mgr-diskprediction-local 17.2.5-1~bpo11+1 >>> > all diskprediction-local module for ceph-mgr >>> > ii ceph-mgr-k8sevents 17.2.5-1~bpo11+1 >>> > all kubernetes events module for ceph-mgr >>> > ii ceph-mgr-modules-core 17.2.5-1~bpo11+1 >>> > all ceph manager modules which are always enabled >>> > ii ceph-mon 17.2.5-1~bpo11+1 >>> > amd64 monitor server for the ceph storage system >>> > ii ceph-mon-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-mon >>> > ii ceph-osd 17.2.5-1~bpo11+1 >>> > amd64 OSD server for the ceph storage system >>> > ii ceph-osd-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-osd >>> > ii ceph-volume 17.2.5-1~bpo11+1 >>> > all tool to facilidate OSD deployment >>> > ii cephadm 17.2.5-1~bpo11+1 >>> > amd64 cephadm utility to bootstrap ceph daemons with >>> > systemd and containers >>> > ii libcephfs2 17.2.5-1~bpo11+1 >>> > amd64 Ceph distributed file system client library >>> > ii libcephfs2-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for libcephfs2 >>> > ii libsqlite3-mod-ceph 17.2.5-1~bpo11+1 >>> > amd64 SQLite3 VFS for Ceph >>> > ii libsqlite3-mod-ceph-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for libsqlite3-mod-ceph >>> > ii python3-ceph-argparse 17.2.5-1~bpo11+1 >>> > all Python 3 utility libraries for Ceph CLI >>> > ii python3-ceph-common 17.2.5-1~bpo11+1 >>> > all Python 3 utility libraries for Ceph >>> > ii python3-cephfs 17.2.5-1~bpo11+1 >>> > amd64 Python 3 libraries for the Ceph libcephfs >>> library >>> > ii python3-cephfs-dbg 17.2.5-1~bpo11+1 >>> > amd64 Python 3 libraries for the Ceph libcephfs >>> library >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users(a)ceph.io >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> >>> -- >>> Cheers, >>> Venky >>> >>>

1 0

CephFS warning: clients laggy due to laggy OSDs
by Janek Bevendorff 26 Sep '23

26 Sep '23

Hey all, Since the upgrade to Ceph 16.2.14, I keep seeing the following warning: 10 client(s) laggy due to laggy OSDs ceph health detail shows it as: [WRN] MDS_CLIENTS_LAGGY: 10 client(s) laggy due to laggy OSDs mds.***(mds.3): Client *** is laggy; not evicted because some OSD(s) is/are laggy more of this... When I restart the client(s) or the affected MDS daemons, the message goes away and then comes back after a while. ceph osd perf does not list any laggy OSDs (a few with 10-60ms ping, but overwhelmingly < 1ms), so I'm on a total loss what this even means. I have never seen this message before nor was I able to find anything about it. Do you have any idea what this message actually means and how I can get rid of it? Thanks Janek

4 9

September Ceph Science Virtual User Group
by Kevin Hrpcek 25 Sep '23

25 Sep '23

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday September 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. NOTE: The change to using Jitsi for the meeting. We are no longer using the bluejeans meeting links. The ceph calendar event does not yet reflect this and has the wrong day as well. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20230927 Virtual event details: September 27th, 2023 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

1 0

rgw: strong consistency for (bucket) policy settings?
by Matthias Ferdinand 25 Sep '23

25 Sep '23

Hi, while I don't currently use rgw, I still am curious about consistency guarantees. Usually, S3 has strong read-after-write consistency guarantees (for requests that do not overlap). According to https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/ in Ceph this is also true for per-object ACLs. Is there also a strong consistency guarantee for (bucket) policies? The documentation at https://docs.ceph.com/en/latest/radosgw/bucketpolicy/ apparently does not say anything about this. How would multiple rgw instances synchronize a policy change? Is this effective immediate with strong consistency or is there some propagation delay (hopefully on with some upper bound)? Best regards Matthias

3 6

In which cases can the "mon_osd_full_ratio" and the "mon_osd_backfillfull_ratio" be exceeded ?
by Raphael Laguerre 25 Sep '23

25 Sep '23

Hello, In which cases can the "mon_osd_full_ratio" and the "mon_osd_backfillfull_ratio" be exceeded ? More specifically, in case a subset of OSDs fail, if there isn't any more space left in the remaining OSDs to migrate the PGs of the failed OSDs without exceeding either the "mon_osd_full_ratio" or the "mon_osd_backfillfull_ratio of at least one OSD, how will the cluster behave ? Will it exceed the "mon_osd_full_ratio" or the "mon_osd_backfillfull_ratio" of one or more OSDs to ensure an "active+clean" state ? Best regards, Raphaël

1 0