- ceph-users - lists.ceph.io

Dump/Add users yaml/json
by Albert Shih 07 Dec '24

07 Dec '24

Hi everyone, Stupid question....after some test I was able to dump a user caps with ceph auth get --format json but I'wasn't able to find the other way, something ceph auth add fubar.json Is they are any way to add a user (without given a key and with a key). Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 29 nov. 2024 18:12:35 CET

4 7

CephFS: Revert snapshot
by Andre Tann 05 Dec '24

05 Dec '24

Hi all, in CephFS, creating and removing a snapshot is as easy as - mkdir .snap/SNAPSHOT_NAME - rmdir .snap/SNAPSHOT_NAME Both steps are completed in no time, because no data has to be moved around. However, what I cannot find is a way to revert to a snapshot. The only way I see is to rsync … .snap/SNAPSHOT_NAME/ . In this case all the data has to be moved around, with two main disadvantages: - takes forever - consumes double the space than before Is there a better way to do this? Cheers -- Andre Tann

2 1

Weird pg degradation behavior
by Szabo, Istvan (Agoda) 04 Dec '24

04 Dec '24

Hi, I lost a node due to cpu failure so until hardware fix, I let it down as Ceph marked osds out, however I keep it in the tree because in 1-2 days will be online again. This is the current state: health: HEALTH_WARN Degraded data redundancy: 1726/23917349718 objects degraded(0.000%), 1 pg degraded, 1 pg undersized Out of the 9x osd nodes 2x of them have 4x nvme osd (temporarily) and 7x has only 2x (for index pool on 3 replicas) where this pg is located. (The down osd node is a 2x nvme osd one) This is some part of the query of that specific degraded pg: ... "up": [ 233, 202 ], "acting": [ 233, 202 ], "avail_no_missing": [ "233", "202" ], ... The above mentioned osds are on the 2x osd nodes with 4x nvme. What is not clear, I don't see any probing osd or any indicator for the 3rd missing part of that specific pg. This is the pgs_brief: PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 10.f6 active+undersized+degraded [233,202] 233 [233,202] 233 Shouldn't have any indicator what it is looking for? Which osd or some probing osd? (It's quincy 17.2.7). If I purge the down osds on the down server, I guess it would kick off recovery, I'm just curious why don't show anywhere what it is looking for? Ty

1 0

what's the minimum m to keep cluster functioning when 2 OSDs are down?
by Tony Liu 04 Dec '24

04 Dec '24

Hi, With Reef, I'd like to keep the cluster functioning while 2 OSDs are down. The cluster has more than 10 nodes, each of which has 64x 2.5G threads, 256GB memory, 24x NVMe drives and 2x 25GB networking. Will k=4 and m=2 achieve what I want? I know no data loss with m=2 when 2 OSDs are down, but is the cluster still functioning/writable in that case? Or m=3 is required to keep cluster functioning while 2 OSDs are down? In case m=3, what would be the reasonable k? Overhead 1.5 or 1.6 is efficient enough for me. So, k=5 or k=6? Any comments is welcome. Thanks! Tony

2 1

Mailing List Issues
by Kozakis, Anestis 04 Dec '24

04 Dec '24

Apologies for posting this here but was unsure where else I could dot his. I am being made redundant in my current position (Last day is this Friday). I have been able to successfully subscribe using my personal e-mail address to the ceph-announce list, but every time I try to subscribe to ceph-users from the web page, I keep getting an API/Mailman Core error. Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 1

Update host operating system - Ceph version 18.2.4 reef
by alessandro＠universonet.com.br 03 Dec '24

03 Dec '24

Ceph version 18.2.4 reef (cephadm) Hello, We have a cluster running with 6 Ubuntu 20.04 servers and we would like to add another host but with Ubuntu 22.04, will we have any problems? We would like to add new HOST with Ubuntu 22.04 and deactivate the Ubuntu 20.04 ones, our idea would be to update the hosts from Ubuntu 20.04 to 22.04. If anyone can give us some guidance, thank you very much.

1 0

lifecycle processing in multisite
by Christopher Durham 03 Dec '24

03 Dec '24

Hello, I have reef 18.2.4 on Rocky9. I have a multi-site environment with one zone on each side of the multi-site. Replication appears to be working in both directions. My question is, is lifecycle processing independent on each side? I have a lifecycle json installed to delete all objects in a given bucket and prefix after say, 180 days. (6 months). I have noticed that if I run: # radosgw-admin lc process --bucket <bucket> --rgw-debug 20 on the master zone, then the objects this process deletes do not go away on the other side. I expect that perhaps tonight, when lifecycle runs on the slave side, that these objects will disappear there, butwanted to confirm that lifecycle processing is independent on each zone. It appears that the s3api that puts the lifecycle json config ends up pushing it to each zone, so that the lifecycle is the same on all zones But the lifecycle cleanup may not actually execute at exactly the same time on all zones. Am I correct? If not, what should I be seeing? Thanks -Chris

1 0

down OSDs, Bluestore out of space, unable to restart
by John Jasen 02 Dec '24

02 Dec '24

Ceph version 17.2.6 After a power loss event affecting my ceph cluster, I've been putting humpty dumpty back together since. One problem I face is that with objects degraded, rebalancing doesn't run -- and this resulted in several of my fast OSDs filling up. I have 8 OSDs currently down, 100% full (exceeding all the full ratio settings on by default or I toggled to try and keep it together), and when I try to restart them, they fail out. Is there any way to bring these back from the dead? Here's some interesting output from journalctl -xeu on the failed OSD: ceph-osd[2383080]: bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) ceph-osd[2383080]: bluestore(/var/lib/ceph/osd/ceph-242) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... ceph-osd[2389725]: bluefs _allocate allocation failed, needed 0x3000 ceph-6ab85342-53d6-11ee-88a7-e43d1a153e91-osd-242[2389718]: -2> 2024-11-25T18:31:42.070+0000 7f0adfdef540 -1 bluefs _flush_range_F allocated: 0x0 offset: 0x0 length: 0x230f ceph-osd[2389725]: bluefs _flush_range_F allocated: 0x0 offset: 0x0 length: 0x230f Followed quickly by an abort: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: In funct> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: 3380: ce> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x559bf4361d2f] 2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x7a9) [0x559bf4b225f9] 3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) [0x559bf4b22812] 4: (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] 5: (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] 6: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] 7: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] 8: (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] 9: (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions const*, rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] 10: (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, std::allocator<rocksdb::VersionSet::ManifestWriter> >&, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> 11: (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> 12: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocksdb::ColumnFamilyOptions const> 13: (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] 14: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x13f1) [0x559bf50d3f21] 15: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::Colu> 16: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFa> 17: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x77a) [0x559bf503766a] 18: (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] 19: (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] 20: (BlueStore::_mount()+0x396) [0x559bf4a795d6] 21: (OSD::init()+0x556) [0x559bf44a0eb6] 22: main() 23: __libc_start_main() 24: _start() *** Caught signal (Aborted) ** in thread 7f0adfdef540 thread_name:ceph-osd ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0addff1cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x559bf4361def] 5: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x7a9) [0x559bf4b225f9] 6: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) [0x559bf4b22812] 7: (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] 8: (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] 9: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] 10: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] 11: (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] 12: (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions const*, rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] 13: (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, std::allocator<rocksdb::VersionSet::ManifestWriter> >&, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> 14: (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> 15: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocksdb::ColumnFamilyOptions const> 16: (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] 17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x13f1) [0x559bf50d3f21] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::Colu> 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFa> 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x77a) [0x559bf503766a] 21: (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] 22: (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] 23: (BlueStore::_mount()+0x396) [0x559bf4a795d6] 24: (OSD::init()+0x556) [0x559bf44a0eb6] 25: main() 26: __libc_start_main() 27: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

5 21

mount path missing for subvolume
by bruno.pessanha＠gmail.com 02 Dec '24

02 Dec '24

I'm using: ``` # ceph version ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) ``` I get the following error when trying to run: # ceph fs subvolume info volume01 vol01 group01 Error ENOENT: mount path missing for subvolume 'vol01' The volume exists and it's mounted on the clients: # ceph fs subvolume ls volume01 group01 [ { "name": "vol02" }, { "name": "vol01" } ] Would you know what could be happening here? Regards, Bruno

1 0

Ceph mirror failing on /archive/el6/x86_64/ceph-0.67.10-0.el6.x86_64.rpm
by Rouven Seifert 02 Dec '24

02 Dec '24

Hi there, we are operating a mirror of downloads.ceph.com that is used as package source for some internal hosts. Recently the rsync script began failing with the following error: rsync: send_files failed to open "/archive/el6/x86_64/ceph-0.67.10-0.el6.x86_64.rpm" (in ceph): Permission denied (13) Who would be the best to report this to? Best regards, Rouven -- Rouven Seifert Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) E-Mail: rouven.seifert(a)agdsn.de https://agdsn.de

1 0