- ceph-users - lists.ceph.io

Upgrading Ceph Cluster OS
by Nima AbolhassanBeigi 14 May '24

14 May '24

Hi, We want to upgrade the OS version of our production ceph cluster by reinstalling the OS on the server. Is there any way to upgrade the OS without having to redeploy the OSDs? We looked at the contents of /var/lib/ceph/osd/ceph-*, the block directory is linked to a lvm device. If we reinstall the OS, will there be any problem with this linked directory? Best Regards, Nima

4 4

Determine client/inode/dnode source of massive explosion in CephFS metadata pool usage (Red Hat Nautilus CephFS)
by Paul Browne 14 May '24

14 May '24

Hello Ceph users, We've recently seen a very massive uptick in the stored capacity of our CephFS metadata pool, 150X the raw stored capacity used in a very short timeframe of only 48 hours or so. The number of stored objects rose by ~1.5 million or so in that timeframe (attached PNG shows the increase) What I'd really like to be able to determine, but haven't yet figured out how, is to map these newly stored objects (over this limited time window) to inodes/dnodes in the filesystem and from there to individual namespaces being used in the filesystem. This should then allow me to track back the increased usage to specific projects using the filesystem for research data storage and give them a mild warning about possibly exhausting the available metadata pool capacity. Would anyone know if there's any capability in CephFs to do something like this, specifically in Nautilus (being run here as Red Hat Ceph Storage 4)? We've scheduled upgrades to later RHCS releases, but I'd like the cluster and CephFS state to be in a better place first if possible. Thanks, Paul Browne [cid:e87ad248-6621-4e9d-948b-da4428f8dbb8] ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 4

label or pseudo name for cephfs volume path
by Adiga, Anantha 14 May '24

14 May '24

Hi, Under the circumstance that a ceph fs subvolume has to be recreated , the uuid will change and we have to change all sources that reference the volume path. Is there a way to provide a label /tag to the volume path that can be used for pv_root_path so that we do not have to change any references OR any suggestion to address this need ? Spec containing subvolume path: k8s_rook_pv: - name: inventory-pv subvolumegroup: cephfs_data_pool_ec21_subvolumegroup subvolume: cluster_inventory_subvolume data_pool: cephfs_data_pool_ec21 size: 10Gi pvc_name: prometheus-inventory-pvc pv_root_path: /volumes/cephfs_data_pool_ec21_subvolumegroup/cluster_inventory_subvolume/ecfdf977-1fdb-4474-b8dd-c3bedb42620e mode: ReadWriteMany Thank you, Anantha

2 1

Ceph User + Community Meeting and Survey [May 23]
by Noah Lehman 13 May '24

13 May '24

Hi Ceph users and devs, We've announced the next community meeting and survey on Ceph social media and would appreciate help promoting. The next meeting will be Thursday, May 23rd at 10PM EDT. Details can be found here <https://www.meetup.com/ceph-user-group/events/300883526/?utm_campaign=share…> . Ceph twitter <https://twitter.com/Ceph/status/1788243792589427010>/xCeph linkedin <https://www.linkedin.com/feed/update/urn:li:share:7194009505439883264> *Language:* Our next Ceph User + Developer Monthly Meeting is coming May 23! The goal of these meetings is to elicit feedback from the users, companies, and organizations who use #Ceph in their production environments. Take the survey and share your voice: https://t.co/xGAN5FlGDu Thanks! Noah

2 2

Multisite: metadata behind on shards
by Szabo, Istvan (Agoda) 13 May '24

13 May '24

Hi, Wonder what is the mechanism behind the sync mechanism because I need to restart all the gateways every 2 days on the remote sites to keep those it in sync. (Octopus 15.2.7) Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: SPDK with cephadm and reef
by xiaowenhao111 12 May '24

12 May '24

1 0

Ceph reef and (slow) backfilling - how to speed it up
by Götz Reinicke 12 May '24

12 May '24

Dear ceph community, I’ve a ceph cluster which got upgraded from nautilus/pacific/…to reef over time. Now I added two new nodes to an existing EC pool as I did with the previous versions of ceph. Now I face the fact, that the previous „backfilling tuning“ I’v used by increasing injectargs --osd-max-backfills=XX --osd-recovery-max-active=YY dose not work anymore. With adjusting thous parameters the backfill was running with up to 2k +- objects/s. As I’m not (yet) familiar with the reef opiont the only speed up so far I found is „ceph config set osd osd_mclock_profile high_recovery_ops“ which currently runs the backfill with up to 600 opbjects/s. My question: What is a best (simple) way to speed that backfill up ? I’v tried to understand the custom profiles (?) but without success - and did not apply anything other yet. Thanks for feedback and suggestions ! Best regards . Götz

7 10

Reef: Dashboard: Object Gateway Graphs have no Data
by Dave Hall 10 May '24

10 May '24

Hello. We're running a containerized deployment of Reef with a focus on RGW. We noticed that while the Grafana graphs for other categories - OSDs, Pools, etc - have data, the graphs for the Object Gateway category are empty. I did some looking last week and found reference to something about an unintended limit on the number of graphs that could be rendered. However, I did not find reference to a fix. Is it a setting that needs to be changed, or is it a piece of code that is pending for the next fix release? A second - possibly related question: There is an informational message on the Object Gateway panel about how the default realm is not set. We're not using multi-site sync, so we haven't set the realm, but we're wondering if we should set a default realm just to keep the code happy. We're also wondering if this is related in any way to the empty graphs described above. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

2 1

MDS crash in interval_set: FAILED ceph_assert(p->first <= start)
by Dejan Lesjak 10 May '24

10 May '24

Hello, We have cephfs with two active MDS. Currently rank 1 is repeatedly crashing with FAILED ceph_assert(p->first <= start) in md_log_replay thread. Is there any way to work around this and get to accesible file system or should we start with disaster recovery? It seems similar to https://tracker.ceph.com/issues/61009 Crash info: { "assert_condition": "p->first <= start", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h", "assert_func": "void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]", "assert_line": 568, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fcdaaf8a640 time 2024-05-08T00:26:22.049974+0200\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start)\n", "assert_thread_name": "md_log_replay", "backtrace": [ "/lib64/libc.so.6(+0x54db0) [0x7fcdb7a54db0]", "/lib64/libc.so.6(+0xa154c) [0x7fcdb7aa154c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcdb83610ff]", "/usr/lib64/ceph/libceph-common.so.2(+0x161263) [0x7fcdb8361263]", "/usr/bin/ceph-mds(+0x1f3b0e) [0x55a5904a9b0e]", "/usr/bin/ceph-mds(+0x1f3b55) [0x55a5904a9b55]", "(EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4b9d) [0x55a5906e1c8d]", "(EUpdate::replay(MDSRank*)+0x5d) [0x55a5906eacbd]", "(MDLog::_replay_thread()+0x7a1) [0x55a590694af1]", "/usr/bin/ceph-mds(+0x1460f1) [0x55a5903fc0f1]", "/lib64/libc.so.6(+0x9f802) [0x7fcdb7a9f802]", "/lib64/libc.so.6(+0x3f450) [0x7fcdb7a3f450]" ], "ceph_version": "18.2.2", "crash_id": "2024-05-07T22:26:22.050652Z_8be89ffb-bb87-4832-9339-57f8bd29f766", "entity_name": "mds.spod19", "os_id": "almalinux", "os_name": "AlmaLinux", "os_version": "9.3 (Shamrock Pampas Cat)", "os_version_id": "9.3", "process_name": "ceph-mds", "stack_sig": "3d0a2ca9b3c7678bf69efc20fff42b588c63f8be1832e1e0c28c99bafc082c15", "timestamp": "2024-05-07T22:26:22.050652Z", "utsname_hostname": "spod19.ijs.si", "utsname_machine": "x86_64", "utsname_release": "5.14.0-362.8.1.el9_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Tue Nov 7 14:54:22 EST 2023" } Cheers, Dejan

3 5

Forcing Posix Permissions On New CephFS Files
by duluxoz 09 May '24

09 May '24

Hi All, I've gone and gotten myself into a "can't see the forest for the trees" state, so I'm hoping someone can take pity on me and answer a really dumb Q. So I've got a CephFS system happily bubbling along and a bunch of (linux) workstations connected to a number of common shares/folders. To take a single one of these folders as an example ("music") the sub-folders and files of that share all belong to root:music with permissions of 2770 (folders) and 0660 (files). The "music" folder is then connected to (as per the Ceph Doco: mount.ceph) via each workstation's fstab file - all good, all working, everyone's happy. What I'm trying to achieve is that when a new piece of music (a file) is uploaded to the Ceph Cluster the file inherits the music share's default ownership (root:music) and permissions (0660). What is happening at the moment is I'm getting permissions of 644 (and 755 for new folders). I've been looking for a way to do what I want but, as I said, I've gone and gotten myself thoroughly mixed-up. Could someone please point me in the right direction on how to achieve what I'm after - thanks Cheers Dulux-Oz

3 2