- ceph-users - lists.ceph.io

Upgrade of OS and ceph during recovery
by Rok Jaklič 27 Nov '24

27 Nov '24

Hi, right now the cluster is doing recovery for the last two weeks and it seems it will be doing so for the next week or so also. Meanwhile a new quincy update came, which fixes some of the things for us but we would need to upgrade to AlmaLinux 9. Has anyone done maintainace or upgrade of nodes during recovery? Kind regards, Rok

1 0

CephFS empty files in a Frankenstein system
by Linas Vepstas 27 Nov '24

27 Nov '24

Don't laugh. I am experimenting with Ceph in an enthusiast, small-office, home-office setting. Yes, this is not the conventional use case, but I think Ceph almost is, almost could be used for this. Do I need to explain why? These kinds of people (i.e. me) already run RAID. And maybe CIFS/Samba or NFS. On half-a-dozen or more machines. And RAID is OK, I guess, as long as your mirrors are in sync. And if you have three disks in a mirror, do they vote as to who has the best data? Not mdraid, no. And fsck takes hours on a multi-terabyte system with old disks. There's no CRC on data, just metadata. Maybe there's silent corruption in the background. Who knows. And being admin for CIFS/NFS in a small network is not exactly a fulfilling experience. So, hey, you know what? Perhaps CephFS can replace these. Perhaps CephFS is better. Perhaps CephFS is ready for this. I mean, I like ext4fs, but only because btrfs hates me. I'm loyal. But I'm here now. Seriously, I think that, with just a little bit of polishing and automation, Ceph could be deployed in the small-office/home-office setting. Don't laugh. This could happen. Sadly, I am seeing a tiny bit of data corruption. I said don't laugh. I'm resource constrained, server and disk constrained, and time-unconstrained. So I thought I'd bootstrap up from nothing. Manually.. First, one server, with two OSD's from two partitions on two different disks. Works great! For the record, this should be no worse than RAID mirroring on one host, and so is 100% entirely acceptable in the home/small-office environment. I move (not copy, but move) a few 100 GB of data I can afford to lose. Of course there are "warning: degraded+undersize" warnings, but hardly a surprise. I add a second server, and a 3rd OSD. The initial system is running Ceph "reef", because it's running Debian testing which has reef in it. The second system has ceph "pacific", because it's Debian stable, and that's what Debian stable currently packages. (Yes, pacific is past EOL. But it's on Debian stable, so....) Next version of Debian stable won't show up till next summer. I thought reef and pacific might not talk to one-another, but they do, and happily so. Excellent! +1 ceph! The system starts rebalancing onto the third OSD. After running all night, it gets to 2/3rds or 3/4 clean, and maybe 1/4th degraded-undersize. Surprising at first, because I had size/min_size set to 3/2 and I had 3 OSD's so .. ??? Later I understood that CRUSH wants three hosts, but whatever. I experiment with changing size/min_size to 2/1 and the warnings go away! Going back to 3/2 and they reappear. I am somewhat unhappy: before switching to 2/1, I saw most PG's were on most OSD's. Good! Switching to 2/1 made many of them disappear (!) and switching back to 3/2 did not make them reappear. The OSD's seem mildly underpopulated, and seem to remain that way. Huh. Who am I to say, but I would have liked to have seen one copy of *every* PG on *every* OSD, because that's what size=3 should mean, right? Even if there are only two servers? Or should I select a different crush algo? (I have not played with this, yet). I would sleep better if all three disks had a copy, because that would be an actual improvement over bog-standard raid mirroring. Anyway, I add a 4th OSD on the second server (so that there are 4 total) and I add a third server with zero OSD's (so that ceph-mon can create a proper quorum.) And by now I have over a TB of data on there (that I can afford to lose). So that's the backstory. Here's the problem. Last night, I move (not copy, but move) a 33 GB directory onto CephFS. This morning I see: mv: cannot overwrite directory '/dest/.snap' with non-directory '/source/.snap' wtf? I cd to /dest/.snap and indeed it is a directory. And it's empty. The source .snap is a file, 736 bytes in size. Directory timestamp and source-file timestamp are identical. I attempt a hand repair. I cd to the parent dir, do rmdir .snap and rmdir says "no such directory". wtf? So then `ls -la` and indeed, there is no such directory. But I can still cd into it! Huh. I cannot rmdir it, I cannot cp or mv it, because "it doesn't exist". If I cd into it, I cannot create any files in it, I get "permission denied", even as root. So it's a directory, it won't show in the listing of the parent dir. I can cd to it, but I cannot put anything in it. This, to me, says "CephFS bug". Other culprits could have been: (a) bad disk (b) corruption during copy, i.e. bad fs stat during copy due to (b1) bad kernel (b2) bad SATA traffic (b3) corrupt kernel page table, buffer, block cache, etc. (b4) bad data on the source disk. But all this is in the past: I can look at the source file: it's there, it's readable, it has valid contents, (It's ascii text. It looks just fine.) Clearly some transient error perturbed things. However, I now have a CephFS with a directory that I cannot remove, and I cannot place anything into it, and I cannot create a file with the same name as the directory. So that's not good. Sometime after I hit send on this email, ceph will automatically run scrub and/or deep scrub, and maybe this problem will fix itself. If there's something I should do to preserve this broken-ness for future posterity debugging, tell me now before it's too late. While I'm on this topic: The failed `mv` means that the source was still there, and so I was able to run `diff -rq --no-dereference` of the original source and the copy. I discovered that there were 5 files that were borked. They all had the same timestamps as the original, but the content was all zeros (as reported by od) I found that I could repair these by hand-copying the original over them, and the result is fine. Yes, this is a frankenstein system. Yes, it has 3 mons and three mgrs and three mds running on three machines, but OSD's only on two of them. I'll add more OSD's maybe tomorrow. But meanwhile, I have a system I can lose sleep over. The five files that were corrupted were corrupted *silently*, I would never have known if not for the weird-dir bug allowing me to go do the diff and find the corruption. This was on a copy of 33GB containing 412K files. I'd previously moved over a TB of files. Are any of these damaged? God knows. I was sloppy, I did not keep the originals, I did not do a diff. I trusted CephFS to do the right thing, and it seems it didn't. That trust is shattered. How often does ext4fs do something like this? I don't know. I've used ext2/3/4fs for 25 years on maybe a hundred different machines, and never lost data (that wasn't user error, i.e. rm-r* in the wrong directory) This is with 25 years of consumer-grade, low-quality pata/sata and sometimes scsi gear plugged into consumer-grade boxes cobbled from parts bought on newegg or maybe refurbished from Dell. (Yeah I also get to use huge high-end machines, but this is "for example") If there was dataloss during these decades, I'm not aware of it. Was there any? Laws of probability say "yes". Once upon a time, I managed to corrupt an ext2fs disk so that e2fsck could not fix it. Yes, *hit happens. But I've been using Ceph for barely a week, and its 2024 so it should be mature, so I'm not thrilled that I hit a bug so early and so easily. WTF. I did read the paper https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" which is an awesome paper, by the way, kudos to the authors. It does have one mis-representation, though. It says, paraphrasing: "it typically takes ten years for a file system to mature, and we here at CephFS/bluestore did it in only two (glow)" That paper was published in 2019 and it's now 2024, and off-the-shelf CephFS is clearly and blatantly buggy. Ceph is still awesome, but you cannot blame this bug on crappy hardware or a frankenstein sysconfig. Either the corner cases work, or they don't, and the authors of other filesystems take a decade or two to get through all of those corner cases. It would seem that CephFS has not evaded this iron law. Sorry for ending on such a downer, but hey... I'm one of those perfectionists that wreck things for the good-enough people. So it goes. -- Linas -- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.

3 2

How to modify the destination pool name in the rbd-mirror configuration?
by David Yang 27 Nov '24

27 Nov '24

Hello, everyone. Under normal circumstances, we synchronize from PoolA of ClusterA to PoolA of ClusterB (same pool name), which is also easy to configure. Now the requirements are as follows: ClusterA/Pool synchronizes to BackCluster/PoolA ClusterB/Pool synchronizes to BackCluster/PoolB After reading rbd-mirror and other documents, I did not find the configuration of different pool names.

2 4

Re: 4k IOPS: miserable performance in All-SSD cluster
by Peter Linder 27 Nov '24

27 Nov '24

In my experience, ceph will add around 1ms even if only on localhost. If this is in the client code or on the OSD's, I dont really know. I don't even know the precise reason, but the latency is there nevertheless. Perhaps you can find the reason here among the tradeoffs ceph and similar systems have to make to ensure consistency even if a partition can happen at any time: https://en.wikipedia.org/wiki/PACELC_theorem With size=3, a write will go first to the primary OSD for the PG, (0,1ms), then from there to two more PGs (in parallell), so 0,2ms more total RTT. Then back to the client, 0,1ms. That is, very roughly, 1,4ms even if storage latency is 0 which it never is even for ssds. If you set size=1, you can skip the step where the primary OSD replicates to the 2 replicas, but you still have cephs internal latency as well as the network latency to reach the primary OSD for whatever PG the object will belong to which could be on any server. So expect a small improvement but not too much. With that said, a single thread will not exceed 1000 iops ever in a typical setup. /Peter Den 2024-11-26 kl. 21:09, skrev Martin Gerhard Loschwitz: > that would mean 2-3ms latency between hosts hanging above each other > in the same rack connected to the same switches. > > Ping shows 0,2ms of latency though for all three affected clusters. > > So roughly 5000 iops. We can certainly add Ceph latency to that, but > that would mean Ceph eats 99% of the available performance, wouldn't it? > > Also, that wouldn't explain why we're seeing a bit of improvement with > size=1 for a specific pool but not a massive improvement, given that > at least half of the latency is taken out of the equation in that case. > > Best regards > Martin > > Peter Linder <peter.linder(a)fiberdirekt.se> schrieb am Di. 26. Nov. > 2024 um 20:52: > > With qd=1 (queue depth?) and a single thread, this isn't totally > unreasonable. > > Ceph will have an internal latency of around 1ms or so, add some > network > to that and an operation can take 2-3ms. With a single operation in > flight all the time, this means 333-500 operations per second. With > hdds, even fewer. > > What happens if you try again with many more threads? > > > Den 2024-11-25 kl. 15:22, skrev Martin Gerhard Loschwitz: > > Folks, > > > > I am getting somewhat desperate debugging multiple setups here > within the same environment. Three clusters, two SSD-only, one > HDD-only, and what they all have in common is abysmal 4k IOPS > performance when measuring with „rados bench“. Abysmal means: In > an All-SSD cluster I will get roughly 400 IOPS over more than 250 > devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on > the low side of things to me. > > > > In the second cluster, also All-SSD based, I get roughly 120 4k > IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter > both with substantially fewer devices, granted. But even with 20 > HDDs, 68 4k IOPS seems like a very bad value to me. > > > > I’ve tried to rule out everything I know of: BIOS > misconfigurations, HBA problems, networking trouble (I am seeing > comparably bad values with a size=1 pool) and so further and so > on. But to no avail. Has anybody dealt with something similar on > Dell hardware or in general? What could cause such extremely bad > benchmark results? > > > > I measure with rados bench and qd=1 at 4k block size. „ceph tell > osd bench“ with 4k blocks yields 30k+ IOPS for every single device > in the big cluster, and all that leads to is 400 IOPS in total > when writing to it? Even with no replication in place? That looks > a bit off, doesn't it? Any help will be greatly appreciated, thank > you very much in advance. Even a pointer to the right direction > would be held in high esteem right now. Thank you very much in > advance! > > > > Best regards > > Martin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

4 5

v17.2.8 Quincy released
by Yuri Weinstein 26 Nov '24

26 Nov '24

We're happy to announce the 8th backport release in the Quincy series. https://ceph.io/en/news/blog/2024/v17-2-8-quincy-released/ v17.2.8 will have RPM/centos 9 packages instead of RPM/centos 8 built. v17.2.8 container images, now based on CentOS 9, may be incompatible on older kernels (e.g., Ubuntu 18.04) due to differences in thread creation methods. Users upgrading to v17.2.8 container images with older OS versions may encounter crashes during `pthread_create`. However, we recommend upgrading your OS to avoid this unsupported combination. Users should expect to see the el8 rpm subdirectory empty and the "dnf" commands are expected to fail with 17.2.8. They can choose to use 17.2.8 RPM packages for centos 8/el8 provided by CERN as a community member or continue to stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 These CERN packages come with no warranty and have not been tested. The software in them has been tested by Ceph according to [platforms](https://docs.ceph.com/en/latest/start/os-recommendations/#platfo…. The repository for el8 builds is hosted by CERN on [Linux @ CERN](https://linuxsoft.cern.ch/repos/ceph-ext-quincy8el-stable/). The public part of the GPG key used to sign the packages is available at [RPM-GPG-KEY-Ceph-Community](https://linuxsoft.cern.ch/repos/RPM-GPG-KEY-Ceph-Community). Notable Changes --------------- * RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to being prone to false negative results. It's safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * RBD: The option ``--image-id`` has been added to `rbd children` CLI command, so it can be run for images in the trash. * RBD: `RBD_IMAGE_OPTION_CLONE_FORMAT` option has been exposed in Python bindings via `clone_format` optional parameter to `clone`, `deep_copy` and `migration_prepare` methods. * RBD: `RBD_IMAGE_OPTION_FLATTEN` option has been exposed in Python bindings via `flatten` optional parameter to `deep_copy` and `migration_prepare` methods. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.8.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: f817ceb7f187defb1d021d6328fa833eb8e943b3

3 3

RGW Daemons Crash After Adding Secondary Zone with Archive module
by mahnoosh shahidi 26 Nov '24

26 Nov '24

Hi everyone, I encountered an issue while setting up a multisite configuration in Ceph. After adding the second secondary zone and enabling the archive module, the RGW daemons started crashing repeatedly after running the period update command. The crash log shows the following error: **** Caught signal (Aborted) ** * *in thread 7fc7d20c7640 thread_name:data-sync * *ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) * *1: /lib64/libc.so.6(+0x3e6f0) [0x7fc7f5c916f0] * *2: /lib64/libc.so.6(+0x8b94c) [0x7fc7f5cde94c] * *3: raise() * *4: abort() * *5: /usr/bin/radosgw(+0x30e668) [0x55983a811668] * *6: /usr/bin/radosgw(+0x9236c7) [0x55983ae266c7] * *7: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x125) [0x55983aa484b5] * *8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x489) [0x55983aa49529] * *9: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x55983aa4a31d] * *10: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x3e0) [0x55983ae36c20] * *11: /usr/bin/radosgw(+0x705e88) [0x55983ac08e88] * *12: (RGWRadosThread::Worker::entry()+0xb3) [0x55983ac0c863] * *13: /lib64/libc.so.6(+0x89c02) [0x7fc7f5cdcc02] * *14: /lib64/libc.so.6(+0x10ec40) [0x7fc7f5d61c40] * *NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. * Any idea about this? Thank you.

1 0

Multisite RGW-SYNC error: failed to remove omap key from error repo
by Szabo, Istvan (Agoda) 26 Nov '24

26 Nov '24

Hi, I see in our logs continuously these kind of messaages: RGW-SYNC:data:sync:shard[7]:entry: ERROR: failed to remove omap key from error repo (hkg.rgw.log:datalog.sync-status.shard.61c9d940-fde4-4bed-9389-edc8d7741817.7.retry retcode=-1 And we started to receive large omaps in the log pool which we trim and scrub the pg to get rid of it, however this is not the root cause. What I can see in this line is the remote site id: 61c9d940-fde4-... Which has 1 bucket to sync and it looks like syncing so don't understand what is this error. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

CephFS 16.2.10 problem
by Alexey.Tsivinsky＠baikalelectronics.ru 26 Nov '24

26 Nov '24

Good morning! It's a novelty for our situation. The list of cluster locks now contains entries for the addresses of the servers on which the mds is running. The lifetime of the records is a day. For now, we decided to wait until the records are deleted and see how the cluster behaves. Best regards! Alexey Tsivinsky e-mail:a.tsivinsky@baikalelectronics.com

1 0

Cephalocon Update - New Users Workshop and Power Users Session
by Dan van der Ster 26 Nov '24

26 Nov '24

Dear Fellow Cephalopods, Cephalocon 2024 hosted by CERN is just around the corner! In case you missed it, the full schedule is now live: https://events.linuxfoundation.org/cephalocon/ The Program Committee received a record-breaking 176 submissions, carefully selecting 60 speakers to present an exciting lineup. Attendees will have the chance to dive into the latest updates from Ceph Developers and engage with fellow users. We're thrilled to report that registration is approaching our target of 300 attendees, and interest is higher than anticipated! Our hosts are working hard to accommodate even more participants. If you haven't registered yet and plan to join us, don't delay---secure your spot today! Additionally, two pre-Cephalocon events are open for attendees: - New to Ceph? Join the Ceph New Users Workshop - https://indico.cern.ch/e/ceph-new-users-workshop It’s a great opportunity for beginners to get up to speed. - Experienced Ceph User? Participate in the Power Users afternoon session at the Developers Summit - https://indico.cern.ch/e/ceph-developer-summit These pre-events are free but require a valid Cephalocon registration. We look forward to seeing you there! Warm regards, Dan, on behalf of the Cephalocon Organizers -- Dan van der Ster CTO @ CLYSO

4 5

How to synchronize pools with the same name in multiple clusters to multiple pools in one cluster
by David Yang 26 Nov '24

26 Nov '24

We have ceph clusters in multiple regions to provide rbd services. We are currently preparing a remote backup plan, which is to synchronize pools with the same name in each region to different pools in one cluster. For example: Cluster A Pool synchronized to backup cluster poolA Cluster B Pool synchronized to backup cluster poolB I read the rbd-mirror documentation, but I don't understand how to start synchronization using bootstrap mode.

1 0