- ceph-users - lists.ceph.io

OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Boris 27 Jan '26

27 Jan '26

Hi, we are currently facing an issue, that suddenly none of the OSDs will start after the container started with the new versions. This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, but took really long (like 5 hours) and then every single OSD after that needed a host reboot to bring the disk back up and continue the update. We've stopped after 6 tries. And one disk never came back up. We removed and zapped the OSD. The orchestrator picked the available disk and recreated it. It came up within seconds. We have around 90 clusters and this happened only on a single one. All others updates within two hours without any issues. The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of output from udevadm, lsblk and nsenter The activation container (ceph-UUID-osd-N-activate) gets killed after a couple of minutes. It also looks like the block and block.db links in /var/lib/ceph/UUID/osd.N/ are not correctly set. When we restart the daemons that needed a host restart, the OSD doesn't come up and needs a host restart. All OSDs are encrypted. Does anyone got some ideas how to debug further?

1 0

upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 26 Jan '26

26 Jan '26

Hi, Is there a way to use the new upmap-read balancer mode when I have CephFS kernel clients connected? Our cluster runs Squid and all connected clients are on kernel 6.11-6.14, so they should be new enough. However, they identify themselves as luminous (I could not find clear documentation about this, but this seems to be hard-coded and by design), so I cannot run ceph osd set-require-min-compat-client reef, which is required for the balancer module to work. I have 500 clients connected and migrating them all to FUSE would be a headache. Is there a way around that? Some of my pools have extremely high read balance scores and we're also struggling with performance issues as a result. Janek

1 0

January Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 22 Jan '26

22 Jan '26

Hey folks, We will be having a Ceph science/research/big cluster call on Tuesday January 27th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: January 27th, 2026 9am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20260127 Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Re: rgw - migrate data from replicated to ec pool
by Anthony D'Atri 22 Jan '26

22 Jan '26

Thanks, Casey, I was very excited to learn of this at Cephalocon. At the time it seemed like a proposal, hearing that work has begun is fantastic. This should AFAICT allow retrofitting a first SSD data pool into an existing CephFS deployment, though I’ll be curious if the pool indirection overhead erodes the benefits of doing so. > On Jan 22, 2026, at 9:04 AM, Casey Bodley via ceph-users <ceph-users(a)ceph.io> wrote: > > storage classes and/or lifecycle are not ideal for this because, while > they can add a new pool, they don't provide a way to retire the old > one. removing/replacing the default STANDARD storage class is > especially difficult because, as Janne suggested, the STANDARD storage > class pool stores all 'head objects' which act like symlinks for > object data in other storage class pools. lifecycle transitions will > only move object data around, not their head objects > > luckily, the rados team has started work on a "pool migration" > feature. sharing links to the design doc and early draft pr: > https://docs.ceph.com/en/latest/dev/pool-migration-design/ > https://github.com/ceph/ceph/pull/65746 > > On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users > <ceph-users(a)ceph.io> wrote: >> >> >>> On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: >>> >>> Please correct me if I am wrong here (because I might totally be), but >>> doesn't this leave some kind of "softlink" object in the old pool and >>> "only" move the content over to the new pool? >> >> Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not >> >> Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] >> In this ticket you can grab copy-paste XML for what I'm talking about >> >> >> >> [1] https://tracker.ceph.com/issues/50974 >> k >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw - migrate data from replicated to ec pool
by Anthony D'Atri 22 Jan '26

22 Jan '26

> ceph-users <ceph-users(a)ceph.io>: >>> On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: >>> >>> we want to migrate the data from a replicated to an EC pool. >>> Is there a way to do it without downtime? >> >> Your clients can move objects from one storage class to another, example >> s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" > > Please correct me if I am wrong here (because I might totally be), but > doesn't this leave some kind of "softlink" object in the old pool and > "only" move the content over to the new pool? Not to my knowledge. You can however accomplish the same thing with an LC policy, which would likely be way more efficient. I must note, though, that storage class names are arbitrary to RGW, but not to some clients / libraries, which expect specific server behavior from GLACIER, or won’t work at all for an SC named, say, ETHELMERMAN. It’s best to stick to AWS names as documented here: https://docs.ceph.com/en/latest/radosgw/placement/#using-storage-classes Today, you can use rbd-mirror or live migration to move volumes between pools, but at some point the client attachment I think needs to be refreshed. With CephFS you attach the new data pool, setfattr a directory for the file layout to use the pool, and cp (not mv) each file onto it. There are scripts available online. Since this is a client operation that copies all the data then deletes the old, it is resource-intensive and not transparent to clients, at least not until such a time that the directory names are swapped, and not for any open files. As with RGW head objects, which are always in the default storage class, CephFS benefits from the first data pool being on fast SSDs for backtraces, especially when there are a zillion tiny files. This cannot AFAIK be retrofitted (at least not before Umbrella), but one could create a new CephFS with new pools and migrate files onto it. > there was some kind of situation where you still need to retain the old pool. If transparent RADOS pool migration makes it into Umbrella, that will be true. But likely one will be able to pare down pg_num for that pool and it won’t be a big deal. > Perhaps this is related to layering ceph pools > (also one way in the past to move data from pool A to B while changing > EC values or repl, not recommended anymore) which is why I am so > unsure. If anyone wants to test/verify I am happy to be proven wrong. > > If I am wrong, then https://docs.ceph.com/en/latest/radosgw/placement/ > has some good info on how to make more than one ceph pool available > for rgw usage and set another one as the new default for users/buckets > created after the change and so forth. Thanks! > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw - migrate data from replicated to ec pool
by Boris 22 Jan '26

22 Jan '26

Hi Casey, that is exactly what I was looking for. I had all the ideas that were mentioned here, but I wanted it to be transparent for the customer without them noticing. With the pool migration feature, this would be really great. Thanks all for your input :) Am Do., 22. Jan. 2026 um 15:06 Uhr schrieb Casey Bodley via ceph-users < ceph-users(a)ceph.io>: > storage classes and/or lifecycle are not ideal for this because, while > they can add a new pool, they don't provide a way to retire the old > one. removing/replacing the default STANDARD storage class is > especially difficult because, as Janne suggested, the STANDARD storage > class pool stores all 'head objects' which act like symlinks for > object data in other storage class pools. lifecycle transitions will > only move object data around, not their head objects > > luckily, the rados team has started work on a "pool migration" > feature. sharing links to the design doc and early draft pr: > https://docs.ceph.com/en/latest/dev/pool-migration-design/ > https://github.com/ceph/ceph/pull/65746 > > On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > > > > On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > > > Please correct me if I am wrong here (because I might totally be), but > > > doesn't this leave some kind of "softlink" object in the old pool and > > > "only" move the content over to the new pool? > > > > Depends on client (s5cmd/aws cli program), mostly. What exactly this > command actually do - copy data over client or not > > > > Also, your clients may use LifeCycle Policy to move objects to another > class. In your case this should work, cause since implementation Ceph can't > move objects between classes in shared pool, see #50974 [1] > > In this ticket you can grab copy-paste XML for what I'm talking about > > > > > > > > [1] https://tracker.ceph.com/issues/50974 > > k > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: rgw - migrate data from replicated to ec pool
by Casey Bodley 22 Jan '26

22 Jan '26

storage classes and/or lifecycle are not ideal for this because, while they can add a new pool, they don't provide a way to retire the old one. removing/replacing the default STANDARD storage class is especially difficult because, as Janne suggested, the STANDARD storage class pool stores all 'head objects' which act like symlinks for object data in other storage class pools. lifecycle transitions will only move object data around, not their head objects luckily, the rados team has started work on a "pool migration" feature. sharing links to the design doc and early draft pr: https://docs.ceph.com/en/latest/dev/pool-migration-design/ https://github.com/ceph/ceph/pull/65746 On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users <ceph-users(a)ceph.io> wrote: > > > > On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > > > Please correct me if I am wrong here (because I might totally be), but > > doesn't this leave some kind of "softlink" object in the old pool and > > "only" move the content over to the new pool? > > Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not > > Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] > In this ticket you can grab copy-paste XML for what I'm talking about > > > > [1] https://tracker.ceph.com/issues/50974 > k > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: rgw - migrate data from replicated to ec pool
by Konstantin Shalygin 22 Jan '26

22 Jan '26

> On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > Please correct me if I am wrong here (because I might totally be), but > doesn't this leave some kind of "softlink" object in the old pool and > "only" move the content over to the new pool? Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] In this ticket you can grab copy-paste XML for what I'm talking about [1] https://tracker.ceph.com/issues/50974 k

1 0

Re: rgw - migrate data from replicated to ec pool
by Janne Johansson 22 Jan '26

22 Jan '26

Den tors 22 jan. 2026 kl 09:26 skrev Konstantin Shalygin via ceph-users <ceph-users(a)ceph.io>: > > On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > > > we want to migrate the data from a replicated to an EC pool. > > Is there a way to do it without downtime? > > Your clients can move objects from one storage class to another, example > s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" Please correct me if I am wrong here (because I might totally be), but doesn't this leave some kind of "softlink" object in the old pool and "only" move the content over to the new pool? I seem to recall something like this when you move objects between storage pools/placements in ceph s3, and it could totally be about something else so don't take my word for it, but there is a nagging feeling in my mind there was some kind of situation where you still need to retain the old pool. Perhaps this is related to layering ceph pools (also one way in the past to move data from pool A to B while changing EC values or repl, not recommended anymore) which is why I am so unsure. If anyone wants to test/verify I am happy to be proven wrong. If I am wrong, then https://docs.ceph.com/en/latest/radosgw/placement/ has some good info on how to make more than one ceph pool available for rgw usage and set another one as the new default for users/buckets created after the change and so forth. -- May the most significant bit of your life be positive.

1 0

Re: rgw - migrate data from replicated to ec pool
by Konstantin Shalygin 22 Jan '26

22 Jan '26

Hi, > On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > we want to migrate the data from a replicated to an EC pool. > Is there a way to do it without downtime? Your clients can move objects from one storage class to another, example s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" k

1 0