- ceph-users - lists.ceph.io

January Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 23 Jan '26

23 Jan '26

Hey folks, We will be having a Ceph science/research/big cluster call on Tuesday January 27th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: January 27th, 2026 9am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20260127 Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Re: rgw - migrate data from replicated to ec pool
by Anthony D'Atri 22 Jan '26

22 Jan '26

Thanks, Casey, I was very excited to learn of this at Cephalocon. At the time it seemed like a proposal, hearing that work has begun is fantastic. This should AFAICT allow retrofitting a first SSD data pool into an existing CephFS deployment, though I’ll be curious if the pool indirection overhead erodes the benefits of doing so. > On Jan 22, 2026, at 9:04 AM, Casey Bodley via ceph-users <ceph-users(a)ceph.io> wrote: > > storage classes and/or lifecycle are not ideal for this because, while > they can add a new pool, they don't provide a way to retire the old > one. removing/replacing the default STANDARD storage class is > especially difficult because, as Janne suggested, the STANDARD storage > class pool stores all 'head objects' which act like symlinks for > object data in other storage class pools. lifecycle transitions will > only move object data around, not their head objects > > luckily, the rados team has started work on a "pool migration" > feature. sharing links to the design doc and early draft pr: > https://docs.ceph.com/en/latest/dev/pool-migration-design/ > https://github.com/ceph/ceph/pull/65746 > > On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users > <ceph-users(a)ceph.io> wrote: >> >> >>> On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: >>> >>> Please correct me if I am wrong here (because I might totally be), but >>> doesn't this leave some kind of "softlink" object in the old pool and >>> "only" move the content over to the new pool? >> >> Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not >> >> Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] >> In this ticket you can grab copy-paste XML for what I'm talking about >> >> >> >> [1] https://tracker.ceph.com/issues/50974 >> k >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw - migrate data from replicated to ec pool
by Anthony D'Atri 22 Jan '26

22 Jan '26

> ceph-users <ceph-users(a)ceph.io>: >>> On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: >>> >>> we want to migrate the data from a replicated to an EC pool. >>> Is there a way to do it without downtime? >> >> Your clients can move objects from one storage class to another, example >> s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" > > Please correct me if I am wrong here (because I might totally be), but > doesn't this leave some kind of "softlink" object in the old pool and > "only" move the content over to the new pool? Not to my knowledge. You can however accomplish the same thing with an LC policy, which would likely be way more efficient. I must note, though, that storage class names are arbitrary to RGW, but not to some clients / libraries, which expect specific server behavior from GLACIER, or won’t work at all for an SC named, say, ETHELMERMAN. It’s best to stick to AWS names as documented here: https://docs.ceph.com/en/latest/radosgw/placement/#using-storage-classes Today, you can use rbd-mirror or live migration to move volumes between pools, but at some point the client attachment I think needs to be refreshed. With CephFS you attach the new data pool, setfattr a directory for the file layout to use the pool, and cp (not mv) each file onto it. There are scripts available online. Since this is a client operation that copies all the data then deletes the old, it is resource-intensive and not transparent to clients, at least not until such a time that the directory names are swapped, and not for any open files. As with RGW head objects, which are always in the default storage class, CephFS benefits from the first data pool being on fast SSDs for backtraces, especially when there are a zillion tiny files. This cannot AFAIK be retrofitted (at least not before Umbrella), but one could create a new CephFS with new pools and migrate files onto it. > there was some kind of situation where you still need to retain the old pool. If transparent RADOS pool migration makes it into Umbrella, that will be true. But likely one will be able to pare down pg_num for that pool and it won’t be a big deal. > Perhaps this is related to layering ceph pools > (also one way in the past to move data from pool A to B while changing > EC values or repl, not recommended anymore) which is why I am so > unsure. If anyone wants to test/verify I am happy to be proven wrong. > > If I am wrong, then https://docs.ceph.com/en/latest/radosgw/placement/ > has some good info on how to make more than one ceph pool available > for rgw usage and set another one as the new default for users/buckets > created after the change and so forth. Thanks! > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw - migrate data from replicated to ec pool
by Boris 22 Jan '26

22 Jan '26

Hi Casey, that is exactly what I was looking for. I had all the ideas that were mentioned here, but I wanted it to be transparent for the customer without them noticing. With the pool migration feature, this would be really great. Thanks all for your input :) Am Do., 22. Jan. 2026 um 15:06 Uhr schrieb Casey Bodley via ceph-users < ceph-users(a)ceph.io>: > storage classes and/or lifecycle are not ideal for this because, while > they can add a new pool, they don't provide a way to retire the old > one. removing/replacing the default STANDARD storage class is > especially difficult because, as Janne suggested, the STANDARD storage > class pool stores all 'head objects' which act like symlinks for > object data in other storage class pools. lifecycle transitions will > only move object data around, not their head objects > > luckily, the rados team has started work on a "pool migration" > feature. sharing links to the design doc and early draft pr: > https://docs.ceph.com/en/latest/dev/pool-migration-design/ > https://github.com/ceph/ceph/pull/65746 > > On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > > > > On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > > > Please correct me if I am wrong here (because I might totally be), but > > > doesn't this leave some kind of "softlink" object in the old pool and > > > "only" move the content over to the new pool? > > > > Depends on client (s5cmd/aws cli program), mostly. What exactly this > command actually do - copy data over client or not > > > > Also, your clients may use LifeCycle Policy to move objects to another > class. In your case this should work, cause since implementation Ceph can't > move objects between classes in shared pool, see #50974 [1] > > In this ticket you can grab copy-paste XML for what I'm talking about > > > > > > > > [1] https://tracker.ceph.com/issues/50974 > > k > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: rgw - migrate data from replicated to ec pool
by Casey Bodley 22 Jan '26

22 Jan '26

storage classes and/or lifecycle are not ideal for this because, while they can add a new pool, they don't provide a way to retire the old one. removing/replacing the default STANDARD storage class is especially difficult because, as Janne suggested, the STANDARD storage class pool stores all 'head objects' which act like symlinks for object data in other storage class pools. lifecycle transitions will only move object data around, not their head objects luckily, the rados team has started work on a "pool migration" feature. sharing links to the design doc and early draft pr: https://docs.ceph.com/en/latest/dev/pool-migration-design/ https://github.com/ceph/ceph/pull/65746 On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users <ceph-users(a)ceph.io> wrote: > > > > On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > > > Please correct me if I am wrong here (because I might totally be), but > > doesn't this leave some kind of "softlink" object in the old pool and > > "only" move the content over to the new pool? > > Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not > > Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] > In this ticket you can grab copy-paste XML for what I'm talking about > > > > [1] https://tracker.ceph.com/issues/50974 > k > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: rgw - migrate data from replicated to ec pool
by Konstantin Shalygin 22 Jan '26

22 Jan '26

> On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > Please correct me if I am wrong here (because I might totally be), but > doesn't this leave some kind of "softlink" object in the old pool and > "only" move the content over to the new pool? Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] In this ticket you can grab copy-paste XML for what I'm talking about [1] https://tracker.ceph.com/issues/50974 k

1 0

Re: rgw - migrate data from replicated to ec pool
by Janne Johansson 22 Jan '26

22 Jan '26

Den tors 22 jan. 2026 kl 09:26 skrev Konstantin Shalygin via ceph-users <ceph-users(a)ceph.io>: > > On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > > > we want to migrate the data from a replicated to an EC pool. > > Is there a way to do it without downtime? > > Your clients can move objects from one storage class to another, example > s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" Please correct me if I am wrong here (because I might totally be), but doesn't this leave some kind of "softlink" object in the old pool and "only" move the content over to the new pool? I seem to recall something like this when you move objects between storage pools/placements in ceph s3, and it could totally be about something else so don't take my word for it, but there is a nagging feeling in my mind there was some kind of situation where you still need to retain the old pool. Perhaps this is related to layering ceph pools (also one way in the past to move data from pool A to B while changing EC values or repl, not recommended anymore) which is why I am so unsure. If anyone wants to test/verify I am happy to be proven wrong. If I am wrong, then https://docs.ceph.com/en/latest/radosgw/placement/ has some good info on how to make more than one ceph pool available for rgw usage and set another one as the new default for users/buckets created after the change and so forth. -- May the most significant bit of your life be positive.

1 0

Re: rgw - migrate data from replicated to ec pool
by Konstantin Shalygin 22 Jan '26

22 Jan '26

Hi, > On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > we want to migrate the data from a replicated to an EC pool. > Is there a way to do it without downtime? Your clients can move objects from one storage class to another, example s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" k

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Alfonso Escribano Merino 22 Jan '26

22 Jan '26

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Joshua Blanch 21 Jan '26

21 Jan '26

We ran into that allow_ec_optimization on a legacy pool and scrubbing issue recently on 20.2.0. Wrote down some notes here: https://docs.clyso.com/blog/2026/04/13/tentacle-v20.2.0-ec-bug/ Specifically it is these trackers which will be in the v20.2.1 release allow_ec issue: https://tracker.ceph.com/issues/71642 scrubbing issue: https://tracker.ceph.com/issues/73184 https://tracker.ceph.com/issues/73260 On Wed, 21 Jan 2026 at 11:17, Eric Dold via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Reto, hi Bill > > I found this PR that was merged after v20.2.0 was tagged. > https://github.com/ceph/ceph/pull/66021 > > I'm running Tentacle on Ubuntu Jammy without cephadm, with EC 6+2 and > FastEC enabled. > > I'm getting inconsistent PGs with errors like these: > 3.336s0 shard 25(5) soid 3:6cc82c0a:::10002107475.0000002c:head : size > 700416 != size 696320 from auth oi > 3:6cc82c0a:::10002107475.0000002c:head(2830345'2540087 > osd.41.2825431:3886213 dirty|data_digest s 4194304 uv 17841689 dd 8f22c13e > alloc_hint [0 0 0]) > 3.336s0 shard 13(4) soid 3:6cc1c47e:::10002585665.0000045e:10279 : size > 700416 != size 696320 from auth oi > 3:6cc1c47e:::10002585665.0000045e:10279(2905787'3123318 > osd.41.2896446:3729885 dirty|data_digest s 4194304 uv 23445132 dd 365ee4af > alloc_hint [0 0 0] shard_versions={1: 2904402'3115959, 2: 2904402'3115959, > 3: 2904402'3115959, 4: 2904402'3115959, 5: 2904402'3115959}) > 3.336s0 shard 13(4) soid 3:6cc1f637:::20000007ea9.000008fb:bd : size 700416 > != size 696320 from auth oi > 3:6cc1f637:::20000007ea9.000008fb:bd(45314'680679 client.124476.0:17244371 > dirty|data_digest s 4194304 uv 274244 dd 9f754009 alloc_hint [0 0 0]) > 3.336 shard 13(4) soid 3:6cc82c71:::1000098dda0.00000237:head : candidate > size 700416 info size 696320 mismatch > 3.336 shard 60(1) soid 3:6cde2c17:::20001b3c880.00000000:head : candidate > size 4096 info size 0 mismatch > 3.336 shard 13(4) soid 3:6cde2eb0:::100000d68b9.0000000b:head : candidate > size 622592 info size 618496 mismatch > > All these errors have something to do with size. > > When I restart the OSDs the cluster is healthy until some scrub picks this > up again. > The data seems ok. > > It seems like the fix is not in the tentacle release. > Is there a future release with fixes for this planned? > > Regards > Eric > > > On Fri, 5 Dec 2025 at 16:34, Reto Gysi via ceph-users <ceph-users(a)ceph.io> > wrote: > > > Hi Bill > > > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > > bill_scales(a)uk.ibm.com>: > > > > > Hi Reto, > > > > > > Sorry to hear about your problems with turning on ec optimizations. > I've > > > led the team that developed this feature, so we are keen to help > > understand > > > what happened to your system. > > > > > > Your configuration looks fine as far as being supported with ec > > > optimizations. The daemons (mons, osds and mgr) need to be running > > tentacle > > > code to use this feature, there is no requirement to update any of your > > > clients. > > > > > > Do you have any more examples of the inconsistent objects logs that you > > > can share with me? > > > > > > There was a bug that we fixed late in the development cycle which was > > > scrubbing incorrectly reporting a size mismatch for objects written > prior > > > to turning on optimizations. This is because prior to turning on > > > optimizations objects are padded to a multiple of the stripe_width, > > > afterwards objects are no longer padded. The scrub code was > occasionally > > > getting confused and incorrectly reporting an inconsistency. In this > case > > > the scrub was reporting false positive errors - there was nothing wrong > > > with the data and no problems accessing the data. I notice the log you > > > shared in the email was for a size mismatch, I'm interested whether all > > the > > > logs were for mismatched sizes. We will do some further work to confirm > > > that the fix is in the 20.2.0 tentacle release and that there are no > > > problems with the back port. > > > > > > > As far I could see the errors where all about the mismatched sizes. > > > > > > > > > > You also mention that you saw some OSD crashes. Do you have any further > > > information about these? > > > > > > > I just could cause another crash with starting up a windows VM, and > doing a > > WIndows file history backup to a drive that is on an RBD Image on pool > > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > > > <disk type="network" device="disk"> > > <driver name="qemu" type="raw" cache="writethrough" io="threads" > > discard="unmap" detect_zeroes="unmap"/> > > <auth username="admin"> > > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > > </auth> > > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > > <host name="zephir" port="6789"/> > > </source> > > <target dev="vdd" bus="virtio"/> > > <alias name="virtio-disk3"/> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > > function="0x0"/> > > </disk> > > > > root@zephir:~# ceph -s > > cluster: > > id: 27923302-87a5-11ec-ac5b-976d21a49941 > > health: HEALTH_WARN > > 2 osds down > > Reduced data availability: 49 pgs inactive > > Degraded data redundancy: 15722192/79848340 objects degraded > > (19.690%), 246 pgs degraded, 247 pgs undersized > > > > services: > > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) > [leader: > > zephir] > > mgr: zephir.enywvy(active, since 21h), standbys: > debian.nusuye > > mds: 3/3 daemons up, 3 standby > > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > > pgs > > cephfs-mirror: 2 daemons active (2 hosts) > > rbd-mirror: 2 daemons active (2 hosts) > > rgw: 1 daemon active (1 hosts, 1 zones) > > tcmu-runner: 5 portals active (2 hosts) > > > > data: > > volumes: 3/3 healthy > > pools: 25 pools, 450 pgs > > objects: 17.52M objects, 62 TiB > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > pgs: 10.889% pgs not active > > 15722192/79848340 objects degraded (19.690%) > > 197 active+undersized+degraded > > 183 active+clean > > 49 undersized+degraded+peered > > 15 active+clean+scrubbing > > 5 active+clean+scrubbing+deep > > 1 active+undersized > > > > io: > > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > > > root@zephir:~# ceph osd status > > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > > 0 debian 11.2T 5289G 0 0 0 0 exists,up > > 1 zephir 11.6T 4846G 0 0 0 0 exists > > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > > 3 zephir 11.0T 5497G 0 0 0 0 exists > > 4 zephir 159G 1376G 0 0 2 0 exists,up > > 5 zephir 120G 1415G 0 0 0 0 exists,up > > 6 debian 11.3T 5200G 0 0 1 0 exists,up > > 7 zephir 305G 1230G 0 0 0 0 exists,up > > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > > 10 zephir 235G 1300G 0 0 0 0 exists,up > > 11 zephir 223G 1312G 0 0 0 0 exists,up > > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > > 13 debian 118G 631G 0 0 3 8 exists,up > > 14 debian 345G 1190G 0 0 0 0 exists,up > > 15 debian 11.4T 5079G 0 0 0 0 exists,up > > 16 debian 554G 3029G 0 0 3 8 exists,up > > 17 debian 12.5T 5874G 0 0 1 0 exists,up > > root@zephir:~# > > > > I attached the log of osd.1 to > > https://filebin.net/jwvs6kuqrc7hx8id > > > > Best Regards, > > > > Reto > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0