Hi Reto, hi Bill
I found this PR that was merged after v20.2.0 was tagged.
https://github.com/ceph/ceph/pull/66021
I'm running Tentacle on Ubuntu Jammy without cephadm, with EC 6+2 and
FastEC enabled.
I'm getting inconsistent PGs with errors like these:
3.336s0 shard 25(5) soid 3:6cc82c0a:::10002107475.0000002c:head : size
700416 != size 696320 from auth oi
3:6cc82c0a:::10002107475.0000002c:head(2830345'2540087
osd.41.2825431:3886213 dirty|data_digest s 4194304 uv 17841689 dd 8f22c13e
alloc_hint [0 0 0])
3.336s0 shard 13(4) soid 3:6cc1c47e:::10002585665.0000045e:10279 : size
700416 != size 696320 from auth oi
3:6cc1c47e:::10002585665.0000045e:10279(2905787'3123318
osd.41.2896446:3729885 dirty|data_digest s 4194304 uv 23445132 dd 365ee4af
alloc_hint [0 0 0] shard_versions={1: 2904402'3115959, 2: 2904402'3115959,
3: 2904402'3115959, 4: 2904402'3115959, 5: 2904402'3115959})
3.336s0 shard 13(4) soid 3:6cc1f637:::20000007ea9.000008fb:bd : size 700416
!= size 696320 from auth oi
3:6cc1f637:::20000007ea9.000008fb:bd(45314'680679 client.124476.0:17244371
dirty|data_digest s 4194304 uv 274244 dd 9f754009 alloc_hint [0 0 0])
3.336 shard 13(4) soid 3:6cc82c71:::1000098dda0.00000237:head : candidate
size 700416 info size 696320 mismatch
3.336 shard 60(1) soid 3:6cde2c17:::20001b3c880.00000000:head : candidate
size 4096 info size 0 mismatch
3.336 shard 13(4) soid 3:6cde2eb0:::100000d68b9.0000000b:head : candidate
size 622592 info size 618496 mismatch
All these errors have something to do with size.
When I restart the OSDs the cluster is healthy until some scrub picks this
up again.
The data seems ok.
It seems like the fix is not in the tentacle release.
Is there a future release with fixes for this planned?
Regards
Eric
On Fri, 5 Dec 2025 at 16:34, Reto Gysi via ceph-users <ceph-users@ceph.io>
wrote:
Hi Bill
Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales <
bill_scales@uk.ibm.com>:
Hi Reto,
Sorry to hear about your problems with turning on ec optimizations.
I've
led the team that developed this feature, so we are keen to help
understand
what happened to your system.
Your configuration looks fine as far as being supported with ec
optimizations. The daemons (mons, osds and mgr) need to be running
tentacle
code to use this feature, there is no requirement to update any of your
clients.
Do you have any more examples of the inconsistent objects logs that you
can share with me?
There was a bug that we fixed late in the development cycle which was
scrubbing incorrectly reporting a size mismatch for objects written
prior
to turning on optimizations. This is because prior to turning on
optimizations objects are padded to a multiple of the stripe_width,
afterwards objects are no longer padded. The scrub code was
occasionally
getting confused and incorrectly reporting an inconsistency. In this
case
the scrub was reporting false positive errors - there was nothing wrong
with the data and no problems accessing the data. I notice the log you
shared in the email was for a size mismatch, I'm interested whether all
the
logs were for mismatched sizes. We will do some further work to confirm
that the fix is in the 20.2.0 tentacle release and that there are no
problems with the back port.
As far I could see the errors where all about the mismatched sizes.
You also mention that you saw some OSD crashes. Do you have any further
information about these?
I just could cause another crash with starting up a windows VM, and
doing a
WIndows file history backup to a drive that is on an RBD Image on pool
rbd_ecpool where I had the allow_ec_optimization flag enabled.
<disk type="network" device="disk">
<driver name="qemu" type="raw" cache="writethrough" io="threads"
discard="unmap" detect_zeroes="unmap"/>
<auth username="admin">
<secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/>
</auth>
<source protocol="rbd" name="rbd/game_windows_backup_drive" index="3">
<host name="zephir" port="6789"/>
</source>
<target dev="vdd" bus="virtio"/>
<alias name="virtio-disk3"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02"
function="0x0"/>
</disk>
root@zephir:~# ceph -s
cluster:
id: 27923302-87a5-11ec-ac5b-976d21a49941
health: HEALTH_WARN
2 osds down
Reduced data availability: 49 pgs inactive
Degraded data redundancy: 15722192/79848340 objects degraded
(19.690%), 246 pgs degraded, 247 pgs undersized
services:
mon: 3 daemons, quorum zephir,debian,raspi (age 20h)
[leader:
zephir]
mgr: zephir.enywvy(active, since 21h), standbys:
debian.nusuye
mds: 3/3 daemons up, 3 standby
osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped
pgs
cephfs-mirror: 2 daemons active (2 hosts)
rbd-mirror: 2 daemons active (2 hosts)
rgw: 1 daemon active (1 hosts, 1 zones)
tcmu-runner: 5 portals active (2 hosts)
data:
volumes: 3/3 healthy
pools: 25 pools, 450 pgs
objects: 17.52M objects, 62 TiB
usage: 116 TiB used, 63 TiB / 179 TiB avail
pgs: 10.889% pgs not active
15722192/79848340 objects degraded (19.690%)
197 active+undersized+degraded
183 active+clean
49 undersized+degraded+peered
15 active+clean+scrubbing
5 active+clean+scrubbing+deep
1 active+undersized
io:
client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr
root@zephir:~# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 debian 11.2T 5289G 0 0 0 0 exists,up
1 zephir 11.6T 4846G 0 0 0 0 exists
2 zephir 10.7T 5827G 0 0 2 4 exists,up
3 zephir 11.0T 5497G 0 0 0 0 exists
4 zephir 159G 1376G 0 0 2 0 exists,up
5 zephir 120G 1415G 0 0 0 0 exists,up
6 debian 11.3T 5200G 0 0 1 0 exists,up
7 zephir 305G 1230G 0 0 0 0 exists,up
8 zephir 10.9T 5594G 0 0 0 0 exists,up
9 zephir 11.9T 4616G 0 0 0 0 exists,up
10 zephir 235G 1300G 0 0 0 0 exists,up
11 zephir 223G 1312G 0 0 0 0 exists,up
12 zephir 11.0T 5479G 0 0 0 0 exists,up
13 debian 118G 631G 0 0 3 8 exists,up
14 debian 345G 1190G 0 0 0 0 exists,up
15 debian 11.4T 5079G 0 0 0 0 exists,up
16 debian 554G 3029G 0 0 3 8 exists,up
17 debian 12.5T 5874G 0 0 1 0 exists,up
root@zephir:~#
I attached the log of osd.1 to
https://filebin.net/jwvs6kuqrc7hx8id
Best Regards,
Reto
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
