- ceph-users - lists.ceph.io

cephfs-data-scan orphan objects while mds active?
by Olli Rajala 22 May '24

22 May '24

Hi, I suspect that I have some orphan objects on a data pool after quite haphazardly evicting and removing a cache pool after deleting 17TB of files from cephfs. I have forward scrubbed the mds and the filesystem is in clean state. This is a production system and I'm curious if it would be safe to run cephfs-data-scan scan_extents and scan_inodes while the fs is online? Does it help if I give a custom tag while forward scrubbing and then use --filter-tag on the backward scans? ...or is there some other way to check and cleanup orphans? tnx, --------------------------- Olli Rajala - Lead TD Anima Vitae Ltd. www.anima.fi ---------------------------

2 4

Re: How network latency affects ceph performance really with NVME only storage?
by Stefan Bauer 22 May '24

22 May '24

Hi Anthony and others, thank you for your reply. To be honest, I'm not even looking for a solution, i just wanted to ask if latency affects the performance at all in my case and how others handle this ;) One of our partners delivered a solution with a latency-optimized profile for tuned-daemon. Now the latency is much better: apt install tuned tuned-adm profile network-latency # ping 10.1.4.13 PING 10.1.4.13 (10.1.4.13) 56(84) bytes of data. 64 bytes from 10.1.4.13: icmp_seq=1 ttl=64 time=0.047 ms 64 bytes from 10.1.4.13: icmp_seq=2 ttl=64 time=0.028 ms 64 bytes from 10.1.4.13: icmp_seq=3 ttl=64 time=0.025 ms 64 bytes from 10.1.4.13: icmp_seq=4 ttl=64 time=0.020 ms 64 bytes from 10.1.4.13: icmp_seq=5 ttl=64 time=0.023 ms 64 bytes from 10.1.4.13: icmp_seq=6 ttl=64 time=0.026 ms 64 bytes from 10.1.4.13: icmp_seq=7 ttl=64 time=0.024 ms 64 bytes from 10.1.4.13: icmp_seq=8 ttl=64 time=0.023 ms 64 bytes from 10.1.4.13: icmp_seq=9 ttl=64 time=0.033 ms 64 bytes from 10.1.4.13: icmp_seq=10 ttl=64 time=0.021 ms ^C --- 10.1.4.13 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9001ms rtt min/avg/max/mdev = 0.020/0.027/0.047/0.007 ms Am 21.05.24 um 15:08 schrieb Anthony D'Atri: > Check the netmask on your interfaces, is it possible that you're sending inter-node traffic up and back down needlessly? > >> On May 21, 2024, at 06:02, Stefan Bauer <sb(a)plzk.de> wrote: >> >> Dear Users, >> >> i recently setup a new ceph 3 node cluster. Network is meshed between all nodes (2 x 25G with DAC). >> Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20) >> >> The latency with ping tests between the nodes shows: >> >> # ping 10.1.3.13 >> PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data. >> 64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms >> 64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms >> 64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms >> 64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms >> 64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms >> 64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms >> 64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms >> 64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms >> 64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms >> 64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms >> 64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms >> --- 10.1.3.13 ping statistics --- >> 11 packets transmitted, 11 received, 0% packet loss, time 10242ms >> rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms >> >> >> On another cluster i have much better values, with 10G SFP+ and fibre-cables: >> >> 64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms >> 64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms >> 64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms >> 64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms >> 64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms >> 64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms >> 64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms >> 64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms >> 64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms >> 64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms >> 64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms >> 64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms >> ^C >> --- long-ipv6-ip ping statistics --- >> 53 packets transmitted, 53 received, 0% packet loss, time 53260ms >> rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms >> >> If i want best performance, does the latency difference matter at all? Should i change DAC to SFP-transceivers wwith fibre-cables to improve overall ceph performance or is this nitpicking? >> >> Thanks a lot. >> >> Stefan >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Mit freundlichen Grüßen Stefan Bauer Schulstraße 5 83308 Trostberg 0179-1194767

2 3

Cephfs over internet
by Marcus 22 May '24

22 May '24

Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode. Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0] Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong? Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs. From my understanding from the documenation this would be the way to set this up with ceph exposed to internet. Is there something that we are missing or something that would make the setup more secure? Many thanks in advance Marcus

11 18

Problems adding a new host via orchestration.
by Gary Molenkamp 21 May '24

21 May '24

Happy Friday all. I was hoping someone could point me in the right direction or clarify any limitations that could be impacting an issue I am having. I'm struggling to add a new set of hosts to my ceph cluster using cephadm and orchestration. When trying to add a host: "ceph orch host add <hostname> 172.31.102.41 --labels _admin" returns: "Error EINVAL: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory" I've verified that the ceph ssh key works to the remote host, host's name matches that returned from `hostname`, python3 is installed, and "/usr/sbin/cephadm prepare-host" on the new hosts returns "host is ok". In addition, the cluster ssh key works between hosts and the existing hosts are able to ssh in using the ceph key. The existing ceph cluster is Pacific release using docker based containerization on RockyLinux8 base OS. The new hosts are RockyLinux9 based, with the cephadm being installed from Quincy release: ./cephadm add-repo --release quincy ./cephadm install I did try installing cephadm from the Pacific release by changing the repo to el8, but that did not work either. Is there a limitation is mixing RL8 and RL9 container hosts under Pacific? Does this same limitation exist under Quincy? Is there a python version dependency? The reason for RL9 on the new hosts is to stage upgrading the OS's for the cluster. I did this under Octopus for moving from Centos7 to RL8. Thanks and I appreciate any feedback/pointers. Gary I've added the log trace here in case that helps (from `ceph log last cephadm`) 2024-02-02T14:22:32.610048+0000 mgr.storage01.oonvfl (mgr.441023307) 4957871 : cephadm [ERR] Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1524, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1370, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 35, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 46, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 133, in makegateway io = gateway_io.create_io(spec, execmodel=self.execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 121, in create_io io = Popen2IOMaster(args, execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 21, in __init__ self.popen = p = execmodel.PopenPiped(args) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 184, in PopenPiped return self.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) File "/lib64/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/lib64/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1528, in _remote_connection raise execnet.gateway_bootstrap.HostNotFound(msg) execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2709, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2574, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1517, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1498, in _check_valid_addr error_ok=True, no_fsid=True) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1326, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1558, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 9

rbd-mirror failed to query services: (13) Permission denied
by Stefan Kooman 21 May '24

21 May '24

Hi, We are testing rbd-mirroring. There seems to be a permission error with the rbd-mirror user. Using this user to query the mirror pool status gives: failed to query services: (13) Permission denied And results in the following output: health: UNKNOWN daemon health: UNKNOWN image health: OK images: 3 total 2 replaying 1 stopped So, this command: rbd --id rbd-mirror mirror pool status rbd So basically the health and daemon health cannot be obtained due to permission errors, but status about images can. When the command is run with admin permissions the health and daemon health are returned without issue. I tested this on Reef 18.2.2. Is this expected behavior? If not, I will create a tracker ticket for it. Gr. Stefan

3 5

dkim on this mailing list
by Marc 21 May '24

21 May '24

Just to confirm if I am messing up my mailserver configs. But currently all messages from this mailing list should generate a dkim pass status?

2 1

Please discuss about Slow Peering
by 서민우 21 May '24

21 May '24

Env: - OS: Ubuntu 20.04 - Ceph Version: Octopus 15.0.0.1 - OSD Disk: 2.9TB NVMe - BlockStorage (Replication 3) Symptom: - Peering when OSD's node up is very slow. Peering speed varies from PG to PG, and some PG may even take 10 seconds. But, there is no log for 10 seconds. - I checked the effect of client VM's. Actually, Slow queries of mysql occur at the same time. There are Ceph OSD logs of both Best and Worst. Best Peering Case (0.5 Seconds) 2024-04-11T15:32:44.693+0900 7f108b522700 1 osd.7 pg_epoch: 27368 pg[6.8] state<Start>: transitioning to Primary 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27371 pg[6.8] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27371 pg[6.8] start_peering_interval up [7,6,11] -> [6,11], acting [7,6,11] -> [6,11], acting_primary 7 -> 6, up_primary 7 -> 6, role 0 -> -1, features acting 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27377 pg[6.8] state<Start>: transitioning to Primary 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27377 pg[6.8] start_peering_interval up [6,11] -> [7,6,11], acting [6,11] -> [7,6,11], acting_primary 6 -> 7, up_primary 6 -> 7, role -1 -> 0, features acting Worst Peering Case (11.6 Seconds) 2024-04-11T15:32:45.169+0900 7f108b522700 1 osd.7 pg_epoch: 27377 pg[30.20] state<Start>: transitioning to Stray 2024-04-11T15:32:45.169+0900 7f108b522700 1 osd.7 pg_epoch: 27377 pg[30.20] start_peering_interval up [0,1] -> [0,7,1], acting [0,1] -> [0,7,1], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 2024-04-11T15:32:46.173+0900 7f108b522700 1 osd.7 pg_epoch: 27378 pg[30.20] state<Start>: transitioning to Stray 2024-04-11T15:32:46.173+0900 7f108b522700 1 osd.7 pg_epoch: 27378 pg[30.20] start_peering_interval up [0,7,1] -> [0,7,1], acting [0,7,1] -> [0,1], acting_primary 0 -> 0, up_primary 0 -> 0, role 1 -> -1, features acting 2024-04-11T15:32:57.794+0900 7f108b522700 1 osd.7 pg_epoch: 27390 pg[30.20] state<Start>: transitioning to Stray 2024-04-11T15:32:57.794+0900 7f108b522700 1 osd.7 pg_epoch: 27390 pg[30.20] start_peering_interval up [0,7,1] -> [0,7,1], acting [0,1] -> [0,7,1], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting *I wish to know about* - Why some PG's take 10 seconds until Peering finishes. - Why Ceph log is quiet during peering. - Is this symptom intended in Ceph. *And please give some advice,* - Is there any way to improve peering speed? - Or, Is there a way to not affect the client when peering occurs? P.S - I checked the symptoms in the following environments. -> Octopus Version, Reef Version, Cephadm, Ceph-Ansible

3 7

How network latency affects ceph performance really with NVME only storage?
by Stefan Bauer 21 May '24

21 May '24

Dear Users, i recently setup a new ceph 3 node cluster. Network is meshed between all nodes (2 x 25G with DAC). Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20) The latency with ping tests between the nodes shows: # ping 10.1.3.13 PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data. 64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms 64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms 64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms 64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms 64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms 64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms 64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms 64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms 64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms 64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms 64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms --- 10.1.3.13 ping statistics --- 11 packets transmitted, 11 received, 0% packet loss, time 10242ms rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms On another cluster i have much better values, with 10G SFP+ and fibre-cables: 64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms 64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms 64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms 64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms 64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms 64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms 64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms 64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms 64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms 64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms 64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms 64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms ^C --- long-ipv6-ip ping statistics --- 53 packets transmitted, 53 received, 0% packet loss, time 53260ms rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms If i want best performance, does the latency difference matter at all? Should i change DAC to SFP-transceivers wwith fibre-cables to improve overall ceph performance or is this nitpicking? Thanks a lot. Stefan

1 0

lost+found is corrupted.
by Malcolm Haak 21 May '24

21 May '24

Hi all, I've almost got my ceph back to normal after a triple drive failure. But it seems my lost+found folder is corrupted. I've followed the process in https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#disaster-… However doing an online scrub, as there is still other damage, fails as it appears my lost+found inode is corrupted. Any advice is appreciated The only thing I can think of is at the beginning of one of the steps it asks if I want to re-create something and, from what I could read from other emails to the list, I thought answering no was the correct answer but I am now wondering if I did want to recreate those entries. (specifically it asks at the cephfs-data-scan init point) Thanks in advance, ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Arch linux 3 node cluster 33 OSD's

1 0

CEPH quincy 17.2.5 with Erasure Code
by Andrea Martra 20 May '24

20 May '24

Hi everyone, I'm managing a Ceph Quincy 17.2.5 cluster, waiting to upgrade it to version 17.2.7, composed and configured as follows: - 16 identical nodes 256 GB RAM, 32 CPU Cores (64 threads), 12 x rotary HDD (BLOCK) + 4 x Sata SSD (RocksDB/WAL) - Erasure Code 11+4 (Jerasure) - 10 x S3 RGW on dedicated nodes (5 physical nodes) - 3 x full SSD dedicated nodes for replicated S3 pools - 2 x 10 Gbit Public network (LACP) + 2 x 10 Gbit cluster network (LACP) - On all nodes: Ubuntu 20.04.4 LTS Operating System updated - Ceph deployed on containers on Docker CE (docker-ce 5:20.10.17~3-0~ubuntu-focal). All pools, except the EC data pool, are configured with replication 3 and stored in dedicated SSD devices on 3 dedicated nodes to guarantee the necessary performance. We encountered a constant and random problem relating to the availability of access to bucket data and many slow_ops relating to rotary OSDs (data-pools) not caused by saturation of physical devices nor by the availability of CPU/RAM on all nodes. It sometimes happens that slow_ops caused by some requests to some PCs in a random way are reported. The cluster is currently in the recovery/rebalance phase for the reconstruction of 3 HDDs that we had to recreate from scratch (all 3 HDDs are physically on the same node). By doing some analysis of the events we verified the following from the status of some OSDs impacted by slow_ops: 20/05/2024 10:19 • "description": "osd_op(client.186021790.0:57620 29.258s0 29:1a5928ea:::31497ca8-e7d6-4e53-b150-91f9ac02ac67.246100.6329_storage%2ffirstMemories%2f1010543%2f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e481205)", "initiated_at": "2024-05-16T15:03:06.015956+0000", "age": 963.83795990199997, "duration": 963.83819621700002, "type_data": { "flag_point": "delayed", "client_info": { "client": "client.186021790", "client_addr": "10.151.11.11:0/3913909849", "tid": 57620 }, "events": [ { "event": "initiated", "time": "2024-05-16T15:03:06.015956+0000", "duration": 0 }, { "event": "throttled", "time": "2024-05-16T15:03:06.015956+0000", "duration": 0 }, { "event": "header_read", "time": "2024-05-16T15:03:06.015954+0000", "duration": 4294967295.9999986 }, { "event": "all_read", "time": "2024-05-16T15:03:06.015961+0000", "duration": 7.2300000000000002e-06 }, { "event": "dispatched", "time": "2024-05-16T15:03:06.015962+0000", "duration": 1.063e-06 }, { "event": "queued_for_pg", "time": "2024-05-16T15:03:06.015966+0000", "duration": 3.332e-06 }, { "event": "reached_pg", "time": "2024-05-16T15:03:06.015992+0000", "duration": 2.6078e-05 }, { "event": "waiting for readable", "time": "2024-05-16T15:03:06.016002+0000", "duration": 1.0348e-05 } ] } } ], "num_ops": 6 } ########### { "event": "reached_pg", "time": "2024-05-16T12:43:11.694220+0000", "duration": 480.97258642200001 }, In essence the operations remain suspended in this condition: "event": "waiting for readable" trying to access some PGs. We intend to update the version as soon as the recovery/rebalance is completed. Does anyone have any idea what checks I could do to analyze the problem more thoroughly? I can't define whether the problem could be the use of the EC or whether the data written in some buckets is in a "non-standard" condition, which causes the access to wait for some reason. Thank you all for your kindness. Greetings Andrea Martra -- -- Andrea Martra +39 393 9048451

1 0