- ceph-users - lists.ceph.io

Phantom hosts
by Tim Holloway 09 Jul '24

09 Jul '24

It's getting worse. As many may be aware, the venerable CentOS 7 OS is hitting end-of-life in a matter of days. The easiest way to upgrade my serves has been to simply create an alternate disk with the new OS, turn my provisioning system loose on it, yank the old OS system disk and jack in the new one. However, Ceph is another matter. For that part, the simplest thing to do is to destroy the Ceph node(s) on the affected box, do the OS upgrade, then re-create the nodes. But now I have even MORE strays. The OSD on my box lives on in Ceph in the dashboard host view even though the documented removal procedures were followed and the VM itself was destroyed. Further, this last node is an RGW node and I cannot remove it from the RGW configuration. It not only shows on the dashboard, it also lists as still active on the command line and as entries in the config database no matter what I do. I really need some solution to this, as it's a major chokepoint in the upgrade process Tim

3 2

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

CephFS snapshots and kernel cephfs continuous I/O
by caskd 08 Jul '24

08 Jul '24

Hello ceph users and devs, I've set up CephFS as a storage which ingests data for multiple hours and snap-schedule hourly. This however causes problems in the case of clients which cannot handle the EAGAIN (Resource busy/temporary unavailable) the kernel gives back during snapshotting. Is there any plans or option to change this behaviour to just wait until the snapshot is done and continue IO as nothing happened? Thanks in advance. -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

2 3

Slow osd ops on large arm cluster
by Adam Prycki 08 Jul '24

08 Jul '24

Hello, we are having issues with slow ops on our large ARM hpc ceph cluster. Cluster runs on 18.2.0 and ubutnu 20.04 MONs, MGRs and MDSs had to be moved to intel servers because of poor single core performance on our arm servers. Our main cephfs data pool is on 54 serwers in 9 racks with 1458 HDDs in total. (OSDs without block.db on ssd) Cephfs data pool is configured as erasure coded pool with k=6 m=2 and rack level replication. Pool has about 16k PGs with average pg per osd at ~90. We have had good experience with EC cephfs on 3,5 times smaller intel ceph cluster. But this arm deployment is becoming problematic. We started experiencing issues since one of the users started to generate sequential RW traffic at at about 5GiB/s. Single OSD with slow ops was enough to create laggy PG and crash application generating this traffic. We've even had issue where osd with slow ops was lagged for 6 hours and required manual restart. Now we are experiencing slow ops even at much lower read only traffic ~400MiB/s Here is an example of slow ops on OSD: { "ops": [ { "description": "osd_op(client.255949991.0:92728602 4.d22s0 4:44b3390a:::1000b640ddc.0000039b:head [read 3633152~8192] snapc 0=[] ondisk+read+known_if_redirected e1117246)", "initiated_at": "2024-07-08T10:19:58.469537+0000", "age": 507.242936848, "duration": 507.24298854800003, "type_data": { "flag_point": "started", "client_info": { "client": "client.255949991", "client_addr": "x.x.x.x:0/887459214", "tid": 92728602 }, "events": [ { "event": "initiated", "time": "2024-07-08T10:19:58.469537+0000", "duration": 0 }, { "event": "throttled", "time": "2024-07-08T10:19:58.469537+0000", "duration": 0 }, { "event": "header_read", "time": "2024-07-08T10:19:58.469535+0000", "duration": 4294967295.9999981 }, { "event": "all_read", "time": "2024-07-08T10:19:58.469571+0000", "duration": 3.5859999999999999e-05 }, { "event": "dispatched", "time": "2024-07-08T10:19:58.469573+0000", "duration": 2.08e-06 }, { "event": "queued_for_pg", "time": "2024-07-08T10:19:58.469586+0000", "duration": 1.2721000000000001e-05 }, { "event": "reached_pg", "time": "2024-07-08T10:19:58.485132+0000", "duration": 0.015546048999999999 }, { "event": "started", "time": "2024-07-08T10:19:58.485147+0000", "duration": 1.5160000000000001e-05 } ] } }, HDD with this OSD is not busy. Arm cores on these servers are slow but no process reaches full 100% core usage. I think we may have the same issue as one described here: https://www.mail-archive.com/ceph-users@ceph.io/msg13273.html I've tried to reduce osd_pool_default_read_lease_ratio form 0.8 to 0.2 I've tried to reduce osd_heartbeat_grace from 20 to 10. It should lower read_lease_interval from 16 to 2 but it didn't help. Still see a lot of slow ops. Could you give me tips what I could tune to fix this issue? Could this be an issue with large number of EC PGs on large cluster with weak CPUs? Best regards Adam Prycki

1 0

Re: pg's stuck activating on osd create
by Eugen Block 08 Jul '24

08 Jul '24

Hi, it depends a bit on the actual OSD layout on the node and your procedure, but there's a chance you might have hit the overdose. But I would expect it to be logged in the OSD logs, two years ago in a Nautilus cluster the message looked like this: > maybe_wait_for_max_pg withhold creation of pg ... According to github in 16.2.15 it could look like this: > maybe_wait_for_max_pg hit max pg, dropping ... But I'm not sure, I haven't seen that in newer clusters (yet). Regards, Eugen Zitat von Richard Bade <hitrich(a)gmail.com>: > Hi Everyone, > I had an issue last night when I was bringing online some osds that I > was rebuilding. When the osds created and came online 15pgs got stuck > in activating. The first osd (osd.112) seemed to come online ok, but > the second one (osd.113) triggered the issue. All the pgs in > activating included osd.112 in the pg map and I resolved it by doing > pg-upmap-items to map the pg back from osd.112 to where it currently > was but it was painful having 10min of stuck i/o os an rbd pool with > vm's running. > > Some details about the cluster: > Pacific 16.2.15, upgraded from Nautilus fairly recently and Luminos > back in the past. All osds were rebuilt on bluestore in Nautilus, as > were the mons. > The disks in question are Intel DC P4510 8TB nvme. I'm rebuilding them > as I had previously had 4x2TB osd's per disk and now wanted to > consolidate down to one osd per disk. > There's around 300 osd's in the pool with 16384 pgs which means that > the 2TB osds had 157pgs on them. However this means that the 8TB osds > have 615pgs on them and I'm wondering if this is maybe the cause of > the problem. > > There are no warnings about too many pgs per osd in the logs or ceph status. > I have the default value of 250 for mon_max_pg_per_osd and default > value of 3.0 for osd_max_pg_per_osd_hard_ratio. > > My plan is to reduce the number of pgs in the pool but I want to > understand and prove what happened here. > Is it likely I've hit pg overdose protection? If I have, how would I > tell as I can't see anything in the cluster logs. > > Thanks, > Rich > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Sanity check
by Adam Witwicki 08 Jul '24

08 Jul '24

Hello, Can someone please let me know what failure domain my erasure code pool is, osd or host? We thought we testing this by turning off 2 hosts, we have had one host offline recently and the cluster was still serving clients - did we get lucky? ceph osd pool get <pool> crush_rule crush_rule: ecpool ceph osd pool get <pool> erasure_code_profile erasure_code_profile: 6-2 rule ecpool { id 3 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } ceph osd erasure-code-profile get 6-2 crush-device-class=hdd crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 octopus Regards Adam

2 1

Pacific 16.2.15 `osd noin`
by Zakhar Kirpichenko 08 Jul '24

08 Jul '24

Hi, I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`: cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`: services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin # ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000 Is this expected behavior? Do I misunderstand the purpose of the `noin` option? Best regards, Zakhar

5 7

Re: CephFS constant high write I/O to the metadata pool
by Olli Rajala 04 Jul '24

04 Jul '24

Hi - mostly as a note to future me and if anyone else looking for the same issue... I finally solved this a couple of months ago. No idea what is wrong with Ceph but the root cause that was triggering this MDS issue was that I had several workstations and a couple servers where the updatedb of "locate" was getting run by daily cron exactly the same time every night causing high momentary strain on the MDS which then somehow screwed up the metadata caching and flushing creating this cumulative write io. The thing to note here is that there's a difference with "locate" and "mlocate" packages. The default config (on Ubuntu atleast) of updatedb for "mlocate" does skip scanning cephfs filesystems but not so for "locate" which happily ventures onto all of your cephfs mounts :| --------------------------- Olli Rajala - Lead TD Anima Vitae Ltd. www.anima.fi --------------------------- On Wed, Dec 14, 2022 at 7:41 PM Olli Rajala <olli.rajala(a)anima.fi> wrote: > Hi, > > One thing I now noticed in the mds logs is that there's a ton of entries > like this: > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d345,d346] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d345,d346] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d343,d344] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d343,d344] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d341,d342] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d341,d342] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d33f,d340] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d33f,d340] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache projecting to > [d33d,d33e] n(v1638 rc2022-12-11T18:20:49.317400+0200 b787972591 > 694=484+210) > 2022-12-11T18:20:49.321+0200 7fdd0edde700 20 mds.0.cache result > [d33d,d33e] n(v1638 rc2022-12-11T18:20:49.321400+0200 b787972591 > 695=484+211) > > ...and after dropping the caches considerably less of those - normal, > abnormal, typical, atypical? ...or is that just something that starts > happening after the cache gets filled? > > Tnx, > --------------------------- > Olli Rajala - Lead TD > Anima Vitae Ltd. > www.anima.fi > --------------------------- > > > On Sun, Dec 11, 2022 at 9:07 PM Olli Rajala <olli.rajala(a)anima.fi> wrote: > >> Hi, >> >> I'm still totally lost with this issue. And now lately I've had a couple >> of incidents where the write bw has suddenly jumped to even crazier levels. >> See the graph here: >> https://gist.github.com/olliRJL/3e97e15a37e8e801a785a1bd5358120d >> >> The points where it drops to something manageable again are when I have >> dropped the mds caches. Usually after the drop there is steady rise but now >> these sudden jumps are something new and even more scary :E >> >> Here's a fresh 2sec level 20 mds log: >> https://gist.github.com/olliRJL/074bec65787085e70db8af0ec35f8148 >> >> Any help and ideas greatly appreciated. Is there any tool or procedure to >> safely check or rebuild the mds data? ...if this behaviour could be caused >> by some hidden issue with the data itself. >> >> Tnx, >> --------------------------- >> Olli Rajala - Lead TD >> Anima Vitae Ltd. >> www.anima.fi >> --------------------------- >> >> >> On Fri, Nov 11, 2022 at 9:14 AM Venky Shankar <vshankar(a)redhat.com> >> wrote: >> >>> On Fri, Nov 11, 2022 at 3:06 AM Olli Rajala <olli.rajala(a)anima.fi> >>> wrote: >>> > >>> > Hi Venky, >>> > >>> > I have indeed observed the output of the different sections of perf >>> dump like so: >>> > watch -n 1 ceph tell mds.`hostname` perf dump objecter >>> > watch -n 1 ceph tell mds.`hostname` perf dump mds_cache >>> > ...etc... >>> > >>> > ...but without any proper understanding of what is a normal rate for >>> some number to go up it's really difficult to make anything from that. >>> > >>> > btw - is there some convenient way to capture this kind of temporal >>> output for others to view. Sure, I could just dump once a second to a file >>> or sequential files but is there some tool or convention that is easy to >>> look at and analyze? >>> >>> Not really - you'd have to do it yourself. >>> >>> > >>> > Tnx, >>> > --------------------------- >>> > Olli Rajala - Lead TD >>> > Anima Vitae Ltd. >>> > www.anima.fi >>> > --------------------------- >>> > >>> > >>> > On Thu, Nov 10, 2022 at 8:18 AM Venky Shankar <vshankar(a)redhat.com> >>> wrote: >>> >> >>> >> Hi Olli, >>> >> >>> >> On Mon, Oct 17, 2022 at 1:08 PM Olli Rajala <olli.rajala(a)anima.fi> >>> wrote: >>> >> > >>> >> > Hi Patrick, >>> >> > >>> >> > With "objecter_ops" did you mean "ceph tell mds.pve-core-1 ops" >>> and/or >>> >> > "ceph tell mds.pve-core-1 objecter_requests"? Both these show very >>> few >>> >> > requests/ops - many times just returning empty lists. I'm pretty >>> sure that >>> >> > this I/O isn't generated by any clients - I've earlier tried to >>> isolate >>> >> > this by shutting down all cephfs clients and this didn't have any >>> >> > noticeable effect. >>> >> > >>> >> > I tried to watch what is going on with that "perf dump" but to be >>> honest >>> >> > all I can see is some numbers going up in the different sections :) >>> >> > ...don't have a clue what to focus on and how to interpret that. >>> >> > >>> >> > Here's a perf dump if you or anyone could make something out of >>> that: >>> >> > https://gist.github.com/olliRJL/43c10173aafd82be22c080a9cd28e673 >>> >> >>> >> You'd need to capture this over a period of time to see what ops might >>> >> be going through and what the mds is doing. >>> >> >>> >> > >>> >> > Tnx! >>> >> > o. >>> >> > >>> >> > --------------------------- >>> >> > Olli Rajala - Lead TD >>> >> > Anima Vitae Ltd. >>> >> > www.anima.fi >>> >> > --------------------------- >>> >> > >>> >> > >>> >> > On Fri, Oct 14, 2022 at 8:32 PM Patrick Donnelly < >>> pdonnell(a)redhat.com> >>> >> > wrote: >>> >> > >>> >> > > Hello Olli, >>> >> > > >>> >> > > On Thu, Oct 13, 2022 at 5:01 AM Olli Rajala <olli.rajala(a)anima.fi> >>> wrote: >>> >> > > > >>> >> > > > Hi, >>> >> > > > >>> >> > > > I'm seeing constant 25-50MB/s writes to the metadata pool even >>> when all >>> >> > > > clients and the cluster is idling and in clean state. This >>> surely can't >>> >> > > be >>> >> > > > normal? >>> >> > > > >>> >> > > > There's no apparent issues with the performance of the cluster >>> but this >>> >> > > > write rate seems excessive and I don't know where to look for >>> the >>> >> > > culprit. >>> >> > > > >>> >> > > > The setup is Ceph 16.2.9 running in hyperconverged 3 node core >>> cluster >>> >> > > and >>> >> > > > 6 hdd osd nodes. >>> >> > > > >>> >> > > > Here's typical status when pretty much all clients are idling. >>> Most of >>> >> > > that >>> >> > > > write bandwidth and maybe fifth of the write iops is hitting the >>> >> > > > metadata pool. >>> >> > > > >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > root@pve-core-1:~# ceph -s >>> >> > > > cluster: >>> >> > > > id: 2088b4b1-8de1-44d4-956e-aa3d3afff77f >>> >> > > > health: HEALTH_OK >>> >> > > > >>> >> > > > services: >>> >> > > > mon: 3 daemons, quorum pve-core-1,pve-core-2,pve-core-3 >>> (age 2w) >>> >> > > > mgr: pve-core-1(active, since 4w), standbys: pve-core-2, >>> pve-core-3 >>> >> > > > mds: 1/1 daemons up, 2 standby >>> >> > > > osd: 48 osds: 48 up (since 5h), 48 in (since 4M) >>> >> > > > >>> >> > > > data: >>> >> > > > volumes: 1/1 healthy >>> >> > > > pools: 10 pools, 625 pgs >>> >> > > > objects: 70.06M objects, 46 TiB >>> >> > > > usage: 95 TiB used, 182 TiB / 278 TiB avail >>> >> > > > pgs: 625 active+clean >>> >> > > > >>> >> > > > io: >>> >> > > > client: 45 KiB/s rd, 38 MiB/s wr, 6 op/s rd, 287 op/s wr >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > >>> >> > > > Here's some daemonperf dump: >>> >> > > > >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > root@pve-core-1:~# ceph daemonperf mds.`hostname -s` >>> >> > > > >>> >> > > >>> ----------------------------------------mds----------------------------------------- >>> >> > > > --mds_cache--- ------mds_log------ -mds_mem- >>> -------mds_server------- >>> >> > > mds_ >>> >> > > > -----objecter------ purg >>> >> > > > req rlat fwd inos caps exi imi hifc crev cgra ctru cfsa >>> cfa hcc >>> >> > > hccd >>> >> > > > hccr prcr|stry recy recd|subm evts segs repl|ino dn |hcr >>> hcs hsr cre >>> >> > > > cat |sess|actv rd wr rdwr|purg| >>> >> > > > 40 0 0 767k 78k 0 0 0 1 6 1 0 >>> 0 5 5 >>> >> > > > 3 7 |1.1k 0 0 | 17 3.7k 134 0 |767k 767k| 40 5 >>> 0 0 >>> >> > > > 0 |110 | 4 2 21 0 | 2 >>> >> > > > 57 2 0 767k 78k 0 0 0 3 16 3 0 >>> 0 11 11 >>> >> > > > 0 17 |1.1k 0 0 | 45 3.7k 137 0 |767k 767k| 57 8 >>> 0 0 >>> >> > > > 0 |110 | 0 2 28 0 | 4 >>> >> > > > 57 4 0 767k 78k 0 0 0 4 34 4 0 >>> 0 34 33 >>> >> > > > 2 26 |1.0k 0 0 |134 3.9k 139 0 |767k 767k| 57 13 >>> 0 0 >>> >> > > > 0 |110 | 0 2 112 0 | 19 >>> >> > > > 67 3 0 767k 78k 0 0 0 6 32 6 0 >>> 0 22 22 >>> >> > > > 0 32 |1.1k 0 0 | 78 3.9k 141 0 |767k 768k| 67 4 >>> 0 0 >>> >> > > > 0 |110 | 0 2 56 0 | 2 >>> >> > > > >>> >> > > >>> --------------------------------------------------------------------------------------------------- >>> >> > > > Any ideas where to look at? >>> >> > > >>> >> > > Check the perf dump output of the mds: >>> >> > > >>> >> > > ceph tell mds.<fs_name>:0 perf dump >>> >> > > >>> >> > > over a period of time to identify what's going on. You can also >>> look >>> >> > > at the objecter_ops (another tell command) for the MDS. >>> >> > > >>> >> > > -- >>> >> > > Patrick Donnelly, Ph.D. >>> >> > > He / Him / His >>> >> > > Principal Software Engineer >>> >> > > Red Hat, Inc. >>> >> > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>> >> > > >>> >> > > >>> >> > _______________________________________________ >>> >> > ceph-users mailing list -- ceph-users(a)ceph.io >>> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> > >>> >> >>> >> >>> >> -- >>> >> Cheers, >>> >> Venky >>> >> >>> >>> >>> -- >>> Cheers, >>> Venky >>> >>>

3 3

[RGW] Bucket synchronization in multi-zonegroup
by Huy Nguyen 04 Jul '24

04 Jul '24

Hi community, I tried "radosgw-admin sync group" but it seems to only work between zones within the same zonegroup. So does radosgw support sync a bucket between regions/zonegroups? Or could it be done in any other way? Thanks

1 0

multipart file in broken state
by Matthew Darwin 04 Jul '24

04 Jul '24

When trying to clean up multi-part files, I get the following error: $ rclone backend cleanup s3:bucket-name 2024/07/04 02:42:19 ERROR : S3 bucket bucket-name: failed to remove pending multipart upload for bucket "bucket-name" key "0a424a15dee6fecb241130e9e4e49d99ed120f05/outputs/0121490000-0121491000.output.zst" dated 2024-06-20 09:28:34.112 +0000 UTC (329h13m44.854458863s ago): NoSuchUpload: status code: 404, request id: tx00000bd994059ba3d77af-0066860c0a-ce710f2-s3-1, host id: Why do files get in this state? How should they be fixed? Any hints? I have multiple files in the same state. This is ceph quincy.

1 0