- ceph-users - lists.ceph.io

Shutdown nautilus cluster, start stuck in peering
by Marc Roos 30 Apr '20

30 Apr '20

Shutdown nautilus cluster, start stuck in peering I followed some manual here to do before turning all nodes off: ceph osd set norecover ; ceph osd set norebalance ; ceph osd set nobackfill ; ceph osd set nodown ; ceph osd set pause And after reboot almost al pg's were stuck in peering after unsetting these. What to do now????

1 1

New ceph cluster - cephx disabled, now without access
by Tom Verhaeg 30 Apr '20

30 Apr '20

Hi, I just installed a new cluster with cephadm (https://docs.ceph.com/docs/master/cephadm/install/) all was working fine until I disabled cephx using the following commands: ceph config set global auth_cluster_required none ceph config set global auth_service_required none ceph config set global auth_client_required none Now I keep running into: [root@cph1]# ceph health [errno 13] RADOS permission denied (error connecting to the cluster) However, in my /etc/ceph/ceph.conf there is no mention of the configuration keys mentioned above. Same goes for the cluster folder in: /var/lib/ceph/d5d206f4-8a4f-11ea-9d04-ac1f6bf8b2d3/mgr.cph1.srvnl.nl.qwnssx/config How can I remove this setting now? Obviously all ceph * related commands run into the same RADOS permission denied. Thanks, Tom

1 0

manually configure radosgw
by Patrick Dowler 29 Apr '20

29 Apr '20

I am trying to manually create a radosgw instance for a small development installation. I was able to muddle through and get a working mon, mgr, and osd (x2), but the docs for radosgw are based on ceph-deploy which is not part of the octopus release. The host systems are all lxc/lxd containers with centos 7 and the http://download.ceph.com/rpm-octopus/el7/$basearch repos. The radosgw setip that I did was essentially creating a keyring following the steps in https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/ht… as far as I could, but I have to admit that even after getting mon, mgr, and a couple of osds working I still don't grok the keyring usage... is there a more conceptual doc that explains how they are used and what has to be on each host? ** start rgw0 (systemctl start ceph-radosgw@rgw0) ** The failure on rgw0 host (journalctl -f -u ceph-radosgw(a)rgw0.service) Apr 27 17:00:20 rgw0 systemd[1]: Started Ceph rados gateway. Apr 27 17:00:20 rgw0 radosgw[8668]: 2020-04-27T17:00:20.791+0000 7f330173d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] Apr 27 17:00:20 rgw0 radosgw[8668]: failed to fetch mon config (--no-mon-config to skip) Apr 27 17:00:20 rgw0 systemd[1]: ceph-radosgw(a)rgw0.service: main process exited, code=exited, status=1/FAILURE [root@cephadm ~]# ceph mon dump dumped monmap epoch 2 epoch 2 fsid f286f47e-7043-4dca-aeb0-03fda1910378 last_changed 2020-04-20T16:23:49.311252+0000 created 2020-04-20T16:18:21.389527+0000 min_mon_release 15 (octopus) 0: [v2:10.121.149.102:3300/0,v1:10.121.149.102:6789/0] mon.mon0 ** on mon0 I also see a lot of this in the logs: 2020-04-22T00:29:41.293+0000 7f35cb1da700 0 cephx server client.rgw0: handle_request failed to decode CephXAuthenticate: buffer::end_of_buffer Any help would be appreciated. -- Patrick Dowler Canadian Astronomy Data Centre Victoria, BC, Canada

4 4

Newbie Question: CRUSH and Librados Profiling
by Bobby 29 Apr '20

29 Apr '20

Hi, It is a newbie question. I would be really thankful if you can answer it please. I want to compile the Ceph source code. Because I want to profile *Librados* and *CRUSH* function stacks. Please verify if this is the right track I am following: - I have cloned the Ceph from Ceph git repository - I have installed the build code dependencies from script *install-deps.sh* - Because I would like to use the* gdb debug* client program later, the client program will depend on the librados library, so I must compile ceph in debug mode. Therefore I would modify the parameters of ceph cmake in *do_cmake.sh* script accordingly. - Then I compile *do_cmake* *- *In build I run* make - j 32* *- *To start the developer mode, I run *make vstart.* *- *In the developer mode I can write *READ* and *WRITE* tests...compile these tests and then use some profiling tool to call the compiled executable to profile the function stacks. Is this the correct way for *CPU profiling*? Please let me know if it is fine or is there something more also. Bobby !

1 1

Lock errors in iscsi gateway
by Simone Lazzaris 29 Apr '20

29 Apr '20

Hi; I've build two iscsi gateway for our (small) ceph cluster.The cluster is a nautilus installation, 4 nodes with 9x4TB each, and it's working fine. We mainly use it via s3 object storage interface, but I've deployed also some rbd block devices and a cephfs filesystem. Now I'm trying to connect it to my xenserver installation. Xenserver doesn't speak rados, so I've build the iscsi gateways. Right now they are self-hosted on the xenserver, with plan to move them into physical boxes if/when needed. The gateways are build on centos8, tcmu-runner just cloned from git (I think it's 1.5.2). I've been able to connect them to our six nodes xenserver cluster, and now I'm trying to use it. When I attempt a migration of a VM disk, on the new iscsi volume, I've got these messages on the logfile that I find very worrying: Apr 27 17:32:21 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:22 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:22 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:25 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:25 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:26 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:26 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:27 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:27 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:28 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:28 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:29 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:29 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:30 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:30 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:31 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:31 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:32 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:32 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:33 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:33 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:34 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:34 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:36 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. After a while the migration fails, and I keep seend the error on the logs: Apr 27 17:36:01 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:06 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:08 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:09 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:16 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:21 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:21 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:26 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:28 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:29 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:36 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Any hints? Is this a bug? -- *Simone Lazzaris* *Qcom S.p.A. a socio unico* simone.lazzaris(a)qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook[4]* [5]

3 7

Problems getting ceph-iscsi to work
by Ron Gage 29 Apr '20

29 Apr '20

Hi everyone! I have been working for the past week or so trying to get ceph-iscsi to work - Octopus release. Even just getting a single node working would be a major victory in this battle but so far, victory has proven elusive. My setup: a pair of Dell Optiplex 7010 desktops, each with 16 gig of memory and 1 boot drive (USB 3) and 3 SATA drives (500 Gb SSHD drives). No RAID controllers anywhere. Yes, I know that 3 nodes is the recommended minimum number for a production system - this isn't production (this is just seeing if the darned thing will even work). I am using Centos 8.1.1911 for the OS (4.18.0 kernel) with a basic or minimal installation (no X-Window). Single Gigabit ethernet per node. I have 2 MON and 2 Mgr installed and working, and I have a total of 6 OSDs working. I created the RBD pool (named "rbd" per the published instructions), creating it initially with 256 PGs (autoscale decided that 32 was a better choice - whatever). The cluster is green and all 6 OSDs are green (up and in). All deployment is via cephadm and all containers are running via podman. Here is where things start to fall apart. I was able to find RPM packages for targetcli and python-rtslib (called python3-rtslib) but was not able to find tcmu-runner nor ceph-iscsi packages. OK, no big deal. Time to head over to the manual install guide. I was able to build tcmu-runner, install it and apparently it is running (systemctl says it is active) so that appears to be OK. The problem is getting rbd-target-gw and rbd-target-api to work. They appear to build OK and of course, I am able to get them registered with systemd. They universally fail when trying to run them (systemctl start rbd-target-gw or systemctl start rbd-target-api). Both report failure. Looking in journalctl -xe shows no hints at all regarding why they failed (only that they did). Looking in /var/log/rbd-target-api/ show nothing at all (no files). Likewise in /var/log/rbd-target-gw/ (no files). HELP!! Now, some possibly germane questions: 1) are any other Ceph services required for ceph-iscsi to work like RADOSgw? 2) since there are no apparent packages available for ceph-iscsi, can anything be inferred to the production-readiness of the subsystem? 3) are there any known errata or missing steps in the instructions for getting ceph-iscsi to work? Thanks! Ron Gage

2 3

How to debug ssh: ceph orch host add ceph01 10.10.1.1
by Ml Ml 29 Apr '20

29 Apr '20

Hello List, i did: root@ceph01:~# ceph cephadm set-ssh-config -i /tmp/ssh_conf root@ceph01:~# cat /tmp/ssh_conf Host * User root StrictHostKeyChecking no UserKnownHostsFile /dev/null root@ceph01:~# ceph config-key set mgr/cephadm/ssh_identity_key -i /root/.ssh/id_rsa set mgr/cephadm/ssh_identity_key root@ceph01:~# ceph config-key set mgr/cephadm/ssh_identity_pub -i /root/.ssh/id_rsa.pub set mgr/cephadm/ssh_identity_pub But i get: root@ceph01:~# ceph orch host add ceph01 10.10.1.1 Error ENOENT: Failed to connect to ceph01 (10.10.1.1). Check that the host is reachable and accepts connections using the cephadm SSH key root@ceph01:~# ceph config-key get mgr/cephadm/ssh_identity_key => this shows my private key How can i debug this? root@ceph01:~# ssh 10.10.1.1 or root@ceph01:~# ssh ceph01 work without a prompt or key error. I am using 15.2.0. Thanks, Michael

3 3

CDS Pacific: Dashboard planning summary
by Lenz Grimmer 29 Apr '20

29 Apr '20

Hi all, a few weeks ago, a number of virtual Ceph Developer Summit meetings took place as a replacement for the in-person summit that was planned as part of Cephalocon in Seoul: https://pad.ceph.com/p/cds-pacific The Ceph Dashboard team also participated in these and held three video conference meetings to lay out our plans for the Pacific release. For details, please take a look at our notes at this Etherpad: https://pad.ceph.com/p/ceph-dashboard-pacific-priorities We tried to identify a few "themes", outlining individual tasks which we keep track of in the tracker.ceph.com bug tracker. The tracker issues should be used for discussing and defining the tasks at hand. A key theme for the upcoming Ceph Pacific release is the intention to further deepen and enhance the integration and support with cephadm and the orchestrator. For Ceph octopus, we tried focusing on the most common day-2 operation which is OSD management, but going forward we would like to also support the deployment and management of all other Ceph-related services that can be rolled out via cephadm and the orchestrator. In a hopefully not so distant future, we would like to be able to use the dashboard as a kind of "graphical installer", that guides the user through the entire installation deployment process of a Ceph cluster from scratch (well, almost: starting from an initial Mon+Mgr deployment). Another key theme is closing feature gaps: the various services of a Ceph cluster like RBD or RGW are constantly evolving and getting new features, so we always are trying to catch up with the latest developments there. We're also looking into enhancing our monitoring/alerting support and integration with Grafana and Prometheus. Last but not least, we always try to enhance and improve existing functionality and work on better usability and user experience. This also includes bigger refactoring work or updating key components that the dashboard depends on. As always, we would like the dashboard to be an application that Ceph administrators like and actually *want* to use to perform their jobs, so we are very keen on getting your feedback here! If there is anything you are missing or if you find any part of the dashboard to be confusing or not helpful, we'd like to know about it! Please get in touch with us to share your impressions and ideas. The best way to do this is to join the #ceph-dashboard IRC channel on OFTC or by filing a bug report via the tracker: https://tracker.ceph.com/projects/mgr/issues/new Thank you, Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

Upgrading to Octopus
by Simon Sutter 29 Apr '20

29 Apr '20

Hello everybody In octopus there are some interesting looking features, so I tried to upgrading my Centos 7 test nodes, according to: https://docs.ceph.com/docs/master/releases/octopus/ Everything went fine and the cluster is healthy. To test out the new dashboard functions, I tried to install it, but there are missing dependencies: yum install ceph-mgr-dashboard.noarch ..... --> Finished Dependency Resolution Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-routes Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-jwt Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-cherrypy Installing them with pip3 does of course make no difference, because those are yum dependencies. Does anyone know a workaround? Do I have to upgrade to Centos 8 for this to work? Thanks in advance, Simon

5 12

Re: RGW and the orphans
by Katarzyna Myrek 29 Apr '20

29 Apr '20

Hi @Eric Ivancich my cluster has some history and trash gathered over the years. Most (terabytes) is from https://tracker.ceph.com/issues/43756. I was able to reproduce the problem on my LAB and it is for sure connected with https://tracker.ceph.com/issues/43756. When you are on a version older than 14.2.8 you would need to apply lifecycle policy which tries to abort interrupted multiparts older than x days. And when the bucket index is sharded then the broken/un-cancellable MPs are born. To test it I can use s3cmd. My LAB cluster was upgraded to 14.2.8 to make sure the new version does not do cleanup automagically. Here is my procedure (I truncated my personal data): *s3cmd --access_key= --secret_key= --host= --host-bucket= multipart s3://kate-mp-issue* *s3://kate-mp-issue/Initiated Path Id2020-04-06T07:48:55.323Z s3://kate-mp-issue/bottest_20200406T074855.img 2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS* *s3cmd --access_key= --secret_key= --host= --host-bucket= abortmp s3://kate-mp-issue/bottest_20200406T074855.img 2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS* *ERROR: S3 error: 404 (NoSuchUpload)* *RGW logs:* 2020-04-29 07:24:23.126 7fb21b819700 1 ====== starting new request req=0x7fb21b8128d0 ===== 2020-04-29 07:24:23.126 7fb21b819700 1 ====== req done req=0x7fb21b8128d0 op status=0 http_status=200 latency=0s ====== 2020-04-29 07:24:23.126 7fb21b819700 1 civetweb: 0x381a000: IP - - [29/Apr/2020:07:24:22 +0000] "GET /kate-mp-issue/?location HTTP/1.1" 200 275 - - 2020-04-29 07:24:23.202 7fb21b819700 1 ====== starting new request req=0x7fb21b8128d0 ===== 2020-04-29 07:24:23.202 7fb21b819700 1 ====== req done req=0x7fb21b8128d0 op status=-2009 *http_status=404* latency=0s ====== 2020-04-29 07:24:23.202 7fb21b819700 1 civetweb: 0x381a000: Ip - - [29/Apr/2020:07:24:22 +0000] "DELETE /kate-mp-issue/bottest_20200406T074855.img?uploadId=2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS HTTP/1.1" *404* 439 - - So basically I need to remove those ghost entries from the list of interrupted multiparts and clean up objects which are left overs. As far as I understand I would need to go over every object in the pool with `rados ls`, then compare the output with `radosgw-admin bi list (done for every bucket)` and with new command `radosgw-admin radoslist` and remove objects which are on 1 but not on 2 and 3. Plus clean up the interrupted multipart list. Is that correct? @EDH - Manuel Rios <mriosfer(a)easydatahost.com> Is that your method also? I really need to clean up the terbaytes of the leftovers, because my prod cluster is getting full. And now buying anything is not an option (harsh times due to pandemy). Kind regards / Pozdrawiam, Katarzyna Myrek Kind regards / Pozdrawiam, Katarzyna Myrek wt., 28 kwi 2020 o 19:45 EDH - Manuel Rios <mriosfer(a)easydatahost.com> napisał(a): > Im prettty sure that you got the same issue than we already reported : > > https://tracker.ceph.com/issues/43756 > > Garbage and garbage stored into our OSD without be able to cleanup wasting > a lot of space. > > As you can see its solved in the new versions but... the last versión > didnt have any "scrub" or similar system to fix the garbage generated in > the past versions. > > As result , even big companies got their RGW plattform with tons of TB > wasted. > > Eric, Is there a way to ask you to develop(RGW Team) a system to clean our > rgw clusters like rgw bucket scrub? > > I talked with Cbodley and he explained how to it manually but the process > is extremely complex. > > We already calculated that at least a 25% of our rgw cluster is garbage > (100TB), and our options right now: > > - Deploy a new cluster a move rgw Users one by one with their buckets with > an external copy, hopping in the last nautilus version this not happen > again (Not usefull option and not transparent) > - Buy disk and disk waiting for a solution as External Tool (No sure to > continue this way) > - Hire external developers with knowleage of ceph and create a private > tool for that. (Developers with Ceph Core/Rgw knowleage will be no easy to > find) > > Here: ceph version 14.2.8 > > > > -----Mensaje original----- > De: Eric Ivancich <ivancich(a)redhat.com> > Enviado el: martes, 28 de abril de 2020 18:39 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl> > CC: ceph-users(a)ceph.io > Asunto: [ceph-users] Re: RGW and the orphans > > Hi Katarzyna, > > Incomplete multipart uploads are not considered orphans. > > With respect to the 404s…. Which version of ceph are you running? What > tooling are you using to list and cancel? Can you provide a console > transcript of the listing and cancelling? > > Thanks, > > Eric > > -- > J. Eric Ivancich > he / him / his > Red Hat Storage > Ann Arbor, Michigan, USA > > > On Apr 28, 2020, at 2:57 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi all > > > > I am afraid that there is even more thrash available - running > > rgw-orphan-list does not find everything. Like I still have broken > > multiparts -> when I do s3cmd multipart I get a list of > > "pending/interrupted multiparts". When I try to cancel such multipart > > I get 404. > > > > Does anyone have a method for cleanup of such things? Or even a list > > of tasks which should be run regularly on clusters with rgw ? > > > > > > Kind regards / Pozdrawiam, > > Katarzyna Myrek > > > > > > wt., 21 kwi 2020 o 09:57 Janne Johansson <icepic.dz(a)gmail.com> > napisał(a): > >> > >> Den tis 21 apr. 2020 kl 07:29 skrev Eric Ivancich <ivancich(a)redhat.com > >: > >>> > >>> Please be certain to read the associated docs in both: > >>> > >>> doc/radosgw/orphans.rst > >>> doc/man/8/rgw-orphan-list.rst > >>> > >>> so you understand the limitations and potential pitfalls. Generally > this tool will be a precursor to a large delete job, so understanding > what’s going on is important. > >>> I look forward to your report! And please feel free to post additional > questions in this forum. > >>> > >> > >> Where are those? > >> https://github.com/ceph/ceph/tree/master/doc/man/8 > >> https://github.com/ceph/ceph/tree/master/doc/radosgw > >> don't seem to contain them in master. Nor in nautilus branch or octopus. > >> > >> This whole issue feels weird, rgw (or its users) produces dead > >> fragments of mulitparts, orphans and whatnot that needs cleaning up > sooner or later and the info we get is that the old cleaner isn't meant to > be used, it hasn't worked for a long while, there is no fixed version, > perhaps there is a script somewhere with caveats. This (slightly > frustrated) issue is of course on top of "bi trim" > >> "bilog trim" > >> "mdlog trim" > >> "usage trim" > >> > >> "datalog trim" > >> > >> "sync error trim" > >> > >> "gc process" > >> > >> "reshard stale-instances rm" > >> > >> > >> > >> that we rgw admins are supposed to know when to run, how often, what > their quirks are and so on. > >> > >> > >> 'Docs' for rgw means "datalog trim" --help says "trims the datalog", > and the long version on the web would be "this operation trims the datalog" > or something that doesn't add anything more. > >> > >> > >> > >> > >> -- > >> > >> "Grumpy cat was an optimist" > >> > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io >

1 0