June 2020 - ceph-users - lists.ceph.io

Re: rgw : unable to find part(s) of aborted multipart upload of [object].meta

by EDH - Manuel Rios

Hi Amit, Yes in the non-ec pool there’re like 600 files .meta but i dont know if its safe move to data pool. Does anyone know if there is a way to generate a synthetic .meta only so that the delete command is capable of deleting the file? Regards Manuel De: Amit Ghadge <amitg.b14(a)gmail.com> Enviado el: lunes, 29 de junio de 2020 6:14 Para: EDH - Manuel Rios <mriosfer(a)easydatahost.com> Asunto: Re: [ceph-users] rgw : unable to find part(s) of aborted multipart upload of [object].meta You can also check in default.rgw.buckets.non-ec pool for unmerged multipart or bucket indexes that need to be fix. On Mon, Jun 29, 2020 at 5:56 AM EDH - Manuel Rios <mriosfer(a)easydatahost.com<mailto:mriosfer@easydatahost.com>> wrote: Hi Dev's With the failures of the previous versions in the buckets due to the shardings. We have started a copy of the buckets to new buckets to clean our ceph cluster. After synchronizing the bucket with the AWS cli, we are in the phase of deleting the old buckets. We have tried unsuccessfully: radosgw-admin bucket rm --bucket = XXXX --purge-objects Result is a loop that shows with the debug: "NOTE: unable to find part (s) of aborted multipart upload of [object] .meta" After seeing this failure we have tried to clean up using "rados". For this we have listed all the objects belonging to the pool bucket using its marker_id. Once done, we send them a script to delete massively, with "rados -p [rgw-pool.data] rm [object]" The result of all is similar to the following: rados -p default.rgw.buckets.data rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc error removing default.rgw.buckets.data>48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc: (2) No such file or directory Any idea or better way to clean the cluster of these objects? Our estimates are 100TB of wrong objects. Regards Manuel _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

3 years, 9 months

1
0
0 0

mgr log shows a lot of ms_handle_reset messages

by XuYun

Hi, We are running Ceph nautilus (14.2.10 now), every mgr reports a ‘ms_handle_reset’ message every second. Would it be a normal behavior? 2020-06-28 16:35:43.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:44.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:45.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:46.581 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:47.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 Br, Xu Yun

3 years, 9 months

2
4
0 0

rgw : unable to find part(s) of aborted multipart upload of [object].meta

by EDH - Manuel Rios

Hi Dev's With the failures of the previous versions in the buckets due to the shardings. We have started a copy of the buckets to new buckets to clean our ceph cluster. After synchronizing the bucket with the AWS cli, we are in the phase of deleting the old buckets. We have tried unsuccessfully: radosgw-admin bucket rm --bucket = XXXX --purge-objects Result is a loop that shows with the debug: "NOTE: unable to find part (s) of aborted multipart upload of [object] .meta" After seeing this failure we have tried to clean up using "rados". For this we have listed all the objects belonging to the pool bucket using its marker_id. Once done, we send them a script to delete massively, with "rados -p [rgw-pool.data] rm [object]" The result of all is similar to the following: rados -p default.rgw.buckets.data rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc error removing default.rgw.buckets.data>48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc: (2) No such file or directory Any idea or better way to clean the cluster of these objects? Our estimates are 100TB of wrong objects. Regards Manuel

3 years, 9 months

1
0
0 0

fault tolerant about erasure code pool

by Zhenshi Zhou

Hi all, I'm going to deploy a cluster with erasure code pool for cold storage. There are 3 servers for me to set up the cluster, 12 OSDs on each server. Does that mean the data is secure while 1/3 OSDs of the cluster is down, or only 2 of the OSDs is down , if I set the ec profile with k=4 and m=2. Thanks

3 years, 9 months

7
12
0 0

High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

by Benoît Knecht

Hi, We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs: - TOSHIBA MG07ACA14TE [1] - HGST HUH721212ALE604 [2] They're all bluestore OSDs with no separate DB+WAL and part of the same pool. We noticed that while the HGST OSDs have a commit latency of about 15ms, the Toshiba OSDs hover around 150ms (these values come from the `ceph_osd_commit_latency_ms` metric in Prometheus). On paper, it seems like those drives have very similar specs, so it's not clear to me why we're seeing such a large difference when it comes to commit latency. Has anyone had any experience with those Toshiba drives? Or looking at the specs, do you spot anything suspicious? And if you're running a Ceph cluster with various disk brands/models, have you ever noticed some of them standing out when looking at `ceph_osd_commit_latency_ms`? Thanks in advance for your feedback. Cheers, -- Ben [1]: https://toshiba.semicon-storage.com/content/dam/toshiba-ss/asia-pacific/doc… [2]: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/p…

3 years, 9 months

15
27
0 0

bluestore_throttle_bytes

by Seena Fallah

Hi all. Can someone explain or reference a doc about the bluestore_throttle_bytes option? I don't find any docs for this config. Thanks.

3 years, 9 months

1
0
0 0

CephFS: What is the maximum number of files per directory

by Martin Palma

Hi, What is the maximum number of files per directory? I could find the answer in the docs. Best, Martin

3 years, 9 months

4
3
0 0

re Centos8 / octopus installation question

by Mazzystr

Can anyone explain why ktdryer's dev repo is still landing on production installs? [root@centos8 ~]# ./cephadm add-repo --release octopus INFO:root:Writing repo to /etc/yum.repos.d/ceph.repo... INFO:cephadm:Enabling EPEL... INFO:cephadm:Enabling supplementary copr repo ktdreyer/ceph-el8... [root@centos8 ~]# ls /etc/yum.repos.d/ CentOS-AppStream.repo CentOS-Debuginfo.repo CentOS-Media.repo ceph.repo epel.repo CentOS-Base.repo CentOS-Extras.repo CentOS-PowerTools.repo _copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8.repo epel-testing-modular.repo CentOS-centosplus.repo CentOS-fasttrack.repo CentOS-Sources.repo epel-modular.repo epel-testing.repo CentOS-CR.repo CentOS-HA.repo CentOS-Vault.repo epel-playground.repo [root@centos8 ~]# cat /etc/yum.repos.d/_copr\:copr.fedorainfracloud.org \:ktdreyer\:ceph-el8.repo [copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8] name=Copr repo for ceph-el8 owned by ktdreyer baseurl= https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/epel-8… type=rpm-md skip_if_unavailable=True gpgcheck=1 gpgkey= https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey… repo_gpgcheck=0 enabled=1 enabled_metadata=1 Thanks, /Chris C

3 years, 9 months

1
0
0 0

Pointers in __crush_do_rule__ function of CRUSH mapper file

by Bobby

Hi all, I have a question regarding pointer variables used in the __crush_do_rule__ function of CRUSH __mapper.c__. Can someone please help me understand the purpose of following four pointer variables inside __crush_do_rule__: int *b = a + result_max; int *c = b + result_max; int *w = a; int *o = b; The function __crush_do_rule__ is below: /** * crush_do_rule - calculate a mapping with the given input and rule * @map: the crush_map * @ruleno: the rule id * @x: hash input * @result: pointer to result vector * @result_max: maximum result size * @weight: weight vector (for map leaves) * @weight_max: size of weight vector * @cwin: Pointer to at least map->working_size bytes of memory or NULL. */ int crush_do_rule(const struct crush_map *map, int ruleno, int x, int *result, int result_max, const __u32 *weight, int weight_max, void *cwin, const struct crush_choose_arg *choose_args) { int result_len; struct crush_work *cw = cwin; int *a = (int *)((char *)cw + map->working_size); int *b = a + result_max; int *c = b + result_max; int *w = a; int *o = b; int recurse_to_leaf; int wsize = 0; int osize; int *tmp; const struct crush_rule *rule; __u32 step; int i, j; int numrep; int out_size; Thanks Bobby !

3 years, 9 months

1
0
0 0

v14.2.10 Nautilus released

by Abhishek Lekshmanan

We're happy to announce the tenth release in the Nautilus series. In addition to fixing a security-related bug in RGW, this release brings a number of bugfixes across all major components of Ceph. We recommend that all Nautilus users upgrade to this release. For a detailed changelog please refer to the ceph release blog at: https://ceph.io/releases/v14-2-10-nautilus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * RGW: Bucket notifications now support Kafka endpoints. This requires librdkafka of version 0.9.2 and up. Note that Ubuntu 16.04.6 LTS (Xenial Xerus) has an older version of librdkafka, and would require an update to the library. * The pool parameter `target_size_ratio`, used by the pg autoscaler, has changed meaning. It is now normalized across pools, rather than specifying an absolute ratio. For details, see :ref:`pg-autoscaler`. If you have set target size ratios on any pools, you may want to set these pools to autoscale `warn` mode to avoid data movement during the upgrade:: ceph osd pool set <pool-name> pg_autoscale_mode warn * The behaviour of the `-o` argument to the rados tool has been reverted to its orignal behaviour of indicating an output file. This reverts it to a more consistent behaviour when compared to other tools. Specifying object size is now accomplished by using an upper case O `-O`. * The format of MDSs in `ceph fs dump` has changed. * Ceph will issue a health warning if a RADOS pool's `size` is set to 1 or in other words the pool is configured with no redundancy. This can be fixed by setting the pool size to the minimum recommended value with:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false * RGW: bucket listing performance on sharded bucket indexes has been notably improved by heuristically -- and significantly, in many cases -- reducing the number of entries requested from each bucket index shard. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: b340acf629a010a74d90da5782a2c5fe0b54ac20 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

3 years, 9 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users June 2020