- ceph-users - lists.ceph.io

v13.2.9 Mimic released
by Abhishek Lekshmanan 17 Apr '20

17 Apr '20

We're glad to announce the availability of the ninth and very likely the last stable release in the Ceph Mimic stable release series. This release fixes bugs across all components and also contains a RGW security fix. We recommend all mimic users to upgrade to this version. We thank everyone for making this release a possibility. Notable Changes --------------- * CVE-2020-1760: Fixed XSS due to RGW GetObject header-splitting * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap balancing has been removed. Instead use the mgr balancer config `upmap_max_deviation` which now is an integer number of PGs of deviation from the target PGs per OSD. This can be set with a command like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default `upmap_max_deviation` is 1. There are situations where crush rules would not allow a pool to ever have completely balanced PGs. For example, if crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of the racks. In those cases, the configuration value can be increased. * The `cephfs-data-scan scan_links` command now automatically repair inotables and snaptable. For the full changelog please refer to the official release blog entry at https://ceph.io/releases/v13-2-9-mimic-released Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-13.2.9.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 58a2a9b31fd08d8bb3089fce0e312331502ff945 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer

1 0

Re: RGW and the orphans
by Katarzyna Myrek 16 Apr '20

16 Apr '20

Hi Thanks for the quick response. To be honest my cluster is getting full because of that trash and I am at the point where I have to do the removal manually ;/. Kind regards / Pozdrawiam, Katarzyna Myrek czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios <mriosfer(a)easydatahost.com> napisał(a): > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> > Enviado el: jueves, 16 de abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

MDS: cache pressure warnings with Ganesha exports
by Stolte, Felix 16 Apr '20

16 Apr '20

Hey folks, I keep getting ceph health warnings about clients failing to respond to cache pressure. They always refer to sessions from ganesha exports. I've read all threads regarding this issue, but none of my changes resolved it. What I’ve done so far: Ganesha.conf: MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } Attr_Expiration_Time = 0 in every export mds_cache_memory_limit = 17179869184 on MDS Servers I even set “client_oc = false” on the ganesha server, but this doesn’t seem to be applied My setup is ceph version 14.2.8 (All Server and clients. One active MDS. Ganesha 2.8.3 runs on a dedicated Server. ceph daemon mds.<active_mds> dump_mempools (filtered out the empty pools): "mempool": { "by_pool": { "bloom_filter": { "items": 1414723, "bytes": 1414723 }, "buffer_anon": { "items": 180992, "bytes": 2404306271 }, "buffer_meta": { "items": 178660, "bytes": 15722080 }, "osdmap": { "items": 4121, "bytes": 75912 }, "mds_co": { "items": 221728924, "bytes": 16320868177 }, }, "total": { "items": 223507420, "bytes": 18742387163 Every hint how to resolve the issue are welcome. If more information is needed, I am glad to provide it. Regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

3 7

RGW and the orphans
by Katarzyna Myrek 16 Apr '20

16 Apr '20

Hi Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. Does anyone have a good method for that? My cluster mostly has orphans from multipart uploads. Kind regards / Pozdrawiam, Katarzyna Myrek

2 1

radosgw multisite
by Ignazio Cassano 16 Apr '20

16 Apr '20

Hello All, Reading the documentation I created a multisite with luminous version. I would like to know if it sync in one way only. Using s3cmd if I put a file in a bucket on the primary zone I can see the file in the same bucket on the secondary zone. If I put a file in the bucket on secondary zone I cannot see the file on the primary. Is this the correct behaviour ? Thanks & Regards Ignazio

2 3

Re: New to ceph / Very unbalanced cluster
by Reed Dier 16 Apr '20

16 Apr '20

Copying the ML, because I forgot to reply-all. Reed > On Apr 15, 2020, at 3:58 PM, Reed Dier <reed.dier(a)focusvq.com> wrote: > > The problem is almost certainly stemming from unbalanced OSD distribution among your hosts, and assuming you are using a default 3x replication across hosts crush rule set. > > You are limited by your smallest bin size. > > In this case you have a 750GB HDD as the only OSD on node1, so when it wants 3 copies across 3 hosts, there are only ~750GB of space that can fulfill this requirement. > > Having lots of different size OSDs and differing OSDs in your topology is going to lead to issues of under/over utilization. > >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 21.54213 root default >> -3 0.75679 host node1 >> -5 5.39328 host node2 >> -10 15.39206 host node3 > > You either need to redistribute your OSDs across your hosts, or possibly rethink your disk strategy. > You could move osd.5 to node1, and osd.0 to node2, which would give you roughly 6TiB of usable hdd space across your three nodes. > > Reed > >> On Apr 15, 2020, at 10:50 AM, Simon Sutter <ssutter(a)hosttech.ch <mailto:ssutter@hosttech.ch>> wrote: >> >> Hello everybody, >> >> >> >> I'm very new to ceph and installed a testenvironment (nautilus). >> >> The current goal of this cluster is, to be a short period backup. >> >> For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?). >> >> I created for my cephfs two pools, one for metadata and one for storage data. >> >> >> >> I have three nodes and the ceph osd tree looks like this: >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 21.54213 root default >> -3 0.75679 host node1 >> 0 hdd 0.75679 osd.0 up 0.00636 1.00000 >> -5 5.39328 host node2 >> 1 hdd 2.66429 osd.1 up 0.65007 1.00000 >> 3 hdd 2.72899 osd.3 up 0.65007 1.00000 >> -10 15.39206 host node3 >> 5 hdd 7.27739 osd.5 up 1.00000 1.00000 >> 6 hdd 7.27739 osd.6 up 1.00000 1.00000 >> 2 ssd 0.38249 osd.2 up 1.00000 1.00000 >> 4 ssd 0.45479 osd.4 up 1.00000 1.00000 >> >> >> The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview: >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS >> 0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up >> 1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up >> 3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB 8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up >> 5 hdd 7.27739 1.00000 7.3 TiB 390 GiB 389 GiB 8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up >> 6 hdd 7.27739 1.00000 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up >> 2 ssd 0.38249 1.00000 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up >> 4 ssd 0.45479 1.00000 466 GiB 28 GiB 27 GiB 4 KiB 1024 MiB 438 GiB 6.03 0.52 4 up >> TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57 >> MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87 >> >> To counteract this, I tried to turn on the balancer module. >> >> The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects: >> >> 144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%) >> >> >> >> So my question is: is ceph supposed to do that? >> Why are all those objects misplaced? Because of those 112 PGs on osd0? >> Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512 >> >> >> >> Thank you very much >> Simon Sutter >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >

2 1

Issues with RGW PUT performance after upgrade to 14.2.8
by Katarzyna Myrek 16 Apr '20

16 Apr '20

Hi After upgrading to 14.2.8 i can see that PUT operations are significantly slower. GET and DELETE still have the same performance. I double checked OSD nodes and I cannot find anything suspicious there. No extreme iowaits etc. Anyone have the same problem? Kind regards / Pozdrawiam, Katarzyna Myrek

1 0

Cephadm and rados gateways
by Edward Garcia 16 Apr '20

16 Apr '20

Hello, I have a question. I’m trying to deploy a rados gateway and the container keeps crashing in less than a second on the host. Ceph/cephadm keeps trying to recreate it and its an endless loop. Looking at the log the gateway container is failing in trying to bind to port 80. How do I configure the settings for the gateway to change the port? I see on the documentation page that the settings are configured via the monitor configuration but what setting exactly? I tried adding rgw_frontends to ceph.conf and with cephadm config set. I initially created the realm, group, and some just like the cephadm page stated but every time I ceph orch apply rgw ... when I check with ceph config dump I see the rgw service defaults back to port 80.

1 0

Deletion of objects and garbage collector
by Priya Sehgal 15 Apr '20

15 Apr '20

Hi, I read at many places that when an object is deleted from ceph it is queued for deletion with the garbage collector (GC). However, when I delete objects of various sizes (both less than 4MB and large sized greater than 4MB i.e. MPU) I always find gc list as empty. I tried disabling GC also to make sure that it does not run and delete it, yet I did not find any object when I run the command. Here's the output: *radosgw-admin gc list* [ { "tag": "2~KsaJkJwSGeuVzeKpkHAe_5vJ3JqZmKc", "time": "2020-04-10 18:25:23.0.769037s", "objs": [] } ] NOTE: I tried to delete an object as on 14th April and 15th April. I issued "s3cmd del" command. Ceph Version I am using in Luminous. Please let me know how object deletes work. -- Regards, Priya

2 4

New to ceph / Very unbalanced cluster
by Simon Sutter 15 Apr '20

15 Apr '20

Hello everybody, I'm very new to ceph and installed a testenvironment (nautilus). The current goal of this cluster is, to be a short period backup. For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?). I created for my cephfs two pools, one for metadata and one for storage data. I have three nodes and the ceph osd tree looks like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.54213 root default -3 0.75679 host node1 0 hdd 0.75679 osd.0 up 0.00636 1.00000 -5 5.39328 host node2 1 hdd 2.66429 osd.1 up 0.65007 1.00000 3 hdd 2.72899 osd.3 up 0.65007 1.00000 -10 15.39206 host node3 5 hdd 7.27739 osd.5 up 1.00000 1.00000 6 hdd 7.27739 osd.6 up 1.00000 1.00000 2 ssd 0.38249 osd.2 up 1.00000 1.00000 4 ssd 0.45479 osd.4 up 1.00000 1.00000 The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up 1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up 3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB 8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up 5 hdd 7.27739 1.00000 7.3 TiB 390 GiB 389 GiB 8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up 6 hdd 7.27739 1.00000 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up 2 ssd 0.38249 1.00000 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up 4 ssd 0.45479 1.00000 466 GiB 28 GiB 27 GiB 4 KiB 1024 MiB 438 GiB 6.03 0.52 4 up TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57 MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87 To counteract this, I tried to turn on the balancer module. The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects: 144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%) So my question is: is ceph supposed to do that? Why are all those objects misplaced? Because of those 112 PGs on osd0? Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512 Thank you very much Simon Sutter

1 0