- ceph-users - lists.ceph.io

Strange behavior for crush buckets of erasure-profile
by tdados＠hotmail.com 19 Dec '19

19 Dec '19

So I wanted to report a crush rule/ec profile strange behaviour regarding radosgw items which i am not sure if it's a bug or it's supposed to work that way. I am trying to implement the below scenario in my home lab: By default there is a "default" erasure-code-profile with the below settings: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 From the above we see that it uses the root bucket. Now ofcourse you would want to create your own ec-profile with custom algorithm/crush buckets etc Let's say for example we create two new ec profiles. One with specific crush-root = ssd-performance2 and one with the crush-root=default (there are no disks there according to ceph osd tree-> end of page) ceph osd erasure-code-profile set test-ec crush-device-class= crush-failure-domain=host crush-root=ssd-performance2 jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 ceph osd erasure-code-profile set test-ec2 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 Now let's create the associated crush rules to use these profiles: ceph osd crush rule create-erasure erasure-test-rule1 test-ec ceph osd crush rule create-erasure erasure-test-rule2 test-ec2 Now let's say you have a radosgw server that has started and by default it creates the 5 default radosgwpools(supposed you have uploaded some data as well): default.rgw.buckets.data default.rgw.buckets.index default.rgw.control default.rgw.log default.rgw.meta Now if you grep these pools with ceph osd dump you will see that all of them are using replicated rules but we want to use erasure for the radosgw data pool. So let's migrate the default.rgw.buckets.data pool to a erasure-coded one. 1) We shutdown the radosgw-server so that we don't allow any requests coming in. 2) ceph osd pool rename default.rgw.buckets.data default.rgw.buckets.data-old 3) ceph osd pool create default.rgw.buckets.data 8 8 erasure test-ec erasure-test-rule - > We use the newly created erasure crush rule with the profile we created and use the ssd-performance2 root bucket 4) rados cppool default.rgw.buckets.data-old default.rgw.buckets.data 5) Start radosgw server again At this point i can see the old objects and i can upload new objects in radosgw and everything is working fine. Now i see this strange behavior after i do the below: We set the default.rgw.buckets.data to use the other erasure crush rule (This is using the root bucket=default which doesn't have any disks): ceph osd pool set default.rgw.buckets.data crush_rule erasure-test-rule2 Bug1? You could still browse the data but any attempt to upload/download hangs there with the below log messages: 2019-12-18 17:07:07.037 7f05a1ece700 0 ERROR: client_io->complete_request() returned Input/output error 2019-12-18 17:07:07.037 7f05a1ece700 2 req 712 0.004s s3:list_buckets op status= Monitor nodes don't display anything and seems that new items cannot be saved (which is correct as it doesn't know where to save them) but at least Monitor nodes should display something as a warning or there must be crush check before to see if the rule can be applied? Reverting back the rule to erasure-test-rule works fine again ================================= Bug 2? If you modify the erasure-test-rule profile to use a null crush bucket (like erasure-test-rule2) then this is not being parsed and identified by the crush rule. Seems crush rules skips that part Example: ceph osd erasure-code-profile set test-ec crush-root=default --force At this point nothing happens and radosgw is working fine. Which it shouldn't as it should see that the data cannot be saved anywhere. Unless it keeps the crush root bucket from the crush rules and not from the erasure coded profiles...even if you force apply/change it to the erasure profile like above. ================================= Bug 3? You don't know which rule is using which erasure-code-profile from ceph osd dump. You only see that this pool is using crush rule number 1 but if you dump this crush rule it doesn't mention which erasure-code profile is using, other than which item_name eg = root bucket Even with the telemetry on with latest release and if you do "ceph telemetry show basic" with below you see there is no crush-root being mentioned. So is the crush rule > erasure_code_profile regarding parsing of the crush_root buckets? { "min_size": 2, "erasure_code_profile": { "crush-failure-domain": "host", "k": "2", "technique": "reed_sol_van", "m": "1", "plugin": "jerasure" }, "pg_autoscale_mode": "warn", "pool": 860, "size": 3, "cache_mode": "none", "target_max_objects": 0, "pg_num": 8, "pgp_num": 8, "target_max_bytes": 0, "type": "erasure" } root@ceph-mon01:~# ceph osd crush rule dump erasure-test-rule { "rule_id": 2, "rule_name": "erasure-test-rule", "ruleset": 2, "type": 3, "min_size": 3, "max_size": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "ssd-performance2" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } root@ceph-mon01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -37 0.18398 root really-low -40 0.09799 host ceph-osd01-really-low 11 hdd 0.09799 osd.11 up 1.00000 1.00000 -41 0.04799 host ceph-osd02-really-low 1 hdd 0.01900 osd.1 up 1.00000 1.00000 9 hdd 0.02899 osd.9 up 1.00000 1.00000 -42 0.03799 host ceph-osd03-really-low 6 hdd 0.01900 osd.6 up 1.00000 1.00000 7 hdd 0.01900 osd.7 up 1.00000 1.00000 -23 10.67598 root spinning-rust -20 2.04900 rack rack1 -3 2.04900 host ceph-osd01 3 hdd 0.04900 osd.3 up 0.95001 1.00000 22 hdd 1.00000 osd.22 up 0.90002 1.00000 17 ssd 1.00000 osd.17 up 1.00000 1.00000 -25 3.07799 rack rack2 -5 3.07799 host ceph-osd02 4 hdd 0.04900 osd.4 up 1.00000 1.00000 8 hdd 0.02899 osd.8 up 1.00000 1.00000 23 hdd 1.00000 osd.23 up 1.00000 1.00000 25 hdd 1.00000 osd.25 up 1.00000 1.00000 12 ssd 1.00000 osd.12 up 1.00000 1.00000 -28 3.54900 rack rack3 -7 3.54900 host ceph-osd03 0 hdd 1.00000 osd.0 up 0.90002 1.00000 5 hdd 0.04900 osd.5 up 1.00000 1.00000 30 hdd 0.50000 osd.30 up 1.00000 1.00000 21 ssd 1.00000 osd.21 up 0.95001 1.00000 24 ssd 1.00000 osd.24 up 1.00000 1.00000 -55 2.00000 rack rack4 -49 2.00000 host ceph-osd04 26 hdd 1.00000 osd.26 up 1.00000 1.00000 27 hdd 1.00000 osd.27 up 1.00000 1.00000 -2 9.10799 root ssd-performance2 -32 2.09799 host ceph-osd01-ssd 2 ssd 0.09799 osd.2 up 1.00000 1.00000 13 ssd 1.00000 osd.13 up 1.00000 1.00000 16 ssd 1.00000 osd.16 up 1.00000 1.00000 -31 3.00000 host ceph-osd02-ssd 14 ssd 1.00000 osd.14 up 1.00000 1.00000 18 ssd 1.00000 osd.18 up 1.00000 1.00000 19 ssd 1.00000 osd.19 up 1.00000 1.00000 -9 2.00999 host ceph-osd03-ssd 10 ssd 0.00999 osd.10 up 0.90002 1.00000 15 ssd 1.00000 osd.15 up 1.00000 1.00000 20 ssd 1.00000 osd.20 up 1.00000 1.00000 -52 2.00000 host ceph-osd04-ssd 28 ssd 1.00000 osd.28 up 1.00000 1.00000 29 ssd 1.00000 osd.29 up 1.00000 1.00000 -1 0 root default root@ceph-mon01:~# Thanks, Anastasios

1 0

re-balancing resulting in unexpected availability issues
by steve.nolen＠rstudio.com 18 Dec '19

18 Dec '19

Hi! We've found ourselves in state with our ceph cluster that we haven't seen before, and are looking for a bit of expertise to chime in. We're running a (potentially unusually laid out) moderately large luminous-based ceph cluster in a public cloud, with 234*8TB OSDs, with a single osd per cloud instance. Here's a snippet of our ceph status: services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon4 mgr: ceph-mon2(active), standbys: ceph-mon1, ceph-mon4 osd: 234 osds: 234 up, 231 in data: pools: 5 pools, 7968 pgs objects: 136.35M objects, 382TiB usage: 1.09PiB used, 713TiB / 1.79PiB avail pgs: 7924 active+clean 44 active+clean+scrubbing+deep Our layout is spread across 6 availability zones (with the majority of osds in three, us-east-1a,us-east-1b and us-east-1e). We've recently decided that a spread across six azs is unnecessary and potentially a detriment to performance, so we are working towards shifting our workload out of 3 of the 6 azs, so that we are evenly placed in 3 azs. Relevant Recent events: 1. We expanded by 24 osds to handle additional capacity needs as well as to provide the capacity necessary to remove all osds in us-east-1f. 2. After the re-balance related to #1 finished we expanded by an additional 12 osds as a follow-on for the #1 change. 3. On the same day as #2, we also issued `ceph crush move` commands to move the location of oldest 20 osds which had previously not been configured with a "datacenter" bucket denoting their availability zone. The re-balancing related to #3 caused quite a change in our cluster, resulting in hours with degraded pgs, and waves of "slow requests" from many of the osds as data shifted. Three of the osds which had their location moved are also wildly more full than the other osds (being in a nearfull, >85% utilized state where the other 231 osds are in the range of 58-63% utilized). A `ceph balancer optimize` plan shows no changes to be made by the balancer to rectify this. Because of their nearfull status, we have marked those three osds as "out". During the resulting re-balancing we experienced more "slow requests" piling up, with some pgs dipping into a peering or activating state (which obviously causes some user-visible trouble). Where we are headed: * We have yet to set the crush map to use a replicated_datacenter rule now that our osds conform to these buckets. * We need to take action to remove/replace the three osds which were over-utilized. * We need to remove the us-east-1f osds that provoked the event in #1 (of which there are 16). What I'm hoping for a bit of direction on: * A recommendation on the order of these operations that would result in the least user-visible time. * Some hints or thoughts on why we are seeing flapping pg availability since the change in #3. * General thoughts on our layout. It has been rather useful for unbounded rapid growth but it seems very likely the sheer count of osds/instances is optimal at this point. Obviously I'm happy to provide any additional information that might be helpful that I've overlooked. Thanks so much for looking! Steve

1 0

Use Wireshark to analysis ceph network package
by Xu Chen 18 Dec '19

18 Dec '19

Hi guys, I want to use tcpdump and Wireshark to capture and analysis packages between clients and ceph cluster. But the protocol only shows tcp, no ceph, so I can not read the data between client and cluster. The wireshark version is 3.07. Hope your help. Thank you.

1 0

list CephFS snapshots
by Lars Täuber 18 Dec '19

18 Dec '19

Hi! Is there a mean to list all snapshots existing in a (subdir of) Cephfs? I can't use the find dommand to look for the ".snap" dirs. I'd like to remove certain (or all) snapshots within a CephFS. But how do I find them? Thanks, Lars

4 9

what's meaning of "cache_hit_rate": 0.000000 in "ceph daemon mds.<x> dump loads" output?
by Sang, Oliver 18 Dec '19

18 Dec '19

hi all, seek your help for this. we are using luminous 12.2.12 and we enabled 3 active mds. when I running "ceph daemon mds.<x> dump loads" on any active mds, I always saw such like below "mds_load": { "mds.0": { "request_rate": 526.045993, "cache_hit_rate": 0.000000, ... "mds.1": { "request_rate": 169.845956, "cache_hit_rate": 0.000000, ... "mds.2": { "request_rate": 300.511478, "cache_hit_rate": 0.000000, don't understand what's this 'cache_hit_rate' and why it seems always 0? we actually set a bigger mds_cache_memory_limit than default such like 16G in /etc/ceph/ceph.conf. are they related? I checked https://docs.ceph.com/docs/luminous/cephfs/mds-config-ref/ , but didn't get further clue from the description of "mds cache *" settings.

1 0

Ceph and centos 8
by fleg＠lpnhe.in2p3.fr 17 Dec '19

17 Dec '19

Hi, We have a ceph+cephfs cluster runing nautilus version 14.2.4 We have debian buster/ubuntu bionic clients mounting cephfs in kernel mode without problems. We now want to mount cephfs from our new centos 8 clients. Unfortunately, ceph-common is needed but there are no packages available for el8 (only el7). And no way to install the el7 packages on centos 8 (missing deps). Thus, despite the fact that centos 8 have a 4.18 kernel (required to use quota, snapshots etc...), it seems impossible to mount in kernel mode (good perfs) and we still have to use the so slow fuse mode. Is it possible to workaround this problem ? Or when is it planned to provides (even as beta) the ceph packages for centos 8 ? Thanks.

4 5

MGR log reports error related to Ceph Dashboard: SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727)
by Thomas Schneider 17 Dec '19

17 Dec '19

Hello, I have enabled MGR module "Dashboard" and created a self-signed cert: root@ld5506:~# ceph dashboard create-self-signed-cert Self-signed certificate created Checking the MGR log I get multiple errors for any action in the dashboard with Ceph version 14.2.4.1 (596a387fb278758406deabf997735a1f706660c9) nautilus (stable): 2019-12-17 08:32:30.203 7f70942eb700 0 mgr[dashboard] [17/Dec/2019:08:32:30] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727) What is causing this error? How can I fix it? THX

1 0

bluestore worries
by Frank R 16 Dec '19

16 Dec '19

Hi all, I am thinking about converting a Filestore cluster to Bluestore. The OSD nodes have 16X4TB 7200 SATA OSDs with NVME write journals. The NVME drives should be large enough to house ~30G DB/WAL OSDs. I am worried that I will see a significant performance hit when the deferred writes to the NVME journals are eliminated with Bluestore. Has anyone converted a similar setup to Bluestore? If so, what was the performance impact. thx Frank

3 2

Re: ceph osd pool ls detail 'removed_snaps' on empty pool?
by Gregory Farnum 16 Dec '19

16 Dec '19

On Mon, Dec 16, 2019 at 11:34 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > Hi Gregory, > > I saw ceph -s showing 'snaptrim'(?), but I have still have these > 'removed_snaps' listed on this pool (also on other pools, I don't > remember creating/deleting them) A 'ceph tell mds.c scrub start /test/ > recursive repair' did not remove those. Can/should/how I remove these? > > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] ceph osd pool ls detail 'removed_snaps' on empty > pool? > > > I have removed_snaps listed on pools that I am not using. They are > mostly for doing some performance testing, so I cannot imagine ever > creating snapshots in them. > > > pool 33 'fs_data.ssd' replicated size 3 min_size 1 crush_rule 5 > object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn > last_change 68413 lfor 0/0/65853 flags hashpspool,selfmanaged_snaps > stripe_width 0 application cephfs > removed_snaps > [567~1,56b~1,56f~1,57f~1,583~1,587~1,58b~1,58f~1,591~1,593~1,595~1,597~1 > ,599~1,59b~1,59d~1,59f~1,5a1~1,5a3~1,5a5~1,5a7~1, These are RADOS snapshots, so CephFS may have removed them but the OSDs still need to trim all the data; cephfs commands won't do anything to or with them. Once the OSD have removed the snapshot (it can take a while, depending on settings and other cluster activity) they'll report it back to the monitor, and it will remove them from the list of removed snaps. (It may not bother removing them if there aren't enough other live snapshots, I forget.) Either way, this list is basically harmless so you shouldn't worry about it. > > > [@ ]# rados df| egrep '^POOL|fs_data.ssd' > POOL_NAME USED OBJECTS CLONES COPIES > MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS > WR USED COMPR UNDER COMPR > fs_data.ssd 0 B 0 0 0 > > 0 0 0 0 0 B 0 0 B 0 > B 0 B > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

How safe is k=2, m=1, min_size=2?
by Dan van der Ster 16 Dec '19

16 Dec '19

Hi all, We have said in the past that an EC pool should have min_size=k+1, for the same reasons that a replica 3 pool needs min_size=2. And we've heard several stories about replica 3, min_size=1 leading to incomplete PGs. Taking a quick poll -- did anyone ever suffer an outage on a pool with k=2, m=1, min_size=2? Thanks! Dan

2 1