- ceph-users - lists.ceph.io

Announcing Ceph Buenos Aires 2019 on Oct 16th at Museo de Informatica
by Victoria Martinez de la Cruz 25 Sep '19

25 Sep '19

Hi all, I'm happy to announce that next Oct 16th we will have the Ceph Day Argentina in Buenos Aires. The event will be held in the Museo de Informatica de Argentina, so apart from hearing the latest features from core developers, real use cases from our users and usage experiences from customers and partners, you will be able to enjoy and contribute to this fantastic museum that holds a great collection of vintage hardware. If you are a local and/or you are in the area, we really hope you can join us! CFP is open in https://forms.zohopublic.com/thingee/form/CephDayArgentina2019/formperma/yf… Ceph Day Buenos Aires site is available in https://ceph.io/cephdays/ceph-day-argentina-2019/ Cheers, Victoria

1 0

Wrong %USED and MAX AVAIL stats for pool
by nalexandrov＠innologica.com 25 Sep '19

25 Sep '19

Hi everyone, We are running Nautilus 14.2.2 with 6 nodes and a total of 44 OSDs, all are 2TB spinning disks. # ceph osd count-metadata osd_objectstore "bluestore": 44 # ceph osd pool get one size size: 3 # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 33 TiB 47 TiB 47 TiB 58.26 TOTAL 80 TiB 33 TiB 47 TiB 47 TiB 58.26 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.06M 47 TiB 68.48 7.1 TiB bench 5 250 MiB 67 250 MiB 0 21 TiB Why pool's stats are showing incorrect values for %USED and MAX AVAIL? They should be much bigger. The first 24 OSDs was created on jewell release and the osd_objectstore was 'filestore'. While we were with mimic release, we added 20 more 'bluestore' OSDs. The first 24 was destroyed and recreated as 'bluestore'. After the upgrade from mimic release, all the OSD's was updated with ceph-bluestore-tool repair. The incorrect values appeared after the upgrade from 14.2.1 to 14.2.2. Any help will be appreciated :) BR, NAlexandrov

2 1

OSD rebalancing issue - should drives be distributed equally over all nodes
by Thomas 25 Sep '19

25 Sep '19

Hi, I'm facing several issues with my ceph cluster (2x MDS, 6x ODS). Here I would like to focus on the issue with pgs backfill_toofull. I assume this is related to the fact that the data distribution on my OSDs is not balanced. This is the current ceph status: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 MDSs report slow metadata IOs 78 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 2 pgs inactive, 2 pgs peering Degraded data redundancy: 304136/153251211 objects degraded (0.198%), 57 pgs degraded, 57 pgs undersized Degraded data redundancy (low space): 265 pgs backfill_toofull 3 pools have too many placement groups 74 slow requests are blocked > 32 sec 80 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 98m) mgr: ld5505(active, since 3d), standbys: ld5506, ld5507 mds: pve_cephfs:1 {0=ld3976=up:active} 1 up:standby osd: 368 osds: 368 up, 367 in; 302 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.08M objects, 195 TiB usage: 590 TiB used, 563 TiB / 1.1 PiB avail pgs: 0.023% pgs not active 304136/153251211 objects degraded (0.198%) 1672190/153251211 objects misplaced (1.091%) 8564 active+clean 196 active+remapped+backfill_toofull 57 active+undersized+degraded+remapped+backfill_toofull 35 active+remapped+backfill_wait 12 active+remapped+backfill_wait+backfill_toofull 2 active+remapped+backfilling 2 peering io: recovery: 18 MiB/s, 4 objects/s Currently I'm using 6 OSD nodes. Node A 48x 1.6TB HDD Node B 48x 1.6TB HDD Node C 48x 1.6TB HDD Node D 48x 1.6TB HDD Node E 48x 7.2TB HDD Node F 48x 7.2TB HDD Question: Is it advisable to distribute the drives equally over all nodes? If yes, how should this be executed w/o ceph disruption? Regards Thomas

3 5

verify_upmap number of buckets 5 exceeds desired 4
by Eric Dold 25 Sep '19

25 Sep '19

Hello, I'm running ceph 14.2.3 on six hosts with each four osds. I did recently upgrade this from four hosts. The cluster is running fine. But i get this in my logs: Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700 -1 verify_upmap number of buckets 5 exceeds desired 4 Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700 -1 verify_upmap number of buckets 5 exceeds desired 4 Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700 -1 verify_upmap number of buckets 5 exceeds desired 4 It looks like the balancer is not doing any work. Here are some infos about the cluster: ceph1 ~ # ceph osd crush rule ls replicated_rule cephfs_ec ceph1 ~ # ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ceph1 ~ # ceph osd crush rule dump cephfs_ec { "rule_id": 1, "rule_name": "cephfs_ec", "ruleset": 1, "type": 3, "min_size": 8, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 4, "type": "host" }, { "op": "choose_indep", "num": 2, "type": "osd" }, { "op": "emit" } ] } ceph1 ~ # ceph osd erasure-code-profile ls default isa_62 ceph1 ~ # ceph osd erasure-code-profile get default k=2 m=1 plugin=jerasure technique=reed_sol_van ceph1 ~ # ceph osd erasure-code-profile get isa_62 crush-device-class= crush-failure-domain=osd crush-root=default k=6 m=2 plugin=isa technique=reed_sol_van The idea with four hosts was that the ec profile should take two osds on each host for the eight buckets. Now with six hosts i guess two hosts will have tow buckets on two osds and four hosts will have each one bucket for a piece of data. Any idea how to resolve this? Regards Eric

1 2

configuration of Ceph-ISCSI gateway
by Gesiel Galvão Bernardes 24 Sep '19

24 Sep '19

Hi everyone, I'm configurating ISCSI gateway in Ceph Mimic (13.2.6) using ceph manual: https://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/ But i stopped in this problem: In manual says: "Set the client’s CHAP username to myiscsiusername and password to myiscsipassword: > /iscsi-target...at:rh7-client> auth chap=myiscsiusername/myiscsipassword" But I receive this response: /iscsi-target...at:rh7-client> auth chap=myiscsitest/myiscsitestpasswd Unexpected keyword parameter 'chap'. The options disponibles are: /iscsi-target...at:rh7-client> auth ? To set authentication, specify username=<user> password=<password> [mutual_username]=<user> [mutual_password]=<password> But if configure as asks: auth username=myiscsitest password=myiscsitestpasswd Failed to update the client's auth: Invalid password I tried with high password complexibility, but the problem persists. My questions: - How is the correct mode for configure authentication? - How contribute for update of documentation? A bug report has opened* for broken information of instalation of ceph-iscsi-gw, but was closed without update of documentation:https://github.com/ceph/ceph-ansible/issues/2707 Regards Gesiel Bernardeds

2 1

RGW orphaned shadow objects
by P. O. 24 Sep '19

24 Sep '19

Hi All, I have a question about "orphaned" objects in default.rgw.buckets.data pool. Few days ago i ran "radosgw-admin orphans find ..." [dc-1 root@mon-1 tmp]$ radosgw-admin orphans list-jobs [ "orphans-find-1" ] Today I checked the result. I listed orphaned objects by command: $# for i in `rados -p default.rgw.log ls |grep orphan.scan.orphans-find-1.rados`; do rados -p default.rgw.log listomapkeys $i; done > orphaned_objects.txt There are a lot: xxx.__shadow_.yyy objects. Is it possible to check if these __shadow_ are orphaned (can be removed) or belongs to any good object? How to check if a shadow object is still in use and what object it belongs to? Some of them are very old. For example: default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_1 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_2 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_4 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_5 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_6 mtime 2017-08-16 04:01:49.000000, size 4194304 Best regards, PO

2 1

Health error: 1 MDSs report slow metadata IOs, 1 MDSs report slow requests
by Thomas 24 Sep '19

24 Sep '19

Hi, ceph health reports 1 MDSs report slow metadata IOs 1 MDSs report slow requests This is the complete output of ceph -s: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 MDSs report slow metadata IOs 1 MDSs report slow requests 72 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 33 pgs inactive, 32 pgs peering Degraded data redundancy: 123285/153918525 objects degraded (0.080%), 27 pgs degraded, 27 pgs undersized Degraded data redundancy (low space): 116 pgs backfill_toofull 3 pools have too many placement groups 54 slow requests are blocked > 32 sec 179 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 21h) mgr: ld5507(active, since 21h), standbys: ld5506, ld5505 mds: pve_cephfs:1 {0=ld3955=up:active} 1 up:standby osd: 368 osds: 368 up, 368 in; 140 remapped pgs data: pools: 6 pools, 8872 pgs objects: 51.31M objects, 196 TiB usage: 591 TiB used, 561 TiB / 1.1 PiB avail pgs: 0.372% pgs not active 123285/153918525 objects degraded (0.080%) 621911/153918525 objects misplaced (0.404%) 8714 active+clean 90 active+remapped+backfill_toofull 26 active+undersized+degraded+remapped+backfill_toofull 16 peering 16 remapped+peering 7 active+remapped+backfill_wait 1 activating 1 active+recovery_wait+degraded 1 active+recovery_wait+undersized+remapped In the log I find these relevant entries: 2019-09-24 13:24:37.073695 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18618.873983 secs 2019-09-24 13:24:42.073757 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18623.874055 secs 2019-09-24 13:24:47.073852 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18628.874149 secs 2019-09-24 13:24:52.073941 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18633.874237 secs 2019-09-24 13:24:57.074073 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18638.874354 secs 2019-09-24 13:25:02.074118 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18643.874415 secs Cephfs is residing on a pool "hdd" with dedicated HDDs (4x 17 1.6TB). This pool is used for RBDs, too. Question: How can I identify the 2 slow requests? And how can I kill these requests? Regards Thomas

4 3

How to reduce or control memory usage during recovery?
by Amudhan P 24 Sep '19

24 Sep '19

Hi, I am using ceph mimic in a small test setup using the below configuration. OS: ubuntu 18.04 1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1. When running a script doing multiple folders creation ceph started throwing error late IO due to high metadata workload. once after folder creation complete PG's degraded and I am waiting for PG to complete recovery but my OSD's starting to crash due to OOM and restarting after some time. Now my question is I can wait for recovery to complete but how do I stop OOM and OSD crash? basically want to know the way to control memory usage during recovery and make it stable. I have also set very low PG metadata_pool 8 and data_pool 16. I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8. Attached msg from "kern.log" from one of the node and snippet of error msg in this mail. ---------error msg snippet ---------- -bash: fork: Cannot allocate memory Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0 Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727 (ceph-osd) score 489 or sacrifice child Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250 regards Amudhan

5 8

Re: Nautilus : ceph dashboard ssl not working
by Volker Theile 24 Sep '19

24 Sep '19

https://www.thegeekdiary.com/centos-rhel-67-why-the-files-in-tmp-directory-… Am 24.09.19 um 14:53 schrieb Lenz Grimmer: > On 9/24/19 1:37 PM, Miha Verlic wrote: > >> I've got slightly different problem. After a few days of running fine, >> dashboard stops working because it is apparently seeking for wrong >> certificate file in /tmp. If I restart ceph-mgr it starts to work again. > Does the restart trigger the creation of a similar-looking file in /tmp? > I wonder if there's some kind of cron job that cleans up the /tmp > directory every now and then... > > Lenz > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Volker Theile Software Engineer | Ceph | openATTIC SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany GF: Felix Imendörffer, HRB 247165 (AG München) Phone: +49 173 5876879 E-Mail: vtheile(a)suse.com

1 0

Configuration of Ceph-ISCSI gateway
by Gesiel Galvão Bernardes 24 Sep '19

24 Sep '19

Hi everyone, I'm configurating ISCSI gateway in Ceph Mimic (13.2.6) using ceph manual: https://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/ But i stopped in this problem: In manual says: "Set the client’s CHAP username to myiscsiusername and password to myiscsipassword: > /iscsi-target...at:rh7-client> auth chap=myiscsiusername/myiscsipassword" But I receive this response: /iscsi-target...at:rh7-client> auth chap=myiscsitest/myiscsitestpasswd Unexpected keyword parameter 'chap'. The options disponibles are: /iscsi-target...at:rh7-client> auth ? To set authentication, specify username=<user> password=<password> [mutual_username]=<user> [mutual_password]=<password> But if configure as asks: auth username=myiscsitest password=myiscsitestpasswd Failed to update the client's auth: Invalid password I tried with high password complexibility, but the problem persists. My questions: - How is the correct mode for configure authentication? - How contribute for update of documentation? A bug report has opened* for broken information of instalation of ceph-iscsi-gw, but was closed without update of documentation:https://github.com/ceph/ceph-ansible/issues/2707 Regards Gesiel Bernardeds

1 0