- ceph-users - lists.ceph.io

stale+active+clean PG
by tomislav.raseta＠gmail.com 15 May '20

15 May '20

Dear all, We're running Ceph Luminous and we've recently hit an issue with some OSD's (autoout states, IO/CPU overload) which unfortunately resulted with one placement group with the state "stale+active+clean", it's a placement group from .rgw.root pool: 1.15 0 0 0 0 0 0 1 1 stale+active+clean 2020-05-11 23:22:51.396288 40'1 2142:152 [3,2,6] 3 [3,2,6] 3 40'1 2020-04-22 00:46:05.904418 40'1 2020-04-20 20:18:13.371396 0 I guess there is no active replica of that object anywhere on the cluster. Restarting osd.3, osd.2 or osd.6 daemons does not help. I've used ceph-objectstore-tool and successfully exported placement group from osd.3, osd.2 and osd.6 and tried to import it on a completely different OSD, the exports differ in filesize slightly, but the osd.3 wihch was the latest primary is the biggest so I've tried to import it on a different OSD, when starting up I see the following (this is from osd.1): 2020-05-14 21:43:19.779740 7f7880ac3700 1 osd.1 pg_epoch: 2459 pg[1.15( v 40'1 (0'0,40'1] local-lis/les=2073/2074 n=0 ec=73/39 lis/c 2073/2073 les/c/f 2074/2074/633 2145/39/2145) [] r=-1 lpr=2455 crt=40'1 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray I see from previous pg dumps (several weeks before while it was still active+clean) that it was 115 bytes with zero objects in it but I am not sure how to interpret that. As this is a pg from .rgw.root pool, I cannot get any response from the cluster when accessing (everything timeouts). What is the correct course of action with this pg? Any help would be greatly appriciated. Thanks, Tomislav

1 0

Using rbd-mirror in existing pools
by Kees Meijs | Nefos 15 May '20

15 May '20

Hi list, Thanks again for pointing me towards rbd-mirror! I've read documentation, old mailing list posts, blog posts and some additional guides. Seems like the tool to help me through my data migration. Given one-way synchronisation and image-based (so, not pool based) configuration, it's still unclear to me how the mirroring will cope with an existing target pool, already consisting of (a lot of) images. Has someone done this already? It feels quite scary with doom scenario's like "cleaning up" the target pool and such in mind... To sum up: my goal is to mirror clusterA/somepool with some specific images to clusterB/someotherpool where already other images reside. The mirrored images should be kept in sync and the other images should be left alone completely. Cheers, Kees -- https://nefos.nl/contact Nefos IT bv Ambachtsweg 25 (industrienummer 4217) 5627 BZ Eindhoven Nederland KvK 66494931 /Aanwezig op maandag, dinsdag, woensdag en vrijdag/

5 7

Re: Cluster network and public network
by Nghia Viet Tran 14 May '20

14 May '20

Hi Martin, Thanks for your response. You mean one network interface for only MON hosts or for the whole cluster including OSD hosts? I’m confusing now because there are some projects that only use one public network for the whole cluster. That means the rebalancing, replicating objects and heartbeats from OSD hosts would affects the performance of Ceph client. From: Martin Verges <martin.verges(a)croit.io> Date: Friday, May 8, 2020 at 16:20 To: Nghia Viet Tran <Nghia.Viet.Tran(a)mgm-tp.com> Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Cluster network and public network Hello Nghia, just use one network interface card and use frontend and backend traffic on the same. No problem with that. If you have a dual port card, use both ports as an LACP channel and maybe separate it using VLANs if you want to, but not required as well. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io<mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Fr., 8. Mai 2020 um 09:29 Uhr schrieb Nghia Viet Tran <Nghia.Viet.Tran(a)mgm-tp.com<mailto:Nghia.Viet.Tran@mgm-tp.com>>: Hi everyone, I have a question about the network setup. From the document, It’s recommended to have 2 NICs per hosts as described in below picture [Diagram] In the picture, OSD hosts will connect to the Cluster network for replicate and heartbeat between OSDs, therefore, we definitely need 2 NICs for it. But seems there are no connections between Ceph MON and Cluster network. Can we install 1 NIC on Ceph MON then? I appreciated any comments! Thank you! -- Nghia Viet Tran (Mr) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

12 20

ceph-ansible replicated crush rule
by Marc Boisis 14 May '20

14 May '20

Hello, With ceph-ansible the default replicated crush rule is : { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } And I would like to have this: { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 2, "max_size": 4, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } How can I do this within ceph-ansible playbook ? I try : crush_rule_replicated: name: replicated_rule root: default ruleset: 0 type: replicated min_size: 2 max_size: 4 step: - take default - chooseleaf firstn 0 type rack - emit default: true crush_rules: - "{{ crush_rule_replicated }}" task path: roles/ceph-osd/tasks/crush_rules.yml:32 try to change it but it fail: stdout_lines: - '' - '{"rule_id":0,"rule_name":"replicated_rule","ruleset":0,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-1,"item_name":"default"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}' Anyone have an idea ? Regards Marc

1 0

Re: virtual machines crashes after upgrade to octopus
by Erwin Lubbers 14 May '20

14 May '20

Hi, Did anyone found a way to resolve the problem? I'm seeing the same on a clean Octopus Ceph installation on Ubuntu 18 with an Octopus compiled KVM server running on CentOS 7.8. The KVM machine shows: [ 7682.233684] fn-radosclient[6060]: segfault at 2b19 ip 00007f8165cc0a50 sp 00007f81397f6490 error 4 in librbd.so.1.12.0[7f8165ab4000+537000] Ceph is healthy and stable for a few weeks and I did not get these messages while running on KVM compiled with Luminous libraries. Regards, Erwin

4 6

Bucket - radosgw-admin reshard process
by CUZA Frédéric 14 May '20

14 May '20

Hi everyone, I am facing an issue with bucket resharding. It started with a warning on my ceph cluster health : [root@ceph_monitor01 ~]# ceph -s cluster: id: 2da0734-2521-1p7r-8b4c-4a265219e807 health: HEALTH_WARN 1 large omap objects Finds out I had a problem with a bucket : "buckets": [ { "bucket": "bucket-elementary-1", "tenant": "", "num_objects": 615915, "num_shards": 3, "objects_per_shard": 205305, "fill_status": "OVER 100.000000%" }, I was wondering why the dynamic sharding wasn't doing its job but find out it was in the sharding queue : [root@ceph_monitor01 ~]# radosgw-admin reshard list [ { "time": "2020-05-14 09:42:10.905080Z", "tenant": "", "bucket_name": " bucket-elementary-1", "bucket_id": "97c1cfac-009f-4f7d-8d9d-9097c322c606.51988974.133", "new_instance_id": "", "old_num_shards": 3, "new_num_shards": 12 } ] As i wanted to process on taks in this queue I tried to run : radosgw-admin reshard process But turns out in an error : [root@ceph_monitor01 ~]# radosgw-admin reshard process ERROR: failed to process reshard logs, error=(22) Invalid argument 2020-05-14 14:15:10.225362 7f99b0437dc0 0 RGWReshardLock::lock failed to acquire lock on bucket-college-35:97c1cfac-009f-4f7d-8d9d-9097c322c606.51988974.133 ret=-22 2020-05-14 14:15:10.225376 7f99b0437dc0 0 process_single_logshardERROR in reshard_bucket bucket-elementary-1:(22) Invalid argument Tried to cancel it tu do it manually but had the same error [root@ceph_monitor01 ~]# radosgw-admin reshard cancel --bucket bucket-elementary-1 Error canceling bucket bucket-elementary-1 resharding: (22) Invalid argument 2020-05-14 14:16:42.196023 7fa0b1655dc0 0 RGWReshardLock::lock failed to acquire lock on bucket-elementary-1:97c1cfac-009f-4f7d-8d9d-9097c322c606.51988974.133 ret=-22 I found this post searching for an answer : https://tracker.ceph.com/issues/39970 But it seems that it will not help me since my whole cluster (monitors / data nodes / rgw) is on : [root@ceph_monitor01 ~]# ceph --version ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable) This is what I have on the status of this bucket : [root@ceph_monitor01 ~]# radosgw-admin reshard status --bucket= bucket-elementary-1 [ { "reshard_status": "CLS_RGW_RESHARD_NONE", "new_bucket_instance_id": "", "num_shards": 18446744073709551615 }, { "reshard_status": "CLS_RGW_RESHARD_NONE", "new_bucket_instance_id": "", "num_shards": 18446744073709551615 }, { "reshard_status": "CLS_RGW_RESHARD_NONE", "new_bucket_instance_id": "", "num_shards": 18446744073709551615 } ] Any help would be appreciated. Regards,

1 0

Re: RGW and the orphans
by EDH - Manuel Rios 14 May '20

14 May '20

Hi Eric Any update about that? Cluster status are critical and there's no a simple tool or cli provided in current releases that help to maintain our S3 Clusters Healthy. Right now with the multipart/sharding bugs looks like a bunch of scrap. Regards Manuel -----Mensaje original----- De: EDH - Manuel Rios <mriosfer(a)easydatahost.com> Enviado el: martes, 5 de mayo de 2020 15:34 Para: Katarzyna Myrek <katarzyna(a)myrek.pl>; Eric Ivancich <ivancich(a)redhat.com> CC: ceph-users(a)ceph.io Asunto: [ceph-users] Re: RGW and the orphans Hi Eric, Expected version to be included your tool in Nautilus? Maybe next reléase? Best Regards Manuel -----Mensaje original----- De: Katarzyna Myrek <katarzyna(a)myrek.pl> Enviado el: lunes, 20 de abril de 2020 12:19 Para: Eric Ivancich <ivancich(a)redhat.com> CC: EDH - Manuel Rios <mriosfer(a)easydatahost.com>; ceph-users(a)ceph.io Asunto: Re: [ceph-users] RGW and the orphans Hi Eric, I will try your tool this week on lab clusters. Will get back to you when I get the results. Kind regards / Pozdrawiam, Katarzyna Myrek pt., 17 kwi 2020 o 21:12 Eric Ivancich <ivancich(a)redhat.com> napisał(a): > > On Apr 17, 2020, at 9:38 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > Hi Eric, > > Would it be possible to use it with an older cluster version (like > running new radosgw-admin in the container, connecting to the cluster > on 14.2.X)? > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > > I did mention the nautilus backport PR in a separate reply (https://github.com/ceph/ceph/pull/34127) > > You can try the master version and see. To the best of my recollection the code porting did not involve any at-rest data structures. Instead it involved internal reorganization of the code. I suspect it would work, but if you try it, please report back what you find. Of course this is currently an experimental feature and care (e.g., sanity checking) should be taken before using the list produced to feed into a massive delete process. > > Eric > > -- > J. Eric Ivancich > > he / him / his > Red Hat Storage > Ann Arbor, Michigan, USA > > > > czw., 16 kwi 2020 o 19:58 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi Eric, > > > > Are there any ETA for get those script backported maybe in 14.2.10? > > > > Regards > > Manuel > > > > > > De: Eric Ivancich <ivancich(a)redhat.com> Enviado el: jueves, 16 de > abril de 2020 19:05 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl>; EDH - Manuel Rios > <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] RGW and the orphans > > > > There is currently a PR for an “orphans list” capability. I’m currently working on the testing side to make sure it’s part of our teuthology suite. > > > > See: https://github.com/ceph/ceph/pull/34148 > > > > Eric > > > > > > On Apr 16, 2020, at 9:26 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi > > Thanks for the quick response. > > To be honest my cluster is getting full because of that trash and I am > at the point where I have to do the removal manually ;/. > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> Enviado el: jueves, 16 de > abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph meltdown, need help
by Frank Schilder 14 May '20

14 May '20

Hi all, a lot of OSDs crashed in our cluster. Mimic 13.2.8. Current status included below. All daemons are running, no OSD process crashed. Can I start marking OSDs in and up to get them back talking to each other? Please advice on next steps. Thanks!! [root@gnosis ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN 2 MDSs report slow metadata IOs 1 MDSs report slow requests nodown,noout,norecover flag(s) set 125 osds down 3 hosts (48 osds) down Reduced data availability: 2221 pgs inactive, 1943 pgs down, 190 pgs peering, 13 pgs stale Degraded data redundancy: 5134396/500993581 objects degraded (1.025%), 296 pgs degraded, 299 pgs undersized 9622 slow ops, oldest one blocked for 2913 sec, daemons [osd.0,osd.100,osd.101,osd.112,osd.118,osd.133,osd.136,osd.142,osd.144,osd.145]... have slow ops. services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-02(active), standbys: ceph-03, ceph-01 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 90 up, 215 in; 230 remapped pgs flags nodown,noout,norecover data: pools: 10 pools, 2545 pgs objects: 62.61 M objects, 144 TiB usage: 219 TiB used, 1.6 PiB / 1.8 PiB avail pgs: 1.729% pgs unknown 85.540% pgs not active 5134396/500993581 objects degraded (1.025%) 1796 down 226 active+undersized+degraded 147 down+remapped 140 peering 65 active+clean 44 unknown 38 undersized+degraded+peered 38 remapped+peering 17 active+undersized+degraded+remapped+backfill_wait 12 stale+peering 12 active+undersized+degraded+remapped+backfilling 4 active+undersized+remapped 2 remapped 2 undersized+degraded+remapped+peered 1 stale 1 undersized+degraded+remapped+backfilling+peered io: client: 26 KiB/s rd, 206 KiB/s wr, 21 op/s rd, 50 op/s wr [root@gnosis ~]# ceph health detail HEALTH_WARN 2 MDSs report slow metadata IOs; 1 MDSs report slow requests; nodown,noout,norecover flag(s) set; 125 osds down; 3 hosts (48 osds) down; Reduced data availability: 2219 pgs inactive, 1943 pgs down, 188 pgs peering, 13 pgs stale; Degraded data redundancy: 5214696/500993589 objects degraded (1.041%), 298 pgs degraded, 299 pgs undersized; 9788 slow ops, oldest one blocked for 2953 sec, daemons [osd.0,osd.100,osd.101,osd.112,osd.118,osd.133,osd.136,osd.142,osd.144,osd.145]... have slow ops. MDS_SLOW_METADATA_IO 2 MDSs report slow metadata IOs mdsceph-08(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 2940 secs mdsceph-12(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 2942 secs MDS_SLOW_REQUEST 1 MDSs report slow requests mdsceph-08(mds.0): 100 slow requests are blocked > 30 secs OSDMAP_FLAGS nodown,noout,norecover flag(s) set OSD_DOWN 125 osds down osd.0 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.6 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.7 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.8 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.16 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.18 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.19 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.21 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.31 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.37 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.38 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.48 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.51 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-22) is down osd.53 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.55 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.62 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.67 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.72 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.75 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.78 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.79 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.80 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.81 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.82 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.83 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.88 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.89 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.92 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.93 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.95 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.96 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.97 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.100 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.104 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.105 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.107 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.108 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.109 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.111 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.113 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.114 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.116 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.117 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.119 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.122 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.123 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.124 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.125 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.126 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.128 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.131 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.134 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.139 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.140 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.141 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.145 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.149 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.151 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.152 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.153 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.154 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.155 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.156 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.157 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-05) is down osd.159 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.161 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.162 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.164 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.165 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.166 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.167 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.171 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.172 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.174 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.176 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.177 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.179 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.182 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-06) is down osd.183 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.184 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.186 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.187 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.190 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.191 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.194 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.195 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.196 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.199 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.200 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.201 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.202 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.203 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.204 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.208 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.210 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.212 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.213 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.214 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.215 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.216 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.218 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.219 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.221 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.224 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.226 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.228 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.230 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.233 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.236 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.238 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.247 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.248 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.254 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.256 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.259 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.260 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.262 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.266 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.267 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.272 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.274 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.275 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.276 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-22) is down osd.281 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-22) is down osd.285 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-05) is down OSD_HOST_DOWN 3 hosts (48 osds) down host ceph-11 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1) (16 osds) is down host ceph-10 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1) (16 osds) is down host ceph-13 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1) (16 osds) is down PG_AVAILABILITY Reduced data availability: 2219 pgs inactive, 1943 pgs down, 188 pgs peering, 13 pgs stale pg 14.513 is stuck inactive for 1681.564244, current state down, last acting [2147483647,2147483647,2147483647,2147483647,2147483647,143,2147483647,2147483647,2147483647,2147483647] pg 14.514 is down, acting [193,2147483647,2147483647,2147483647,2147483647,118,2147483647,2147483647,2147483647,2147483647] pg 14.515 is down, acting [2147483647,2147483647,2147483647,211,133,135,2147483647,2147483647,2147483647,2147483647] pg 14.516 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,205,2147483647] pg 14.517 is down, acting [2147483647,2147483647,5,2147483647,2147483647,2147483647,2147483647,2147483647,61,112] pg 14.518 is down, acting [2147483647,198,2147483647,2147483647,2147483647,2147483647,4,185,2147483647,2147483647] pg 14.519 is down, acting [2147483647,2147483647,68,2147483647,2147483647,2147483647,2147483647,185,2147483647,94] pg 14.51a is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,101,2147483647] pg 14.51b is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,197,2147483647,2147483647,2147483647,2147483647] pg 14.51c is down, acting [193,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,197] pg 14.51d is down, acting [2147483647,2147483647,61,2147483647,77,2147483647,2147483647,2147483647,112,2147483647] pg 14.51e is down, acting [2147483647,2147483647,2147483647,2147483647,112,2147483647,2147483647,193,2147483647,2147483647] pg 14.51f is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,94,2147483647,2147483647] pg 14.520 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,207,2147483647,101,133,2147483647] pg 14.521 is down, acting [205,2147483647,133,2147483647,2147483647,2147483647,2147483647,4,2147483647,193] pg 14.522 is down, acting [101,2147483647,2147483647,11,197,2147483647,136,94,2147483647,2147483647] pg 14.523 is down, acting [2147483647,2147483647,2147483647,118,2147483647,71,2147483647,2147483647,2147483647,2147483647] pg 14.524 is down, acting [2147483647,111,2147483647,2147483647,2147483647,8,2147483647,112,2147483647,2147483647] pg 14.525 is down, acting [2147483647,2147483647,2147483647,142,2147483647,61,2147483647,2147483647,2147483647,2147483647] pg 14.526 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,61,193,2147483647,2147483647,2147483647] pg 14.527 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,109,2147483647,2147483647] pg 14.528 is down, acting [2147483647,133,2147483647,2147483647,2147483647,2147483647,4,2147483647,2147483647,2147483647] pg 14.529 is down, acting [2147483647,112,2147483647,2147483647,2147483647,2147483647,185,2147483647,118,2147483647] pg 14.52a is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,136,2147483647,135,2147483647,2147483647] pg 14.52b is down, acting [2147483647,2147483647,2147483647,112,142,211,2147483647,2147483647,2147483647,2147483647] pg 14.52c is down, acting [185,2147483647,198,2147483647,118,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.52d is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,5,2147483647,2147483647,2147483647] pg 14.52e is down, acting [71,101,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,142,2147483647] pg 14.52f is down, acting [198,2147483647,2147483647,2147483647,2147483647,11,2147483647,2147483647,118,2147483647] pg 14.530 is down, acting [142,2147483647,2147483647,2147483647,133,2147483647,2147483647,2147483647,2147483647,112] pg 14.531 is down, acting [2147483647,142,2147483647,2147483647,2147483647,185,2147483647,2147483647,2147483647,2147483647] pg 14.532 is down, acting [135,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,136,118] pg 14.533 is down, acting [2147483647,77,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.534 is down, acting [2147483647,2147483647,2147483647,185,118,2147483647,2147483647,207,2147483647,2147483647] pg 14.535 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,136,142,133,2147483647] pg 14.536 is down, acting [2147483647,11,2147483647,2147483647,136,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.537 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,77,2147483647] pg 14.538 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,205,2147483647,2147483647] pg 14.539 is down, acting [2147483647,2147483647,2147483647,198,2147483647,2147483647,4,2147483647,2147483647,2147483647] pg 14.53a is down, acting [2147483647,11,136,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.53b is down, acting [2147483647,2147483647,2147483647,2147483647,112,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.53c is down, acting [2147483647,2147483647,2147483647,71,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.53d is down, acting [2147483647,2147483647,2147483647,185,2147483647,2147483647,2147483647,2147483647,2147483647,136] pg 14.53e is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,112,185] pg 14.53f is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,185,2147483647,2147483647,2147483647] pg 14.540 is down, acting [205,2147483647,2147483647,2147483647,2147483647,2147483647,142,2147483647,112,77] pg 14.541 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,197,211,2147483647,2147483647,2147483647] pg 14.542 is down, acting [112,2147483647,101,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.543 is down, acting [111,2147483647,2147483647,2147483647,2147483647,101,2147483647,2147483647,2147483647,2147483647] pg 14.544 is down, acting [4,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,205] pg 14.545 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,142,5,2147483647,2147483647,2147483647] PG_DEGRADED Degraded data redundancy: 5214696/500993589 objects degraded (1.041%), 298 pgs degraded, 299 pgs undersized pg 1.29 is stuck undersized for 2075.633328, current state active+undersized+degraded, last acting [253,258] pg 1.2a is stuck undersized for 1642.864920, current state active+undersized+degraded, last acting [252,255] pg 1.2b is stuck undersized for 2355.149928, current state active+undersized+degraded+remapped+backfill_wait, last acting [240,268] pg 1.2c is stuck undersized for 1459.277329, current state active+undersized+degraded, last acting [241,273] pg 1.2d is stuck undersized for 803.339131, current state undersized+degraded+peered, last acting [282] pg 2.25 is active+undersized+degraded, acting [253,2147483647,2147483647,258,261,273,277,243] pg 2.28 is stuck undersized for 803.340163, current state active+undersized+degraded, last acting [282,241,246,2147483647,273,252,2147483647,268] pg 2.29 is stuck undersized for 803.341160, current state active+undersized+degraded, last acting [240,258,277,264,2147483647,2147483647,271,250] pg 2.2a is stuck undersized for 1447.684978, current state active+undersized+degraded+remapped+backfilling, last acting [252,270,2147483647,261,2147483647,255,287,264] pg 2.2e is stuck undersized for 2030.849944, current state active+undersized+degraded, last acting [264,2147483647,251,245,257,286,261,258] pg 2.51 is stuck undersized for 1459.274671, current state active+undersized+degraded+remapped+backfilling, last acting [270,2147483647,2147483647,265,241,243,240,252] pg 2.52 is stuck undersized for 2030.850897, current state active+undersized+degraded+remapped+backfilling, last acting [240,2147483647,270,265,269,280,278,2147483647] pg 2.53 is stuck undersized for 1459.273517, current state active+undersized+degraded, last acting [261,2147483647,280,282,2147483647,245,243,241] pg 2.61 is stuck undersized for 2075.633140, current state active+undersized+degraded+remapped+backfilling, last acting [269,2147483647,258,286,270,255,2147483647,264] pg 2.62 is stuck undersized for 803.340577, current state active+undersized+degraded, last acting [2147483647,253,258,2147483647,250,287,264,284] pg 2.66 is stuck undersized for 803.341231, current state active+undersized+degraded, last acting [264,280,265,255,257,269,2147483647,270] pg 2.6c is stuck undersized for 963.369539, current state active+undersized+degraded, last acting [286,269,278,251,2147483647,273,2147483647,280] pg 2.70 is stuck undersized for 873.662725, current state active+undersized+degraded, last acting [2147483647,268,255,273,253,265,278,2147483647] pg 2.74 is stuck undersized for 2075.632312, current state active+undersized+degraded+remapped+backfilling, last acting [240,242,2147483647,245,243,269,2147483647,265] pg 3.24 is stuck undersized for 1570.800184, current state active+undersized+degraded, last acting [235,263] pg 3.25 is stuck undersized for 733.673503, current state undersized+degraded+peered, last acting [232] pg 3.28 is stuck undersized for 2610.307886, current state active+undersized+degraded, last acting [263,84] pg 3.2a is stuck undersized for 1214.710839, current state active+undersized+degraded, last acting [181,232] pg 3.2b is stuck undersized for 2075.630671, current state active+undersized+degraded, last acting [63,144] pg 3.52 is stuck undersized for 1570.777598, current state active+undersized+degraded, last acting [158,237] pg 3.54 is stuck undersized for 1350.257189, current state active+undersized+degraded, last acting [239,74] pg 3.55 is stuck undersized for 2592.642531, current state active+undersized+degraded, last acting [157,233] pg 3.5a is stuck undersized for 2075.608257, current state undersized+degraded+peered, last acting [168] pg 3.5c is stuck undersized for 733.674836, current state active+undersized+degraded, last acting [263,234] pg 3.5d is stuck undersized for 2610.307220, current state active+undersized+degraded, last acting [180,84] pg 3.5e is stuck undersized for 1710.756037, current state undersized+degraded+peered, last acting [146] pg 3.61 is stuck undersized for 1080.210021, current state active+undersized+degraded, last acting [168,239] pg 3.62 is stuck undersized for 831.217622, current state active+undersized+degraded, last acting [84,263] pg 3.63 is stuck undersized for 733.674204, current state active+undersized+degraded, last acting [263,232] pg 3.65 is stuck undersized for 1570.790824, current state active+undersized+degraded, last acting [63,84] pg 3.66 is stuck undersized for 733.682973, current state undersized+degraded+peered, last acting [63] pg 3.68 is stuck undersized for 1570.624462, current state active+undersized+degraded, last acting [229,148] pg 3.69 is stuck undersized for 1350.316213, current state undersized+degraded+peered, last acting [235] pg 3.6b is stuck undersized for 783.813654, current state undersized+degraded+peered, last acting [63] pg 3.6c is stuck undersized for 783.819083, current state undersized+degraded+peered, last acting [229] pg 3.6f is stuck undersized for 2610.321349, current state active+undersized+degraded, last acting [232,158] pg 3.72 is stuck undersized for 1350.358149, current state active+undersized+degraded, last acting [229,74] pg 3.73 is stuck undersized for 1570.788310, current state undersized+degraded+peered, last acting [234] pg 11.20 is stuck undersized for 733.682510, current state active+undersized+degraded, last acting [2147483647,239,87,2147483647,158,237,63,76] pg 11.26 is stuck undersized for 1914.334332, current state active+undersized+degraded, last acting [2147483647,237,2147483647,263,158,148,181,180] pg 11.2d is stuck undersized for 1350.365988, current state active+undersized+degraded, last acting [2147483647,2147483647,73,229,86,158,169,84] pg 11.54 is stuck undersized for 1914.398125, current state active+undersized+degraded, last acting [231,169,2147483647,229,84,85,237,63] pg 11.5b is stuck undersized for 2047.980719, current state active+undersized+degraded, last acting [86,237,168,263,144,1,229,2147483647] pg 11.5e is stuck undersized for 873.643661, current state active+undersized+degraded, last acting [181,2147483647,229,158,231,1,169,2147483647] pg 11.62 is stuck undersized for 1144.491696, current state active+undersized+degraded, last acting [2147483647,85,235,74,63,234,181,2147483647] pg 11.6f is stuck undersized for 873.646628, current state active+undersized+degraded, last acting [234,3,2147483647,158,180,63,2147483647,181] SLOW_OPS 9788 slow ops, oldest one blocked for 2953 sec, daemons [osd.0,osd.100,osd.101,osd.112,osd.118,osd.133,osd.136,osd.142,osd.144,osd.145]... have slow ops. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

6 19

Re: Memory usage of OSD
by Igor Fedotov 14 May '20

14 May '20

Rafal, just to mention - stupid allocator is known to cause high memory usage in certain scenarios but it uses bluestore_alloc mempool. Thanks, Igor On 5/13/2020 6:52 PM, Rafał Wądołowski wrote: > Mark, > Unfortunetly I closed terminal with mempool. But there was a lot of bytes used by bluestore_cache_other. That was the highest value (about 85%). The onode cache takes about 10%. PGlog and osdmaps was okey, low values. I saw some ideas that maybe compression_mode force in pool can make a mess. > One more thing, we are running stupid allocator. Right now I am decrease the osd_memory_target to 3GiB and will wait if ram problem occurs. > > > > > Regards, > > Rafał Wądołowski > > ________________________________ > From: Mark Nelson <mnelson(a)redhat.com> > Sent: Wednesday, May 13, 2020 3:30 PM > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Memory usage of OSD > > On 5/13/20 12:43 AM, RafaĹ' WÄ...doĹ'owski wrote: >> Hi, >> I noticed strange situation in one of our clusters. The OSD deamons are taking too much RAM. >> We are running 12.2.12 and have default configuration of osd_memory_target (4GiB). >> Heap dump shows: >> >> osd.2969 dumping heap profile now. >> ------------------------------------------------ >> MALLOC: 6381526944 ( 6085.9 MiB) Bytes in use by application >> MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist >> MALLOC: + 173373288 ( 165.3 MiB) Bytes in central cache freelist >> MALLOC: + 17163520 ( 16.4 MiB) Bytes in transfer cache freelist >> MALLOC: + 95339512 ( 90.9 MiB) Bytes in thread cache freelists >> MALLOC: + 28995744 ( 27.7 MiB) Bytes in malloc metadata >> MALLOC: ------------ >> MALLOC: = 6696399008 ( 6386.2 MiB) Actual memory used (physical + swap) >> MALLOC: + 218267648 ( 208.2 MiB) Bytes released to OS (aka unmapped) >> MALLOC: ------------ >> MALLOC: = 6914666656 ( 6594.3 MiB) Virtual address space used >> MALLOC: >> MALLOC: 408276 Spans in use >> MALLOC: 75 Thread heaps in use >> MALLOC: 8192 Tcmalloc page size >> ------------------------------------------------ >> Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). >> Bytes released to the OS take up virtual address space but no physical memory. >> >> IMO "Bytes in use by application" should be less than osd_memory_target. Am I correct? >> I checked heap dump with google-pprof and got following results. >> Total: 149.4 MB >> 60.5 40.5% 40.5% 60.5 40.5% rocksdb::UncompressBlockContentsForCompressionType >> 34.2 22.9% 63.4% 34.2 22.9% ceph::buffer::create_aligned_in_mempool >> 11.9 7.9% 71.3% 12.1 8.1% std::_Rb_tree::_M_emplace_hint_unique >> 10.7 7.1% 78.5% 71.2 47.7% rocksdb::ReadBlockContents >> >> Does it mean that most of RAM is used by rocksdb? > > It looks like your heap dump is only accounting for 149.4MB of the > memory so probably not representative across the whole ~6.5G. Instead > could you try dumping the mempools via "ceph daemon osd.2969 dump_mempools"? > > >> How can I take a deeper look into memory usage ? > > Beyond looking at the mempools, you can see the bluestore cache > allocation information by either enabling debug bluestore and debug > priority_cache_manager 5, or potentially looking at the PCM perf > counters (I'm not sure if those were in 14.2.12 though). Between the > heap data, mempool data, and priority cache records, it should become > clearer what's going on. > > > Mark > > >> >> Regards, >> >> RafaĹ' WÄ...doĹ'owski >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

iscsi issues with ceph (Nautilus) + tcmu-runner
by Phil Regnauld 14 May '20

14 May '20

So, we've been running with iscsi enabled (tcmu-runner) on our Nautilus ceph cluster for a couple of weeks, and started using it with our vsphere cluster. Things looked good so we put it in production, but yesterday morning we experienced a freeze of all iSCSO I/O one of the ESXi nodes, and the only way to recover was a reboot of the client node (and the VMs with it). At first I thought this was a glitch on the VMware side, as we saw lots of warnings, but after digging in the kernel logs on the ceph side, we saw quite a few of the messages below, leading up to the freeze. We've got two iscsi gateways enabled, and have set things up as per https://docs.ceph.com/docs/mimic/rbd/iscsi-initiators/ Kernel is 4.19 / Debian 10. This is single 10G link for the ESXi nodes, with separate vlan and dedicated vmkernel nic on the storage vlan to talk to the iscsi gateways on the ceph side. Ceph is 10 nodes with a mix of HDD and SSD. The affected pool is replication on HDD w/NVMe block db in front. No overrun / errors on the interfaces, and no mtu mismatch - not that I believe this is connected to the error below. No errors/warnings on the ceph side. Anyone seen this before ? Can provide more details/logs as required. If this happens again, we'll be moving the VMs to an NFS backed vm store (even if iscsi is probably preferred) until we can find a solution to go into full prod. Cheers, Phil [Tue May 12 05:15:45 2020] WARNING: CPU: 8 PID: 2448784 at kernel/workqueue.c:2917 __flush_work.cold.54+0x1f/0x29 [Tue May 12 05:15:45 2020] Modules linked in: fuse cbc ceph libceph libcrc32c crc32c_generic fscache target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_user uio target_core_mod configfs binfmt_misc 8021q garp stp mrp llc bonding intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast cryptd ipmi_ssif ttm intel_cstate drm_kms_helper intel_uncore mei_me iTCO_wdt drm pcc_cpufreq intel_rapl_perf joydev pcspkr sg iTCO_vendor_support mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter evdev dm_mod sunrpc ip_tables x_tables autofs4 squashfs zstd_decompress xxhash loop overlay hid_generic usbhid hid sd_mod ahci xhci_pci ehci_pci libahci ehci_hcd xhci_hcd libata ixgbe igb nvme mxm_wmi lpc_ich [Tue May 12 05:15:45 2020] dca usbcore i2c_i801 scsi_mod mfd_core mdio usb_common nvme_core crc32c_intel i2c_algo_bit wmi button [Tue May 12 05:15:45 2020] CPU: 8 PID: 2448784 Comm: kworker/u32:0 Tainted: G W 4.19.0-8-amd64 #1 Debian 4.19.98-1 [Tue May 12 05:15:45 2020] Hardware name: Supermicro SYS-6018R-TD8/X10DDW-i, BIOS 3.2 12/16/2019 [Tue May 12 05:15:45 2020] Workqueue: tmr-user target_tmr_work [target_core_mod] [Tue May 12 05:15:45 2020] RIP: 0010:__flush_work.cold.54+0x1f/0x29 [Tue May 12 05:15:45 2020] Code: 69 2c 04 00 0f 0b e9 4a d3 ff ff 48 c7 c7 d8 f9 83 be e8 56 2c 04 00 0f 0b e9 41 d6 ff ff 48 c7 c7 d8 f9 83 be e8 43 2c 04 00 <0f> 0b 45 31 ed e9 2b d6 ff ff 49 8d b4 24 b0 00 00 00 48 c7 c7 b8 [Tue May 12 05:15:45 2020] RSP: 0018:ffffa0180b537d38 EFLAGS: 00010246 [Tue May 12 05:15:45 2020] RAX: 0000000000000024 RBX: ffff94f619c3fad0 RCX: 0000000000000006 [Tue May 12 05:15:45 2020] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff94f77fa166b0 [Tue May 12 05:15:45 2020] RBP: ffff94f619c3fad0 R08: 00000000000013ea R09: 0000000000aaaaaa [Tue May 12 05:15:45 2020] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 [Tue May 12 05:15:45 2020] R13: 0000000000000001 R14: ffffffffbda98400 R15: ffff94f5a3ecc088 [Tue May 12 05:15:45 2020] FS: 0000000000000000(0000) GS:ffff94f77fa00000(0000) knlGS:0000000000000000 [Tue May 12 05:15:45 2020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue May 12 05:15:45 2020] CR2: 0000555d72f0f000 CR3: 000000048240a005 CR4: 00000000003606e0 [Tue May 12 05:15:45 2020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [Tue May 12 05:15:45 2020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [Tue May 12 05:15:45 2020] Call Trace: [Tue May 12 05:15:45 2020] ? __irq_work_queue_local+0x50/0x60 [Tue May 12 05:15:45 2020] ? irq_work_queue+0x46/0x50 [Tue May 12 05:15:45 2020] ? wake_up_klogd+0x30/0x40 [Tue May 12 05:15:45 2020] ? vprintk_emit+0x215/0x270 [Tue May 12 05:15:45 2020] ? get_work_pool+0x40/0x40 [Tue May 12 05:15:45 2020] __cancel_work_timer+0x10a/0x190 [Tue May 12 05:15:45 2020] ? printk+0x58/0x6f [Tue May 12 05:15:45 2020] core_tmr_abort_task+0xd6/0x130 [target_core_mod] [Tue May 12 05:15:45 2020] target_tmr_work+0xc4/0x140 [target_core_mod] [Tue May 12 05:15:45 2020] process_one_work+0x1a7/0x3a0 [Tue May 12 05:15:45 2020] worker_thread+0x30/0x390 [Tue May 12 05:15:45 2020] ? create_worker+0x1a0/0x1a0 [Tue May 12 05:15:45 2020] kthread+0x112/0x130 [Tue May 12 05:15:45 2020] ? kthread_bind+0x30/0x30 [Tue May 12 05:15:45 2020] ret_from_fork+0x1f/0x40 [Tue May 12 05:15:45 2020] ---[ end trace 3835b5fe0aa99038 ]--- [Tue May 12 05:15:45 2020] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 94585265 [Tue May 12 05:15:45 2020] ABORT_TASK: Found referenced iSCSI task_tag: 94585266 [Tue May 12 05:15:45 2020] ------------[ cut here ]------------

2 2