August 2019 - ceph-users

by Ramanathan S

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 weeks

3
3
0 0

osd_pg_create causing slow requests in Nautilus

by Bryan Stillwell

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

4 years

6
16
0 0

official ceph.com buster builds?

by Chad W Seys

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

4 years, 1 month

2
2
0 0

MDS failing under load with large cache sizes

by Janek Bevendorff

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 years, 3 months

4
33
0 0

Identify rbd snapshot

by Marc Roos

I have this error. I have found the rbd image with the block_name_prefix:1f114174b0dc51, how can identify what snapshot this is? (Is it a snapshot?) 2019-08-29 16:16:49.255183 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing 2019-08-29 16:24:54.806912 7f9b3f061700 -1 log_channel(cluster) log [ERR] : 17.36 deep-scrub 1 errors

4 years, 7 months

2
1
0 0

645% Clean PG's in Dashboard

by c.lilja＠falseprivacy.org

Hi, I've upgraded to Nautilus from Mimic a while ago and enabled the pg_autoscaler. When pg_autoscaler was activated I got a HEALTH_WARN regarding: POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_ratio 0.000 on pools [] Both target_size_bytes and target_size_ratio on all the pools are set to 0, so I started to wonder why this error message appear. My autoscale-status looks like this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 16708M 4.0 34465G 0.0019 1.0 8 warn cephfs_data_reduced 15506G 2.0 34465G 0.8998 1.0 375 warn cephfs_data 6451G 3.0 34465G 0.5616 1.0 250 warn So the ratio in total is 1.4633.. Isn't 1.0 of the combined ratio of all pools equal of full? I also enabled the Dashboard and saw that the PG Status showed "645% clean" PG's. This cluster was originally installed with version Jewel, so may it be any legacy setting or such that causing this?

4 years, 7 months

2
1
0 0

Re: FileStore OSD, journal direct symlinked, permission troubles.

by Marco Gaiarin

Riprendo quanto scritto nel suo messaggio del 29/08/2019... > Another possibilty is to convert the MBR to GPT (sgdisk --mbrtogpt) and > give the partition its UID (also sgdisk). Then it could be linked by > its uuid. and, in another email: > And I forgot that you can also re-create the journal by itself. I can't > recall the command ATM though. Ahem, i stated the jornal disk are also the OS disks, and i'm using old server, so i think that converting to GPT will lead to an unbootable node... But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter... Also, i've done: adduser ceph disk and partition devices are '660 root:disk': why still i get 'permission denied'? > Or if you are not in need of filestore OSDs, re-create them as bluestore > ones. AFAICS, Ceph has laid more focus on bluestore and it might be > better to do a conversion sooner than later. (my opinion) Not for now; bluestore migration need a bit more time/study/knowledge... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

4 years, 7 months

2
6
0 0

Out of memory

by Sylvain PORTIER

Hi, On my ceph osd servers I have lot of "out of memory messages". My servers are configured with : - 32 G of memory - 11 HDD (3,5 T each) (+ 2 HDD for the system) And the error messages are : /[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// / Top result : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd Is there a way to limit or reduce the memory usage of each osd deamon ? Thank you, Regards, Sylvain PORTIER. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus

4 years, 7 months

4
5
0 0

Heavily-linked lists.ceph.com pipermail archive now appears to lead to 404s

by Florian Haas

Hi, is there any chance the list admins could copy the pipermail archive from lists.ceph.com over to lists.ceph.io? It seems to contain an awful lot of messages referred elsewhere by their archive URL, many (all?) of which appear to now lead to 404s. Example: google "Set existing pools to use hdd device class only". The top hit is a link to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html: $ curl -IL http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html HTTP/1.1 301 Moved Permanently Server: nginx/1.10.3 (Ubuntu) Date: Thu, 29 Aug 2019 12:48:13 GMT Content-Type: text/html Content-Length: 194 Connection: keep-alive Location: https://lists.ceph.io/pipermail/ceph-users-ceph.com/2018-August/029078.html Strict-Transport-Security: max-age=31536000 HTTP/1.1 404 Not Found Server: nginx Date: Thu, 29 Aug 2019 12:48:14 GMT Content-Type: text/html; charset=utf-8 Content-Length: 3774 Connection: keep-alive X-Frame-Options: SAMEORIGIN Vary: Accept-Language, Cookie Content-Language: en Or maybe this is just a redirect rule that needs to be cleverer or more specific, rather than the apparent catch-all .com/.io redirect? Cheers, Florian

4 years, 7 months

5
7
1 0

ceph fs crashes on simple fio test

by Frank Schilder

I need to harden our ceph cluster to satisfy the following properties: Assuming all hardware is functioning properly, 1) Cluster health has highest priority. Heartbeats have priority over client requests. 2) The cluster does not accept more IO than the OSDs can handle. The only exception might be a configurable burst option. 3) Client IO is accepted as long as it does not compromise 1. 4) Ideally, there is fair sharing of the cluster's IO budget between clients (like deadline or completely fair scheduling). Rogue clients should not get priority just because they push a lot. Unfortunately, with default settings our (any?) cluster prefers client IO over cluster health, which opens up for a simple but serious non-privileged client attack on cluster health: I observed a serious issue on our cluster when running a simple fio test that does 4K random writes on 100GB files (see details below). What I observe is that within a few seconds the cluster goes to health_warn with the MDS reporting slow meta data IO and behind on trimming. What is not shown in ceph health detail is, that all OSDs report thousands of slow ops and the counter increases really fast (I include some snippets below). This goes rapidly to the point where OSDs start missing heartbeats and start flapping, some PGs become inactive+degraded and start peering regularly. CPU load, memory and even network load were rather low (one-figure % CPU load). Nothing out of the ordinary. The mons had no issues. After the fio test completed, the OSDs slowly crunched through the backlog and managed to complete all OPS. The complete processing of the ops of a 30 second fio test took ca. 10 minutes. However, even then the cluster did not come back healthy as essential messages between daemons seem to have been lost. It is not a long stretch to assume that one can destroy a ceph fs beyond repair when running this test or an application performing the same IO pattern from multiple clients for several hours. We have a 500 node cluster as clients and I'm afraid that even ordinary IO might trigger this scenario in unlucky circumstances. Since our cluster is only half-trusted (we have root access to clients, but no control over user IO patterns), we are in need to harden our cluster against such destructive IO patterns as much as possible. To me it looks like the cluster is accepting way more IO than the OSDs can handle. Ideally, what I would like to do is configure effective rate limiting on (rogue) clients depending on how much OPS they have in flight already. I would expect that there are tunables for MDS/OSD daemons that control how much IO requests a client can submit/OSD will accept before throttling IO. In particular, I would like to prioritize heartbeats to prevent load-induced OSD flapping. How can I tune the cluster to satisfy the conditions outlined at the top of this e-mail? There were recent threads with similar topics, in particular, "MDS failing under load with large cache sizes" and others reporting unstable MDS daemons under load. However, I believe they were mostly related to cache trimming issues due to large amounts of files created. This is not the case here, its just 4 files with lots of random IO from a single client. A bit of information about our cluster and observations: The cluster is bluestore-only with 8+2 EC fs data pool on spinning disks and an 3(2) replicated fs meta data pool on SSD. We have 8 OSD hosts with 2 shards per host. Each host has 4SSDs and 12 10TB HDDs SAS 12GB with 4k block size. Network is 2x10G bonded for client and 2x10G bonded for replication. The replication network will be extended to 4x10G soon. We are aware that currently the network bandwidth greatly exceeds what the spinning disks can handle. It is dimensioned for adding more disks in the future. The fio job script is: [global] name=fio-rand-write directory=/home/fio # /home is on ceph fs filename_format=tmp/fio-$jobname-${HOSTNAME}-$jobnum-$filenum rw=randwrite bs=4K numjobs=4 time_based=1 runtime=30 [file1] size=100G ioengine=sync Its one of the examples in the fio source repo with small modifications (file pattern, run time). Shortly after starting this fio job from a client connected with single 10G line, all OSDs start reporting slow ops. Picking one, the log messages look like this: Aug 22 12:06:22 ceph-09 ceph-osd: 2019-08-22 10:06:22.151 7f399d1bd700 -1 osd.125 3778 get_health_metrics reporting 146 slow ops, oldest is osd_op(client.1178165.0:1670940 5.3fs0 5:fcc65fe3:::10000f403bc.0000063e:head [write 307200~4096,write 315392~8192,write 380928~12288,write 405504~12288,write 421888~4096,write 458752~4096,write 466944~4096,write 475136~4096,write 487424~8192,write 512000~4096,write 524288~4096,write 565248~4096,write 589824~8192,write 622592~4096,write 651264~4096,write 724992~12288] snapc 12e=[] ondisk+write+known_if_redirected e3778) ... Aug 22 12:06:49 ceph-09 journal: 2019-08-22 10:06:49.415 7f399d1bd700 -1 osd.125 3779 get_health_metrics reporting 4595 slow ops, olde st is osd_op(client.1178165.0:1686216 5.acs0 5:354e96d5:::10000f403bc.00000bc1:head [write 2359296~4096,write 2375680~4096,write 23838 72~4096,write 2404352~4096,write 2428928~8192,write 2469888~4096,write 2490368~8192,write 2514944~4096,write 2527232~4096,write 253542 4~4096,write 2588672~4096,write 2600960~4096,write 2621440~8192,write 2658304~4096,write 2715648~8192,write 2727936~4096] snapc 12e=[] ondisk+write+known_if_redirected e3778) ... Aug 22 12:12:57 ceph-09 journal: 2019-08-22 10:12:57.650 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 8419 slow ops, oldest is osd_op(client.1178165.0:2009417 5.3fs0 5:fcdcf2bd:::10000f47e2d.00001501:head [write 2236416~4096,write 2256896~4096,write 2265088~4096,write 2301952~4096,write 2322432~4096,write 2355200~4096,write 2371584~4096,write 2387968~4096,write 2449408~4096,write 2486272~4096,write 2547712~8192,write 2617344~4096,write 2809856~4096,write 3018752~4096,write 3194880~4096,write 3223552~4096] snapc 12e=[] ondisk+write+known_if_redirected e3839) Aug 22 12:12:58 ceph-09 ceph-osd: 2019-08-22 10:12:58.681 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 8862 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+write+known_if_redirected+full_force e3839) ... Aug 22 12:13:27 ceph-09 journal: 2019-08-22 10:13:27.691 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 13795 slow ops, oldest is osd_op(mds.0.16909:69577573 5.e6s0 5.d8994de6 (undecoded) ondisk+write+known_if_redirected+full_force e3839) ... Aug 22 12:13:59 ceph-09 ceph-osd: 2019-08-22 10:13:59.762 7f399d1bd700 -1 osd.125 3900 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+retry+write+known_if_redirected+full_force e3875) ... Aug 22 12:14:46 ceph-09 journal: 2019-08-22 10:14:46.569 7f399d1bd700 -1 osd.125 3916 get_health_metrics reporting 969 slow ops, oldes t is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+retry+write+known_if_redirected+full_force e3875) Aug 22 12:14:47 ceph-09 ceph-osd: 2019-08-22 10:14:47.617 7f399d1bd700 -1 osd.125 3935 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5:7711c499:::10000f4798b.00000000:head [create,setxattr parent (289),setxattr layout (30)] snap c 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+full_force e3875) ... Aug 22 12:14:53 ceph-09 journal: 2019-08-22 10:14:53.675 7f399d1bd700 -1 osd.125 3939 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5:7711c499:::10000f4798b.00000000:head [create,setxattr parent (289),setxattr layout (30)] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+full_force e3875) This is the last log message, the OSD seems to have executed all OPS at this point or shortly after. The cluster state went to health_ok - at least shortly. At about 12:45 (yes, lunch break) we looked at the cluster again and it was back in health_warn with the following status output: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN 1 MDSs report slow metadata IOs 2 MDSs behind on trimming services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-02, ceph-03 mds: con-fs-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 192 osds: 192 up, 192 in data: pools: 7 pools, 790 pgs objects: 9.01 M objects, 16 TiB usage: 20 TiB used, 1.3 PiB / 1.3 PiB avail pgs: 790 active+clean io: client: 1.9 MiB/s rd, 21 MiB/s wr, 60 op/s rd, 721 op/s wr [root@ceph-01 ~]# ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; 2 MDSs behind on trimming MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mdsceph-08(mds.0): 2 slow metadata IOs are blocked > 30 secs, oldest blocked for 3275 secs MDS_TRIM 2 MDSs behind on trimming mdsceph-08(mds.0): Behind on trimming (1778/128) max_segments: 128, num_segments: 1778 mdsceph-12(mds.0): Behind on trimming (1780/128) max_segments: 128, num_segments: 1780 Num_segments was increasing constantly. Apparently, an operation got stuck and never completed. We hunted a little bit and found [root@ceph-mds:ceph-08 /]# ceph daemon mds.ceph-08 objecter_requests { "ops": [ { "tid": 71206302, "pg": "4.952611b7", "osd": 19, "object_id": "200.0001be74", "object_locator": "@4", "target_object_id": "200.0001be74", "target_object_locator": "@4", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 11:12:41.0.565912s", "osd_ops": [ "write 890478~1923" ] }, { "tid": 71206303, "pg": "4.952611b7", "osd": 19, "object_id": "200.0001be74", "object_locator": "@4", "target_object_id": "200.0001be74", "target_object_locator": "@4", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 11:12:41.0.566236s", "osd_ops": [ "write 892401~1931" ] }, { "tid": 71206301, "pg": "5.eeb918c6", "osd": 67, "object_id": "10000f26f67.00000000", "object_locator": "@5", "target_object_id": "10000f26f67.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "12e=[]", "mtime": "1970-01-01 00:00:00.000000s", "osd_ops": [ "trimtrunc 81854@573" ] }, { "tid": 69577573, "pg": "5.d8994de6", "osd": 125, "object_id": "10000f479c9.00000000", "object_locator": "@5", "target_object_id": "10000f479c9.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.256058s", "osd_ops": [ "create", "setxattr parent (319)", "setxattr layout (30)" ] }, { "tid": 69577598, "pg": "5.deb003e6", "osd": 125, "object_id": "10000f479da.00000000", "object_locator": "@5", "target_object_id": "10000f479da.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.258824s", "osd_ops": [ "create", "setxattr parent (288)", "setxattr layout (30)" ] }, { "tid": 71206300, "pg": "5.5cd5b20b", "osd": 163, "object_id": "10000f01396.00000000", "object_locator": "@5", "target_object_id": "10000f01396.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "12e=[]", "mtime": "1970-01-01 00:00:00.000000s", "osd_ops": [ "trimtrunc 208782@573" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } Notice the ops from the 1970's. Checking ops and dump_blocked_ops on osd.19 showed that these lists were empty. So, we decided to restart osd.19 and it cleared out most of the stuck requests, but did not clear the health warnings: [root@ceph-mds:ceph-08 /]# ceph daemon mds.ceph-08 objecter_requests { "ops": [ { "tid": 69577573, "pg": "5.d8994de6", "osd": 125, "object_id": "10000f479c9.00000000", "object_locator": "@5", "target_object_id": "10000f479c9.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.256058s", "osd_ops": [ "create", "setxattr parent (319)", "setxattr layout (30)" ] }, { "tid": 69577598, "pg": "5.deb003e6", "osd": 125, "object_id": "10000f479da.00000000", "object_locator": "@5", "target_object_id": "10000f479da.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.258824s", "osd_ops": [ "create", "setxattr parent (288)", "setxattr layout (30)" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } Restarting osd.125 finally resolved the health issues. However, the client I run fio on had lost connection to ceph due to this incident, which is really annoying. This client is the head node of our HPC cluster and it was not possible to restore ceph fs access without reboot. This is an additional bad fallout as all users will loose access to our HPC cluster when this happens (/home is on the ceph fs). I dumped the dump_historic_slow_ops of osd.125 in case anyone can use this information. I might be able to repeat this experiment, but cannot promise anything. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 years, 7 months

5
14
0 0

2024

2023

2022

2021

2020

2019

ceph-users August 2019