August 2019 - ceph-users

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

Identify rbd snapshot
by Marc Roos 21 Sep '19

21 Sep '19

I have this error. I have found the rbd image with the block_name_prefix:1f114174b0dc51, how can identify what snapshot this is? (Is it a snapshot?) 2019-08-29 16:16:49.255183 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing 2019-08-29 16:24:54.806912 7f9b3f061700 -1 log_channel(cluster) log [ERR] : 17.36 deep-scrub 1 errors

2 1

645% Clean PG's in Dashboard
by c.lilja＠falseprivacy.org 12 Sep '19

12 Sep '19

Hi, I've upgraded to Nautilus from Mimic a while ago and enabled the pg_autoscaler. When pg_autoscaler was activated I got a HEALTH_WARN regarding: POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_ratio 0.000 on pools [] Both target_size_bytes and target_size_ratio on all the pools are set to 0, so I started to wonder why this error message appear. My autoscale-status looks like this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 16708M 4.0 34465G 0.0019 1.0 8 warn cephfs_data_reduced 15506G 2.0 34465G 0.8998 1.0 375 warn cephfs_data 6451G 3.0 34465G 0.5616 1.0 250 warn So the ratio in total is 1.4633.. Isn't 1.0 of the combined ratio of all pools equal of full? I also enabled the Dashboard and saw that the PG Status showed "645% clean" PG's. This cluster was originally installed with version Jewel, so may it be any legacy setting or such that causing this?

2 1

Re: FileStore OSD, journal direct symlinked, permission troubles.
by Marco Gaiarin 11 Sep '19

11 Sep '19

Riprendo quanto scritto nel suo messaggio del 29/08/2019... > Another possibilty is to convert the MBR to GPT (sgdisk --mbrtogpt) and > give the partition its UID (also sgdisk). Then it could be linked by > its uuid. and, in another email: > And I forgot that you can also re-create the journal by itself. I can't > recall the command ATM though. Ahem, i stated the jornal disk are also the OS disks, and i'm using old server, so i think that converting to GPT will lead to an unbootable node... But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter... Also, i've done: adduser ceph disk and partition devices are '660 root:disk': why still i get 'permission denied'? > Or if you are not in need of filestore OSDs, re-create them as bluestore > ones. AFAICS, Ceph has laid more focus on bluestore and it might be > better to do a conversion sooner than later. (my opinion) Not for now; bluestore migration need a bit more time/study/knowledge... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

2 6

Out of memory
by Sylvain PORTIER 09 Sep '19

09 Sep '19

Hi, On my ceph osd servers I have lot of "out of memory messages". My servers are configured with : - 32 G of memory - 11 HDD (3,5 T each) (+ 2 HDD for the system) And the error messages are : /[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// / Top result : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd Is there a way to limit or reduce the memory usage of each osd deamon ? Thank you, Regards, Sylvain PORTIER. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus

4 5

Heavily-linked lists.ceph.com pipermail archive now appears to lead to 404s
by Florian Haas 05 Sep '19

05 Sep '19

Hi, is there any chance the list admins could copy the pipermail archive from lists.ceph.com over to lists.ceph.io? It seems to contain an awful lot of messages referred elsewhere by their archive URL, many (all?) of which appear to now lead to 404s. Example: google "Set existing pools to use hdd device class only". The top hit is a link to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html: $ curl -IL http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html HTTP/1.1 301 Moved Permanently Server: nginx/1.10.3 (Ubuntu) Date: Thu, 29 Aug 2019 12:48:13 GMT Content-Type: text/html Content-Length: 194 Connection: keep-alive Location: https://lists.ceph.io/pipermail/ceph-users-ceph.com/2018-August/029078.html Strict-Transport-Security: max-age=31536000 HTTP/1.1 404 Not Found Server: nginx Date: Thu, 29 Aug 2019 12:48:14 GMT Content-Type: text/html; charset=utf-8 Content-Length: 3774 Connection: keep-alive X-Frame-Options: SAMEORIGIN Vary: Accept-Language, Cookie Content-Language: en Or maybe this is just a redirect rule that needs to be cleverer or more specific, rather than the apparent catch-all .com/.io redirect? Cheers, Florian

5 7

ceph fs crashes on simple fio test
by Frank Schilder 05 Sep '19

05 Sep '19

I need to harden our ceph cluster to satisfy the following properties: Assuming all hardware is functioning properly, 1) Cluster health has highest priority. Heartbeats have priority over client requests. 2) The cluster does not accept more IO than the OSDs can handle. The only exception might be a configurable burst option. 3) Client IO is accepted as long as it does not compromise 1. 4) Ideally, there is fair sharing of the cluster's IO budget between clients (like deadline or completely fair scheduling). Rogue clients should not get priority just because they push a lot. Unfortunately, with default settings our (any?) cluster prefers client IO over cluster health, which opens up for a simple but serious non-privileged client attack on cluster health: I observed a serious issue on our cluster when running a simple fio test that does 4K random writes on 100GB files (see details below). What I observe is that within a few seconds the cluster goes to health_warn with the MDS reporting slow meta data IO and behind on trimming. What is not shown in ceph health detail is, that all OSDs report thousands of slow ops and the counter increases really fast (I include some snippets below). This goes rapidly to the point where OSDs start missing heartbeats and start flapping, some PGs become inactive+degraded and start peering regularly. CPU load, memory and even network load were rather low (one-figure % CPU load). Nothing out of the ordinary. The mons had no issues. After the fio test completed, the OSDs slowly crunched through the backlog and managed to complete all OPS. The complete processing of the ops of a 30 second fio test took ca. 10 minutes. However, even then the cluster did not come back healthy as essential messages between daemons seem to have been lost. It is not a long stretch to assume that one can destroy a ceph fs beyond repair when running this test or an application performing the same IO pattern from multiple clients for several hours. We have a 500 node cluster as clients and I'm afraid that even ordinary IO might trigger this scenario in unlucky circumstances. Since our cluster is only half-trusted (we have root access to clients, but no control over user IO patterns), we are in need to harden our cluster against such destructive IO patterns as much as possible. To me it looks like the cluster is accepting way more IO than the OSDs can handle. Ideally, what I would like to do is configure effective rate limiting on (rogue) clients depending on how much OPS they have in flight already. I would expect that there are tunables for MDS/OSD daemons that control how much IO requests a client can submit/OSD will accept before throttling IO. In particular, I would like to prioritize heartbeats to prevent load-induced OSD flapping. How can I tune the cluster to satisfy the conditions outlined at the top of this e-mail? There were recent threads with similar topics, in particular, "MDS failing under load with large cache sizes" and others reporting unstable MDS daemons under load. However, I believe they were mostly related to cache trimming issues due to large amounts of files created. This is not the case here, its just 4 files with lots of random IO from a single client. A bit of information about our cluster and observations: The cluster is bluestore-only with 8+2 EC fs data pool on spinning disks and an 3(2) replicated fs meta data pool on SSD. We have 8 OSD hosts with 2 shards per host. Each host has 4SSDs and 12 10TB HDDs SAS 12GB with 4k block size. Network is 2x10G bonded for client and 2x10G bonded for replication. The replication network will be extended to 4x10G soon. We are aware that currently the network bandwidth greatly exceeds what the spinning disks can handle. It is dimensioned for adding more disks in the future. The fio job script is: [global] name=fio-rand-write directory=/home/fio # /home is on ceph fs filename_format=tmp/fio-$jobname-${HOSTNAME}-$jobnum-$filenum rw=randwrite bs=4K numjobs=4 time_based=1 runtime=30 [file1] size=100G ioengine=sync Its one of the examples in the fio source repo with small modifications (file pattern, run time). Shortly after starting this fio job from a client connected with single 10G line, all OSDs start reporting slow ops. Picking one, the log messages look like this: Aug 22 12:06:22 ceph-09 ceph-osd: 2019-08-22 10:06:22.151 7f399d1bd700 -1 osd.125 3778 get_health_metrics reporting 146 slow ops, oldest is osd_op(client.1178165.0:1670940 5.3fs0 5:fcc65fe3:::10000f403bc.0000063e:head [write 307200~4096,write 315392~8192,write 380928~12288,write 405504~12288,write 421888~4096,write 458752~4096,write 466944~4096,write 475136~4096,write 487424~8192,write 512000~4096,write 524288~4096,write 565248~4096,write 589824~8192,write 622592~4096,write 651264~4096,write 724992~12288] snapc 12e=[] ondisk+write+known_if_redirected e3778) ... Aug 22 12:06:49 ceph-09 journal: 2019-08-22 10:06:49.415 7f399d1bd700 -1 osd.125 3779 get_health_metrics reporting 4595 slow ops, olde st is osd_op(client.1178165.0:1686216 5.acs0 5:354e96d5:::10000f403bc.00000bc1:head [write 2359296~4096,write 2375680~4096,write 23838 72~4096,write 2404352~4096,write 2428928~8192,write 2469888~4096,write 2490368~8192,write 2514944~4096,write 2527232~4096,write 253542 4~4096,write 2588672~4096,write 2600960~4096,write 2621440~8192,write 2658304~4096,write 2715648~8192,write 2727936~4096] snapc 12e=[] ondisk+write+known_if_redirected e3778) ... Aug 22 12:12:57 ceph-09 journal: 2019-08-22 10:12:57.650 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 8419 slow ops, oldest is osd_op(client.1178165.0:2009417 5.3fs0 5:fcdcf2bd:::10000f47e2d.00001501:head [write 2236416~4096,write 2256896~4096,write 2265088~4096,write 2301952~4096,write 2322432~4096,write 2355200~4096,write 2371584~4096,write 2387968~4096,write 2449408~4096,write 2486272~4096,write 2547712~8192,write 2617344~4096,write 2809856~4096,write 3018752~4096,write 3194880~4096,write 3223552~4096] snapc 12e=[] ondisk+write+known_if_redirected e3839) Aug 22 12:12:58 ceph-09 ceph-osd: 2019-08-22 10:12:58.681 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 8862 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+write+known_if_redirected+full_force e3839) ... Aug 22 12:13:27 ceph-09 journal: 2019-08-22 10:13:27.691 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 13795 slow ops, oldest is osd_op(mds.0.16909:69577573 5.e6s0 5.d8994de6 (undecoded) ondisk+write+known_if_redirected+full_force e3839) ... Aug 22 12:13:59 ceph-09 ceph-osd: 2019-08-22 10:13:59.762 7f399d1bd700 -1 osd.125 3900 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+retry+write+known_if_redirected+full_force e3875) ... Aug 22 12:14:46 ceph-09 journal: 2019-08-22 10:14:46.569 7f399d1bd700 -1 osd.125 3916 get_health_metrics reporting 969 slow ops, oldes t is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+retry+write+known_if_redirected+full_force e3875) Aug 22 12:14:47 ceph-09 ceph-osd: 2019-08-22 10:14:47.617 7f399d1bd700 -1 osd.125 3935 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5:7711c499:::10000f4798b.00000000:head [create,setxattr parent (289),setxattr layout (30)] snap c 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+full_force e3875) ... Aug 22 12:14:53 ceph-09 journal: 2019-08-22 10:14:53.675 7f399d1bd700 -1 osd.125 3939 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5:7711c499:::10000f4798b.00000000:head [create,setxattr parent (289),setxattr layout (30)] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+full_force e3875) This is the last log message, the OSD seems to have executed all OPS at this point or shortly after. The cluster state went to health_ok - at least shortly. At about 12:45 (yes, lunch break) we looked at the cluster again and it was back in health_warn with the following status output: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN 1 MDSs report slow metadata IOs 2 MDSs behind on trimming services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-02, ceph-03 mds: con-fs-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 192 osds: 192 up, 192 in data: pools: 7 pools, 790 pgs objects: 9.01 M objects, 16 TiB usage: 20 TiB used, 1.3 PiB / 1.3 PiB avail pgs: 790 active+clean io: client: 1.9 MiB/s rd, 21 MiB/s wr, 60 op/s rd, 721 op/s wr [root@ceph-01 ~]# ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; 2 MDSs behind on trimming MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mdsceph-08(mds.0): 2 slow metadata IOs are blocked > 30 secs, oldest blocked for 3275 secs MDS_TRIM 2 MDSs behind on trimming mdsceph-08(mds.0): Behind on trimming (1778/128) max_segments: 128, num_segments: 1778 mdsceph-12(mds.0): Behind on trimming (1780/128) max_segments: 128, num_segments: 1780 Num_segments was increasing constantly. Apparently, an operation got stuck and never completed. We hunted a little bit and found [root@ceph-mds:ceph-08 /]# ceph daemon mds.ceph-08 objecter_requests { "ops": [ { "tid": 71206302, "pg": "4.952611b7", "osd": 19, "object_id": "200.0001be74", "object_locator": "@4", "target_object_id": "200.0001be74", "target_object_locator": "@4", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 11:12:41.0.565912s", "osd_ops": [ "write 890478~1923" ] }, { "tid": 71206303, "pg": "4.952611b7", "osd": 19, "object_id": "200.0001be74", "object_locator": "@4", "target_object_id": "200.0001be74", "target_object_locator": "@4", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 11:12:41.0.566236s", "osd_ops": [ "write 892401~1931" ] }, { "tid": 71206301, "pg": "5.eeb918c6", "osd": 67, "object_id": "10000f26f67.00000000", "object_locator": "@5", "target_object_id": "10000f26f67.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "12e=[]", "mtime": "1970-01-01 00:00:00.000000s", "osd_ops": [ "trimtrunc 81854@573" ] }, { "tid": 69577573, "pg": "5.d8994de6", "osd": 125, "object_id": "10000f479c9.00000000", "object_locator": "@5", "target_object_id": "10000f479c9.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.256058s", "osd_ops": [ "create", "setxattr parent (319)", "setxattr layout (30)" ] }, { "tid": 69577598, "pg": "5.deb003e6", "osd": 125, "object_id": "10000f479da.00000000", "object_locator": "@5", "target_object_id": "10000f479da.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.258824s", "osd_ops": [ "create", "setxattr parent (288)", "setxattr layout (30)" ] }, { "tid": 71206300, "pg": "5.5cd5b20b", "osd": 163, "object_id": "10000f01396.00000000", "object_locator": "@5", "target_object_id": "10000f01396.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "12e=[]", "mtime": "1970-01-01 00:00:00.000000s", "osd_ops": [ "trimtrunc 208782@573" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } Notice the ops from the 1970's. Checking ops and dump_blocked_ops on osd.19 showed that these lists were empty. So, we decided to restart osd.19 and it cleared out most of the stuck requests, but did not clear the health warnings: [root@ceph-mds:ceph-08 /]# ceph daemon mds.ceph-08 objecter_requests { "ops": [ { "tid": 69577573, "pg": "5.d8994de6", "osd": 125, "object_id": "10000f479c9.00000000", "object_locator": "@5", "target_object_id": "10000f479c9.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.256058s", "osd_ops": [ "create", "setxattr parent (319)", "setxattr layout (30)" ] }, { "tid": 69577598, "pg": "5.deb003e6", "osd": 125, "object_id": "10000f479da.00000000", "object_locator": "@5", "target_object_id": "10000f479da.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.258824s", "osd_ops": [ "create", "setxattr parent (288)", "setxattr layout (30)" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } Restarting osd.125 finally resolved the health issues. However, the client I run fio on had lost connection to ceph due to this incident, which is really annoying. This client is the head node of our HPC cluster and it was not possible to restore ceph fs access without reboot. This is an additional bad fallout as all users will loose access to our HPC cluster when this happens (/home is on the ceph fs). I dumped the dump_historic_slow_ops of osd.125 in case anyone can use this information. I might be able to repeat this experiment, but cannot promise anything. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 14

ceph mons stuck in electing state
by Nick 04 Sep '19

04 Sep '19

Hello, I have an old ceph 0.94.10 cluster that had 10 storage nodes with one extra management node used for running commands on the cluster. Over time we'd had some hardware failures on some of the storage nodes, so we're down to 6, with ceph-mon running on the management server and 4 of the storage nodes. We attempted deploying a ceph.conf change and restarted ceph-mon and ceph-osd services, but the cluster went down on us. We found all the ceph-mons are stuck in the electing state, I can't get any response from any ceph commands but I found I can contact the daemon directly and get this information (hostnames removed for privacy reasons): root@<mgmt1>:~# ceph daemon mon.<mgmt1> mon_status { "name": "<mgmt1>", "rank": 0, "state": "electing", "election_epoch": 4327, "quorum": [], "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 10, "fsid": "69611c75-200f-4861-8709-8a0adc64a1c9", "modified": "2019-08-23 08:20:57.620147", "created": "0.000000", "mons": [ { "rank": 0, "name": "<mgmt1>", "addr": "[fdc4:8570:e14c:132d::15]:6789\/0" }, { "rank": 1, "name": "<mon1>", "addr": "[fdc4:8570:e14c:132d::16]:6789\/0" }, { "rank": 2, "name": "<mon2>", "addr": "[fdc4:8570:e14c:132d::28]:6789\/0" }, { "rank": 3, "name": "<mon3>", "addr": "[fdc4:8570:e14c:132d::29]:6789\/0" }, { "rank": 4, "name": "<mon4>", "addr": "[fdc4:8570:e14c:132d::151]:6789\/0" } ] } } Is there any way to force the cluster back into a quorum even if it's just one mon running to start it up? I've tried exporting the mgmt's monmap and injecting it into the other nodes, but it didn't make any difference. Thanks!

4 5

removing/flattening a bucket without data movement?
by Zoltan Arnold Nagy 02 Sep '19

02 Sep '19

Hi folks, Originally our osd tree looked like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2073.15186 root default -14 176.63100 rack s01-rack -19 176.63100 host s01 <snip osds> -15 171.29900 rack s02-rack -20 171.29900 host s02 <snip osds> etc. You get the idea. It was a legacy thing as we've been upgrading this cluster since probably firefly, and started with way less hardware. The crush rule was set up like this originally: step take default step chooseleaf firstn 0 type rack which we have modified to step take default step chooseleaf firstn 0 type host taking advantage of chooseleaf's behavior (eg searching in depth instead of just a single level). Now we thought we could get rid of the rack buckets simply by moving the host buckets to the root using "ceph osd crush move s01 root=default", however this resulted in a bunch of data movement. Swapping the IDs manually in the crushmap seems to work (verified via crushtool's --compare), eg. changing the ID of s01 to s01-rack's and vice versa, including all shadow trees. Looking around I saw that there is a swap-bucket command but that does not swap the IDs just bucket contents, so would result in data movement. Other than manually editing the crushmap is there a better way to achieve this? Is this way the most optimal? Cheers, Zoltan

3 7

Safe to reboot host?
by Brett Chancellor 31 Aug '19

31 Aug '19

Before I write something that's already been done, are there any built in utilities or tools that can tell me if it's safe to reboot a host? I'm looking for something better than just checking the health status, but rather checking pg status and ensuring that a reboot wouldn't take any undersized PGs offline. Thanks. -Brett

2 1

Howto define OSD weight in Crush map
by 74cmonty＠gmail.com 31 Aug '19

31 Aug '19

Hi, after adding an OSD to Ceph it is adviseable to create a relevant entry in Crush map using a weight size depending on disk size. Example: ceph osd crush set osd.<id> <weight> root=default host=<hostname> Question: How is the weight defined depending on disk size? Which algorithm can be used to calculate the weight? From my first Ceph installation (luminous) the Crush map entry for - HDD device with 1.80TB size (output of lsscsi -s): weight was 1.627229 - NVMI device with 3.20TB size (output of lsscsi -s): weight was 2.910889 THX

2 1

Error: err=rados: File too large caller=cephstorage_linux.go:231
by tapas＠wolk.com 30 Aug '19

30 Aug '19

Hi, I am trying to upload image size=>190 MB got error "err=rados: File too large caller=cephstorage_linux.go:231" Please help Thanks!!!! Tapas

1 0

Re: backfill_toofull after adding new OSDs
by Frank Schilder 30 Aug '19

30 Aug '19

I observe the same issue after adding two new OSD hosts to an almost empty mimic cluster. > Let's try to restrict discussion to the original thread > "backfill_toofull while OSDs are not full" and get a tracker opened up > for this issue. Is this the issue you are referring to: https://tracker.ceph.com/issues/41255 ? I have a number of larger rebalance operations ahead and will probably see this for a couple of days. If there is any information (logs etc.) I can provide, please let me know. Status right now is: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_ERR 15227159/90990337 objects misplaced (16.735%) Degraded data redundancy (low space): 64 pgs backfill_toofull too few PGs per OSD (29 < min 30) services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs-1/1/1 up {0=ceph-12=up:active}, 1 up:standby-replay osd: 208 osds: 208 up, 208 in; 273 remapped pgs data: pools: 7 pools, 790 pgs objects: 9.45 M objects, 17 TiB usage: 21 TiB used, 1.4 PiB / 1.4 PiB avail pgs: 15227159/90990337 objects misplaced (16.735%) 517 active+clean 190 active+remapped+backfill_wait 64 active+remapped+backfill_wait+backfill_toofull 19 active+remapped+backfilling io: client: 893 KiB/s rd, 6.3 MiB/s wr, 208 op/s rd, 306 op/s wr recovery: 298 MiB/s, 156 objects/s Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Howto add DB (aka RockDB) device to existing OSD on HDD
by 74cmonty＠gmail.com 30 Aug '19

30 Aug '19

Hi, I have created OSD on HDD w/o putting DB on faster drive. In order to improve performance I have now a single SSD drive with 3.8TB. I modified /etc/ceph/ceph.conf by adding this in [global]: bluestore_block_db_size = 53687091200 This should create RockDB with size 50GB. Then I tried to move DB to a new device (SSD) that is not formatted: root@ld5505:~# ceph-bluestore-tool bluefs-bdev-new-db –-path /var/lib/ceph/osd/ceph-76 --dev-target /dev/sdbk too many positional options have been specified on the command line Checking the content of /var/lib/ceph/osd/ceph-76 it appears that there's no link to block.db: root@ld5505:~# ls -l /var/lib/ceph/osd/ceph-76/ insgesamt 52 -rw-r--r-- 1 ceph ceph 418 Aug 27 11:08 activate.monmap lrwxrwxrwx 1 ceph ceph 93 Aug 27 11:08 block -> /dev/ceph-8cd045dc-9eb2-47ad-9668-116cf425a66a/osd-block-9c51bde1-3c75-4767-8808-f7e7b58b8f97 -rw-r--r-- 1 ceph ceph 2 Aug 27 11:08 bluefs -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 ceph_fsid -rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 fsid -rw------- 1 ceph ceph 56 Aug 27 11:08 keyring -rw-r--r-- 1 ceph ceph 8 Aug 27 11:08 kv_backend -rw-r--r-- 1 ceph ceph 21 Aug 27 11:08 magic -rw-r--r-- 1 ceph ceph 4 Aug 27 11:08 mkfs_done -rw-r--r-- 1 ceph ceph 41 Aug 27 11:08 osd_key -rw-r--r-- 1 ceph ceph 6 Aug 27 11:08 ready -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 require_osd_release -rw-r--r-- 1 ceph ceph 10 Aug 27 11:08 type -rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 whoami root@ld5505:~# more /var/lib/ceph/osd/ceph-76/bluefs 1 Questions: How can I add DB device for every single existing OSD to this new SSD drive? How can I increase the DB size later in case it's insufficient? THX

2 1

help
by Amudhan P 30 Aug '19

30 Aug '19

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning . "ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%) "ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+ pg 2.3 is stuck undersized for 1076.456113, current state active+undersized+ pg 2.7 is stuck undersized for 1076.456342, current state active+undersized+ pg 2.8 is stuck undersized for 1076.455920, current state active+undersized+ pg 2.a is stuck undersized for 1076.486412, current state active+undersized+ pg 2.b is stuck undersized for 1076.485975, current state active+undersized+ pg 2.f is stuck undersized for 1076.486953, current state active+undersized+ pg 2.10 is stuck undersized for 1076.486763, current state active+undersized pg 2.12 is stuck undersized for 1076.486539, current state active+undersized pg 2.13 is stuck undersized for 1075.419199, current state active+undersized pg 2.17 is stuck undersized for 1076.455424, current state active+undersized pg 2.18 is stuck undersized for 1075.419639, current state active+undersized pg 2.1a is stuck undersized for 1076.455966, current state active+undersized pg 2.1b is stuck undersized for 1076.486677, current state active+undersized pg 2.1f is stuck undersized for 1076.455572, current state active+undersized how to bring it health status OK regards Amudhan

5 10

Which CephFS clients send a compressible hint?
by Erwin Bogaard 30 Aug '19

30 Aug '19

We're mainly using CephFS using the Centos/Rhel 7 kernel client and I'm pondering if I should go for bluestore compression mode" = passive or aggressive with this client to get compression on (preferably) only compressible objects. Is there any list of CephFS clients that send compressible hints? If not, is there any other way to detect this, other than checking a client by just writing some compressible and some uncompressible data? The documentation is rather sparse on this matter, as far as I can see.

2 1

Deleted snapshot still having error, how to fix (pg repair is not working)
by Marc Roos 30 Aug '19

30 Aug '19

I was a little bit afraid I would be deleting this snapshot without result. How do I fix this error (pg repair is not working) pg 17.36 is active+clean+inconsistent, acting [7,29,12] 2019-08-30 10:40:04.580470 7f9b3f061700 -1 log_channel(cluster) log [ERR] : repair 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing

1 0

645% Clean PG's in Dashboard
by c.lilja＠falseprivacy.org 30 Aug '19

30 Aug '19

Hi, I've upgraded to Nautilus from Mimic a while ago and enabled the pg_autoscaler. When pg_autoscaler was activated I got a HEALTH_WARN regarding: POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_ratio 0.000 on pools [] Both target_size_bytes and target_size_ratio on all the pools are set to 0, so I started to wonder why this error message appear. My autoscale-status looks like this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 16708M 4.0 34465G 0.0019 1.0 8 warn cephfs_data_reduced 15506G 2.0 34465G 0.8998 1.0 375 warn cephfs_data 6451G 3.0 34465G 0.5616 1.0 250 warn So the ratio in total is 1.4633.. Isn't 1.0 of the combined ratio of all pools equal of full? I also enabled the Dashboard and saw that the PG Status showed "645% clean" PG's. This cluster was originally installed with version Jewel, so may it be any legacy setting or such that causing this?

1 0

Re: active+remapped+backfilling with objects misplaced
by David Casier 30 Aug '19

30 Aug '19

Hi, First, do not panic :) Secondly, verify that the number of pg pools is adapted to the need and the cluster. Third, if I understood correctly, there was a too small number of pg in the pool and the data is essentially on a PG => wait for the data to be distributed correctly. If possible, there is pg-upmap for a finer distribution of PGs.

2 1

Re: Identify rbd snapshot
by Ilya Dryomov 30 Aug '19

30 Aug '19

On Thu, Aug 29, 2019 at 11:20 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I have this error. I have found the rbd image with the > block_name_prefix:1f114174b0dc51, how can identify what snapshot this > is? (Is it a snapshot?) > > 2019-08-29 16:16:49.255183 7f9b3f061700 -1 log_channel(cluster) log > [ERR] : deep-scrub 17.36 > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected > clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing > 2019-08-29 16:24:54.806912 7f9b3f061700 -1 log_channel(cluster) log > [ERR] : 17.36 deep-scrub 1 errors If you already found the image, just "rbd snap ls <image name>". Is there a snapshot with SNAPID 4? Thanks, Ilya

2 1

admin socket for OpenStack client vanishes
by Georg Fleig 30 Aug '19

30 Aug '19

Hi, I am trying to get detailed information about the RBD images used by OpenStack (r/w operations, throughput, ..). On the mailing list I found instructions that this is possible using an admin socket of the client [1]. So I enabled the socket on one of my hosts according to [2]. The manual states that the socket should be there once I restart the VM. At some point it actually does appear, but it vanishes within a second or two. If I keep monitoring the directory I see it appearing for roughly 1-2 seconds per minute. The socket looks like this: root@compute01:/var/run/ceph/guests# ls -l srwxr-xr-x 1 cinder cinder 0 Aug 29 17:54 ceph-client.cinder.2772108.94507439454256.asok Does anyone know what I am doing wrong? Or is there another way to get information about which RBD image is causing the most load on a cluster? Regards, Georg [1] http://webcache.googleusercontent.com/search?q=cache%3Ahttp%3A%2F%2Flists.c… By the way, the mail archive from before 2019 seems to be inaccessible. Using google cache as a fallback. [2] https://docs.ceph.com/docs/mimic/rbd/rbd-openstack/#configuring-nova

1 0

Danish ceph users
by Torben Hørup 29 Aug '19

29 Aug '19

Hi A colleague and I are talking about making an event in Denmark for the danish ceph community, and we would like to get a feeling of how many ceph users are there in Denmark and hereof who would be interested in a Danish ceph event ? Regards, Torben

4 3

Re: Ceph + SAMBA (vfs_ceph)
by David Disseldorp 29 Aug '19

29 Aug '19

[resend to ceph.io addr] Hi Salsa, The vfs_ceph "ceph: user_id" parameter specifies the CephX user ID which Samba (via libcephfs) uses to authenticate with the Ceph cluster. These credentials are completely separate to the SMB user credentials which you provide with smbclient, etc. It seems that you might be confusing them in your setup. As documented in the Samba vfs_ceph manpage, "kernel share modes = no" must currently be configured for CephFS shares. Also, the "cephfs" share name in your smb.conf doesn't match the "cephfs_a" name you're giving to smbclient. I'd suggest configuring a locally backed Samba share first and get that working with your Windows clients before moving over to the CephFS backed share. Cheers, David

1 0

Failure to start ceph-mon in docker
by Frank Schilder 29 Aug '19

29 Aug '19

Hi Robert, this sounds a bit worse than I thought. I remember that RedHat stopped packaging Ubuntu containers. The default for the public containers is now Centos, which I'm running on all my hosts (I also use ceph containers). If Ubuntu indeed reserves a different default UID for ceph and UID 167 is available, then you should probably create/change the ceph user with GID=UID=167 on all hosts before installing anything ceph related. Otherwise there might be more to re-own than just /var/lib/ceph (e.g /var/log/ceph). The ansible tasks I included earlier will do this user creation. Everything should work fine after that. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert LeBlanc <robert(a)leblancnet.us> Sent: 29 August 2019 16:41 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] Failure to start ceph-mon in docker Frank, Thank you for the explanation, these are freshly installed machines and did not have ceph on them. I checked one of the other OSD nodes and there is no ceph user in /etc/passwd, nor is UID 167 allocated to any user. I did install ceph-common from the 18.04 repos before realizing that deploying ceph in containers did not update the host's /etc/apt/sources.list (or add an entry in /etc/apt/sources.list.d/). I manually added the repo for nautilus and upgraded the packages. So, I don't know if that had anything to do with it. Maybe Ubuntu packages ceph under UID 64045 and upgrading to the Ceph distributed packages didn't change the UID. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Aug 29, 2019 at 12:33 AM Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> wrote: Hi Robert, this is a bit less trivial than it might look right now. The ceph user is usually created by installing the package ceph-common. By default it will use id 167. If the ceph user already exists, I would assume it will use the existing user to allow an operator to avoid UID collisions (if 167 is used already). If you use docker, the ceph UID on the host and inside the container should match (or need to be translated). If they don't, you will have a lot of fun re-owning stuff all the time, because deployments will use the symbolic name ceph, which has different UIDs on the host and inside the container in your case. I would recommend removing this discrepancy as soon as possible: 1) Find out why there was a ceph user with UID different from 167 before installation of ceph-common. Did you create it by hand? Was UID 167 allocated already? 2) If you can safely change the GID and UID of ceph to 167, just do groupmod+usermod with new GID and UID. 3) If 167 is used already by another service, you will have to map the UIDs between host and container. To prevent ansible from deploying dockerized ceph with mismatching user ID for ceph, add these tasks to an appropriate part of your deployment (general host preparation or so): - name: "Create group 'ceph'." group: name: ceph gid: 167 local: yes state: present system: yes - name: "Create user 'ceph'." user: name: ceph password: "!" comment: "ceph-container daemons" uid: 167 group: ceph shell: "/sbin/nologin" home: "/var/lib/ceph" create_home: no local: yes state: present system: yes This should err if a group and user ceph already exist with IDs different from 167. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: ceph-users <ceph-users-bounces(a)lists.ceph.com<mailto:ceph-users-bounces@lists.ceph.com>> on behalf of Robert LeBlanc <robert(a)leblancnet.us<mailto:robert@leblancnet.us>> Sent: 28 August 2019 23:23:06 To: ceph-users Subject: Re: [ceph-users] Failure to start ceph-mon in docker Turns out /var/lib/ceph was ceph.ceph and not 167.167, chowning it made things work. I guess only monitor needs that permission, rgw,mgr,osd are all happy without needing it to be 167.167. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Aug 28, 2019 at 1:45 PM Robert LeBlanc <robert(a)leblancnet.us<mailto:robert@leblancnet.us><mailto:robert@leblancnet.us<mailto:robert@leblancnet.us>>> wrote: We are trying to set up a new Nautilus cluster using ceph-ansible with containers. We got things deployed, but I couldn't run `ceph s` on the host so decided to `apt install ceph-common and installed the Luminous version from Ubuntu 18.04. For some reason the docker container that was running the monitor restarted and won't restart. I added the repo for Nautilus and upgraded ceph-common, but the problem persists. The Manager and OSD docker containers don't seem to be affected at all. I see this in the journal: Aug 28 20:40:55 sun-gcs02-osd01 systemd[1]: Starting Ceph Monitor... Aug 28 20:40:55 sun-gcs02-osd01 docker[2926]: Error: No such container: ceph-mon-sun-gcs02-osd01 Aug 28 20:40:55 sun-gcs02-osd01 systemd[1]: Started Ceph Monitor. Aug 28 20:40:55 sun-gcs02-osd01 docker[2949]: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Aug 28 20:40:56 sun-gcs02-osd01 docker[2949]: 2019-08-28 20:40:56 /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster... Aug 28 20:40:56 sun-gcs02-osd01 docker[2949]: warning: line 41: 'osd_memory_target' in section 'osd' redefined Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: 2019-08-28 20:41:03 /opt/ceph-container/bin/entrypoint.sh: /etc/ceph/ceph.conf is already memory tuned Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: 2019-08-28 20:41:03 /opt/ceph-container/bin/entrypoint.sh: SUCCESS Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: exec: PID 368: spawning /usr/bin/ceph-mon --cluster ceph --default-log-to-file=false --default-mon-cluster-log-to-file=false --setuser ceph --setgroup ceph -d --mon-cluster-log-to-stderr --log-stderr-prefix=debug -i sun-gcs02-osd01 --mon-data /var/lib/ceph/mon/ceph-sun-gcs02-osd01 --public-addr 10.65.101.21 Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: exec: Waiting 368 to quit Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: warning: line 41: 'osd_memory_target' in section 'osd' redefined Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 0 set uid:gid to 167:167 (ceph:ceph) Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 0 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-mon, pid 368 Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 -1 stat(/var/lib/ceph/mon/ceph-sun-gcs02-osd01) (13) Permission denied Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 -1 error accessing monitor data directory at '/var/lib/ceph/mon/ceph-sun-gcs02-osd01': (13) Permission denied Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: managing teardown after SIGCHLD Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: Waiting PID 368 to terminate Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: Process 368 is terminated Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: Bye Bye, container will die with return code -1 Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: if you don't want me to die and have access to a shell to debug this situation, next time run me with '-e DEBUG=stayalive' Aug 28 20:41:04 sun-gcs02-osd01 systemd[1]: ceph-mon(a)sun-gcs02-osd01.service: Main process exited, code=exited, status=255/n/a Aug 28 20:41:04 sun-gcs02-osd01 systemd[1]: ceph-mon(a)sun-gcs02-osd01.service: Failed with result 'exit-code'. The directories for the monitor are owned by 167.167 and matches the UID.GID that the container reports. oot@sun-gcs02-osd01:~# ls -lhd /var/lib/ceph/ drwxr-x--- 14 ceph ceph 4.0K Jul 30 22:15 /var/lib/ceph/ root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/ total 56K drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-mds drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-mgr drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-osd drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-rbd drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-rbd-mirror drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-rgw drwxr-xr-x 3 167 167 4.0K Jul 30 22:15 mds drwxr-xr-x 3 167 167 4.0K Jul 30 22:15 mgr drwxr-xr-x 3 167 167 4.0K Jul 30 22:15 mon drwxr-xr-x 14 167 167 4.0K Jul 30 22:28 osd drwxr-xr-x 4 167 167 4.0K Aug 1 23:36 radosgw drwxr-xr-x 254 167 167 12K Aug 28 20:44 tmp root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/mon/ total 4.0K drwxr-xr-x 3 167 167 4.0K Jul 30 22:16 ceph-sun-gcs02-osd01 root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/mon/ceph-sun-gcs02-osd01/ total 16K -rw------- 1 167 167 77 Jul 30 22:15 keyring -rw-r--r-- 1 167 167 8 Jul 30 22:15 kv_backend -rw-r--r-- 1 167 167 3 Jul 30 22:16 min_mon_release drwxr-xr-x 2 167 167 4.0K Aug 28 19:16 store.db root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/mon/ceph-sun-gcs02-osd01/store.db/ total 149M -rw-r--r-- 1 167 167 1.7M Aug 28 19:16 050225.log -rw-r--r-- 1 167 167 65M Aug 28 19:16 050227.sst -rw-r--r-- 1 167 167 45M Aug 28 19:16 050228.sst -rw-r--r-- 1 167 167 16 Aug 16 07:40 CURRENT -rw-r--r-- 1 167 167 37 Jul 30 22:15 IDENTITY -rw-r--r-- 1 167 167 0 Jul 30 22:15 LOCK -rw-r--r-- 1 167 167 1.3M Aug 28 19:16 MANIFEST-027846 -rw-r--r-- 1 167 167 4.7K Aug 1 23:38 OPTIONS-002825 -rw-r--r-- 1 167 167 4.7K Aug 16 07:40 OPTIONS-027849 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

1 0

How to map 2 different Openstack users belonging to the same project to 2 distinct radosgw users ?
by Massimo Sgaravatto 29 Aug '19

29 Aug '19

I have a question regarding ceph-openstack integration for object storage. Is there some configuration/hack/workaround that allows to map 2 different users belonging to the same OpenStack project to 2 distinct radosgw users ? I saw this old bug: https://tracker.ceph.com/issues/20570 but it looks like it isn't going to be addressed in the near future Thanks, Massimo

2 2

Ceph pool snapshot mechanism
by Yannick.Martin＠cdiscount.com 29 Aug '19

29 Aug '19

Hello Can you tell me if Ceph snapshot at pool level rely on copy-on-write mechanism ? I found a RedHat help page that said that this is incompatible with RBD snapshot (I don't use it) and taks about "irreversible" ? Can you tell me about the last term ? Regards,

1 0

modifying "osd_memory_target"
by Amudhan P 29 Aug '19

29 Aug '19

Hi, How do i change "osd_memory_target" in ceph command line. regards Amudhan

2 1

How to customize object size
by tapas＠wolk.com 29 Aug '19

29 Aug '19

Hi, I am new to ceph ... i am trying to increase object file size .. i can upload file size upto 128MB .. how can i upload more than 128MB file . i can upload file using this rados --pool z10 put testfile-128M.txt testfile-128M.txt Thats ok when the file size is upto 128MB But its not ok when the file size is >128MB Got this error error putting z13/testfile-129MB.txt: (27) File too large Please help Tapas Jana

2 1

"ceph-users" mailing list!
by Tapas Jana 29 Aug '19

29 Aug '19

tapas@wolkcom

1 0

Ceph Ansible - - name: set grafana_server_addr fact - ipv4
by Lee Norvall 29 Aug '19

29 Aug '19

Hi Ceph: nautilus (14.2.2) NFS-Ganesha v 2.8 ceph-ansible stable 4.0 << git checkout 28th Aug CentOS 7 I am trying to do a fresh installation using Ceph Ansible and I am getting the following error when running the playbook. I have not enabled or config dashboard/grafana/prometheus yet. fatal: [stor1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: No first item, sequence was empty. The error appears to be in '/usr/share/ceph-ansible/roles/ceph-facts/tasks/facts.yml': line 314, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: set grafana_server_addr fact - ipv4 ^ here If I remove [grafana-server] from the /etc/ansible/hosts file I get fatal: [stor1]: FAILED! => changed=false msg: you must add a [grafana-server] group and add at least one node. Any idea what might be causing this?

3 2

Re: FileStore OSD, journal direct symlinked, permission troubles.
by Marco Gaiarin 29 Aug '19

29 Aug '19

Mandi! Alwin Antreich In chel di` si favelave... > > There's something i can do? Thanks. > Did you go through our upgrade guide(s)? Sure! > See the link [0] below, for the > permission changes. They are needed when an upgrade from Hammer to Jewel > is done. Sure! The problem arise in the 'Set partition type' section, because: root@deadpool:~# for l in $(readlink -f /var/lib/ceph/osd/ceph-*/journal); do echo $l; blkid -o udev -p $l; echo ""; done /dev/sda5 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=9c277a97-05 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=5 ID_PART_ENTRY_OFFSET=546877440 ID_PART_ENTRY_SIZE=97654784 ID_PART_ENTRY_DISK=8:0 /dev/sda6 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=9c277a97-06 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=6 ID_PART_ENTRY_OFFSET=644534272 ID_PART_ENTRY_SIZE=97654784 ID_PART_ENTRY_DISK=8:0 /dev/sdb7 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=802474ca-07 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=7 ID_PART_ENTRY_OFFSET=742191104 ID_PART_ENTRY_SIZE=97654784 ID_PART_ENTRY_DISK=8:16 /dev/sdb8 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=802474ca-08 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=8 ID_PART_ENTRY_OFFSET=839847936 ID_PART_ENTRY_SIZE=97853440 ID_PART_ENTRY_DISK=8:16 As stated, partitions are 'DOS'... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

2 2

Re: FileStore OSD, journal direct symlinked, permission troubles.
by Alwin Antreich 29 Aug '19

29 Aug '19

Hello Marco, On Thu, Aug 29, 2019 at 12:55:56PM +0200, Marco Gaiarin wrote: > > I've just finished a double upgrade on my ceph (PVE-based) from hammer > to jewel and from jewel to luminous. > > All went well, apart that... OSD does not restart automatically, > because permission troubles on the journal: > > Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal > Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: 2019-08-28 14:41:55.449886 7fa505a43e00 -1 filestore(/var/lib/ceph/osd/ceph-2) mount(1822): failed to open journal /var/lib/ceph/osd/ceph-2/journal: (13) Permission denied > Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: 2019-08-28 14:41:55.453524 7fa505a43e00 -1 osd.2 0 OSD:init: unable to mount object store > Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: 2019-08-28 14:41:55.453535 7fa505a43e00 -1 #033[0;31m ** ERROR: osd init failed: (13) Permission denied#033[0m > > > A little fast rewind: when i've setup the cluster i've used some 'old' > servers, using a couple of SSD disks as SO and as journal. > Because servers was old, i was forced to partition the boot disk in > DOS, not GPT mode. > > While creating the OSD, i've received some warnings: > > WARNING:ceph-disk:Journal /dev/sdaX was not prepared with ceph-disk. Symlinking directly. > > > Looking at the cluster now, seems to me that osd init scripts try to > idetify journal based on GPT partition label/info, and clearly fail. > > > Not that if i do, on servers that hold OSD: > > for l in $(readlink -f /var/lib/ceph/osd/ceph-*/journal); do chown ceph: $l; done > > OSD start flawlessy. > > > There's something i can do? Thanks. Did you go through our upgrade guide(s)? See the link [0] below, for the permission changes. They are needed when an upgrade from Hammer to Jewel is done. On the wiki you can also find the upgrade guides for PVE 5.x -> 6.x and Luminous -> Nautilus. -- Cheers, Alwin [0] https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel#Set_permission

1 0

pg_autoscale HEALTH_WARN
by James Dingwall 29 Aug '19

29 Aug '19

Hi, I'm running a small nautilus cluster (14.2.2) which was recently upgraded from mimic (13.2.6). After the upgrade I enabled the pg_autoscaler which resulted in most of the pools having their pg count changed. All the remapping has completed but the cluster is still reporting a HEALTH_WARN. I have adjusted the target ratios such that sum < 1.0 but this didn't help. What else can I look at? Thanks, James # ceph -s cluster: id: ... health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio services: mon: 3 daemons, quorum ceph-00,ceph-01,ceph-02 (age 3d) mgr: ceph-01(active, since 6d), standbys: ceph-02, ceph-00 osd: 32 osds: 32 up (since 2d), 32 in (since 2d) rgw: 1 daemon active (rgw-00) data: pools: 14 pools, 1512 pgs objects: 4.17M objects, 16 TiB usage: 47 TiB used, 69 TiB / 116 TiB avail pgs: 1510 active+clean 2 active+clean+scrubbing+deep # ceph osd pool autoscale-status (this might wrap horribly...): POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE loc.rgw.buckets.index 0 3.0 116.1T 0.0000 1.0 4 on vms1 5318G 3.0 116.1T 0.1341 0.2000 1.0 256 on vms2 3419G 3.0 116.1T 0.0862 0.0200 1.0 64 on .rgw.root 3648k 3.0 116.1T 0.0000 1.0 4 on default.rgw.meta 384.0k 3.0 116.1T 0.0000 1.0 4 on lov.rgw.log 384.0k 3.0 116.1T 0.0000 1.0 4 on vms3 35799G 3.0 116.1T 0.9028 0.6000 1.0 1024 on default.rgw.control 0 3.0 116.1T 0.0000 1.0 4 on loc.rgw.meta 768.5k 3.0 116.1T 0.0000 1.0 4 on vms4 2306G 3.0 116.1T 0.0582 0.1000 1.0 128 on loc.rgw.buckets.non-ec 200.4k 3.0 116.1T 0.0000 1.0 4 on loc.rgw.buckets.data 56390M 3.0 116.1T 0.0014 1.0 4 on loc.rgw.control 0 3.0 116.1T 0.0000 1.0 4 on default.rgw.log 0 3.0 116.1T 0.0000 1.0 4 on Zynstra is a private limited company registered in England and Wales (registered number 07864369). Our registered office and Headquarters are at The Innovation Centre, Broad Quay, Bath, BA1 1UD. This email, its contents and any attachments are confidential. If you have received this message in error please delete it from your system and advise the sender immediately.

1 0

the ceph rbd read dd with fio performance diffrent so huge?
by linghucongsong 28 Aug '19

28 Aug '19

The performance with the dd and fio diffrent is so huge? I have 25 OSDS with 8TB hdd. with dd I only get 410KB/s read perfomance,but with fio I get 991.23MB/s read perfomance. like below: Thanks in advance! root@Server-d5754749-cded-4964-8129-ba1accbe86b3:~# time dd of=/dev/zero if=/mnt/testw.dbf bs=4k count=10000 iflag=direct 10000+0 records in 10000+0 records out 40960000 bytes (41 MB, 39 MiB) copied, 99.9445 s, 410 kB/s real 1m39.950s user 0m0.040s sys 0m0.292s root@Server-d5754749-cded-4964-8129-ba1accbe86b3:~# fio --filename=/mnt/test1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=libaio -bs=4k -size=1G -numjobs=30 -runtime=10 -group_reporting -name=mytest mytest: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1 ... fio-2.2.10 Starting 30 threads Jobs: 30 (f=30): [R(30)] [100.0% done] [1149MB/0KB/0KB /s] [294K/0/0 iops] [eta 00m:00s] mytest: (groupid=0, jobs=30): err= 0: pid=5261: Tue Aug 27 13:37:28 2019 read : io=9915.2MB, bw=991.23MB/s, iops=253752, runt= 10003msec slat (usec): min=2, max=200020, avg=39.10, stdev=1454.14 clat (usec): min=1, max=160019, avg=38.57, stdev=1006.99 lat (usec): min=4, max=200022, avg=87.37, stdev=1910.99 clat percentiles (usec): | 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 1], 20.00th=[ 1], | 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1], | 70.00th=[ 1], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2], | 99.00th=[ 612], 99.50th=[ 684], 99.90th=[ 780], 99.95th=[ 1020], | 99.99th=[56064] bw (KB /s): min= 7168, max=46680, per=3.30%, avg=33460.79, stdev=12024.35 lat (usec) : 2=73.62%, 4=22.38%, 10=0.05%, 20=0.03%, 50=0.01% lat (usec) : 100=0.01%, 250=0.03%, 500=1.93%, 750=1.75%, 1000=0.14% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.03%, 250=0.01% cpu : usr=1.83%, sys=4.30%, ctx=104743, majf=0, minf=59 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=2538284/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: io=9915.2MB, aggrb=991.23MB/s, minb=991.23MB/s, maxb=991.23MB/s, mint=10003msec, maxt=10003msec Disk stats (read/write): vdb: ios=98460/0, merge=0/0, ticks=48840/0, in_queue=49144, util=17.28%

3 3

Upgrade procedure on Ubuntu Bionic with stock packages
by Mark Schouten 28 Aug '19

28 Aug '19

Hi, I have a cluster running on Ubuntu Bionic, with stock Ubuntu Ceph packages. When upgrading, I always try to follow the procedure as documented here: https://docs.ceph.com/docs/master/install/upgrading-ceph/ However, the Ubuntu packages restart all daemons upon upgrade, per node. So if I upgrade the first node, it will restart mon, osds, rgw, and mds'es on that node, even though the rest of the cluster is running the old version. I tried upgrading a single package, to see how that goes, but due to dependencies in dpkg, all other packages are upgraded as well. How should I proceed? Thanks, -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

2 3

3 OSD down and unable to start
by Jordi Blasco 28 Aug '19

28 Aug '19

Hello, I've been facing some issues with a single node ceph cluster (mimic). I know an environment like this shouldn't be in production but the server end up dealing with operational workloads for the last 2 years. Some users detected some issues in cephfs; some files not being accessible and hanging the node while trying to list the content of affected folders. I noticed a heavy memory load on the server. Main memory was consumed by cache as well as quite a reasonable swap. The command "ceph health detail" reported some inactive PGs. Those PGs didn't exist. After rebooting the node, an fsck was run in the 3 affected OSDs. ceph-bluestore-tool fsck --deep yes --path /var/lib/ceph/osd/ceph-1/ Unfortunately, all of them crashed with a core dump and now they don't start anymore. The logs report messages like: 2019-08-28 03:00:12.999 7f21d787c240 4 rocksdb: [/build/ceph-13.2.1/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-004059 2019-08-28 03:00:12.999 7f21d787c240 4 rocksdb: [/build/ceph-13.2.1/src/rocksdb/db/db_impl.cc:252] Shutdown: canceling all background work 2019-08-28 03:00:12.999 7f21d787c240 4 rocksdb: [/build/ceph-13.2.1/src/rocksdb/db/db_impl.cc:397] Shutdown complete 2019-08-28 03:00:12.999 7f21d787c240 -1 rocksdb: NotFound: 2019-08-28 03:00:12.999 7f21d787c240 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db erroring opening db: 2019-08-28 03:00:12.999 7f21d787c240 1 bluefs umount 2019-08-28 03:00:12.999 7f21d787c240 1 stupidalloc 0x0x5650c5255800 shutdown 2019-08-28 03:00:12.999 7f21d787c240 1 bdev(0x5650c5604a80 /var/lib/ceph/osd/ceph-0/block) close 2019-08-28 03:00:13.247 7f21d787c240 1 bdev(0x5650c5604700 /var/lib/ceph/osd/ceph-0/block) close 2019-08-28 03:00:13.479 7f21d787c240 -1 osd.0 0 OSD:init: unable to mount object store 2019-08-28 03:00:13.479 7f21d787c240 -1 ** ERROR: osd init failed: (5) Input/output error I'm not sure if the fsck has introduced additional damage. After that, I tried to mark unfound as lost with the following commands: ceph pg 4.1e mark_unfound_lost revert ceph pg 9.1d mark_unfound_lost revert ceph pg 13.3 mark_unfound_lost revert ceph pg 13.e mark_unfound_lost revert Currently, since there are 3 OSD down, there are: 316 unclean PGs 76 inactive PGs root@ceph-s01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 0.43599 root ssd -4 0.43599 disktype ssd_disk 12 ssd 0.43599 osd.12 up 1.00000 1.00000 -1 60.03792 root default -5 60.03792 disktype hdd_disk 0 hdd 0 osd.0 down 1.00000 1.00000 1 hdd 5.45799 osd.1 down 0 1.00000 2 hdd 5.45799 osd.2 up 1.00000 1.00000 3 hdd 5.45799 osd.3 up 1.00000 1.00000 4 hdd 5.45799 osd.4 up 1.00000 1.00000 5 hdd 5.45799 osd.5 up 1.00000 1.00000 6 hdd 5.45799 osd.6 up 1.00000 1.00000 7 hdd 5.45799 osd.7 down 0 1.00000 8 hdd 5.45799 osd.8 up 1.00000 1.00000 9 hdd 5.45799 osd.9 up 1.00000 1.00000 10 hdd 5.45799 osd.10 up 1.00000 1.00000 11 hdd 5.45799 osd.11 up 1.00000 1.00000 Running the following command, a MANIFEST file appeared in the folder db/lost. I guess that the repair moved here. # ceph-bluestore-tool bluefs-export --path /var/lib/ceph/osd/ceph-7 --out-dir osd7/ ... db/LOCK db/MANIFEST-000001 db/OPTIONS-018543 db/OPTIONS-018581 db/lost/ db/lost/MANIFEST-018578 Any ideas? Suggestions? Thank you. Regards, Jordi

1 0

Best way to stop an OSD form coming back online
by Cory Hawkless 28 Aug '19

28 Aug '19

I have an OSD that is throwing sense errors - It's at it's end of life and needs to be replaced. The server is in the datacentre and I won't get there for a few weeks so I've stopped the service (systemctl stop ceph-osd@208) and let the cluster rebalance, all is well. My thinking is that if for some reason the host that OSD208 resides within was to reboot, that OSD would start and become part of the cluster again. So I'd like to prevent this OSD from ever starting again without physically being able to remove it from the server. I was thinking that deleting it's key from the auth list might work. So a ceph osd purge 208 Then when the service tries to start it'll fail with an auth error. Any other suggestions? Cheers, Cory

4 3

meta: lists.ceph.io password reset
by Dan van der Ster 27 Aug '19

27 Aug '19

Sorry to post this to the list, but does this lists.ceph.io password reset work for anyone? https://lists.ceph.io/accounts/password/reset/ For my accounts which are getting mail I have "The e-mail address is not assigned to any user account". Best Regards, Dan

1 1

Re: MON DNS Lookup & Version 2 Protocol
by Ricardo Dias 27 Aug '19

27 Aug '19

Hi Dominic, I just created a feature ticket in the Ceph tracker to keep track of this issue. Here's the ticket: https://tracker.ceph.com/issues/41537 Cheers, Ricardo Dias On 17/07/19 20:06, DHilsbos(a)performair.com wrote: > All; > > I'm trying to firm up my understanding of how Ceph works, and ease of management tools and capabilities. > > I stumbled upon this: http://docs.ceph.com/docs/nautilus/rados/configuration/mon-lookup-dns/ > > It got me wondering; how do you convey protocol version 2 capabilities in this format? > > The examples all list port 6789, which is the port for protocol version 1. Would I add SRV records for port 3300? How does the client distinguish v1 from v2 in this case? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International, Inc. > DHilsbos(a)PerformAir.com > www.PerformAir.com > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Ricardo Dias Senior Software Engineer - Storage Team SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

1 0

ceph's replicas question
by Wesley Peng 27 Aug '19

27 Aug '19

Hi, We have all SSD disks as ceph's backend storage. Consider the cost factor, can we setup the cluster to have only two replicas for objects? thanks & regards Wesley

7 8

krdb upmap compatibility
by Frank R 26 Aug '19

26 Aug '19

It seems that with Linux kernel 4.16.10 krdb clients are seen as Jewel rather than Luminous. Can someone tell me which kernel version will be seen as Luminous as I want to enable the Upmap Balancer.

4 5

active+remapped+backfilling with objects misplaced
by Arash Shams 26 Aug '19

26 Aug '19

Hi everybody Im new to ceph and I have a question related to active+remapped+backfilling and misplaced objects Recently I copied more than 10 million objects to a new cluster with 3 nodes and 6 osds during this migration one of my OSDs got full and health check became ERR I dont know why but ceph started to write every object on only one osd(can I change this behaviour) and after it gots full I try to reweight it by utilization and increase the pgs for 1 pool. cluster became accessible again with status warning and recovery started. I checked the cluster status for 2 days and I found that I always have 1 pg with status active+remapped+backfilling and more than 5% misplaced objects. I thought recovery process takes more days so I leave the cluster to do the recovery in background, till now I have more active+remapped+backfill_wait pgs and more misplaced objects ( about 10%) the question is what should I do ? waiting for recovery to finish ? can I speedup this process ? these servers are in production environment am I in trouble or not ? Kind Regards Thanks

1 0

Luminous and mimic: adding OSD can crash mon(s) and lead to loss of quorum
by Florian Haas 26 Aug '19

26 Aug '19

Hi everyone, there are a couple of bug reports about this in Redmine but only one (unanswered) mailing list message[1] that I could find. So I figured I'd raise the issue here again and copy the original reporters of the bugs (they are BCC'd, because in case they are no longer subscribed it wouldn't be appropriate to share their email addresses with the list). This is about https://tracker.ceph.com/issues/40029, and https://tracker.ceph.com/issues/39978 (the latter of which was recently closed as a duplicate of the former). In short, it appears that at least in luminous and mimic (I haven't tried nautilus yet), it's possible to crash a mon when attempting to add a new OSD as it's trying to inject itself into the crush map under its host bucket, when that host bucket does not exist yet. What's worse is that when the OSD's "ceph osd new" process has thus crashed the leader mon, a new leader is elected and in case the "ceph osd new" process is still running on the OSD node, it will promptly connect to that mon, and kill it too. This then continues until sufficiently many mons have died for quorum to be lost. The recovery steps appear to involve - killing the "ceph osd new" process, - restarting mons until you regain quorum, - and then running "ceph osd purge" to drop the problematic OSD entry from the crushmap and osdmap. The issue can apparently be worked around by adding the host buckets to the crushmap manually before adding the new OSDs, but surely this isn't intended to be a prerequisite, at least not to the point of mons crashing otherwise? Also I am guessing that this is some weird corner case rooted in an unusual combination of contributing factors, because otherwise I am guessing more people would be bitten by this problem. Anyone able to share their thoughts on this one? Have more people run into this? Cheers, Florian [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034880.html — interestingly I could find this message in the pipermail archive but none in the one that my MUA keeps for me. So perhaps that message wasn't delivered to all subscribers, which might be why it has gone unanswered.

2 4

ceph rbd disk performance question
by linghucongsong 24 Aug '19

24 Aug '19

HI all! I use ceph as the openstack VM disk. I have a VM run postgresql. I found the disk on the vm run postgresql is very busy and slow! But the ceph cluster is very healthy and without any slow request. Even the vm disk is very busy, but the ceph cluster is look like very idle. My ceph version is 12.2.8 and the vm disk is ext4 file system. The postgresql vm disk is very busy! look below: avg-cpu: %user %nice %system %iowait %steal %idle 0.53 0.00 1.6 16.55 0.00 81.31 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util vdb 0.00 7425.00 2.0 65.5 40.00 63904.00 940.54 134.27 66966.54 12553.40 69042.94 14.71 100.05 The ceph cluster is very idle. osd commit_latency(ms) apply_latency(ms) 39 0 1 38 0 2 37 0 0 36 0 1 35 0 0 34 0 0 33 0 0 32 0 0 31 0 0 30 0 1 29 0 1 28 0 0 27 0 1 26 0 1 25 0 1 24 0 1 23 0 0 22 0 0 9 0 0 8 0 0 7 0 0 6 0 1 5 0 1 4 0 7 0 0 1 1 0 3 2 0 2 3 0 1 10 0 1 11 0 0 13 0 0 14 0 0 15 0 1 16 0 0 17 0 1 18 0 0 19 0 0 20 0 0 21 0 0 Can anybody tell me why? Thanks in advance!

3 3

Re: Failed to get omap key when mirroring of image is enabled
by Jason Dillaman 23 Aug '19

23 Aug '19

On Fri, Aug 23, 2019 at 7:38 AM Ajitha Robert <ajitharobert01(a)gmail.com> wrote: > > Sir, > > I have a running DR setup with ceph.. but i did the same for another two sites.. Its actually direct L2 connectivity link between sites.. I m getting repeated error > > rbd::mirror::InstanceWatcher: C_NotifyInstanceRequestfinish: resending after timeout That is just indicative that you are having issues talking to your local cluster. Assuming you only have a single rbd-mirror daemon running, it seems like it cannot even sent a message through the OSD to itself within 5 seconds. Perhaps your cluster is too slow to respond? > > its coming continously. so not getting replicated to other site. Whether direct L2 connectivity is a concern?? whether rbd-mirror expects a L3 layer link for two sites? > > On Wed, Jul 24, 2019 at 12:42 AM Ajitha Robert <ajitharobert01(a)gmail.com> wrote: >> >> Thanks for your reply. >> >> Regarding rbd mirroring, Can you please check the logs for rbd image creation. Second[2] one started syncing and no progress further. >> >> 1)Log for manual rbd image creation >> >> http://paste.openstack.org/show/754766/ >> >> >> 2)Log for 16gb volume created from cinder, status in cinder volume is available >> >> http://paste.openstack.org/show/754767/ >> >> >> 3)Log for 100gb volume created from cinder, status in cinder volume is error >> >> http://paste.openstack.org/show/754769/ >> >> >> On Tue, Jul 23, 2019 at 1:13 AM Jason Dillaman <jdillama(a)redhat.com> wrote: >>> >>> On Mon, Jul 22, 2019 at 3:26 PM Ajitha Robert <ajitharobert01(a)gmail.com> wrote: >>> > >>> > Thanks for your reply >>> > >>> > 1) In scenario 1, I didnt attempt to delete the cinder volume. Please find the cinder volume log. >>> > http://paste.openstack.org/show/754731/ >>> >>> It might be better to ping Cinder folks about that one. It doesn't >>> really make sense to me from a quick glance. >>> >>> > >>> > 2) In scenario 2. I will try with debug. But i m having a test setup with one OSD in primary and one OSD in secondary. distance between two ceph clusters is 300 km >>> > >>> > >>> > 3)I have disabled ceph authentication totally for all including rbd-mirror daemon. Also i have deployed the ceph cluster using ceph-ansible. Will these both create any issue to the entire setup >>> >>> Not to my knowledge. >>> >>> > 4)The image which was in syncing mode, showed read only status in secondary. >>> >>> Mirrored images are either primary or non-primary. It is the expected >>> (documented) behaviour that non-primary images are read-only. >>> >>> > 5)In a presentation i found as journaling feature is causing poor performance in IO operations and we can skip the journaling process for mirroring... Is it possible.. By enabling mirroring to entire cinder pool as pool mode instead of mirror mode of rbd mirroring.. And we can skip the replication_enabled is true spec in cinder type.. >>> >>> Journaling is required for RBD mirroring. >>> >>> > >>> > >>> > >>> > On Mon, Jul 22, 2019 at 11:13 PM Jason Dillaman <jdillama(a)redhat.com> wrote: >>> >> >>> >> On Mon, Jul 22, 2019 at 10:49 AM Ajitha Robert <ajitharobert01(a)gmail.com> wrote: >>> >> > >>> >> > No error log in rbd-mirroring except some connection timeout came once, >>> >> > Scenario 1: >>> >> > when I create a bootable volume of 100 GB with a glance image.Image get downloaded and from cinder, volume log throws with "volume is busy deleting volume that has snapshot" . Image was enabled with exclusive lock, journaling, layering, object-map, fast-diff and deep-flatten >>> >> > Cinder volume is in error state but the rbd image is created in primary but not in secondary. >>> >> >>> >> Any chance you know where in Cinder that error is being thrown? A >>> >> quick grep of the code doesn't reveal that error message. If the image >>> >> is being synced to the secondary site when you attempt to delete it, >>> >> it's possible you could hit this issue. Providing debug log messages >>> >> from librbd on the Cinder controller might also be helpful for this. >>> >> >>> >> > Scenario 2: >>> >> > but when i create a 50gb volume with another glance image. Volume get created. and in the backend i could see the rbd images both in primary and secondary >>> >> > >>> >> > From rbd mirror image status i found secondary cluster starts copying , and syncing was struck at around 14 %... It will be in 14 % .. no progress at all. should I set any parameters for this like timeout?? >>> >> > >>> >> > I manually checked rbd --cluster primary object-map check <object-name>.. No results came for the objects and the command was in hanging.. Thats why got worried on the failed to map object key log. I couldnt even rebuild the object map. >>> >> >>> >> It sounds like one or more of your primary OSDs are not reachable from >>> >> the secondary site. If you run w/ "debug rbd-mirror = 20" and "debug >>> >> rbd = 20", you should be able to see the last object it attempted to >>> >> copy. From that, you could use "ceph osd map" to figure out the >>> >> primary OSD for that object. >>> >> >>> >> > the image which was in syncing mode, showed read only status in secondary. >>> >> > >>> >> > >>> >> > >>> >> > On Mon, 22 Jul 2019, 17:36 Jason Dillaman, <jdillama(a)redhat.com> wrote: >>> >> >> >>> >> >> On Sun, Jul 21, 2019 at 8:25 PM Ajitha Robert <ajitharobert01(a)gmail.com> wrote: >>> >> >> > >>> >> >> > I have a rbd mirroring setup with primary and secondary clusters as peers and I have a pool enabled image mode.., In this i created a rbd image , enabled with journaling. >>> >> >> > >>> >> >> > But whenever i enable mirroring on the image, I m getting error in osd.log. I couldnt trace it out. please guide me to solve this error. >>> >> >> > >>> >> >> > I think initially it worked fine. but after ceph process restart. these error coming >>> >> >> > >>> >> >> > >>> >> >> > Secondary.osd.0.log >>> >> >> > >>> >> >> > 2019-07-22 05:36:17.371771 7ffbaa0e9700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:61: failed to get omap key: client_a5c76849-ba16-480a-a96b-ebfdb7f6ac65 >>> >> >> > 2019-07-22 05:36:17.388552 7ffbaa0e9700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than minimum: 0 < 1 >>> >> >> > 2019-07-22 05:36:17.413102 7ffbaa0e9700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:61: failed to get omap key: order >>> >> >> > 2019-07-22 05:36:23.341490 7ffbab8ec700 0 <cls> /build/ceph-12.2.12/src/cls/rbd/cls_rbd.cc:4125: error retrieving image id for global id '9e36b9f8-238e-4a54-a055-19b19447855e': (2) No such file or directory >>> >> >> > >>> >> >> > >>> >> >> > primary-osd.0.log >>> >> >> > >>> >> >> > 2019-07-22 05:16:49.287769 7fae12db1700 0 log_channel(cluster) log [DBG] : 1.b deep-scrub ok >>> >> >> > 2019-07-22 05:16:54.078698 7fae125b0700 0 log_channel(cluster) log [DBG] : 1.1b scrub starts >>> >> >> > 2019-07-22 05:16:54.293839 7fae125b0700 0 log_channel(cluster) log [DBG] : 1.1b scrub ok >>> >> >> > 2019-07-22 05:17:04.055277 7fae12db1700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than minimum: 0 < 1 >>> >> >> > >>> >> >> > 2019-07-22 05:33:21.540986 7fae135b2700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than minimum: 0 < 1 >>> >> >> > 2019-07-22 05:35:27.447820 7fae12db1700 0 <cls> /build/ceph-12.2.12/src/cls/rbd/cls_rbd.cc:4125: error retrieving image id for global id '8a61f694-f650-4ba1-b768-c5e7629ad2e0': (2) No such file or directory >>> >> >> >>> >> >> Those don't look like errors, but the log level should probably be >>> >> >> reduced for those OSD cls methods. If you look at your rbd-mirror >>> >> >> daemon log, do you see any errors? That would be the important place >>> >> >> to look. >>> >> >> >>> >> >> > >>> >> >> > -- >>> >> >> > Regards, >>> >> >> > Ajitha R >>> >> >> > _______________________________________________ >>> >> >> > ceph-users mailing list >>> >> >> > ceph-users(a)lists.ceph.com >>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> Jason >>> >> >>> >> >>> >> >>> >> -- >>> >> Jason >>> > >>> > >>> > >>> > -- >>> > Regards, >>> > Ajitha R >>> >>> >>> >>> -- >>> Jason >> >> >> >> -- >> Regards, >> Ajitha R > > > > -- > Regards, > Ajitha R -- Jason

2 1

RBD, OpenStack Nova, libvirt, qemu-guest-agent, and FIFREEZE: is this working as intended?
by Florian Haas 23 Aug '19

23 Aug '19

Hi everyone, apologies in advance; this will be long. It's also been through a bunch of edits and rewrites, so I don't know how well I'm expressing myself at this stage — please holler if anything is unclear and I'll be happy to try to clarify. I am currently in the process of investigating the behavior of OpenStack Nova instances when being snapshotted and suspended, in conjunction with qemu-guest-agent (qemu-ga). I realize that RBD-backed Nova/libvirt instances are expected to behave differently from file-backed ones, but I think I might have reason to believe that the RBD-backed ones are indeed behaving incorrectly, and I'd like to verify that. So first up, for comparison, let's recap how a Nova/libvirt/KVM instance behaves when it is *not* backed by RBD (such as, it's using a qcow2 file that is on a Nova compute node in /var/lib/nova/instances), is booted from an image with the hw_qemu_guest_agent=yes meta property set, and runs qemu-guest-agent within the guest: - User issues "nova suspend" or "openstack server suspend". - If nova-compute on the compute node decides that the instance has qemu-guest-agent running (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), it sends a guest-sync command over the guest agent VirtIO serial port. This command registers in the qemu-ga log file in the guest. - nova-compute on the compute node sends a libvirt managed-save command. - Nova reports the instance as suspended. - User issues "nova resume" or "openstack server resume". - nova-compute on the compute node sends a libvirt start command. - Again, if nova-compute on the compute node knows that the instance has qemu-guest-agent running, it sends another command over the serial port, namely guest-set-time. This, too, registers in the guest's qemu-ga log. - Nova reports the instance as active (running normally) again. Now, when I instead use a Nova environment that is fully RBD-backed, I see exactly the same behavior as described above. So I know that in principle, nova-compute/qemu-ga communication works in both an RBD-backed and a non-RBD-backed environment. However, things appear to get very different when it comes to snapshots. Again, starting with a file-backed environment: - User issues "nova image-create" or "openstack server image create". - If nova-compute on the compute node decides that the instance can be quiesced (which is the case if it's qemu or kvm, and its image has hw_qemu_guest_agent=yes), then it sends a "guest-fsfreeze-freeze" command over the guest agent VirtIO serial port. - The guest agent inside the guest loops over all mounted filesystems, and issues the FIFREEZE ioctl (which maps to the kernel freeze_super() function). This can be seen in the qemu-ga log file in the guest, and it is also verifiable by using ftrace on the qemu-ga PID and checking for the freeze_super() function call. - nova-compute then takes a live snapshot of the instance. - Once complete, the guest gets a "guest-fsfreeze-thaw" command, and again I can see this in the qemu-ga log, and with ftrace. And now with RBD: - User issues "nova image-create" or "openstack server image create". - The guest-fsfreeze-freeze agent command never happens. Now I can see the info message from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… in my nova-compute log, which confirms that we're attempting a live snapshot. I also do *not* see the warning from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac…, so it looks like the direct_snapshot() call from https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… succeeds. This is defined in https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… and it uses RBD functionality only. Importantly, it never interacts with qemu-ga, so it appears to not worry at all about freezing the filesystem. (Which does seem to contradict https://docs.ceph.com/docs/master/rbd/rbd-openstack/?highlight=uuid#image-p…, by the way, so that may be a documentation bug.) Now here's another interesting part. Were the direct snapshot to fail, if I read https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… and https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac… correctly, the fallback behavior would be as follows: The domain would next be "suspended" (note, again this is Nova suspend, which maps to libvirt managed-save per https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac…) then snapshotted using a libvirt call and resumed again post-snapshot. In which case there would be a guest-sync call on suspend. And it's this part that has me a bit worried. If an RBD backed instance, on a successful snapshot, never freezes its filesystem *and* never does any kind of sync, either, doesn't that mean that such an instance can't be made to produce consistent snapshots? (Particularly in the case of write-back caching, which is recommended and normally safe for RBD/virtio devices.) Or is there some magic within the Qemu RBD storage driver that I am unaware of, that makes any such contortions unnecessary? Thanks in advance for your insights! Cheers, Florian

3 7

Re: Strange Ceph architect with SAN storages
by Brett Chancellor 23 Aug '19

23 Aug '19

It's certainly possible. It makes things a little more complex though. Some questions you may want to consider during the design.. - Is the customer aware this won't preserve any data on the luns they are hoping to reuse. - Is the plan to eventually replace the SAN with JBOD, in the same systems? If so you may want to make your luns look like the eventual drive size and count. - Is the plan to use a few systems with SAN and add standalone systems later? Then you need to calculate expected speeds and divide between failure domains. - Is the plan to use a couple of hosts with SAN to save money, and have the rest be traditional Ceph storage? If so consider putting the SAN hosts all in one failure domain. - Depending on the SAN you may consider aligning your failure domains to different arrays, switches, or even array directors. - Remember to take the hosts network speed into consideration when calculating how many luns to put on each host. Hope that helps. -Brett On Thu, Aug 22, 2019, 4:14 AM Mohsen Mottaghi <mohsenmottaghi(a)outlook.com> wrote: > Hi > > > Yesterday one of our customers asked us a strange request. He asked us to > use SAN as the Ceph storage space to add the SAN storages it currently has > to the cluster and reduce other disk purchase costs. > > > Anybody know can we do this or not?! And if this is possible how we should > start to architect this Strange Ceph?! Is it good or not?! > > > > Thanks for your help. > > Mohsen Mottaghi > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

4 4

Ceph Tech Talk Cancelled for August
by Mike Perez 22 Aug '19

22 Aug '19

Hi all, We're cancelling our Ceph tech talk today. In the mean time check out our archive and consider reaching out to me about giving your own talk in the upcoming months. https://ceph.com/ceph-tech-talks/ -- Mike Perez (thingee)

1 0

Re: About image migration
by Jason Dillaman 22 Aug '19

22 Aug '19

On Wed, Aug 21, 2019 at 9:33 PM Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> wrote: > > I tested and combed the current migration process. If I read and write the new image during the migration process and then use migration_abort, the newly written data will be lost. Do we have a solution to this problem? That's a good point that we didn't consider. I've opened a tracker ticket against the issue [1]. > > -----邮件原件----- > 发件人: Jason Dillaman [mailto:jdillama@redhat.com] > 发送时间: 2019年8月22日 8:38 > 收件人: Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> > 抄送: ceph-users <ceph-users(a)ceph.com> > 主题: Re: About image migration > > On Wed, Aug 21, 2019 at 8:35 PM Zaharo Bai (白战豪)-云数据中心集团 > > <baizhanhao01(a)inspur.com> wrote: > > > > So, what is the current usage scenario for our online migration? I understand that the current online migration of the community has not been truly online, and the upper layer (iscsi target, openstack, etc.) must be required to do some operations such as switching and data maintenance. > > Is there a way to achieve full online migration in the rbd layer, the upper application is not aware, just need to call librbd's CLI or API, or is it necessary to do so, because doing so will inevitably change the architecture of ceph. > > If using RBD under a higher-level layer, the upper layers would need to know about the migration to update their internal data structures to point to the correct (new) image. The only case where this really isn't necessary is when live-migrating an image "in-place" (i.e. you keep it in the same pool w/ the same name). > > > -----邮件原件----- > > 发件人: Jason Dillaman [mailto:jdillama@redhat.com] > > 发送时间: 2019年8月21日 20:44 > > 收件人: Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> > > 抄送: ceph-users <ceph-users(a)ceph.com> > > 主题: Re: About image migration > > > > On Tue, Aug 20, 2019 at 10:04 PM Zaharo Bai (白战豪)-云数据中心集团 > > <baizhanhao01(a)inspur.com> wrote: > > > > > > Hi jason: > > > > > > I have a question I would like to ask you, Is the current image migration and openstack adapted? according to my understanding, openstack’s previous live-migration logic is implemented in cinder, just call librbd rbd_read/write API to do Data migration. > > > > I believe the existing Cinder volume block live-migration is just a wrapper around QEMU's block live-migration functionality, so indeed it would just be a series of RBD read/write API calls between two volumes. The built-in RBD live-migration is similar but it copies snapshots and preserves image sparseness during the migration process. > > Because Cinder is using the QEMU block live-migration functionality, it's not tied into RBD's live-migration. > > > > > > > > > > > > > > Best wishes > > > > > > Zaharo > > > > > > > > -- > > Jason > > > > -- > Jason [1] https://tracker.ceph.com/issues/41394 -- Jason

1 0

Strange Ceph architect with SAN storages
by Mohsen Mottaghi 22 Aug '19

22 Aug '19

Hi Yesterday one of our customers asked us a strange request. He asked us to use SAN as the Ceph storage space to add the SAN storages it currently has to the cluster and reduce other disk purchase costs. Anybody know can we do this or not?! And if this is possible how we should start to architect this Strange Ceph?! Is it good or not?! Thanks for your help. Mohsen Mottaghi

1 0

deep-scrub stat mismatch after PG merge
by Daniel Schreiber 22 Aug '19

22 Aug '19

Hi everyone, few days ago I reduced the number og PGs on a small pool. The cluster runs 14.2.2, was upgraded from jewel to 14.2.1 and the to 14.2.2. I did a ceph-bluestore-tool repair on all OSDs to update statistics. Today I got a scrub error reporting: 4.3 scrub : stat mismatch, got 68/68 objects, 0/0 clones, 68/68 dirty, 2/2 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 545259538/511705106 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. This message appears on the primary OSD only. rados list-inconsistent-obj 4.3 {"epoch":228717,"inconsistents":[]} What can I do to find the reason for the scrub error? Is this a hardware failure or a bug? Is it safe to do a pg repair? Thanks for your help. Daniel -- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444 Fax: +49 371 531 835444

1 0

Re: About image migration
by Jason Dillaman 22 Aug '19

22 Aug '19

On Wed, Aug 21, 2019 at 8:35 PM Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> wrote: > > So, what is the current usage scenario for our online migration? I understand that the current online migration of the community has not been truly online, and the upper layer (iscsi target, openstack, etc.) must be required to do some operations such as switching and data maintenance. > Is there a way to achieve full online migration in the rbd layer, the upper application is not aware, just need to call librbd's CLI or API, or is it necessary to do so, because doing so will inevitably change the architecture of ceph. If using RBD under a higher-level layer, the upper layers would need to know about the migration to update their internal data structures to point to the correct (new) image. The only case where this really isn't necessary is when live-migrating an image "in-place" (i.e. you keep it in the same pool w/ the same name). > -----邮件原件----- > 发件人: Jason Dillaman [mailto:jdillama@redhat.com] > 发送时间: 2019年8月21日 20:44 > 收件人: Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> > 抄送: ceph-users <ceph-users(a)ceph.com> > 主题: Re: About image migration > > On Tue, Aug 20, 2019 at 10:04 PM Zaharo Bai (白战豪)-云数据中心集团 > <baizhanhao01(a)inspur.com> wrote: > > > > Hi jason: > > > > I have a question I would like to ask you, Is the current image migration and openstack adapted? according to my understanding, openstack’s previous live-migration logic is implemented in cinder, just call librbd rbd_read/write API to do Data migration. > > I believe the existing Cinder volume block live-migration is just a wrapper around QEMU's block live-migration functionality, so indeed it would just be a series of RBD read/write API calls between two volumes. The built-in RBD live-migration is similar but it copies snapshots and preserves image sparseness during the migration process. > Because Cinder is using the QEMU block live-migration functionality, it's not tied into RBD's live-migration. > > > > > > > > > Best wishes > > > > Zaharo > > > > -- > Jason -- Jason

1 0

Re: About image migration
by Jason Dillaman 21 Aug '19

21 Aug '19

On Tue, Aug 20, 2019 at 10:04 PM Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> wrote: > > Hi jason: > > I have a question I would like to ask you, Is the current image migration and openstack adapted? according to my understanding, openstack’s previous live-migration logic is implemented in cinder, just call librbd rbd_read/write API to do Data migration. I believe the existing Cinder volume block live-migration is just a wrapper around QEMU's block live-migration functionality, so indeed it would just be a series of RBD read/write API calls between two volumes. The built-in RBD live-migration is similar but it copies snapshots and preserves image sparseness during the migration process. Because Cinder is using the QEMU block live-migration functionality, it's not tied into RBD's live-migration. > > > > Best wishes > > Zaharo -- Jason

1 0

Multisite RGW data corruption (not 14.2.1 curl issue)
by vladimir＠bashkirtsev.com 20 Aug '19

20 Aug '19

Hello, I have setup two separate Ceph clusters with RGW instance each and trying to achieve multisite data synchronization. Primary runs 13.2.5, slave runs 14.2.2 (I have upgraded slave side from 14.2.1 due to known data corruption during transfer due to curl errors). I have emptied slave zone and allowed sync to run from beginning to the end. Then I have recalculated MD5 hashes over original data and over data in slave zones and found that in some cases they do not match. Data corruption is evident. Comparison of data byte for byte shows that some parts of data just moved around (for example file has correct bytes from 0 to 513k and then exactly the same bytes from position of 513k up to 1026k - feels like as some sort of buffer issue). File size is correct. I have read the RGW sources and could not find anything what can cause such sort of behavior however I could not find one single piece of code which I would consider critical: during FetchRemote RGW obtains object's etag from remote RGW but apparently there nothing what recalculates MD5 from actual data and comparing it to etag received to ensure that data was correctly transferred. Even broken data then stored to local cluster and into bucket index. Then there just nothing further to prevent wrong data to reach end user downloading from slave zone. Such check should happen regardless of sync backend in use: objects with broken data should not be stored in bucket index at all. No one needs broken data and it is better just to fail object sync to allow it to try again later instead of storing faulty data and then allowing end users to download it. Is it me who cannot see such sort of check or it is actually missing?! And if it is not there at all I think it should be quite high on TODO list. Regards, Vladimir

2 2

Re: Failing heartbeats when no backfill is running
by Lorenz Kiefner 19 Aug '19

19 Aug '19

Oh no, it's not that bad. It's $ ping -s 65000 dest.inati.on on a VPN connection that has a MTU of 1300 via IPv6. So I suspect that I only get an answer, when all 51 fragments get fully returned. It's clear that big packets with lots of fragments are more affected by packet loss than 64 byte pings. I just (at 9 o'clock in the morning) repeated this ping test and got hardly any drops (less than 1%), even with the size of 64k. So it's really dependent on the time of the day. Seems like some ISPs are dropping some packets, especially in the evening... A few minutes ago I restarted all down-marked OSDs, but they are getting marked down again... Seems like Ceph is tolerable against packet loss (it surely affects performance, but this irrelevant for me). Could erasure coded pools pose some problems? Thank you all for every hint! Lorenz Am 15.08.19 um 08:51 schrieb Janne Johansson: > Den ons 14 aug. 2019 kl 17:46 skrev Lorenz Kiefner > <root+cephusers(a)deinadmin.de <mailto:root%2Bcephusers@deinadmin.de>>: > > Is ceph sensitive to packet loss? On some VPN links I have up to 20% > packet loss on 64k packets but less than 3% on 5k packets in the > evenings. > > > 20% seems crazy high, there must be something really wrong there. > > At 20%, you would get tons of packet timeouts to wait for on all those > lost frames, > then resends of (at least!) those 20% extra, which in turn would lead > to 20% of those > resends getting lost, all while the main streams of data try to move > forward when some > older packet do get over. This is a really bad situation to design for, > > I think you should look for a link solution that doesn't drop that > many packets, instead of changing > the software you try to run over that link, all others will notice > this too and act badly in some way or other. > > Heck, 20% is like taking a math schoolbook and remove all instances of > "3" and "8" and see if kids can learn to count from it. 8-/ > > -- > May the most significant bit of your life be positive.

2 3

Re: rbd image journal performance
by Jason Dillaman 18 Aug '19

18 Aug '19

On Mon, Aug 12, 2019 at 10:03 PM yangjun(a)cmss.chinamobile.com <yangjun(a)cmss.chinamobile.com> wrote: > > Hi Jason, > > I was recently testing the RBD mirror feature(ceph12.2.8), my test environment is a single-node cluster, which including 10 3T hdd OSDs + 800G pcie ssd + bluestore, and the wal and db partition of the OSD is 30G. > The test result of a 100G image is as follows： > > disable journal enable journal decline percentage > iops： 1000 877 12.3% > bw： 402MB/s 129MB/s 67% > > > Why does the bandwidth decline so much after starting journal of the RBD image? I'm very appreciate if you could give me some suggestions for optimization. Thank you very much. The use of the journal requires first writing to the journal and, once committed, writing to the image (i.e. doubling the latency). Therefore, the expected worst-case performance should be around 2x slower [1]. There was a recent bug fix [2] in the master branch that will be backported to older releases which greatly increases small IO journal performance -- since it was nearly 10x slower due to the bug instead of the expected 2x [3]. > ________________________________ > yangjun(a)cmss.chinamobile.com [1] https://www.slideshare.net/JasonDillaman/disaster-recovery-and-ceph-block-s… [2] https://tracker.ceph.com/issues/40072 [3] https://youtu.be/ZifNGprBUTA?t=1687 -- Jason

1 0

Re: Request to guide on ceph-deploy install command for luminuous 12.2.12 release
by Suresh Rama 18 Aug '19

18 Aug '19

Since you deleted the stack this is meaningless. You can simply delete the volumes from pool using the rbd. The proper way to delete is to delete them before destroying the stack. If the stack is alive and you have issues deleting, you can take two approach. 1) run openstack volume delete with --debug to see what happens. 2) use rbd status pool/volume to see is there a watcher to the volume. If yes, then the instence has not been removed/not properly removed. Thanks, Suresh On Fri, Aug 16, 2019, 8:21 PM Nerurkar, Ruchir (Nokia - US/Mountain View) < ruchir.nerurkar(a)nokia.com> wrote: > Hello, > > > > So I am able to install Ceph on CentOS 7.4 and I can successfully > integrate my Openstack testbed with it. > > > > However, I am facing with an issue recently where after deleting stack, my > cinder volumes are not getting deleted and they are getting stuck. > > > > Any idea on this issue? > > > > Best Regards, > > Ruchir Nerurkar > > 857-701-3405 > > > > *From:* Götz Reinicke <goetz.reinicke(a)filmakademie.de> > *Sent:* Monday, August 12, 2019 1:17 PM > *To:* Nerurkar, Ruchir (Nokia - US/Mountain View) < > ruchir.nerurkar(a)nokia.com> > *Cc:* ceph-users(a)ceph.io > *Subject:* Re: [ceph-users] Request to guide on ceph-deploy install > command for luminuous 12.2.12 release > > > > Hi, > > > > Am 07.08.2019 um 20:20 schrieb Nerurkar, Ruchir (Nokia - US/Mountain View) > <ruchir.nerurkar(a)nokia.com>: > > > > Hello, > > > > I work in Nokia as a Software QA engineer and I am trying to install Ceph > on centOS 7.4 version. > > > > Do you have to stick at 7.4? > > > > > > But I am getting this error with the following output: - > > > > vmgoscephcontrollerluminous][WARNIN] Error: Package: > 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph) > > [mvmgoscephcontrollerluminous][WARNIN] Requires: > liblz4.so.1()(64bit) > > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: > 2:ceph-osd-12.2.12-0.el7.x86_64 (Ceph) > > [mvmgoscephcontrollerluminous][WARNIN] Requires: > liblz4.so.1()(64bit) > > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: > 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph) > > [mvmgoscephcontrollerluminous][WARNIN] Requires: > gperftools-libs >= 2.6.1 > > [mvmgoscephcontrollerluminous][WARNIN] Available: > gperftools-libs-2.4-8.el7.i686 (base) > > [mvmgoscephcontrollerluminous][DEBUG ] You could try using --skip-broken > to work around the problem > > [mvmgoscephcontrollerluminous][WARNIN] gperftools-libs = > 2.4-8.el7 > > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: > 2:ceph-mon-12.2.12-0.el7.x86_64 (Ceph) > > [mvmgoscephcontrollerluminous][WARNIN] Requires: > liblz4.so.1()(64bit) > > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: > 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph) > > [mvmgoscephcontrollerluminous][WARNIN] Requires: > liblz4.so.1()(64bit) > > [mvmgoscephcontrollerluminous][DEBUG ] You could try running: rpm -Va > --nofiles --nodigest > > [mvmgoscephcontrollerluminous][ERROR ] RuntimeError: command returned > non-zero exit status: 1 > > [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y > install ceph ceph-radosgw > > > > Does anyone know about this issue like how can I resolve package > dependencies? > > > > My first shot: Update Centos to min. 7.5 (which > includes gperftools-libs-2.6.1) > > > > And install lz4 too. > > > > Regards . Götz > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Unsubscribe Me
by Arun POONIA 17 Aug '19

17 Aug '19

Remove email address ! -- Arun Poonia

1 0

CEPH Cluster Backup - Options on my solution
by Mike O'Connor 17 Aug '19

17 Aug '19

Hi All I've been worried that I did not have a good backup of my cluster and having looked around I could not find anything which did not require local storage. I found Rhian script while looking for a backup solution before a major version upgrade and found that it worked very well. I'm looking for opinions on this method below. This uses stdio to pipe the data directly from rbd to a s3 host, while encrypting the data. I have a few worries. 1. The snap shots, I think I should be deleting the older one and only keeping the the last one or two. 2. Having only one full backup and diff from there seems wrong to me but the amount of data would seem to preclude creating a new full back up on a regular biases. I'm sure that others will pipe up with more issues. Note. The sum.c program is needed because s3 needs to be told about files which are larger than 5Gig and the standard tools I tested all had a limit of 32bits when printing out the size in bytes Cheers Mike #!/bin/bash ### # Original Author: Rhian Resnick - Updated a lot by Mike O'Connor # Purpose: Backup CEPH RBD using snapshots, the files that are created should be stored off the ceph cluster, but you can use ceph storage during the process of backing them up. ### export AWS_PROFILE=wasabi PUBKEY='PublicKey' pool=$1 if [ “$pool” == “” ] then echo Usage: $0 pool exit 1 fi rbd ls $pool | while read vol do if [ $vol == ISO ] then continue fi # Look up latest backup file echo BACKUP ${vol} LASTSNAP=`aws s3 ls s3://dcbackup/$vol/ | sort | tail -n 1 | awk '{print $4}' | cut -d "." -f 1` echo "Last Snap: $vol/$LASTSNAP" # Create a snap, we need this to do the diff NEWSNAP=`date +%y%m%d%H%M` echo "New Snap: $NEWSNAP" echo rbd snap create $pool/$vol@$NEWSNAP rbd snap create $pool/$vol@$NEWSNAP if [ "$LASTSNAP" == "" ] then RBD_SIZE=`rbd diff $pool/$vol | ./sum` echo "rbd export-diff $pool/$vol@$NEWSNAP - | seccure-encrypt ${PUBKEY} | aws s3 cp --expected-size ${RBD_SIZE} - s3://dcbackup/$vol/$NEWSNAP.diff" rbd export-diff $pool/$vol@$NEWSNAP - | seccure-encrypt ${PUBKEY} | aws s3 cp --expected-size ${RBD_SIZE} - s3://dcbackup/$vol/$NEWSNAP.diff else RBD_SIZE=`rbd diff --from-snap $LASTSNAP $pool/$vol | ./sum` echo "rbd export-diff --from-snap $LASTSNAP $pool/$vol@$NEWSNAP - | seccure-encrypt ${PUBKEY} | aws s3 cp --expected-size ${RBD_SIZE} - s3://dcbackup/$vol/$NEWSNAP.diff" rbd export-diff --from-snap $LASTSNAP $pool/$vol@$NEWSNAP - | seccure-encrypt ${PUBKEY} | aws s3 cp - s3://dcbackup/$vol/$NEWSNAP.diff fi echo echo done #include <stdio.h> #include <inttypes.h> #include <string.h> #include <stdlib.h> /* sum column 2 of input with lines like: number number word * skipping the first (header) row * * usage: sum [-v] file * * if -v is specified a count of rows processed (including the header) is also output */ int main(int argc, char *argv[]) { uint64_t sum = 0L; size_t n = 100; unsigned int count=1; char *buf = (char*)malloc(n); getline(&buf,&n,stdin); while (1) { uint64_t offset, bytes; n = fscanf(stdin, "%lld %lld %s\n", &offset, &bytes, buf); if (n != 3) break; ++count; sum += bytes; } if (argc>1 && strcmp(argv[1],"-v")==0) printf("%d %lld\n", count, sum); else printf("%lld\n", sum); }

4 6

Mapped rbd is very slow
by Olivier AUDRY 17 Aug '19

17 Aug '19

hello I got a huge difference of read/write performance on my ceph cluster when I use rbd. rbd bench reach the limit of my cluster (1gbps network) when the performance in mapped rbd is very low. 30MB/s. ceph version : ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) kernel version : 4.19.0-5-amd64 ceph -s cluster: id: fdf6eba0-cde9-401c-8901-751eb1e8b7d2 health: HEALTH_OK services: mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4-k8s001 (age 41h) mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4- k8s001 osd: 10 osds: 10 up (since 41h), 10 in (since 41h) rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2-k8s001) data: pools: 8 pools, 240 pgs objects: 109.14k objects, 229 GiB usage: 694 GiB used, 3.0 TiB / 3.7 TiB avail pgs: 240 active+clean here the bench rbd create --size=10G kube/bench rbd feature disable kube/bench object-map fast-diff deep-flatten rbd -p kube bench kube/bench --io-type write --io-size 8192 --io- threads 256 --io-total 10G --io-pattern seq elapsed: 110 ops: 1310720 ops/sec: 11884.39 bytes/sec: 97356926.64 this reach the 1gbps limit of my network. Looks good. let's test random write rbd -p kube bench kube/bench --io-type write --io-size 8192 --io- threads 256 --io-total 10G --io-pattern rand elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58 this reach the 1gbps limit of my network. Looks good. rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seq elapsed: 40 ops: 1310720 ops/sec: 32712.03 bytes/sec: 267976974.35 this is great perf ! let's try rand rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand elapsed: 29 ops: 1310720 ops/sec: 43859.78 bytes/sec: 359299337.92 this is great perf ! But when I mount the image : rbd map bench --pool kube --name client.admin /sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ cd /mnt/ dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s 34.1MB/s vs 85MB/s ....

6 18

Mapped rbd is very slowe
by Olivier AUDRY 16 Aug '19

16 Aug '19

hello I got a huge difference of read/write performance on my ceph cluster when I use rbd. rbd bench reach the limit of my cluster (1gbps network) when the performance in mapped rbd is very low. 30MB/s. ceph version : ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)kernel version : 4.19.0-5-amd64 ceph -s cluster: id: fdf6eba0-cde9-401c-8901- 751eb1e8b7d2 health: HEALTH_OK services: mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4- k8s001 (age 41h) mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4-k8s001 osd: 10 osds: 10 up (since 41h), 10 in (since 41h) rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2- k8s001) data: pools: 8 pools, 240 pgs objects: 109.14k objects, 229 GiB usage: 694 GiB used, 3.0 TiB / 3.7 TiB avail pgs: 240 active+clean here the bench rbd create --size=10G kube/benchrbd feature disable kube/bench object- map fast-diff deep-flatten rbd -p kube bench kube/bench --io-type write --io-size 8192 --io- threads 256 --io-total 10G --io-pattern seqelapsed: 110 ops: 1310720 ops/sec: 11884.39 bytes/sec: 97356926.64this reach the 1gbps limit of my network. Looks good. let's test random writerbd -p kube bench kube/bench --io-type write -- io-size 8192 --io-threads 256 --io-total 10G --io-pattern randelapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58this reach the 1gbps limit of my network. Looks good. rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seqelapsed: 40 ops: 1310720 ops/sec: 32712.03 bytes/sec: 267976974.35this is great perf ! let's try randrbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern randelapsed: 29 ops: 1310720 ops/sec: 43859.78 bytes/sec: 359299337.92 this is great perf ! But when I mount the image :rbd map bench --pool kube --name client.admin/sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/cd /mnt/dd if=/dev/zero of=test bs=8192k count=100 oflag=direct838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s 34.1MB/s vs 85MB/s .... here the full config { "name": "mon.sbg2-k8s001", "cluster": "ceph", "admin_socket": "/var/run/ceph/ceph-mon.sbg2- k8s001.asok", "admin_socket_mode": "", "auth_client_required": "cephx", "auth_cluster_required": "cephx", "auth_debug": "false", "auth_mon_ticket_ttl": "43200.000000", "auth_service_required": "cephx", "auth_service_ticket_ttl": "3600.000000", "auth_supported": "", "bdev_aio": "true", "bdev_aio_max_queue_depth": "1024", "bdev_aio_poll_ms": "250", "bdev_aio_reap_max": "16", "bdev_async_discard": "false", "bdev_block_size": "4096", "bdev_debug_aio": "false", "bdev_debug_aio_log_age": "5.000000", "bdev_debug_aio_suicide_timeout": "60.000000", "bdev_debug_inflight_ios": "false", "bdev_enable_discard": "false", "bdev_inject_crash": "0", "bdev_inject_crash_flush_delay": "2", "bdev_nvme_retry_count": "- 1", "bdev_nvme_unbind_from_kernel": "false", "bluefs_alloc_size": "1048576", "bluefs_allocator": "bitmap", "bluefs_buffered_io": "true", "bluefs_compact_log_sync": "false", "bluefs_log_compact_min_ratio": "5.000000", "bluefs_log_compact_min_size": "16777216", "bluefs_max_log_runway": "4194304", "bluefs_max_prefetch": "1048576", "bluefs_min_flush_size": "524288", "bluefs_min_log_runway": "1048576", "bluefs_preextend_wal_files": "false", "bluefs_sync_write": "false", "bluestore_2q_cache_kin_ratio": "0.500000", "bluestore_2q_cache_kout_ratio": "0.500000", "bluestore_allocator": "bitmap", "bluestore_bitmapallocator_blocks_per_zone": "1024", "bluestore_bitmapallocator_span_size": "1024", "bluestore_blobid_prealloc": "10240", "bluestore_block_create": "true", "bluestore_block_db_create": "false", "bluestore_block_db_path": "", "bluestore_block_db_size": "0", "bluestore_block_path": "", "bluestore_block_preallocate_file": "false", "bluestore_block_size": "10737418240", "bluestore_block_wal_create": "false", "bluestore_block_wal_path": "", "bluestore_block_wal_size": "100663296", "bluestore_bluefs": "true", "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", "bluestore_bluefs_balance_interval": "1.000000", "bluestore_bluefs_db_compatibility": "true", "bluestore_bluefs_env_mirror": "false", "bluestore_bluefs_gift_ratio": "0.020000", "bluestore_bluefs_max_ratio": "0.900000", "bluestore_bluefs_min": "1073741824", "bluestore_bluefs_min_free": "1073741824", "bluestore_bluefs_min_ratio": "0.020000", "bluestore_bluefs_reclaim_ratio": "0.200000", "bluestore_cache_autotune": "true", "bluestore_cache_autotune_interval": "5.000000", "bluestore_cache_kv_ratio": "0.400000", "bluestore_cache_meta_ratio": "0.400000", "bluestore_cache_size": "0", "bluestore_cache_size_hdd": "1073741824", "bluestore_cache_size_ssd": "3221225472", "bluestore_cache_trim_interval": "0.050000", "bluestore_cache_trim_max_skip_pinned": "64", "bluestore_cache_type": "2q", "bluestore_clone_cow": "true", "bluestore_compression_algorithm": "snappy", "bluestore_compression_max_blob_size": "0", "bluestore_compression_max_blob_size_hdd": "524288", "bluestore_compression_max_blob_size_ssd": "65536", "bluestore_compression_min_blob_size": "0", "bluestore_compression_min_blob_size_hdd": "131072", "bluestore_compression_min_blob_size_ssd": "8192", "bluestore_compression_mode": "none", "bluestore_compression_required_ratio": "0.875000", "bluestore_csum_type": "crc32c", "bluestore_debug_freelist": "false", "bluestore_debug_fsck_abort": "false", "bluestore_debug_inject_bug21040": "false", "bluestore_debug_inject_csum_err_probability": "0.000000", "bluestore_debug_inject_read_err": "false", "bluestore_debug_misc": "false", "bluestore_debug_no_reuse_blocks": "false", "bluestore_debug_omit_block_device_write": "false", "bluestore_debug_omit_kv_commit": "false", "bluestore_debug_permit_any_bdev_label": "false", "bluestore_debug_prefill": "0.000000", "bluestore_debug_prefragment_max": "1048576", "bluestore_debug_random_read_err": "0.000000", "bluestore_debug_randomize_serial_transaction": "0", "bluestore_debug_small_allocations": "0", "bluestore_default_buffered_read": "true", "bluestore_default_buffered_write": "false", "bluestore_deferred_batch_ops": "0", "bluestore_deferred_batch_ops_hdd": "64", "bluestore_deferred_batch_ops_ssd": "16", "bluestore_extent_map_inline_shard_prealloc_size": "256", "bluestore_extent_map_shard_max_size": "1200", "bluestore_extent_map_shard_min_size": "150", "bluestore_extent_map_shard_target_size": "500", "bluestore_extent_map_shard_target_size_slop": "0.200000", "bluestore_freelist_blocks_per_key": "128", "bluestore_fsck_on_mkfs": "true", "bluestore_fsck_on_mkfs_deep": "false", "bluestore_fsck_on_mount": "false", "bluestore_fsck_on_mount_deep": "true", "bluestore_fsck_on_umount": "false", "bluestore_fsck_on_umount_deep": "true", "bluestore_fsck_read_bytes_cap": "67108864", "bluestore_gc_enable_blob_threshold": "0", "bluestore_gc_enable_total_threshold": "0", "bluestore_ignore_data_csum": "false", "bluestore_kvbackend": "rocksdb", "bluestore_log_omap_iterator_age": "1.000000", "bluestore_log_op_age": "5.000000", "bluestore_max_alloc_size": "0", "bluestore_max_blob_size": "0", "bluestore_max_blob_size_hdd": "524288", "bluestore_max_blob_size_ssd": "65536", "bluestore_max_deferred_txc": "32", "bluestore_min_alloc_size": "0", "bluestore_min_alloc_size_hdd": "65536", "bluestore_min_alloc_size_ssd": "16384", "bluestore_nid_prealloc": "1024", "bluestore_no_per_pool_stats_tolerance": "until_repair", "bluestore_prefer_deferred_size": "0", "bluestore_prefer_deferred_size_hdd": "32768", "bluestore_prefer_deferred_size_ssd": "0", "bluestore_retry_disk_reads": "3", "bluestore_rocksdb_cf": "false", "bluestore_rocksdb_cfs": "M= P= L=", "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_ number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,wr itable_file_max_buffer_size=0,compaction_readahead_size=2097152", "b luestore_spdk_coremask": "0x1", "bluestore_spdk_io_sleep": "5", "bluestore_spdk_max_io_completion": "0", "bluestore_spdk_mem": "512", "bluestore_sync_submit_transaction": "false", "bluestore_throttle_bytes": "67108864", "bluestore_throttle_cost_per_io": "0", "bluestore_throttle_cost_per_io_hdd": "670000", "bluestore_throttle_cost_per_io_ssd": "4000", "bluestore_throttle_deferred_bytes": "134217728", "bluestore_warn_on_bluefs_spillover": "true", "bluestore_warn_on_legacy_statfs": "true", "cephx_cluster_require_signatures": "false", "cephx_cluster_require_version": "1", "cephx_require_signatures": "false", "cephx_require_version": "1", "cephx_service_require_signatures": "false", "cephx_service_require_version": "1", "cephx_sign_messages": "true", "chdir": "", "client_acl_type": "", "client_cache_mid": "0.750000", "client_cache_size": "16384", "client_caps_release_delay": "5", "client_check_pool_perm": "true", "client_debug_force_sync_read": "false", "client_debug_getattr_caps": "false", "client_debug_inject_tick_delay": "0", "client_die_on_failed_dentry_invalidate": "true", "client_die_on_failed_remount": "false", "client_dirsize_rbytes": "true", "client_force_lazyio": "false", "client_inject_fixed_oldest_tid": "false", "client_inject_release_failure": "false", "client_max_inline_size": "4096", "client_mds_namespace": "", "client_metadata": "", "client_mount_gid": "-1", "client_mount_timeout": "300.000000", "client_mount_uid": "-1", "client_mountpoint": "/", "client_notify_timeout": "10", "client_oc": "true", "client_oc_max_dirty": "104857600", "client_oc_max_dirty_age": "5.000000", "client_oc_max_objects": "1000", "client_oc_size": "209715200", "client_oc_target_dirty": "8388608", "client_permissions": "true", "client_quota_df": "true", "client_readahead_max_bytes": "0", "client_readahead_max_periods": "4", "client_readahead_min": "131072", "client_reconnect_stale": "false", "client_snapdir": ".snap", "client_tick_interval": "1.000000", "client_trace": "", "client_try_dentry_invalidate": "false", "client_use_faked_inos": "false", "client_use_random_mds": "false", "clog_to_graylog": "false", "clog_to_graylog_host": "127.0.0.1", "clog_to_graylog_port": "12201", "clog_to_monitors": "default=true", "clog_to_syslog": "false", "clog_to_syslog_facility": "default=daemon audit=local0", "clog_to_syslog_level": "info", "cluster_addr": "- ", "cluster_network": "192.168.212.0/24", "cluster_network_interface": "", "compressor_zlib_isal": "false", "compressor_zlib_level": "5", "crash_dir": "/var/lib/ceph/crash", "crush_location": "", "crush_location_hook": "", "crush_location_hook_timeout": "10", "daemonize": "false", "debug_allow_any_pool_priority": "false", "debug_asok": "1/5", "debug_asok_assert_abort": "false", "debug_asserts_on_shutdown": "false", "debug_auth": "1/5", "debug_bdev": "1/3", "debug_bluefs": "1/5", "debug_bluestore": "1/5", "debug_buffer": "0/1", "debug_civetweb": "1/10", "debug_client": "0/5", "debug_compressor": "1/5", "debug_context": "0/1", "debug_crush": "1/1", "debug_crypto": "1/5", "debug_deliberately_leak_memory": "false", "debug_dpdk": "1/5", "debug_eventtrace": "1/5", "debug_filer": "0/1", "debug_filestore": "1/3", "debug_finisher": "1/1", "debug_fuse": "1/5", "debug_heartbeatmap": "1/5", "debug_javaclient": "1/5", "debug_journal": "1/3", "debug_journaler": "0/5", "debug_kinetic": "1/5", "debug_kstore": "1/5", "debug_leveldb": "4/5", "debug_lockdep": "0/1", "debug_mds": "1/5", "debug_mds_balancer": "1/5", "debug_mds_locker": "1/5", "debug_mds_log": "1/5", "debug_mds_log_expire": "1/5", "debug_mds_migrator": "1/5", "debug_memdb": "4/5", "debug_mgr": "1/5", "debug_mgrc": "1/5", "debug_mon": "1/5", "debug_monc": "0/10", "debug_ms": "0/0", "debug_none": "0/5", "debug_objclass": "0/5", "debug_objectcacher": "0/5", "debug_objecter": "0/1", "debug_optracker": "0/5", "debug_osd": "1/5", "debug_paxos": "1/5", "debug_perfcounter": "1/5", "debug_rados": "0/5", "debug_rbd": "0/5", "debug_rbd_mirror": "0/5", "debug_rbd_replay": "0/5", "debug_refs": "0/0", "debug_reserver": "1/1", "debug_rgw": "1/5", "debug_rgw_sync": "1/5", "debug_rocksdb": "4/5", "debug_striper": "0/1", "debug_throttle": "1/1", "debug_timer": "0/1", "debug_tp": "0/5", "debug_xio": "1/5", "device_failure_prediction_mode": "none", "enable_experimental_unrecoverable_data_corrupting_features" : "", "erasure_code_dir": "/usr/lib/ceph/erasure- code", "err_to_graylog": "false", "err_to_stderr": "true", "err_to_syslog": "false", "event_tracing": "false", "fake_statfs_for_testing": "0", "fatal_signal_handlers": "true", "filer_max_purge_ops": "10", "filer_max_truncate_ops": "128", "filestore_apply_finisher_threads": "1", "filestore_blackhole": "false", "filestore_btrfs_clone_range": "true", "filestore_btrfs_snap": "true", "filestore_caller_concurrency": "10", "filestore_collect_device_partition_information": "true", "filestore_commit_timeout": "600.000000", "filestore_debug_inject_read_err": "false", "filestore_debug_omap_check": "false", "filestore_debug_random_read_err": "0.000000", "filestore_debug_verify_split": "false", "filestore_dump_file": "", "filestore_expected_throughput_bytes": "209715200.000000", "filestore_expected_throughput_ops": "200.000000", "filestore_fadvise": "true", "filestore_fail_eio": "true", "filestore_fd_cache_shards": "16", "filestore_fd_cache_size": "128", "filestore_fiemap": "false", "filestore_fiemap_threshold": "4096", "filestore_fsync_flushes_journal_data": "false", "filestore_index_retry_probability": "0.000000", "filestore_inject_stall": "0", "filestore_journal_parallel": "false", "filestore_journal_trailing": "false", "filestore_journal_writeahead": "false", "filestore_kill_at": "0", "filestore_max_alloc_hint_size": "1048576", "filestore_max_inline_xattr_size": "0", "filestore_max_inline_xattr_size_btrfs": "2048", "filestore_max_inline_xattr_size_other": "512", "filestore_max_inline_xattr_size_xfs": "65536", "filestore_max_inline_xattrs": "0", "filestore_max_inline_xattrs_btrfs": "10", "filestore_max_inline_xattrs_other": "2", "filestore_max_inline_xattrs_xfs": "10", "filestore_max_sync_interval": "5.000000", "filestore_max_xattr_value_size": "0", "filestore_max_xattr_value_size_btrfs": "65536", "filestore_max_xattr_value_size_other": "1024", "filestore_max_xattr_value_size_xfs": "65536", "filestore_merge_threshold": "- 10", "filestore_min_sync_interval": "0.010000", "filestore_odsync_write": "false", "filestore_omap_backend": "rocksdb", "filestore_omap_backend_path": "", "filestore_omap_header_cache_size": "1024", "filestore_ondisk_finisher_threads": "1", "filestore_op_thread_suicide_timeout": "180", "filestore_op_thread_timeout": "60", "filestore_op_threads": "2", "filestore_punch_hole": "false", "filestore_queue_high_delay_multiple": "0.000000", "filestore_queue_high_delay_multiple_bytes": "0.000000", "filestore_queue_high_delay_multiple_ops": "0.000000", "filestore_queue_high_threshhold": "0.900000", "filestore_queue_low_threshhold": "0.300000", "filestore_queue_max_bytes": "104857600", "filestore_queue_max_delay_multiple": "0.000000", "filestore_queue_max_delay_multiple_bytes": "0.000000", "filestore_queue_max_delay_multiple_ops": "0.000000", "filestore_queue_max_ops": "50", "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=k NoCompression", "filestore_seek_data_hole": "false", "filestore_sloppy_crc": "false", "filestore_sloppy_crc_block_size": "65536", "filestore_splice": "false", "filestore_split_multiple": "2", "filestore_split_rand_factor": "20", "filestore_update_to": "1000", "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", "filestore_wbthrottle_btrfs_ios_start_flusher": "500", "filestore_wbthrottle_enable": "true", "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", "filestore_wbthrottle_xfs_inodes_start_flusher": "500", "filestore_wbthrottle_xfs_ios_hard_limit": "5000", "filestore_wbthrottle_xfs_ios_start_flusher": "500", "filestore_xfs_extsize": "false", "filestore_zfs_snap": "false", "fio_dir": "/tmp/fio", "fsid": "fdf6eba0-cde9-401c-8901- 751eb1e8b7d2", "fuse_allow_other": "true", "fuse_atomic_o_trunc": "true", "fuse_big_writes": "false", "fuse_debug": "false", "fuse_default_permissions": "false", "fuse_disable_pagecache": "false", "fuse_max_write": "0", "fuse_multithreaded": "true", "fuse_require_active_mds": "true", "fuse_set_user_groups": "true", "fuse_syncfs_on_mksnap": "true", "fuse_use_invalidate_cb": "true", "gss_ktab_client_file": "/var/lib/ceph/mon.sbg2-k8s001/gss_client_mon.sbg2- k8s001.ktab", "gss_target_name": "ceph", "heartbeat_file": "", "heartbeat_inject_failure": "0", "heartbeat_interval": "5", "host": "", "inject_early_sigterm": "false", "journal_aio": "true", "journal_align_min_size": "65536", "journal_block_align": "true", "journal_block_size": "4096", "journal_dio": "true", "journal_discard": "false", "journal_force_aio": "false", "journal_ignore_corruption": "false", "journal_max_write_bytes": "1073714824", "journal_max_write_entries": "10000", "journal_replay_from": "0", "journal_throttle_high_multiple": "0.000000", "journal_throttle_high_threshhold": "0.900000", "journal_throttle_low_threshhold": "0.600000", "journal_throttle_max_multiple": "0.000000", "journal_write_header_frequency": "0", "journal_zero_on_create": "false", "journaler_prefetch_periods": "10", "journaler_prezero_periods": "5", "journaler_write_head_interval": "15", "key": "", "keyfile": "", "keyring": "/var/lib/ceph/mon/ceph-sbg2- k8s001/keyring", "kinetic_hmac_key": "asdfasdf", "kinetic_host": "", "kinetic_port": "8123", "kinetic_use_ssl": "false", "kinetic_user_id": "1", "kstore_backend": "rocksdb", "kstore_default_stripe_size": "65536", "kstore_fsck_on_mount": "false", "kstore_fsck_on_mount_deep": "true", "kstore_max_bytes": "67108864", "kstore_max_ops": "512", "kstore_nid_prealloc": "1024", "kstore_onode_map_size": "1024", "kstore_rocksdb_options": "compression=kNoCompression", "kstore_sync_submit_transaction": "false", "kstore_sync_transaction": "false", "leveldb_block_size": "65536", "leveldb_bloom_size": "0", "leveldb_cache_size": "536870912", "leveldb_compact_on_mount": "false", "leveldb_compression": "false", "leveldb_log": "", "leveldb_log_to_ceph_log": "true", "leveldb_max_open_files": "0", "leveldb_paranoid": "false", "leveldb_write_buffer_size": "33554432", "lockdep": "false", "lockdep_force_backtrace": "false", "log_coarse_timestamps": "true", "log_file": "/var/log/ceph/ceph-mon.sbg2-k8s001.log", "log_flush_on_exit": "false", "log_graylog_host": "127.0.0.1", "log_graylog_port": "12201", "log_max_new": "1000", "log_max_recent": "10000", "log_stderr_prefix": "", "log_stop_at_utilization": "0.970000", "log_to_file": "true", "log_to_graylog": "false", "log_to_stderr": "false", "log_to_syslog": "false", "max_rotating_auth_attempts": "10", "mds_action_on_write_error": "1", "mds_bal_export_pin": "true", "mds_bal_fragment_dirs": "true", "mds_bal_fragment_fast_factor": "1.500000", "mds_bal_fragment_interval": "5", "mds_bal_fragment_size_max": "100000", "mds_bal_idle_threshold": "0.000000", "mds_bal_interval": "10", "mds_bal_max": "- 1", "mds_bal_max_until": "-1", "mds_bal_merge_size": "50", "mds_bal_midchunk": "0.300000", "mds_bal_min_rebalance": "0.100000", "mds_bal_min_start": "0.200000", "mds_bal_minchunk": "0.001000", "mds_bal_mode": "0", "mds_bal_need_max": "1.200000", "mds_bal_need_min": "0.800000", "mds_bal_replicate_threshold": "8000.000000", "mds_bal_sample_interval": "3.000000", "mds_bal_split_bits": "3", "mds_bal_split_rd": "25000.000000", "mds_bal_split_size": "10000", "mds_bal_split_wr": "10000.000000", "mds_bal_target_decay": "10.000000", "mds_bal_unreplicate_threshold": "0.000000", "mds_beacon_grace": "15.000000", "mds_beacon_interval": "4.000000", "mds_cache_memory_limit": "1073741824", "mds_cache_mid": "0.700000", "mds_cache_reservation": "0.050000", "mds_cache_size": "0", "mds_cache_trim_decay_rate": "1.000000", "mds_cache_trim_threshold": "65536", "mds_cap_revoke_eviction_timeout": "0.000000", "mds_client_prealloc_inos": "1000", "mds_client_writeable_range_max_inc_objs": "1024", "mds_damage_table_max_entries": "10000", "mds_data": "/var/lib/ceph/mds/ceph-sbg2-k8s001", "mds_debug_auth_pins": "false", "mds_debug_frag": "false", "mds_debug_scatterstat": "false", "mds_debug_subtrees": "false", "mds_decay_halflife": "5.000000", "mds_default_dir_hash": "2", "mds_dir_keys_per_op": "16384", "mds_dir_max_commit_size": "10", "mds_dirstat_min_interval": "1.000000", "mds_dump_cache_after_rejoin": "false", "mds_dump_cache_on_map": "false", "mds_dump_cache_threshold_file": "0", "mds_dump_cache_threshold_formatter": "1073741824", "mds_early_reply": "true", "mds_enable_op_tracker": "true", "mds_enforce_unique_name": "true", "mds_freeze_tree_timeout": "30.000000", "mds_hack_allow_loading_invalid_metadata": "false", "mds_health_cache_threshold": "1.500000", "mds_health_summarize_threshold": "10", "mds_heartbeat_grace": "15.000000", "mds_inject_migrator_session_race": "false", "mds_inject_traceless_reply_probability": "0.000000", "mds_journal_format": "1", "mds_kill_create_at": "0", "mds_kill_export_at": "0", "mds_kill_import_at": "0", "mds_kill_journal_at": "0", "mds_kill_journal_expire_at": "0", "mds_kill_journal_replay_at": "0", "mds_kill_link_at": "0", "mds_kill_mdstable_at": "0", "mds_kill_openc_at": "0", "mds_kill_rename_at": "0", "mds_log_events_per_segment": "1024", "mds_log_max_events": "-1", "mds_log_max_segments": "128", "mds_log_pause": "false", "mds_log_segment_size": "0", "mds_log_skip_corrupt_events": "false", "mds_max_caps_per_client": "1048576", "mds_max_completed_flushes": "100000", "mds_max_completed_requests": "100000", "mds_max_export_size": "20971520", "mds_max_file_recover": "32", "mds_max_purge_files": "64", "mds_max_purge_ops": "8192", "mds_max_purge_ops_per_pg": "0.500000", "mds_max_retries_on_remount_failure": "5", "mds_max_scrub_ops_in_progress": "5", "mds_max_xattr_pairs_size": "65536", "mds_min_caps_per_client": "100", "mds_mon_shutdown_timeout": "5.000000", "mds_op_complaint_time": "30.000000", "mds_op_history_duration": "600", "mds_op_history_size": "20", "mds_op_log_threshold": "5", "mds_purge_queue_busy_flush_period": "1.000000", "mds_recall_global_max_decay_threshold": "65536", "mds_recall_max_caps": "5000", "mds_recall_max_decay_rate": "2.500000", "mds_recall_max_decay_threshold": "16384", "mds_recall_warning_decay_rate": "60.000000", "mds_recall_warning_threshold": "32768", "mds_reconnect_timeout": "45.000000", "mds_replay_interval": "1.000000", "mds_request_load_average_decay_rate": "60.000000", "mds_root_ino_gid": "0", "mds_root_ino_uid": "0", "mds_scatter_nudge_interval": "5.000000", "mds_session_blacklist_on_evict": "true", "mds_session_blacklist_on_timeout": "true", "mds_sessionmap_keys_per_op": "1024", "mds_shutdown_check": "0", "mds_skip_ino": "0", "mds_snap_max_uid": "4294967294", "mds_snap_min_uid": "0", "mds_snap_rstat": "false", "mds_thrash_exports": "0", "mds_thrash_fragments": "0", "mds_tick_interval": "5.000000", "mds_verify_backtrace": "1", "mds_verify_scatter": "false", "mds_wipe_ino_prealloc": "false", "mds_wipe_sessions": "false", "mempool_debug": "false", "memstore_device_bytes": "1073741824", "memstore_page_set": "false", "memstore_page_size": "65536", "mgr_client_bytes": "134217728", "mgr_client_messages": "512", "mgr_client_service_daemon_unregister_timeout": "1.000000", "mgr_connect_retry_interval": "1.000000", "mgr_data": "/var/lib/ceph/mgr/ceph-sbg2- k8s001", "mgr_debug_aggressive_pg_num_changes": "false", "mgr_initial_modules": "restful iostat", "mgr_mds_bytes": "134217728", "mgr_mds_messages": "128", "mgr_module_path": "/usr/share/ceph/mgr", "mgr_mon_bytes": "134217728", "mgr_mon_messages": "128", "mgr_osd_bytes": "536870912", "mgr_osd_messages": "8192", "mgr_service_beacon_grace": "60.000000", "mgr_stats_period": "5", "mgr_stats_threshold": "5", "mgr_tick_period": "2", "mon_accept_timeout_factor": "2.000000", "mon_allow_pool_delete": "false", "mon_cache_target_full_warn_ratio": "0.660000", "mon_client_bytes": "104857600", "mon_client_hunt_interval": "3.000000", "mon_client_hunt_interval_backoff": "1.500000", "mon_client_hunt_interval_max_multiple": "10.000000", "mon_client_hunt_interval_min_multiple": "1.000000", "mon_client_hunt_parallel": "3", "mon_client_max_log_entries_per_message": "1000", "mon_client_ping_interval": "10.000000", "mon_client_ping_timeout": "30.000000", "mon_clock_drift_allowed": "0.050000", "mon_clock_drift_warn_backoff": "5.000000", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_cluster_log_file_level": "debug", "mon_cluster_log_to_file": "true", "mon_cluster_log_to_graylog": "false", "mon_cluster_log_to_graylog_host": "127.0.0.1", "mon_cluster_log_to_graylog_port": "12201", "mon_cluster_log_to_stderr": "false", "mon_cluster_log_to_syslog": "default=false", "mon_cluster_log_to_syslog_facility": "daemon", "mon_cluster_log_to_syslog_level": "info", "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_config_key_max_entry_size": "65536", "mon_cpu_threads": "4", "mon_crush_min_required_version": "hammer", "mon_daemon_bytes": "419430400", "mon_data": "/var/lib/ceph/mon/ceph-sbg2-k8s001", "mon_data_avail_crit": "5", "mon_data_avail_warn": "30", "mon_data_size_warn": "16106127360", "mon_debug_block_osdmap_trim": "false", "mon_debug_deprecated_as_obsolete": "false", "mon_debug_dump_json": "false", "mon_debug_dump_location": "/var/log/ceph/ceph-mon.sbg2- k8s001.tdump", "mon_debug_dump_transactions": "false", "mon_debug_extra_checks": "false", "mon_debug_no_initial_persistent_features": "false", "mon_debug_no_require_bluestore_for_ec_overwrites": "false", "mon_debug_no_require_mimic": "false", "mon_debug_no_require_nautilus": "false", "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", "mon_delta_reset_interval": "10.000000", "mon_dns_srv_name": "ceph- mon", "mon_election_timeout": "5.000000", "mon_enable_op_tracker": "true", "mon_fake_pool_delete": "false", "mon_force_quorum_join": "false", "mon_globalid_prealloc": "10000", "mon_health_log_update_period": "5", "mon_health_max_detail": "50", "mon_health_to_clog": "true", "mon_health_to_clog_interval": "3600", "mon_health_to_clog_tick_interval": "60.000000", "mon_host": "192.168.211.14", "mon_initial_members": "rbx4-k8s001", "mon_inject_pg_merge_bounce_probability": "0.000000", "mon_inject_sync_get_chunk_delay": "0.000000", "mon_inject_transaction_delay_max": "10.000000", "mon_inject_transaction_delay_probability": "0.000000", "mon_keyvaluedb": "rocksdb", "mon_lease": "5.000000", "mon_lease_ack_timeout_factor": "2.000000", "mon_lease_renew_interval_factor": "0.600000", "mon_log_max_summary": "50", "mon_max_log_entries_per_event": "4096", "mon_max_log_epochs": "500", "mon_max_mdsmap_epochs": "500", "mon_max_mgrmap_epochs": "500", "mon_max_osd": "10000", "mon_max_pg_per_osd": "250", "mon_max_pool_pg_num": "65536", "mon_max_snap_prune_per_epoch": "100", "mon_mds_blacklist_interval": "86400.000000", "mon_mds_force_trim_to": "0", "mon_mds_skip_sanity": "false", "mon_mgr_beacon_grace": "30", "mon_mgr_digest_period": "5", "mon_mgr_inactive_grace": "60", "mon_mgr_mkfs_grace": "120", "mon_mgr_proxy_client_bytes_ratio": "0.300000", "mon_min_osdmap_epochs": "500", "mon_op_complaint_time": "30", "mon_op_history_duration": "600", "mon_op_history_size": "20", "mon_op_history_slow_op_size": "20", "mon_op_history_slow_op_threshold": "10", "mon_op_log_threshold": "5", "mon_osd_adjust_down_out_interval": "true", "mon_osd_adjust_heartbeat_grace": "true", "mon_osd_auto_mark_auto_out_in": "true", "mon_osd_auto_mark_in": "false", "mon_osd_auto_mark_new_in": "true", "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_blacklist_default_expire": "3600.000000", "mon_osd_cache_size": "500", "mon_osd_crush_smoke_test": "true", "mon_osd_destroyed_out_interval": "600", "mon_osd_down_out_interval": "600", "mon_osd_down_out_subtree_limit": "rack", "mon_osd_err_op_age_ratio": "128.000000", "mon_osd_force_trim_to": "0", "mon_osd_full_ratio": "0.950000", "mon_osd_initial_require_min_compat_client": "jewel", "mon_osd_laggy_halflife": "3600", "mon_osd_laggy_max_interval": "300", "mon_osd_laggy_weight": "0.300000", "mon_osd_mapping_pgs_per_chunk": "4096", "mon_osd_max_creating_pgs": "1024", "mon_osd_max_initial_pgs": "1024", "mon_osd_min_down_reporters": "2", "mon_osd_min_in_ratio": "0.750000", "mon_osd_min_up_ratio": "0.300000", "mon_osd_nearfull_ratio": "0.850000", "mon_osd_prime_pg_temp": "true", "mon_osd_prime_pg_temp_max_estimate": "0.250000", "mon_osd_prime_pg_temp_max_time": "0.500000", "mon_osd_report_timeout": "900", "mon_osd_reporter_subtree_level": "host", "mon_osd_snap_trim_queue_warn_on": "32768", "mon_osd_warn_op_age": "32.000000", "mon_osdmap_full_prune_enabled": "true", "mon_osdmap_full_prune_interval": "10", "mon_osdmap_full_prune_min": "10000", "mon_osdmap_full_prune_txsize": "100", "mon_pg_check_down_all_threshold": "0.500000", "mon_pg_stuck_threshold": "60", "mon_pg_warn_max_object_skew": "10.000000", "mon_pg_warn_min_objects": "10000", "mon_pg_warn_min_per_osd": "30", "mon_pg_warn_min_pool_objects": "1000", "mon_pool_quota_crit_threshold": "0", "mon_pool_quota_warn_threshold": "0", "mon_probe_timeout": "2.000000", "mon_reweight_max_change": "0.050000", "mon_reweight_max_osds": "4", "mon_reweight_min_bytes_per_osd": "104857600", "mon_reweight_min_pgs_per_osd": "10", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction _dynamic_level_bytes=true", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_session_timeout": "300", "mon_smart_report_timeout": "5", "mon_stat_smooth_intervals": "6", "mon_subscribe_interval": "86400.000000", "mon_sync_debug": "false", "mon_sync_max_payload_size": "1048576", "mon_sync_provider_kill_at": "0", "mon_sync_requester_kill_at": "0", "mon_sync_timeout": "60.000000", "mon_target_pg_per_osd": "100", "mon_tick_interval": "5", "mon_timecheck_interval": "300.000000", "mon_timecheck_skew_interval": "30.000000", "mon_warn_on_cache_pools_without_hit_sets": "true", "mon_warn_on_crush_straw_calc_version_zero": "true", "mon_warn_on_legacy_crush_tunables": "true", "mon_warn_on_misplaced": "false", "mon_warn_on_msgr2_not_enabled": "true", "mon_warn_on_osd_down_out_interval_zero": "true", "mon_warn_on_pool_no_app": "true", "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "monmap": "", "ms_async_max_op_threads": "5", "ms_async_op_threads": "3", "ms_async_rdma_buffer_size": "131072", "ms_async_rdma_cm": "false", "ms_async_rdma_device_name": "", "ms_async_rdma_dscp": "96", "ms_async_rdma_enable_hugepage": "false", "ms_async_rdma_local_gid": "", "ms_async_rdma_polling_us": "1000", "ms_async_rdma_port_num": "1", "ms_async_rdma_receive_buffers": "32768", "ms_async_rdma_receive_queue_len": "4096", "ms_async_rdma_roce_ver": "1", "ms_async_rdma_send_buffers": "1024", "ms_async_rdma_sl": "3", "ms_async_rdma_support_srq": "true", "ms_async_rdma_type": "ib", "ms_bind_before_connect": "false", "ms_bind_ipv4": "true", "ms_bind_ipv6": "false", "ms_bind_msgr1": "true", "ms_bind_msgr2": "true", "ms_bind_port_max": "7300", "ms_bind_port_min": "6800", "ms_bind_prefer_ipv4": "false", "ms_bind_retry_count": "3", "ms_bind_retry_delay": "5", "ms_client_mode": "crc", "ms_cluster_mode": "crc", "ms_cluster_type": "", "ms_connection_idle_timeout": "900", "ms_connection_ready_timeout": "10", "ms_crc_data": "true", "ms_crc_header": "true", "ms_die_on_bad_msg": "false", "ms_die_on_bug": "false", "ms_die_on_old_message": "false", "ms_die_on_skipped_message": "false", "ms_die_on_unhandled_msg": "false", "ms_dispatch_throttle_bytes": "104857600", "ms_dpdk_coremask": "0xF", "ms_dpdk_debug_allow_loopback": "false", "ms_dpdk_gateway_ipv4_addr": "", "ms_dpdk_host_ipv4_addr": "", "ms_dpdk_hugepages": "", "ms_dpdk_hw_flow_control": "true", "ms_dpdk_hw_queue_weight": "1.000000", "ms_dpdk_lro": "true", "ms_dpdk_memory_channel": "4", "ms_dpdk_netmask_ipv4_addr": "", "ms_dpdk_pmd": "", "ms_dpdk_port_id": "0", "ms_dpdk_rx_buffer_count_per_core": "8192", "ms_dump_corrupt_message_level": "1", "ms_dump_on_send": "false", "ms_initial_backoff": "0.200000", "ms_inject_delay_max": "1.000000", "ms_inject_delay_msg_type": "", "ms_inject_delay_probability": "0.000000", "ms_inject_delay_type": "", "ms_inject_internal_delays": "0.000000", "ms_inject_socket_failures": "0", "ms_learn_addr_from_peer": "true", "ms_max_accept_failures": "4", "ms_max_backoff": "15.000000", "ms_mon_client_mode": "crc", "ms_mon_cluster_mode": "crc", "ms_mon_service_mode": "crc", "ms_msgr2_encrypt_messages": "false", "ms_msgr2_sign_messages": "false", "ms_pq_max_tokens_per_priority": "16777216", "ms_pq_min_cost": "65536", "ms_public_type": "", "ms_rwthread_stack_bytes": "1048576", "ms_service_mode": "crc", "ms_tcp_listen_backlog": "512", "ms_tcp_nodelay": "true", "ms_tcp_prefetch_max_size": "4096", "ms_tcp_rcvbuf": "0", "ms_type": "async+posix", "nss_db_path": "", "objecter_completion_locks_per_session": "32", "objecter_debug_inject_relock_delay": "false", "objecter_inflight_op_bytes": "104857600", "objecter_inflight_ops": "1024", "objecter_inject_no_watch_ping": "false", "objecter_retry_writes_after_first_reply": "false", "objecter_tick_interval": "5.000000", "objecter_timeout": "10.000000", "objectstore_blackhole": "false", "osd_agent_delay_time": "5.000000", "osd_agent_hist_halflife": "1000", "osd_agent_max_low_ops": "2", "osd_agent_max_ops": "4", "osd_agent_min_evict_effort": "0.100000", "osd_agent_quantize_effort": "0.100000", "osd_agent_slop": "0.020000", "osd_allow_recovery_below_min_size": "true", "osd_async_recovery_min_cost": "100", "osd_auto_mark_unfound_lost": "false", "osd_backfill_retry_interval": "30.000000", "osd_backfill_scan_max": "512", "osd_backfill_scan_min": "64", "osd_backoff_on_degraded": "false", "osd_backoff_on_peering": "false", "osd_backoff_on_unfound": "true", "osd_beacon_report_interval": "300", "osd_bench_duration": "30", "osd_bench_large_size_max_throughput": "104857600", "osd_bench_max_block_size": "67108864", "osd_bench_small_size_max_iops": "100", "osd_blkin_trace_all": "false", "osd_calc_pg_upmaps_aggressively": "true", "osd_calc_pg_upmaps_local_fallback_retries": "100", "osd_calc_pg_upmaps_max_stddev": "1.000000", "osd_check_for_log_corruption": "false", "osd_check_max_object_name_len_on_startup": "true", "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw timeindex user version cas", "osd_class_dir": "/usr/lib/rados- classes", "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw timeindex user version cas", "osd_class_update_on_start": "true", "osd_client_message_cap": "100", "osd_client_message_size_cap": "2147483648", "osd_client_op_priority": "63", "osd_client_watch_timeout": "30", "osd_command_max_records": "256", "osd_command_thread_suicide_timeout": "900", "osd_command_thread_timeout": "600", "osd_copyfrom_max_chunk": "8388608", "osd_crush_chooseleaf_type": "1", "osd_crush_initial_weight": "- 1.000000", "osd_crush_update_on_start": "true", "osd_crush_update_weight_set": "true", "osd_data": "/var/lib/ceph/osd/ceph-sbg2- k8s001", "osd_debug_crash_on_ignored_backoff": "false", "osd_debug_deep_scrub_sleep": "0.000000", "osd_debug_drop_ping_duration": "0", "osd_debug_drop_ping_probability": "0.000000", "osd_debug_feed_pullee": "- 1", "osd_debug_inject_copyfrom_error": "false", "osd_debug_inject_dispatch_delay_duration": "0.100000", "osd_debug_inject_dispatch_delay_probability": "0.000000", "osd_debug_misdirected_ops": "false", "osd_debug_no_acting_change": "false", "osd_debug_no_purge_strays": "false", "osd_debug_op_order": "false", "osd_debug_pg_log_writeout": "false", "osd_debug_random_push_read_error": "0.000000", "osd_debug_reject_backfill_probability": "0.000000", "osd_debug_shutdown": "false", "osd_debug_skip_full_check_in_backfill_reservation": "false", "osd_debug_skip_full_check_in_recovery": "false", "osd_debug_verify_cached_snaps": "false", "osd_debug_verify_missing_on_start": "false", "osd_debug_verify_snaps": "false", "osd_debug_verify_stray_on_activate": "false", "osd_deep_scrub_interval": "604800.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "2000000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.150000", "osd_deep_scrub_stride": "131072", "osd_deep_scrub_update_digest_min_age": "7200", "osd_default_data_pool_replay_window": "45", "osd_default_notify_timeout": "30", "osd_delete_sleep": "0.000000", "osd_delete_sleep_hdd": "5.000000", "osd_delete_sleep_hybrid": "2.000000", "osd_delete_sleep_ssd": "0.000000", "osd_discard_disconnected_ops": "true", "osd_enable_op_tracker": "true", "osd_erasure_code_plugins": "jerasure lrc isa", "osd_failsafe_full_ratio": "0.970000", "osd_fast_fail_on_connection_refused": "true", "osd_fast_info": "true", "osd_find_best_info_ignore_history_les": "false", "osd_force_auth_primary_missing_objects": "100", "osd_force_recovery_pg_log_entries_factor": "1.300000", "osd_function_tracing": "false", "osd_heartbeat_grace": "20", "osd_heartbeat_interval": "6", "osd_heartbeat_min_healthy_ratio": "0.330000", "osd_heartbeat_min_peers": "10", "osd_heartbeat_min_size": "2000", "osd_heartbeat_use_min_delay_socket": "false", "osd_hit_set_max_size": "100000", "osd_hit_set_min_size": "1000", "osd_hit_set_namespace": ".ceph- internal", "osd_ignore_stale_divergent_priors": "false", "osd_inject_bad_map_crc_probability": "0.000000", "osd_inject_failure_on_pg_removal": "false", "osd_journal": "/var/lib/ceph/osd/ceph-sbg2- k8s001/journal", "osd_journal_flush_on_shutdown": "true", "osd_journal_size": "5120", "osd_kill_backfill_at": "0", "osd_loop_before_reset_tphandle": "64", "osd_map_cache_size": "1024", "osd_map_dedup": "true", "osd_map_message_max": "40", "osd_map_message_max_bytes": "10485760", "osd_map_share_max_epochs": "40", "osd_max_attr_name_len": "100", "osd_max_attr_size": "0", "osd_max_backfills": "4", "osd_max_markdown_count": "5", "osd_max_markdown_period": "600", "osd_max_object_name_len": "2048", "osd_max_object_namespace_len": "256", "osd_max_object_size": "134217728", "osd_max_omap_bytes_per_request": "1073741824", "osd_max_omap_entries_per_request": "131072", "osd_max_pg_blocked_by": "16", "osd_max_pg_log_entries": "3000", "osd_max_pg_per_osd_hard_ratio": "3.000000", "osd_max_pgls": "1024", "osd_max_push_cost": "8388608", "osd_max_push_objects": "10", "osd_max_scrubs": "1", "osd_max_snap_prune_intervals_per_epoch": "512", "osd_max_trimming_pgs": "2", "osd_max_write_size": "512", "osd_memory_base": "805306368", "osd_memory_cache_min": "134217728", "osd_memory_cache_resize_interval": "1.000000", "osd_memory_expected_fragmentation": "0.150000", "osd_memory_target": "4294967296", "osd_memory_target_cgroup_limit_ratio": "0.800000", "osd_min_pg_log_entries": "3000", "osd_min_recovery_priority": "0", "osd_mon_ack_timeout": "30.000000", "osd_mon_heartbeat_interval": "30", "osd_mon_report_interval": "5", "osd_mon_report_max_in_flight": "2", "osd_mon_shutdown_timeout": "5.000000", "osd_num_op_tracker_shard": "32", "osd_numa_auto_affinity": "true", "osd_numa_node": "- 1", "osd_numa_prefer_iface": "true", "osd_objecter_finishers": "1", "osd_objectstore": "bluestore", "osd_objectstore_fuse": "false", "osd_objectstore_tracing": "false", "osd_op_complaint_time": "30.000000", "osd_op_history_duration": "600", "osd_op_history_size": "20", "osd_op_history_slow_op_size": "20", "osd_op_history_slow_op_threshold": "10.000000", "osd_op_log_threshold": "5", "osd_op_num_shards": "0", "osd_op_num_shards_hdd": "5", "osd_op_num_shards_ssd": "8", "osd_op_num_threads_per_shard": "0", "osd_op_num_threads_per_shard_hdd": "1", "osd_op_num_threads_per_shard_ssd": "2", "osd_op_pq_max_tokens_per_priority": "4194304", "osd_op_pq_min_cost": "65536", "osd_op_queue": "wpq", "osd_op_queue_cut_off": "low", "osd_op_queue_mclock_anticipation_timeout": "0.000000", "osd_op_queue_mclock_client_op_lim": "0.000000", "osd_op_queue_mclock_client_op_res": "1000.000000", "osd_op_queue_mclock_client_op_wgt": "500.000000", "osd_op_queue_mclock_osd_rep_op_lim": "0.000000", "osd_op_queue_mclock_osd_rep_op_res": "1000.000000", "osd_op_queue_mclock_osd_rep_op_wgt": "500.000000", "osd_op_queue_mclock_peering_event_lim": "0.001000", "osd_op_queue_mclock_peering_event_res": "0.000000", "osd_op_queue_mclock_peering_event_wgt": "1.000000", "osd_op_queue_mclock_pg_delete_lim": "0.001000", "osd_op_queue_mclock_pg_delete_res": "0.000000", "osd_op_queue_mclock_pg_delete_wgt": "1.000000", "osd_op_queue_mclock_recov_lim": "0.001000", "osd_op_queue_mclock_recov_res": "0.000000", "osd_op_queue_mclock_recov_wgt": "1.000000", "osd_op_queue_mclock_scrub_lim": "0.001000", "osd_op_queue_mclock_scrub_res": "0.000000", "osd_op_queue_mclock_scrub_wgt": "1.000000", "osd_op_queue_mclock_snap_lim": "0.001000", "osd_op_queue_mclock_snap_res": "0.000000", "osd_op_queue_mclock_snap_wgt": "1.000000", "osd_op_thread_suicide_timeout": "150", "osd_op_thread_timeout": "15", "osd_open_classes_on_start": "true", "osd_os_flags": "0", "osd_peering_op_priority": "255", "osd_pg_delete_cost": "1048576", "osd_pg_delete_priority": "5", "osd_pg_epoch_max_lag_factor": "2.000000", "osd_pg_epoch_persisted_max_stale": "40", "osd_pg_log_dups_tracked": "3000", "osd_pg_log_trim_max": "10000", "osd_pg_log_trim_min": "100", "osd_pg_max_concurrent_snap_trims": "2", "osd_pg_object_context_cache_count": "64", "osd_pg_stat_report_interval_max": "500", "osd_pool_default_cache_max_evict_check_size": "10", "osd_pool_default_cache_min_evict_age": "0", "osd_pool_default_cache_min_flush_age": "0", "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", "osd_pool_default_cache_target_dirty_ratio": "0.400000", "osd_pool_default_cache_target_full_ratio": "0.800000", "osd_pool_default_crush_rule": "- 1", "osd_pool_default_ec_fast_read": "false", "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1", "osd_pool_default_flag_hashpspool": "true", "osd_pool_default_flag_nodelete": "false", "osd_pool_default_flag_nopgchange": "false", "osd_pool_default_flag_nosizechange": "false", "osd_pool_default_flags": "0", "osd_pool_default_hit_set_bloom_fpp": "0.050000", "osd_pool_default_min_size": "2", "osd_pool_default_pg_autoscale_mode": "on", "osd_pool_default_pg_num": "8", "osd_pool_default_pgp_num": "0", "osd_pool_default_size": "3", "osd_pool_default_type": "replicated", "osd_pool_erasure_code_stripe_unit": "4096", "osd_pool_use_gmt_hitset": "true", "osd_push_per_object_cost": "1000", "osd_read_ec_check_for_errors": "false", "osd_recover_clone_overlap": "true", "osd_recover_clone_overlap_limit": "10", "osd_recovery_cost": "20971520", "osd_recovery_delay_start": "0.000000", "osd_recovery_max_active": "10", "osd_recovery_max_chunk": "8388608", "osd_recovery_max_omap_entries_per_chunk": "8096", "osd_recovery_max_single_start": "1", "osd_recovery_op_priority": "4", "osd_recovery_op_warn_multiple": "16", "osd_recovery_priority": "5", "osd_recovery_retry_interval": "30.000000", "osd_recovery_sleep": "0.000000", "osd_recovery_sleep_hdd": "0.100000", "osd_recovery_sleep_hybrid": "0.025000", "osd_recovery_sleep_ssd": "0.000000", "osd_requested_scrub_priority": "120", "osd_rollback_to_cluster_snap": "", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "0", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "24", "osd_scrub_end_week_day": "7", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "604800.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "86400.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.000000", "osd_shutdown_pgref_assert": "false", "osd_skip_data_digest": "false", "osd_smart_report_timeout": "5", "osd_snap_trim_cost": "1048576", "osd_snap_trim_priority": "5", "osd_snap_trim_sleep": "0.000000", "osd_stats_ack_timeout_decay": "0.900000", "osd_stats_ack_timeout_factor": "2.000000", "osd_target_transaction_size": "30", "osd_tier_default_cache_hit_set_count": "4", "osd_tier_default_cache_hit_set_grade_decay_rate": "20", "osd_tier_default_cache_hit_set_period": "1200", "osd_tier_default_cache_hit_set_search_last_n": "1", "osd_tier_default_cache_hit_set_type": "bloom", "osd_tier_default_cache_min_read_recency_for_promote": "1", "osd_tier_default_cache_min_write_recency_for_promote": "1", "osd_tier_default_cache_mode": "writeback", "osd_tier_promote_max_bytes_sec": "5242880", "osd_tier_promote_max_objects_sec": "25", "osd_tracing": "false", "osd_use_stale_snap": "false", "osd_uuid": "00000000-0000-0000-0000- 000000000000", "osd_verify_sparse_read_holes": "false", "osdc_blkin_trace_all": "false", "paxos_kill_at": "0", "paxos_max_join_drift": "10", "paxos_min": "500", "paxos_min_wait": "0.050000", "paxos_propose_interval": "1.000000", "paxos_service_trim_max": "500", "paxos_service_trim_min": "250", "paxos_stash_full_interval": "25", "paxos_trim_max": "500", "paxos_trim_min": "250", "perf": "true", "pid_file": "", "plugin_crypto_accelerator": "crypto_isal", "plugin_dir": "/usr/lib/ceph", "public_addr": "v2:192.168.211.12:0/0", "public_bind_addr": "- ", "public_network": "192.168.211.0/24", "public_network_interface": "", "qat_compressor_enabled": "false", "rados_mon_op_timeout": "0.000000", "rados_osd_op_timeout": "0.000000", "rados_tracing": "false", "rbd_atime_update_interval": "60", "rbd_auto_exclusive_lock_until_manual_request": "true", "rbd_balance_parent_reads": "false", "rbd_balance_snap_reads": "false", "rbd_blacklist_expire_seconds": "0", "rbd_blacklist_on_break_lock": "true", "rbd_blkin_trace_all": "false", "rbd_cache": "true", "rbd_cache_block_writes_upfront": "false", "rbd_cache_max_dirty": "25165824", "rbd_cache_max_dirty_age": "1.000000", "rbd_cache_max_dirty_object": "0", "rbd_cache_size": "33554432", "rbd_cache_target_dirty": "16777216", "rbd_cache_writethrough_until_flush": "true", "rbd_clone_copy_on_read": "false", "rbd_concurrent_management_ops": "10", "rbd_default_clone_format": "auto", "rbd_default_data_pool": "", "rbd_default_features": "61", "rbd_default_format": "2", "rbd_default_map_options": "", "rbd_default_order": "22", "rbd_default_pool": "rbd", "rbd_default_stripe_count": "0", "rbd_default_stripe_unit": "0", "rbd_discard_granularity_bytes": "65536", "rbd_discard_on_zeroed_write_same": "true", "rbd_enable_alloc_hint": "true", "rbd_journal_commit_age": "5.000000", "rbd_journal_max_concurrent_object_sets": "0", "rbd_journal_max_payload_bytes": "16384", "rbd_journal_object_flush_age": "0.000000", "rbd_journal_object_flush_bytes": "0", "rbd_journal_object_flush_interval": "0", "rbd_journal_object_max_in_flight_appends": "0", "rbd_journal_order": "24", "rbd_journal_pool": "", "rbd_journal_splay_width": "4", "rbd_localize_parent_reads": "false", "rbd_localize_snap_reads": "false", "rbd_mirror_concurrent_image_deletions": "1", "rbd_mirror_concurrent_image_syncs": "5", "rbd_mirror_delete_retry_interval": "30.000000", "rbd_mirror_image_policy_migration_throttle": "300", "rbd_mirror_image_policy_rebalance_timeout": "0.000000", "rbd_mirror_image_policy_type": "simple", "rbd_mirror_image_policy_update_throttle_interval": "1.000000", "rbd_mirror_image_state_check_interval": "30", "rbd_mirror_journal_commit_age": "5.000000", "rbd_mirror_journal_max_fetch_bytes": "32768", "rbd_mirror_journal_poll_age": "5.000000", "rbd_mirror_leader_heartbeat_interval": "5", "rbd_mirror_leader_max_acquire_attempts_before_break": "3", "rbd_mirror_leader_max_missed_heartbeats": "2", "rbd_mirror_perf_stats_prio": "5", "rbd_mirror_pool_replayers_refresh_interval": "30", "rbd_mirror_sync_point_update_age": "30.000000", "rbd_mirroring_delete_delay": "0", "rbd_mirroring_replay_delay": "0", "rbd_mirroring_resync_after_disconnect": "false", "rbd_move_to_trash_on_remove": "false", "rbd_move_to_trash_on_remove_expire_seconds": "0", "rbd_mtime_update_interval": "60", "rbd_non_blocking_aio": "true", "rbd_op_thread_timeout": "60", "rbd_op_threads": "1", "rbd_qos_bps_burst": "0", "rbd_qos_bps_limit": "0", "rbd_qos_iops_burst": "0", "rbd_qos_iops_limit": "0", "rbd_qos_read_bps_burst": "0", "rbd_qos_read_bps_limit": "0", "rbd_qos_read_iops_burst": "0", "rbd_qos_read_iops_limit": "0", "rbd_qos_schedule_tick_min": "50", "rbd_qos_write_bps_burst": "0", "rbd_qos_write_bps_limit": "0", "rbd_qos_write_iops_burst": "0", "rbd_qos_write_iops_limit": "0", "rbd_readahead_disable_after_bytes": "52428800", "rbd_readahead_max_bytes": "524288", "rbd_readahead_trigger_requests": "10", "rbd_request_timed_out_seconds": "30", "rbd_skip_partial_discard": "true", "rbd_sparse_read_threshold_bytes": "65536", "rbd_tracing": "false", "rbd_validate_names": "true", "rbd_validate_pool": "true", "restapi_base_url": "", "restapi_log_level": "", "rgw_acl_grants_max_num": "100", "rgw_admin_entry": "admin", "rgw_barbican_url": "", "rgw_bucket_default_quota_max_objects": "- 1", "rgw_bucket_default_quota_max_size": "- 1", "rgw_bucket_index_max_aio": "8", "rgw_bucket_quota_cache_size": "10000", "rgw_bucket_quota_soft_threshold": "0.950000", "rgw_bucket_quota_ttl": "600", "rgw_cache_enabled": "true", "rgw_cache_expiry_interval": "900", "rgw_cache_lru_size": "10000", "rgw_content_length_compat": "false", "rgw_copy_obj_progress": "true", "rgw_copy_obj_progress_every_bytes": "1048576", "rgw_cors_rules_max_num": "100", "rgw_cross_domain_policy": "<allow-access-from domain=\"*\" secure=\"false\" />", "rgw_crypt_default_encryption_key": "", "rgw_crypt_require_ssl": "true", "rgw_crypt_s3_kms_encryption_keys": "", "rgw_crypt_suppress_logs": "true", "rgw_curl_low_speed_limit": "1024", "rgw_curl_low_speed_time": "300", "rgw_curl_wait_timeout_ms": "1000", "rgw_data": "/var/lib/ceph/radosgw/ceph-sbg2- k8s001", "rgw_data_log_changes_size": "1000", "rgw_data_log_num_shards": "128", "rgw_data_log_obj_prefix": "data_log", "rgw_data_log_window": "30", "rgw_data_notify_interval_msec": "200", "rgw_default_realm_info_oid": "default.realm", "rgw_default_region_info_oid": "default.region", "rgw_default_zone_info_oid": "default.zone", "rgw_default_zonegroup_info_oid": "default.zonegroup", "rgw_defer_to_bucket_acls": "", "rgw_delete_multi_obj_max_num": "1000", "rgw_dmclock_admin_lim": "0.000000", "rgw_dmclock_admin_res": "100.000000", "rgw_dmclock_admin_wgt": "100.000000", "rgw_dmclock_auth_lim": "0.000000", "rgw_dmclock_auth_res": "200.000000", "rgw_dmclock_auth_wgt": "100.000000", "rgw_dmclock_data_lim": "0.000000", "rgw_dmclock_data_res": "500.000000", "rgw_dmclock_data_wgt": "500.000000", "rgw_dmclock_metadata_lim": "0.000000", "rgw_dmclock_metadata_res": "500.000000", "rgw_dmclock_metadata_wgt": "500.000000", "rgw_dns_name": "", "rgw_dns_s3website_name": "", "rgw_dynamic_resharding": "true", "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts", "rgw_enable_gc_threads": "true", "rgw_enable_lc_threads": "true", "rgw_enable_ops_log": "false", "rgw_enable_quota_threads": "true", "rgw_enable_static_website": "false", "rgw_enable_usage_log": "false", "rgw_enforce_swift_acls": "true", "rgw_exit_timeout_secs": "120", "rgw_expose_bucket": "false", "rgw_extended_http_attrs": "", "rgw_fcgi_socket_backlog": "1024", "rgw_frontends": "beast port=7480", "rgw_gc_max_concurrent_io": "10", "rgw_gc_max_objs": "32", "rgw_gc_max_trim_chunk": "16", "rgw_gc_obj_min_wait": "7200", "rgw_gc_processor_max_time": "3600", "rgw_gc_processor_period": "3600", "rgw_get_obj_max_req_size": "4194304", "rgw_get_obj_window_size": "16777216", "rgw_healthcheck_disabling_path": "", "rgw_host": "", "rgw_ignore_get_invalid_range": "false", "rgw_init_timeout": "300", "rgw_inject_notify_timeout_probability": "0.000000", "rgw_keystone_accepted_admin_roles": "", "rgw_keystone_accepted_roles": "Member, admin", "rgw_keystone_admin_domain": "", "rgw_keystone_admin_password": "", "rgw_keystone_admin_password_path": "", "rgw_keystone_admin_project": "", "rgw_keystone_admin_tenant": "", "rgw_keystone_admin_token": "", "rgw_keystone_admin_token_path": "", "rgw_keystone_admin_user": "", "rgw_keystone_api_version": "2", "rgw_keystone_barbican_domain": "", "rgw_keystone_barbican_password": "", "rgw_keystone_barbican_project": "", "rgw_keystone_barbican_tenant": "", "rgw_keystone_barbican_user": "", "rgw_keystone_implicit_tenants": "false", "rgw_keystone_revocation_interval": "900", "rgw_keystone_token_cache_size": "10000", "rgw_keystone_url": "", "rgw_keystone_verify_ssl": "true", "rgw_lc_debug_interval": "-1", "rgw_lc_lock_max_time": "60", "rgw_lc_max_objs": "32", "rgw_lc_max_rules": "1000", "rgw_lc_thread_delay": "0", "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", "rgw_ldap_dnattr": "uid", "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", "rgw_ldap_searchfilter": "", "rgw_ldap_secret": "/etc/openldap/secret", "rgw_ldap_uri": "ldaps://<ldap.your.domain>", "rgw_lifecycle_work_time": "00:00- 06:00", "rgw_list_bucket_min_readahead": "1000", "rgw_list_buckets_max_chunk": "1000", "rgw_log_http_headers": "", "rgw_log_nonexistent_bucket": "false", "rgw_log_object_name": "%Y-%m-%d-%H-%i- %n", "rgw_log_object_name_utc": "false", "rgw_max_attr_name_len": "0", "rgw_max_attr_size": "0", "rgw_max_attrs_num_in_req": "0", "rgw_max_chunk_size": "4194304", "rgw_max_concurrent_requests": "1024", "rgw_max_listing_results": "1000", "rgw_max_notify_retries": "3", "rgw_max_objs_per_shard": "100000", "rgw_max_put_param_size": "1048576", "rgw_max_put_size": "5368709120", "rgw_max_slo_entries": "1000", "rgw_md_log_max_shards": "64", "rgw_md_notify_interval_msec": "200", "rgw_mime_types_file": "/etc/mime.types", "rgw_mp_lock_max_time": "600", "rgw_multipart_min_part_size": "5242880", "rgw_multipart_part_upload_limit": "10000", "rgw_nfs_fhcache_partitions": "3", "rgw_nfs_fhcache_size": "2017", "rgw_nfs_lru_lane_hiwat": "911", "rgw_nfs_lru_lanes": "5", "rgw_nfs_max_gc": "300", "rgw_nfs_namespace_expire_secs": "300", "rgw_nfs_write_completion_interval_s": "10", "rgw_num_async_rados_threads": "32", "rgw_num_control_oids": "8", "rgw_num_rados_handles": "1", "rgw_obj_stripe_size": "4194304", "rgw_obj_tombstone_cache_size": "1000", "rgw_objexp_chunk_size": "100", "rgw_objexp_gc_interval": "600", "rgw_objexp_hints_num_shards": "127", "rgw_olh_pending_timeout_sec": "3600", "rgw_op_thread_suicide_timeout": "0", "rgw_op_thread_timeout": "600", "rgw_op_tracing": "false", "rgw_opa_token": "", "rgw_opa_url": "", "rgw_opa_verify_ssl": "true", "rgw_ops_log_data_backlog": "5242880", "rgw_ops_log_rados": "true", "rgw_ops_log_socket_path": "", "rgw_override_bucket_index_max_shards": "0", "rgw_period_latest_epoch_info_oid": ".latest_epoch", "rgw_period_push_interval": "2.000000", "rgw_period_push_interval_max": "30.000000", "rgw_period_root_pool": ".rgw.root", "rgw_port": "", "rgw_print_continue": "true", "rgw_print_prohibited_content_length": "false", "rgw_put_obj_max_window_size": "67108864", "rgw_put_obj_min_window_size": "16777216", "rgw_rados_tracing": "false", "rgw_realm": "", "rgw_realm_root_pool": ".rgw.root", "rgw_region": "", "rgw_region_root_pool": ".rgw.root", "rgw_relaxed_region_enforcement": "false", "rgw_relaxed_s3_bucket_names": "false", "rgw_remote_addr_param": "REMOTE_ADDR", "rgw_request_uri": "", "rgw_reshard_batch_size": "64", "rgw_reshard_bucket_lock_duration": "360", "rgw_reshard_max_aio": "128", "rgw_reshard_num_logs": "16", "rgw_reshard_thread_interval": "600", "rgw_resolve_cname": "false", "rgw_rest_getusage_op_compat": "false", "rgw_run_sync_thread": "true", "rgw_s3_auth_order": "sts, external, local", "rgw_s3_auth_use_keystone": "false", "rgw_s3_auth_use_ldap": "false", "rgw_s3_auth_use_rados": "true", "rgw_s3_auth_use_sts": "false", "rgw_s3_success_create_obj_status": "0", "rgw_safe_max_objects_per_shard": "102400", "rgw_scheduler_type": "throttler", "rgw_script_uri": "", "rgw_service_provider_name": "", "rgw_shard_warning_threshold": "90.000000", "rgw_socket_path": "", "rgw_sts_client_id": "", "rgw_sts_client_secret": "", "rgw_sts_entry": "sts", "rgw_sts_key": "sts", "rgw_sts_max_session_duration": "43200", "rgw_sts_token_introspection_url": "", "rgw_swift_account_in_url": "false", "rgw_swift_auth_entry": "auth", "rgw_swift_auth_url": "", "rgw_swift_custom_header": "", "rgw_swift_enforce_content_length": "false", "rgw_swift_need_stats": "true", "rgw_swift_tenant_name": "", "rgw_swift_token_expiration": "86400", "rgw_swift_url": "", "rgw_swift_url_prefix": "swift", "rgw_swift_versioning_enabled": "false", "rgw_sync_data_inject_err_probability": "0.000000", "rgw_sync_lease_period": "120", "rgw_sync_log_trim_concurrent_buckets": "4", "rgw_sync_log_trim_interval": "1200", "rgw_sync_log_trim_max_buckets": "16", "rgw_sync_log_trim_min_cold_buckets": "4", "rgw_sync_meta_inject_err_probability": "0.000000", "rgw_sync_trace_history_size": "4096", "rgw_sync_trace_per_node_log_size": "32", "rgw_sync_trace_servicemap_update_interval": "10", "rgw_thread_pool_size": "512", "rgw_torrent_comment": "", "rgw_torrent_createby": "", "rgw_torrent_encoding": "", "rgw_torrent_flag": "false", "rgw_torrent_origin": "", "rgw_torrent_sha_unit": "524288", "rgw_torrent_tracker": "", "rgw_trust_forwarded_https": "false", "rgw_usage_log_flush_threshold": "1024", "rgw_usage_log_tick_interval": "30", "rgw_usage_max_shards": "32", "rgw_usage_max_user_shards": "1", "rgw_use_opa_authz": "false", "rgw_user_default_quota_max_objects": "- 1", "rgw_user_default_quota_max_size": "- 1", "rgw_user_max_buckets": "1000", "rgw_user_quota_bucket_sync_interval": "180", "rgw_user_quota_sync_idle_users": "false", "rgw_user_quota_sync_interval": "86400", "rgw_user_quota_sync_wait_time": "86400", "rgw_verify_ssl": "true", "rgw_website_routing_rules_max_num": "50", "rgw_zone": "", "rgw_zone_root_pool": ".rgw.root", "rgw_zonegroup": "", "rgw_zonegroup_root_pool": ".rgw.root", "rocksdb_block_size": "4096", "rocksdb_bloom_bits_per_key": "20", "rocksdb_cache_index_and_filter_blocks": "true", "rocksdb_cache_index_and_filter_blocks_with_high_priority": "true", "rocksdb_cache_row_ratio": "0.000000", "rocksdb_cache_shard_bits": "4", "rocksdb_cache_size": "536870912", "rocksdb_cache_type": "binned_lru", "rocksdb_collect_compaction_stats": "false", "rocksdb_collect_extended_stats": "false", "rocksdb_collect_memory_stats": "false", "rocksdb_enable_rmrange": "false", "rocksdb_index_type": "binary_search", "rocksdb_log_to_ceph_log": "true", "rocksdb_metadata_block_size": "4096", "rocksdb_partition_filters": "false", "rocksdb_perf": "false", "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "true", "rotating_keys_bootstrap_timeout": "30", "rotating_keys_renewal_timeout": "10", "run_dir": "/var/run/ceph", "setgroup": "ceph", "setuser": "ceph", "setuser_match_path": "", "target_max_misplaced_ratio": "0.050000", "threadpool_default_timeout": "60", "threadpool_empty_queue_max_wait": "2", "throttler_perf_counter": "true", "xio_max_conns_per_portal": "32", "xio_max_send_inline": "512", "xio_mp_max_1k": "8192", "xio_mp_max_256": "8192", "xio_mp_max_64": "65536", "xio_mp_max_hint": "4096", "xio_mp_max_page": "4096", "xio_mp_min": "128", "xio_portal_threads": "2", "xio_queue_depth": "128", "xio_trace_mempool": "false", "xio_trace_msgcnt": "false", "xio_trace_xcon": "false", "xio_transport_type": "rdma"}

2 1

deprecating inline_data support for CephFS
by Jeff Layton 16 Aug '19

16 Aug '19

A couple of weeks ago, I sent a request to the mailing list asking whether anyone was using the inline_data support in cephfs: https://docs.ceph.com/docs/mimic/cephfs/experimental-features/#inline-data I got exactly zero responses, so I'm going to formally propose that we move to start deprecating this feature for Octopus. Why deprecate this feature? =========================== While the userland clients have support for both reading and writing, the kernel only has support for reading, and aggressively uninlines everything as soon as it needs to do any writing. That uninlining has some rather nasty potential race conditions too that could cause data corruption. We could work to fix this, and maybe add write support for the kernel, but it adds a lot of complexity to the read and write codepaths in the clients, which are already pretty complex. Given that there isn't a lot of interest in this feature, I think we ought to just pull the plug on it. How should we do this? ====================== We should start by disabling this feature in master for Octopus. In particular, we should stop allowing users to call "fs set inline_data true" on filesystems where it's disabled, and maybe throw a loud warning about the feature being deprecated if the mds is started on a filesystem that has it enabled. We could also consider creating a utility to crawl an existing filesystem and uninline anything there, if there was need for it. Then, in a few release cycles, once we're past the point where someone can upgrade directly from Nautilus (release Q or R?) we'd rip out support for this feature entirely. Thoughts, comments, questions welcome. -- Jeff Layton <jlayton(a)redhat.com>

2 2

Small HDD cluster, switch from Bluestore to Filestore
by Richard Bade 16 Aug '19

16 Aug '19

Hi Everyone, There's been a few threads around about small HDD (spinning disk) clusters and performance on Bluestore. One recently from Christian (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036385.html) was particularly interesting to us as we have a very similar setup to what Christian has and we see similar performance. We have a 6 node cluster each with 12x 4TB SATA HDD, IT mode LSI 3008, wal/db on 33GB NVMe partitions. Each node has a single Xeon Gold 6132 CPU @ 2.60GHz and dual 10GB network. We also use bcache with 1 180GB NVMe partition shared between 6 osd's. Workload is via KVM (Proxmox) I did the same benchmark fio tests as Christian. Here's my results (M for me, C for Christian) direct=0 ======== M -- read : io=6008.0MB, bw=203264KB/s, iops=49, runt= 30267msec C -- read: IOPS=40, BW=163MiB/s (171MB/s)(7556MiB/46320msec) direct=1 ======== M -- read : io=32768MB, bw=1991.4MB/s, iops=497, runt= 16455ms C -- read: IOPS=314, BW=1257MiB/s (1318MB/s)(32.0GiB/26063msec) direct=0 ======== M -- write: io=32768MB, bw=471105KB/s, iops=115, runt= 71225msec C -- write: IOPS=119, BW=479MiB/s (503MB/s)(32.0GiB/68348msec direct=1 ======== M -- write: io=32768MB, bw=479829KB/s, iops=117, runt= 69930msec C -- write: IOPS=139, BW=560MiB/s (587MB/s)(32.0GiB/58519msec) I should probably mention that there was some active workload on the cluster at that time also, around 500iops write and 100MB/s throughput. The main problem that we're having with this cluster is how easy it is for it to hit slow requests and we have one particular vm that ends up doing scsi resets because of the latency. So we're considering switching these osd's to filestore. We have two other clusters using filestore/bcache/ssd journal and the performance seems to be much better on those - taking into account the different sizes. What are peoples thoughts on this size cluster? Is it just not a good fit with bluestore and our type of workload? Also, does anyone have any knowledge on future support for filestore? I'm concerned that we may have to migrate our other clusters off filestore sometime in the future and that'll hurt us with the current performance. Rich

5 7

Re: Request to guide on ceph-deploy install command for luminuous 12.2.12 release
by Götz Reinicke 16 Aug '19

16 Aug '19

Hi, > Am 07.08.2019 um 20:20 schrieb Nerurkar, Ruchir (Nokia - US/Mountain View) <ruchir.nerurkar(a)nokia.com>: > > Hello, > > I work in Nokia as a Software QA engineer and I am trying to install Ceph on centOS 7.4 version. Do you have to stick at 7.4? > But I am getting this error with the following output: - > > vmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph) > [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-osd-12.2.12-0.el7.x86_64 (Ceph) > [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph) > [mvmgoscephcontrollerluminous][WARNIN] Requires: gperftools-libs >= 2.6.1 > [mvmgoscephcontrollerluminous][WARNIN] Available: gperftools-libs-2.4-8.el7.i686 (base) > [mvmgoscephcontrollerluminous][DEBUG ] You could try using --skip-broken to work around the problem > [mvmgoscephcontrollerluminous][WARNIN] gperftools-libs = 2.4-8.el7 > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-mon-12.2.12-0.el7.x86_64 (Ceph) > [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) > [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph) > [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) > [mvmgoscephcontrollerluminous][DEBUG ] You could try running: rpm -Va --nofiles --nodigest > [mvmgoscephcontrollerluminous][ERROR ] RuntimeError: command returned non-zero exit status: 1 > [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y install ceph ceph-radosgw > > Does anyone know about this issue like how can I resolve package dependencies? My first shot: Update Centos to min. 7.5 (which includes gperftools-libs-2.6.1) And install lz4 too. Regards . Götz

2 1

Upgrad luminous -> mimic , any pointers?
by Marc Roos 15 Aug '19

15 Aug '19

I have a fairly dormant ceph luminous cluster on centos7 with stock kernel, and thought about upgrading it before putting it to more use. I can remember some page on the ceph website that had specific instructions mentioning upgrading from luminous. But I can't find it anymore, this page[0] even says I am viewing an unsupported version of ceph?? Any pointers aside from a default upgrade procedure? - I am using rbd, rgw and cephfs (only one mds) - Still using direct block devices (not lvm) - have one 1 pg active+clean+inconsistent since a year or so, that I am getting sentimentally attached to ;) [0] https://docs.ceph.com/docs/mimic/install/upgrading-ceph/#

1 1

Re: Upgrad luminous -> mimic , any pointers?
by Marc Roos 15 Aug '19

15 Aug '19

Pfff, you are right, I don't even know which one is the newest latest, indeed Nautilus -----Original Message----- Subject: Re: [ceph-users] Upgrad luminous -> mimic , any pointers? Why would you go to Mimic instead of Nautilus? > > > > I have a fairly dormant ceph luminous cluster on centos7 with stock > kernel, and thought about upgrading it before putting it to more use. > I can remember some page on the ceph website that had specific > instructions mentioning upgrading from luminous. But I can't find it > anymore, this page[0] even says I am viewing an unsupported version of > ceph?? > > Any pointers aside from a default upgrade procedure? > - I am using rbd, rgw and cephfs (only one mds) > - Still using direct block devices (not lvm) > - have one 1 pg active+clean+inconsistent since a year or so, that I > am getting sentimentally attached to ;) > > > [0] > https://docs.ceph.com/docs/mimic/install/upgrading-ceph/# > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Mgr stability
by Reed Dier 15 Aug '19

15 Aug '19

Hi all, I'm experiencing some stability issues with my ceph-mgr daemons post Nautilus upgrade (13.2.6->14.2.2). On top of Ubuntu 18.04, the mgr's are colocated with the mon's, and 2 of the 3 mon/mgr nodes also colocate an MDS, running a single rank active:standby-replay. The main issue(s) I am seeing are that I will have mgr's silently die, to where ceph -s will report "no active mgr", despite none of the mgr daemons crashing or dying. Typically they end up coming back up/in after some amount of time, but metrics stall, and mgr commands will stall/hang. > $ ceph mgr module ls > { > "enabled_modules": [ > "diskprediction_local", > "influx", > "iostat", > "prometheus", > "rbd_support", > "restful", > "telemetry" > ], I ran the mgr with debug=20 and was unable to find anything terribly useful there, but can try to reproduce things if that seems helpful. The only real non-standard bits of my mgr instances is that I cherry-picked this change <https://github.com/ceph/ceph/pull/28427/files#diff-5add36422933f238d3aaf529…> to module.py to cure an issue with the influx plugin. Typically these stalls happen when the mgr is servicing a command, and lately that command has been the balancer running. For example: > $ time ceph balancer eval $POOL --debug-ms 1 > 2019-08-14 11:36:21.370 7f4990601700 1 Processor -- start > 2019-08-14 11:36:21.370 7f4990601700 1 -- start start > 2019-08-14 11:36:21.370 7f4990601700 1 --2- >> v2:10.0.1.250:3300/0 conn(0x7f4988168a80 0x7f4988169ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect > 2019-08-14 11:36:21.370 7f4990601700 1 --2- >> v2:10.0.1.249:3300/0 conn(0x7f498816f6a0 0x7f4988171b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect > 2019-08-14 11:36:21.370 7f4990601700 1 -- --> v1:10.0.1.250:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498808bbb0 con 0x7f49881684c0 > 2019-08-14 11:36:21.370 7f4990601700 1 -- --> v2:10.0.1.249:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498808ce00 con 0x7f498816f6a0 > 2019-08-14 11:36:21.370 7f4990601700 1 -- --> v2:10.0.1.250:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498808cfb0 con 0x7f4988168a80 > 2019-08-14 11:36:21.370 7f498dbc6700 1 --2- >> v2:10.0.1.250:3300/0 conn(0x7f4988168a80 0x7f4988169ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > 2019-08-14 11:36:21.370 7f498dbc6700 1 --2- >> v2:10.0.1.250:3300/0 conn(0x7f4988168a80 0x7f4988169ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:10.0.1.250:3300/0 says I am v2:10.0.1.250:44952/0 (socket says 10.0.1.250:44952) > 2019-08-14 11:36:21.370 7f498dbc6700 1 -- 10.0.1.250:0/2021657657 learned_addr learned my addr 10.0.1.250:0/2021657657 (peer_addr_for_me v2:10.0.1.250:0/0) > 2019-08-14 11:36:21.370 7f498ebc8700 1 --2- 10.0.1.250:0/2021657657 >> v2:10.0.1.249:3300/0 conn(0x7f498816f6a0 0x7f4988171b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > 2019-08-14 11:36:21.370 7f498dbc6700 1 -- 10.0.1.250:0/2021657657 >> v2:10.0.1.249:3300/0 conn(0x7f498816f6a0 msgr2=0x7f4988171b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down > 2019-08-14 11:36:21.370 7f498dbc6700 1 --2- 10.0.1.250:0/2021657657 >> v2:10.0.1.249:3300/0 conn(0x7f498816f6a0 0x7f4988171b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop > 2019-08-14 11:36:21.370 7f498dbc6700 1 -- 10.0.1.250:0/2021657657 >> v1:10.0.1.250:6789/0 conn(0x7f49881684c0 legacy=0x7f49881692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.370 7f498dbc6700 1 -- 10.0.1.250:0/2021657657 --> v2:10.0.1.250:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498808d1b0 con 0x7f4988168a80 > 2019-08-14 11:36:21.370 7f498dbc6700 1 --2- 10.0.1.250:0/2021657657 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f4988168a80 0x7f4988169ae0 crc :-1 s=READY pgs=82826 cs=0 l=1 rx=0 tx=0).ready entity=mon.1 client_cookie=10837f5a102e309b server_cookie=0 in_seq=0 out_seq=0 > 2019-08-14 11:36:21.370 7f498d3c5700 1 -- 10.0.1.250:0/2021657657 <== mon.1 v2:10.0.1.250:3300/0 1 ==== mon_map magic: 0 v1 ==== 395+0+0 (crc 0 0 0) 0x7f497c0020d0 con 0x7f4988168a80 > 2019-08-14 11:36:21.370 7f498d3c5700 1 -- 10.0.1.250:0/2021657657 <== mon.1 v2:10.0.1.250:3300/0 2 ==== config(6 keys) v1 ==== 207+0+0 (crc 0 0 0) 0x7f497c002410 con 0x7f4988168a80 > 2019-08-14 11:36:21.370 7f498d3c5700 1 -- 10.0.1.250:0/2021657657 <== mon.1 v2:10.0.1.250:3300/0 3 ==== mon_map magic: 0 v1 ==== 395+0+0 (crc 0 0 0) 0x7f497c002b40 con 0x7f4988168a80 > 2019-08-14 11:36:21.370 7f4990601700 1 -- 10.0.1.250:0/2021657657 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f4988168a80 msgr2=0x7f4988169ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.370 7f4990601700 1 --2- 10.0.1.250:0/2021657657 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f4988168a80 0x7f4988169ae0 crc :-1 s=READY pgs=82826 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:36:21.370 7f4990601700 1 -- 10.0.1.250:0/2021657657 shutdown_connections > 2019-08-14 11:36:21.370 7f4990601700 1 --2- 10.0.1.250:0/2021657657 >> v2:10.0.1.249:3300/0 conn(0x7f498816f6a0 0x7f4988171b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop > 2019-08-14 11:36:21.370 7f4990601700 1 --2- 10.0.1.250:0/2021657657 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f4988168a80 0x7f4988169ae0 unknown :-1 s=CLOSED pgs=82826 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:36:21.370 7f4990601700 1 -- 10.0.1.250:0/2021657657 shutdown_connections > 2019-08-14 11:36:21.370 7f4990601700 1 -- 10.0.1.250:0/2021657657 wait complete. > 2019-08-14 11:36:21.370 7f4990601700 1 -- 10.0.1.250:0/2021657657 >> 10.0.1.250:0/2021657657 conn(0x7f4988163a40 msgr2=0x7f4988165ed0 unknown :-1 s=STATE_NONE l=0).mark_down > 2019-08-14 11:36:21.374 7f4990601700 1 Processor -- start > 2019-08-14 11:36:21.374 7f4990601700 1 -- start start > 2019-08-14 11:36:21.374 7f4990601700 1 -- --> v1:10.0.1.249:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880b3f30 con 0x7f49880a8360 > 2019-08-14 11:36:21.374 7f4990601700 1 -- --> v1:10.0.1.250:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880b4170 con 0x7f49880b0030 > 2019-08-14 11:36:21.374 7f4990601700 1 -- --> v1:10.0.1.251:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880b4420 con 0x7f49880a7da0 > 2019-08-14 11:36:21.374 7f498ebc8700 1 --1- >> v1:10.0.1.250:6789/0 conn(0x7f49880b0030 0x7f49880b2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:10.0.1.250:6789/0 says I am v1:10.0.1.250:57466/0 (socket says 10.0.1.250:57466) > 2019-08-14 11:36:21.374 7f498ebc8700 1 -- 10.0.1.250:0/40211647 learned_addr learned my addr 10.0.1.250:0/40211647 (peer_addr_for_me v1:10.0.1.250:0/0) > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.0 v1:10.0.1.249:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197104487 0 0) 0x7f497c007c60 con 0x7f49880a8360 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 --> v1:10.0.1.249:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c002d30 con 0x7f49880a8360 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.0 v1:10.0.1.249:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1101082921 0 0) 0x7f497c007f40 con 0x7f49880a8360 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 --> v1:10.0.1.249:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f496c003a30 con 0x7f49880a8360 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.0 v1:10.0.1.249:6789/0 3 ==== mon_map magic: 0 v1 ==== 395+0+0 (unknown 1022418838 0 0) 0x7f497c0024e0 con 0x7f49880a8360 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 >> v1:10.0.1.251:6789/0 conn(0x7f49880a7da0 legacy=0x7f49880a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 >> v1:10.0.1.250:6789/0 conn(0x7f49880b0030 legacy=0x7f49880b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 >> v1:10.0.1.249:6789/0 conn(0x7f49880a8360 legacy=0x7f49880a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.374 7f497b7fe700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.251:3300/0,v1:10.0.1.251:6789/0] conn(0x7f496c003ed0 0x7f496c0049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect > 2019-08-14 11:36:21.374 7f497b7fe700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f496c004f10 0x7f496c005360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect > 2019-08-14 11:36:21.374 7f497b7fe700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.249:3300/0,v1:10.0.1.249:6789/0] conn(0x7f496c009b80 0x7f496c00bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.249:3300/0,v1:10.0.1.249:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c00c530 con 0x7f496c009b80 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c00c6e0 con 0x7f496c004f10 > 2019-08-14 11:36:21.374 7f497b7fe700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.251:3300/0,v1:10.0.1.251:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c00c890 con 0x7f496c003ed0 > 2019-08-14 11:36:21.374 7f498dbc6700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f496c004f10 0x7f496c005360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > 2019-08-14 11:36:21.374 7f498e3c7700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.251:3300/0,v1:10.0.1.251:6789/0] conn(0x7f496c003ed0 0x7f496c0049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > 2019-08-14 11:36:21.374 7f498ebc8700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.249:3300/0,v1:10.0.1.249:6789/0] conn(0x7f496c009b80 0x7f496c00bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > 2019-08-14 11:36:21.378 7f498dbc6700 1 -- 10.0.1.250:0/40211647 >> [v2:10.0.1.251:3300/0,v1:10.0.1.251:6789/0] conn(0x7f496c003ed0 msgr2=0x7f496c0049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.378 7f498dbc6700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.251:3300/0,v1:10.0.1.251:6789/0] conn(0x7f496c003ed0 0x7f496c0049c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:36:21.378 7f498dbc6700 1 -- 10.0.1.250:0/40211647 >> [v2:10.0.1.249:3300/0,v1:10.0.1.249:6789/0] conn(0x7f496c009b80 msgr2=0x7f496c00bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:36:21.378 7f498dbc6700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.249:3300/0,v1:10.0.1.249:6789/0] conn(0x7f496c009b80 0x7f496c00bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:36:21.378 7f498dbc6700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c008180 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f4990601700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49880b47e0 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f498ebc8700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.249:3300/0,v1:10.0.1.249:6789/0] conn(0x7f496c009b80 0x7f496c00bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! > 2019-08-14 11:36:21.378 7f498ebc8700 1 -- 10.0.1.250:0/40211647 reap_dead start > 2019-08-14 11:36:21.378 7f4990601700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49880a8360 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f498dbc6700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f496c004f10 0x7f496c005360 crc :-1 s=READY pgs=82828 cs=0 l=1 rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 > 2019-08-14 11:36:21.378 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.1 v2:10.0.1.250:3300/0 1 ==== mon_map magic: 0 v1 ==== 395+0+0 (crc 0 0 0) 0x7f49880b47e0 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.1 v2:10.0.1.250:3300/0 2 ==== config(6 keys) v1 ==== 207+0+0 (crc 0 0 0) 0x7f497c0010a0 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.1 v2:10.0.1.250:3300/0 3 ==== mon_map magic: 0 v1 ==== 395+0+0 (crc 0 0 0) 0x7f497c001490 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.1 v2:10.0.1.250:3300/0 4 ==== mgrmap(e 2536) v1 ==== 28602+0+0 (crc 0 0 0) 0x7f497c001690 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f497b7fe700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] conn(0x7f496c028790 0x7f496c02ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect > 2019-08-14 11:36:21.378 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.1 v2:10.0.1.250:3300/0 5 ==== osd_map(172849..172849 src has 171976..172849) v4 ==== 106776+0+0 (crc 0 0 0) 0x7f497c0489d0 con 0x7f496c004f10 > 2019-08-14 11:36:21.378 7f498e3c7700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] conn(0x7f496c028790 0x7f496c02ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > 2019-08-14 11:36:21.378 7f498e3c7700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] conn(0x7f496c028790 0x7f496c02ace0 crc :-1 s=READY pgs=16527 cs=0 l=1 rx=0 tx=0).ready entity=mgr.1731601147 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 > 2019-08-14 11:36:21.382 7f4990601700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49680031e0 con 0x7f496c004f10 > 2019-08-14 11:36:21.390 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mon.1 v2:10.0.1.250:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+101614 (crc 0 0 0) 0x7f497c000e90 con 0x7f496c004f10 > 2019-08-14 11:36:21.586 7f4990601700 1 -- 10.0.1.250:0/40211647 --> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] -- command(tid 0: {"prefix": "balancer eval", "option": "$POOL", "target": ["mgr", ""]}) v1 -- 0x7f4968002150 con 0x7f496c028790 > 2019-08-14 11:37:40.274 7f497b7fe700 1 -- 10.0.1.250:0/40211647 <== mgr.1731601147 v2:10.0.1.250:6800/34061 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+47 (crc 0 0 0) 0x7f4968002150 con 0x7f496c028790 > pool "$POOL" score 0.009701 (lower is better) > 2019-08-14 11:37:40.306 7f4990601700 1 -- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] conn(0x7f496c028790 msgr2=0x7f496c02ace0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:37:40.306 7f4990601700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] conn(0x7f496c028790 0x7f496c02ace0 crc :-1 s=READY pgs=16527 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:37:40.306 7f4990601700 1 -- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f496c004f10 msgr2=0x7f496c005360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down > 2019-08-14 11:37:40.306 7f4990601700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f496c004f10 0x7f496c005360 crc :-1 s=READY pgs=82828 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:37:40.306 7f4990601700 1 -- 10.0.1.250:0/40211647 shutdown_connections > 2019-08-14 11:37:40.306 7f4990601700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:6800/34061,v1:10.0.1.250:6801/34061] conn(0x7f496c028790 0x7f496c02ace0 unknown :-1 s=CLOSED pgs=16527 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:37:40.306 7f4990601700 1 --2- 10.0.1.250:0/40211647 >> [v2:10.0.1.250:3300/0,v1:10.0.1.250:6789/0] conn(0x7f496c004f10 0x7f496c005360 unknown :-1 s=CLOSED pgs=82828 cs=0 l=1 rx=0 tx=0).stop > 2019-08-14 11:37:40.306 7f4990601700 1 -- 10.0.1.250:0/40211647 shutdown_connections > 2019-08-14 11:37:40.306 7f4990601700 1 -- 10.0.1.250:0/40211647 wait complete. > 2019-08-14 11:37:40.306 7f4990601700 1 -- 10.0.1.250:0/40211647 >> 10.0.1.250:0/40211647 conn(0x7f498809d440 msgr2=0x7f498809d8b0 unknown :-1 s=STATE_NONE l=0).mark_down > > real 1m19.023s > user 0m0.288s > sys 0m0.050s While the mgr didn't die, it did take 80 seconds to eval a pool, which typically takes roughly 2-5 seconds to run. And while this command was running, the ceph-mgr processes was sitting around 100% in top. > 2019-08-14 11:36:20.734 7fbd77570700 0 ms_deliver_dispatch: unhandled message 0x55d96ee809c0 mgrreport(mds.db +110-0 packed 1366) v7 from mds.1717053370 v2:10.0.1.251:6800/140458953 > 2019-08-14 11:36:20.734 7fbd8a95a700 1 mgr finish mon failed to return metadata for mds.db: (2) No such file or directory > 2019-08-14 11:36:21.586 7fbd77570700 0 log_channel(audit) log [DBG] : from='client.1731656635 -' entity='client.admin' cmd=[{"prefix": "balancer eval", "option": "$POOL", "target": ["mgr", ""]}]: dispatch > 2019-08-14 11:36:21.734 7fbd77570700 0 ms_deliver_dispatch: unhandled message 0x55d968250a40 mgrreport(mds.db +110-0 packed 1366) v7 from mds.1717053370 v2:10.0.1.251:6800/140458953 > 2019-08-14 11:36:21.734 7fbd8a95a700 1 mgr finish mon failed to return metadata for mds.db: (2) No such file or directory > 2019-08-14 11:36:22.734 7fbd77570700 0 ms_deliver_dispatch: unhandled message 0x55d969297a00 mgrreport(mds.db +110-0 packed 1366) v7 from mds.1717053370 v2:10.0.1.251:6800/140458953 > [snip-repeating] > 2019-08-14 11:37:38.826 7fbd77570700 0 ms_deliver_dispatch: unhandled message 0x55d96b768d80 mgrreport(mds.db +110-0 packed 1366) v7 from mds.1717053370 v2:10.0.1.251:6800/140458953 > 2019-08-14 11:37:38.826 7fbd8a95a700 1 mgr finish mon failed to return metadata for mds.db: (2) No such file or directory > 2019-08-14 11:37:39.574 7fbd77d71700 1 mgr[balancer] Handling command: '{'prefix': 'balancer eval', 'option': '$POOL', 'target': ['mgr', '']}' > 2019-08-14 11:37:40.186 7fbd77570700 0 ms_deliver_dispatch: unhandled message 0x55d967aa23c0 mgrreport(mds.db +110-0 packed 1366) v7 from mds.1717053370 v2:10.0.1.251:6800/140458953 > 2019-08-14 11:37:40.186 7fbd8a95a700 1 mgr finish mon failed to return metadata for mds.db: (2) No such file or directory > 2019-08-14 11:37:41.054 7fbd77570700 0 ms_deliver_dispatch: unhandled message 0x55d96117c9c0 mgrreport(mds.db +110-0 packed 1366) v7 from mds.1717053370 v2:10.0.1.251:6800/140458953 > 2019-08-14 11:37:41.054 7fbd8a95a700 1 mgr finish mon failed to return metadata for mds.db: (2) No such file or directory I filtered out a bunch of messages with the mgr servicing `osd pool stats` requests, which I have running in a watch -n1. And I also snipped a bunch of repeating messages about mds.db, the standby-replay mds, and not sure if that is anything worth caring about. Hoping someone may have some ideas of where to look to try and improve stability with the mgr's. Thanks, Reed

3 4

How to tune the ceph balancer in nautilus
by Manuel Lausch 15 Aug '19

15 Aug '19

Hi, I playing around with the ceph balancer in luminous and nautilus. While tuning some balancer settings I experienced some problems with nautilus. In Luminous I cold configure the max_misplaced value like this: ceph config-key set mgr/balancer/max_misplaced 0.002 With the same command in nautilus I get this Warning: WARNING: it looks like you might be trying to set a ceph-mgr module configuration key. Since Ceph 13.0.0 (Mimic), mgr module configuration is done with `config set`, and new values set using `config-key set` will be ignored. After investigating the nautilus documentation (https://docs.ceph.com/docs/nautilus/rados/operations/balancer/#throttling) I tried this: ceph config set mgr mgr/balancer/max_misplaced 0.002 and get this error: Error EINVAL: unrecognized config option 'mgr/balancer/max_misplaced' My question is, how can I configure this parameters? In general the whole "ceph config" foo confuses me a bit. ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) Regards Manuel

1 0

rgw luminous 12.2.12
by Marc Roos 15 Aug '19

15 Aug '19

FYI I just had an issue with radosgw / civetweb. Wanted to upload 40 MB file, it started with poor transfer speed, which was decreasing over time to 20KB/s when I stopped the transfer. I had to kill radosgw and start it to get 'normal' operation back.

1 0

"Signature check failed" from certain clients
by Peter Sarossy 15 Aug '19

15 Aug '19

hey folks, I spent the past 2 hours digging through the forums and similar sources with no luck.. I use ceph storage for docker stacks, and this issue has taken the whole thing down as I cannot mount their volumes back... Starting yesterday, some of my nodes cannot mount the filesystem and it just hangs on the mount command, while the logs are full of the messages below. It doesn't matter which MDS node is active for some clients it just works, and for others it doesn't. Any hints? 2019-08-15 02:20:55.230 7f6e80e47700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13406 cs=3203 l=0).handle_message_footer Signature check failed 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 Message signature does not match contents. 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1Signature on message: 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 sig: 1045888080092928376 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1Locally calculated signature: 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 sig_check:13982737427498638198 2019-08-15 02:20:56.254 7f6e7fe45700 0 Signature failed. 2019-08-15 02:20:56.254 7f6e7fe45700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13410 cs=3205 l=0).handle_message_footer Signature check failed 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 Message signature does not match contents. 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1Signature on message: 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 sig: 1045888080092928376 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1Locally calculated signature: 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 sig_check:13982737427498638198 2019-08-15 02:20:57.246 7f6e80646700 0 Signature failed. 2019-08-15 02:20:57.246 7f6e80646700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13414 cs=3207 l=0).handle_message_footer Signature check failed Cheers, Peter

2 1

Failing heartbeats when no backfill is running
by Lorenz Kiefner 14 Aug '19

14 Aug '19

Dear ceph-users, I'm having trouble with heartbeats, there are a lot of "heartbeat_check: no reply from..."-messages in my logs when there is no backfilling or repairing running (yes, it's failing when all PGs are active+clean). Only a few OSDs are failing, even when there are several OSDs on the same host. Doesn't look like a network issue to me. When I set the flags "nobackfill" and "norecover" there are no heartbeat issues. My cluster is kind of heterogenous: it's ARMv7 and x86_64, connected mostly via VPN. Some hosts are Debian Stretch, so I'm still using Ceph Luminous (12.2.12). Is there someone having the same issue? What could be the next steps to debug? Any ideas? Thanks for any help! Lorenz

3 4

Re: Fw: Ceph-Deploy
by Paul Emmerich 14 Aug '19

14 Aug '19

You cannot add OSDs to a cluster that is offline. -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Aug 14, 2019 at 5:43 PM Cory Mueller <corymueller(a)hotmail.com> wrote: > > Good Morning, > > > I was wondering if you were aware of a way to add more OSD nodes to an existing OFFLINE cluster. > > Ive tried to use ceph-deploy new > ceph-deploy admin > ceph-deploy install > > and none of them allow me to join the cluster. I tried install the ceph packages beforehand and that still didnt work. The one part that im noticing causing the most issues is the ASC key. > I installed the asc key the same way i did on the previous 5 nodes before, yet when I take it offline to connect it to cluster, it still tries to install one from online which wont work in a offline environment. > > I tried copying the asc key from the other nodes in the cluster that are working, and that didnt change anything either. > > > I was hoping you had a few steps that I should be following to bypass this. > > Thank you in advance > Cory Mueller > > > ________________________________________ > From: Sage Weil <sweil(a)redhat.com> > Sent: August 14, 2019 2:20 PM > To: Cory Mueller > Subject: Re: Ceph-Deploy > > Try emailing ceph-users(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

rbd image usage per osd
by Frank R 14 Aug '19

14 Aug '19

I have an all RBD pool/cluster. I am interested in tracking how much disk space is being used by each RBD image on every OSD drive. The OSDs are Filestore. Does anyone know of any existing scripts that accomplish this task? If not, what commands can be used to generate this info?

2 1

Fw: Ceph-Deploy
by Cory Mueller 14 Aug '19

14 Aug '19

Good Morning, I was wondering if you were aware of a way to add more OSD nodes to an existing OFFLINE cluster. Ive tried to use ceph-deploy new ceph-deploy admin ceph-deploy install and none of them allow me to join the cluster. I tried install the ceph packages beforehand and that still didnt work. The one part that im noticing causing the most issues is the ASC key. I installed the asc key the same way i did on the previous 5 nodes before, yet when I take it offline to connect it to cluster, it still tries to install one from online which wont work in a offline environment. I tried copying the asc key from the other nodes in the cluster that are working, and that didnt change anything either. I was hoping you had a few steps that I should be following to bypass this. Thank you in advance Cory Mueller ________________________________________ From: Sage Weil <sweil(a)redhat.com> Sent: August 14, 2019 2:20 PM To: Cory Mueller Subject: Re: Ceph-Deploy Try emailing ceph-users(a)ceph.io

1 0

BlueStore _txc_add_transaction errors (possibly related to bug #38724)
by Florian Haas 14 Aug '19

14 Aug '19

Hi everyone, it seems there have been several reports in the past related to BlueStore OSDs crashing from unhandled errors in _txc_add_transaction: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-April/034444.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032172.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031960.ht… http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031964.ht… Bug #38724 tracks this, has been fixed in master with https://github.com/ceph/ceph/pull/27929, and is pending backports (and, I dare say, is *probably* misclassified as being only minor, as this does cause potential data loss as soon as it affects enough OSDs simultaneously): https://tracker.ceph.com/issues/38724 We just ran into a similar issue with a couple of BlueStore OSDs that we recently added to a Luminous (12.2.12) cluster that was upgraded from Jewel, and hence, still largely runs on FileStore. I say similar because evidently other people reporting this problem have been running into ENOENT (No such file or directory) or ENOTEMPTY (Directory not empty); for us it's interestingly E2BIG (Argument list too long): https://tracker.ceph.com/issues/38724#note-26 So I'm wondering if someone could shed light on these questions: * Is this the same issue as that which https://github.com/ceph/ceph/pull/27929 fixes? * Thus, since https://github.com/ceph/ceph/pull/29115 (the Nautilus backport for that fix) has been merged, but is not yet included in a release, do *Nautilus* users get a fix in the upcoming 14.2.3 release, and once they update, would this bug go away with no further intervention required? * For users on *Luminous*, since https://tracker.ceph.com/issues/39694 (the Luminous version of 38724) says "non-trivial backport", is it fair to say that a fix might still take a while for that release? * Finally, are Luminous users safe from this bug if they keep using, or revert to, FileStore? Thanks in advance for your thoughts! Please keep Erik CC'd on your reply. Cheers, Florian

4 6

Ceph Tech Talk for August 22nd
by Mike Perez 13 Aug '19

13 Aug '19

Hi everyone, We had a change with our original planned tech talk for August 22nd. This is short notice, but if you have a presentation topic you want to give and discuss with the community, now is a good opportunity. https://ceph.com/ceph-tech-talks/ -- Mike Perez (thingee)

1 0

Planning Ceph User Survey for 2019
by Mike Perez 12 Aug '19

12 Aug '19

Hi everyone, It's that time of the year again for us to form this year's user survey. The user survey gives the Ceph community insight with how people are using Ceph and where we should be spending our efforts. You can see last year's survey in this blog post: https://ceph.com/ceph-blog/ceph-user-survey-2018-results To start, let's use the Ceph pad to look at the previous set of questions and the answer options that will be available. https://pad.ceph.com/p/user-survey-2019 Let the discussing begin! -- Mike Perez (thingee)

1 0

Re: Built-in HA?
by Volodymyr Litovka 09 Aug '19

09 Aug '19

Hi Robert, > Another option is if both RDMA ports are on the same card, then you can do RDMA with a bond. This does not work if you have two separate cards. Yes, we recently talked to Mellanox and their engineers also recommend this way. > As far as your questions go, my guess would be that you would want to have the different NICs in different broadcast domains yes, the idea was to use two public networks on two different NICs with addresses from different subnets. It is possible to set 2+ networks in Ceph configuration, but it’s unclear how Ceph is going to use this configuration. > or set up Source Based Routing and bind the source port on the connection (not the easiest, but allows you to have multiple NICs in the same broadcast domain). I don't have experience with Ceph in this type of configuration. it’s too complicated and, frankly, when you’re trying to reach max performance, Source Based Routing is a bit from another area :-) At the end of the all, we’re going to test bonding of two ports on same NIC. Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison > On Fri, Aug 2, 2019 at 9:41 AM Volodymyr Litovka <doka.ua(a)gmx.com <mailto:doka.ua@gmx.com>> wrote: > Dear colleagues, > > at the moment, we use Ceph in routed environment (OSPF, ECMP) and everything is ok, reliability is high and there is nothing to complain about. But for hardware reasons (to be more precise - RDMA offload), we are faced with the need to operate Ceph directly on physical interfaces. > > According to documentation, "We generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded." > > Q1: Did anybody test and can explain, how Ceph will behave in first scenario (two IPs on the same network)? I think this configuration require just one statement in 'public network' (where both interfaces reside)? How it will distribute traffic between links, how it will detect link failures and how it will switchover? > > Q2: Did anybody test a bit another scenario - both NICs have addresses in different networks and Ceph configuration contain two 'public networks'? Questions are same - how Ceph distributes traffic between links and how it recovers from link failures? > > Thank you. > > -- > Volodymyr Litovka > "Vision without Execution is Hallucination." -- Thomas Edison > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>

1 0

RGW cache issue
by shellyyang1989 08 Aug '19

08 Aug '19

Hi All, Does RGW process have cache? If yes, what's the cache size of a single RGW process? Thanks, Yang

1 0

How to disable RGW log or change RGW log level
by shellyyang1989 08 Aug '19

08 Aug '19

Hi All, The RGW log always reached the disk capacity. Is there approach to disable RGW log, change the log level or change the default log path? Thanks, Yang

2 1

Re: Bluestore caching oddities, again
by Christian Balzer 08 Aug '19

08 Aug '19

Hello again, Getting back to this: On Sun, 4 Aug 2019 10:47:27 +0900 Christian Balzer wrote: > Hello, > > preparing the first production bluestore, nautilus (latest) based cluster > I've run into the same things other people and myself ran into before. > > Firstly HW, 3 nodes with 12 SATA HDDs each, IT mode LSI 3008, wal/db on > 40GB SSD partitions. (boy do I hate the inability of ceph-volume to deal > with raw partitions). > SSDs aren't a bottleneck in any scenario. > Single E5-1650 v3 @ 3.50GHz, cpu isn't a bottleneck in any scenario, less > than 15% of a core per OSD. > > Connection is via 40GB/s infiniband, IPoIB, no issues here as numbers later > will show. > > Clients are KVMs on Epyc based compute nodes, maybe some more speed could > be squeezed out here with different VM configs, but the cpu isn't an issue > in the problem cases. > > > > 1. 4k random I/O can cause degraded PGs > I've run into the same/similar issue as Nathan Fish here: > https://www.spinics.net/lists/ceph-users/msg526 > During the first 2 tests with 4k random I/O I got shortly degraded PGs as > well, with no indication in CPU or SSD utilization accounting for this. > HDDs were of course busy at that time. > Wasn't able to reproduce this so far, but it leaves me less than > confident. > > This happened again yesterday when rsyncing 260GB of average 4MB files into a Ceph image backed VM. Given the nature of this rsync nothing on the ceph nodes was the least bit busy, the HDDs were all below 15% utilization, CPU bored, etc. Still we got: --- 2019-08-07 15:38:23.452580 osd.21 (osd.21) 651 : cluster [DBG] 1.125 starting backfill to osd.9 from (0'0,0'0] MAX to 1297'21584 2019-08-07 15:38:24.454942 mon.ceph-05 (mon.0) 182756 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2019-08-07 15:38:25.396756 mon.ceph-05 (mon.0) 182757 : cluster [DBG] osdmap e1302: 36 total, 36 up, 36 in 2019-08-07 15:38:23.452026 osd.12 (osd.12) 767 : cluster [DBG] 1.105 starting backfill to osd.25 from (0'0,0'0] MAX to 1297'6782 --- Unfortunately all I have in the OSD log is this: --- 2019-08-07 15:38:23.461 7f155e71b700 1 osd.9 pg_epoch: 1299 pg[1.125( empty local-lis/les=0/0 n=0 ec=189/189 lis/c 1286/1286 les/c/f 1287/1287/0 1298/1299/189) [21,9,28]/[21,28,3] r=-1 lpr=1299 pi=[1286,1299)/1 crt=0'0 unknown mbc={}] state<Start>: transitioning to Stray 2019-08-07 15:38:24.353 7f155e71b700 1 osd.9 pg_epoch: 1301 pg[1.125( v 1297'21584 (1246'18584,1297'21584] local-lis/les=1299/1300 n=5 ec=189/189 lis/c 1299/1299 les/c/f 1300/1300/0 1298/1301/189) [21,9,28] r=1 lpr=1301 pi=[1299,1301)/1 luod=0'0 crt=1297'21584 active mbc={}] start_peering_interval up [21,9,28] -> [21,9,28], acting [21,28,3] -> [21,9,28], acting_primary 21 -> 21, up_primary 21 -> 21, role -1 -> 1, features acting 4611087854031667199 upacting 4611087854031667199 2019-08-07 15:38:24.353 7f155e71b700 1 osd.9 pg_epoch: 1301 pg[1.125( v 1297'21584 (1246'18584,1297'21584] local-lis/les=1299/1300 n=5 ec=189/189 lis/c 1299/1299 les/c/f 1300/1300/0 1298/1301/189) [21,9,28] r=1 lpr=1301 pi=[1299,1301)/1 crt=1297'21584 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray --- How can I find out what happened here, given that it might not happen again anytime soon cranking up debug levels now is a tad late. Thanks, Christian -- Christian Balzer Network/Systems Engineer chibi(a)gol.com Rakuten Mobile Inc.

2 2

OpenStack - rbd_store_chunk_size
by Kees Meijs 07 Aug '19

07 Aug '19

Hi list, I'm figuring if we would like rbd_store_chunk_size = 4 or rbd_store_chunk_size = 8 (or maybe something different) on our new OpenStack / Ceph environment. Any opinions on this matter? Cheers, Kees -- https://nefos.nl/contact Nefos IT bv Ambachtsweg 25 (industrienummer 4217) 5627 BZ Eindhoven Nederland KvK 66494931 /Aanwezig op maandag, dinsdag, woensdag en vrijdag/

1 0

Request to guide on ceph-deploy install command for luminuous 12.2.12 release
by Nerurkar, Ruchir (Nokia - US/Mountain View) 07 Aug '19

07 Aug '19

Hello, I work in Nokia as a Software QA engineer and I am trying to install Ceph on centOS 7.4 version. But I am getting this error with the following output: - vmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph) [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-osd-12.2.12-0.el7.x86_64 (Ceph) [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph) [mvmgoscephcontrollerluminous][WARNIN] Requires: gperftools-libs >= 2.6.1 [mvmgoscephcontrollerluminous][WARNIN] Available: gperftools-libs-2.4-8.el7.i686 (base) [mvmgoscephcontrollerluminous][DEBUG ] You could try using --skip-broken to work around the problem [mvmgoscephcontrollerluminous][WARNIN] gperftools-libs = 2.4-8.el7 [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-mon-12.2.12-0.el7.x86_64 (Ceph) [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph) [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit) [mvmgoscephcontrollerluminous][DEBUG ] You could try running: rpm -Va --nofiles --nodigest [mvmgoscephcontrollerluminous][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y install ceph ceph-radosgw Does anyone know about this issue like how can I resolve package dependencies? Best Regards, Ruchir Nerurkar 857-701-3405

1 0

ceph device list empty
by Gary Molenkamp 07 Aug '19

07 Aug '19

I'm testing an upgrade to Nautilus on a development cluster and the command "ceph device ls" is returning an empty list. # ceph device ls DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY # I have walked through the luminous upgrade documentation under https://docs.ceph.com/docs/master/releases/nautilus/#upgrading-from-mimic-o… but I don't see anything pertaining to "activating" device support under Nautilus. The devices are visible to ceph-volume on the OSS nodes. ie: osdev-stor1 ~]# ceph-volume lvm list ====== osd.0 ======= [block] /dev/ceph-f5eb16ec-7074-477b-8f83-ce87c5f74fa3/osd-block-c1de464f-d838-4558-ba75-1c268e538d6b block device /dev/ceph-f5eb16ec-7074-477b-8f83-ce87c5f74fa3/osd-block-c1de464f-d838-4558-ba75-1c268e538d6b block uuid dlbIm6-H5za-001b-C3mQ-EGks-yoed-zoQpoo <snip> devices /dev/sdb ====== osd.2 ======= [block] /dev/ceph-37145a74-6b2b-4519-b72e-2defe11732aa/osd-block-e06c513b-5af3-4bf6-927f-1f0142c59e8a block device /dev/ceph-37145a74-6b2b-4519-b72e-2defe11732aa/osd-block-e06c513b-5af3-4bf6-927f-1f0142c59e8a block uuid egdvpm-3bXx-xmNO-ACzp-nxax-Wka2-81rfNT <snip> devices /dev/sdc Is there a step I missed? Thanks. Gary. -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566

1 0

Re: [ceph-users] OSD's keep crasching after clusterreboot
by Ansgar Jazdzewski 07 Aug '19

07 Aug '19

another update, we now took the more destructive route and removed the cephfs pools (lucky we had only test date in the filesystem) Our hope was that within the startup-process the osd will delete the no longer needed PG, But this is NOT the Case. So we are still have the same issue the only difference is that the PG does not belong to a pool anymore. -360> 2019-08-07 14:52:32.655 7fb14db8de00 5 osd.44 pg_epoch: 196586 pg[23.f8s0(unlocked)] enter Initial -360> 2019-08-07 14:52:32.659 7fb14db8de00 -1 /build/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 7fb14db8de00 time 2019-08-07 14:52:32.660169 /build/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) we now can take one rout and try to delete the pg by hand in the OSD (bluestore) how this can be done? OR we try to upgrade to Nautilus and hope for the beset. any help hints are welcome, have a nice one Ansgar Am Mi., 7. Aug. 2019 um 11:32 Uhr schrieb Ansgar Jazdzewski <a.jazdzewski(a)googlemail.com>: > > Hi, > > as a follow-up: > * a full log of one OSD failing to start https://pastebin.com/T8UQ2rZ6 > * our ec-pool cration in the fist place https://pastebin.com/20cC06Jn > * ceph osd dump and ceph osd erasure-code-profile get cephfs > https://pastebin.com/TRLPaWcH > > as we try to dig more into it, it looks like a bug in the cephfs or > erasure-coding part of ceph. > > Ansgar > > > Am Di., 6. Aug. 2019 um 14:50 Uhr schrieb Ansgar Jazdzewski > <a.jazdzewski(a)googlemail.com>: > > > > hi folks, > > > > we had to move one of our clusters so we had to boot all servers, now > > we found an Error on all OSD with the EC-Pool. > > > > do we miss some opitons, will an upgrade to 13.2.6 help? > > > > > > Thanks, > > Ansgar > > > > 2019-08-06 12:10:16.265 7fb337b83200 -1 > > /build/ceph-13.2.4/src/osd/ECUtil.h: In function > > 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread > > 7fb337b83200 time 2019-08-06 12:10:16.263025 > > /build/ceph-13.2.4/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % > > stripe_size == 0) > > > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic > > (stable) 1: (ceph::ceph_assert_fail(char const, char const, int, char > > const)+0x102) [0x7fb32eeb83c2] 2: (()+0x2e5587) [0x7fb32eeb8587] 3: > > (ECBackend::ECBackend(PGBackend::Listener, coll_t const&, > > boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore, > > CephContext, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned > > long)+0x4de) [0xa4cbbe] 4: (PGBackend::build_pg_backend(pg_pool_t > > const&, std::map<std::cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> >, > > std::cxx11::basic_string<char, std::char_traits<char>, > > std::allocator<char> >, std::less<std::cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> > >, std > > ::allocator<std::pair<std::__cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> > const, > > std::cxx11::basic_string<char, std::char_traits<char>, > > std::allocator<char> > > > > const&, PGBackend::Listener, coll_t, > > boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore, > > CephContext)+0x2f9 ) [0x9474e9] 5: > > (PrimaryLogPG::PrimaryLogPG(OSDService, std::shared_ptr<OSDMap const>, > > PGPool const&, std::map<std::cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> >, > > std::cxx11::basic_string<char, std::char_traits<char>, > > std::allocator<char> >, std::less<std::cxx11::basic_string<char, > > std::char_tra its<char>, std::allocator<char> > >, > > std::allocator<std::pair<std::__cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> > const, > > std::cxx11::basic_string<char, std::char_traits<char>, > > std::allocator<char> > > > > const&, spg_t)+0x138) [0x8f96e8] 6: > > (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x11d3) > > [0x753553] 7: (OSD::load_pgs()+0x4a9) [0x758339] 8: > > (OSD::init()+0xcd3) [0x7619c3] 9: (main()+0x3678) [0x64d6a8] 10: > > (libc_start_main()+0xf0) [0x7fb32ca68830] 11: (_start()+0x29) > > [0x717389] NOTE: a copy of the executable, or objdump -rdS > > <executable> is needed to interpret this.

1 0

Re: [ceph-users] bluestore write iops calculation
by vitalif＠yourcmc.ru 07 Aug '19

07 Aug '19

> I can add RAM ans is there a way to increase rocksdb caching , can I > increase bluestore_cache_size_hdd to higher value to cache rocksdb? In recent releases it's governed by the osd_memory_target parameter. In previous releases it's bluestore_cache_size_hdd. Check release notes to know for sure. > This we have planned to add some SSDs and how many OSD's rocks db we > can add per SSDs and i guess if one SSD is down then all related OSDs > has to be re-installed. Yes. At least you'd better not put all 24 block.db's on a single SSD :) 4-8 HDDs per an SSD is usually fine. Also check db_used_bytes in `ceph daemon osd.0 perf dump` (replace 0 with actual OSD numbers) to figure out how much space your DBs use. If it's below 30gb you're lucky because in that case DBs will fit on 30GB SSD partitions. https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing -- Vitaliy Filippov

1 0

Re: [ceph-users] New CRUSH device class questions
by Paul Emmerich 07 Aug '19

07 Aug '19

On Wed, Aug 7, 2019 at 9:30 AM Robert LeBlanc <robert(a)leblancnet.us> wrote: >> # ceph osd crush rule dump replicated_racks_nvme >> { >> "rule_id": 0, >> "rule_name": "replicated_racks_nvme", >> "ruleset": 0, >> "type": 1, >> "min_size": 1, >> "max_size": 10, >> "steps": [ >> { >> "op": "take", >> "item": -44, >> "item_name": "default~nvme" <------------ >> }, >> { >> "op": "chooseleaf_firstn", >> "num": 0, >> "type": "rack" >> }, >> { >> "op": "emit" >> } >> ] >> } >> ``` > > > Yes, our HDD cluster is much like this, but not Luminous, so we created as separate root with SSD OSD for the metadata and set up a CRUSH rule for the metadata pool to be mapped to SSD. I understand that the CRUSH rule should have a `step take default class ssd` which I don't see in your rule unless the `~` in the item_name means device class. ~ is the internal implementation of device classes. Internally it's still using separate roots, that's how it stays compatible with older clients that don't know about device classes. And since it wasn't mentioned here yet: consider upgrading to Nautilus to benefit from the new and improved accounting for metadata space. You'll be able to see how much space is used for metadata and quotas should work properly for metadata usage. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 > > Thanks > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

out of memory bluestore osds
by Jaime Ibar 07 Aug '19

07 Aug '19

Hi all, we run a Ceph Luminous 12.2.12 cluster, 7 osds servers 12x4TB disks each. Recently we redeployed the osds of one of them using bluestore backend, however, after this, we're facing Out of memory errors(invoked oom-killer) and the OS kills one of the ceph-osd process. The osd is restarted automatically and back online after one minute. We're running Ubuntu 16.04, kernel 4.15.0-55-generic. The server has 32GB of RAM and 4GB of swap partition. All the disks are hdd, no ssd disks. Bluestore settings are the default ones "osd_memory_target": "4294967296" "osd_memory_cache_min": "134217728" "bluestore_cache_size": "0" "bluestore_cache_size_hdd": "1073741824" "bluestore_cache_autotune": "true" As stated in the documentation, bluestore assigns by default 4GB of RAM per osd(1GB of RAM for 1TB). So in this case 48GB of RAM would be needed. Am I right? Are these the minimun requirements for bluestore? In case adding more RAM is not an option, can any of osd_memory_target, osd_memory_cache_min, bluestore_cache_size_hdd be decrease to fit in our server specs? Would this have any impact on performance? Thanks Jaime -- Jaime Ibar High Performance & Research Computing, IS Services Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. http://www.tchpc.tcd.ie/ | jaime(a)tchpc.tcd.ie Tel: +353-1-896-3725

2 1

Re: [ceph-users] 14.2.2 - OSD Crash
by Igor Fedotov 07 Aug '19

07 Aug '19

Hi Manuel, as Brad pointed out timeouts and suicides are rather consequences of some other issues with OSDs. I recall at least two recent relevant tickets: https://tracker.ceph.com/issues/36482 https://tracker.ceph.com/issues/40741 (see last comments) Both had massive and slow reads from RocksDB which caused timeouts.. Visible symptom for both cases was unexpectedly high read I/O from underlying disks (main and/or DB). You can use iotop for inspection.., These were worsened by having significant part of DB at spinners due to spillovers. So wondering what's your layout in this respect: what drives back troublesome OSDs, is there any spillover to slow device, how massive it is? Also could you please inspect your OSD logs for the presence of lines containing "slow operation observed" substring. And share them if any.. Hope this helps. Thanks, Igor On 8/7/2019 2:16 AM, EDH - Manuel Rios Fernandez wrote: > > Hi > > We got a pair of OSD located in node that crash randomly since 14.2.2 > > OS Version : Centos 7.6 > > There’re a ton of lines before crash , I will unespected: > > -- > > 3045> 2019-08-07 00:39:32.013 7fe9a4996700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7fe987e49700' had timed out after 15 > > -3044> 2019-08-07 00:39:32.013 7fe9a3994700 1 heartbeat_map > is_healthy 'OSD::osd_op_tp thread 0x7fe987e49700' had timed out after 15 > > -3043> 2019-08-07 00:39:32.033 7fe9a4195700 1 heartbeat_map > is_healthy 'OSD::osd_op_tp thread 0x7fe987e49700' had timed out after 15 > > -3042> 2019-08-07 00:39:32.033 7fe9a4996700 1 heartbeat_map > is_healthy 'OSD::osd_op_tp thread 0x7fe987e49700' had timed out after 15 > > -- > > ----- > > Some hundred lines of: > > -164> 2019-08-07 00:47:36.628 7fe9a3994700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7fe98964c700' had timed out after 60 > > -163> 2019-08-07 00:47:36.632 7fe9a3994700 1 heartbeat_map > is_healthy 'OSD::osd_op_tp thread 0x7fe98964c700' had timed out after 60 > > -162> 2019-08-07 00:47:36.632 7fe9a3994700 1 heartbeat_map > is_healthy 'OSD::osd_op_tp thread 0x7fe98964c700' had timed out after 60 > > ----- > > -78> 2019-08-07 00:50:51.755 7fe995bfa700 10 monclient: tick > > -77> 2019-08-07 00:50:51.755 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:50:21.756453) > > -76> 2019-08-07 00:51:01.755 7fe995bfa700 10 monclient: tick > > -75> 2019-08-07 00:51:01.755 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:50:31.756604) > > -74> 2019-08-07 00:51:11.755 7fe995bfa700 10 monclient: tick > > -73> 2019-08-07 00:51:11.755 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:50:41.756788) > > -72> 2019-08-07 00:51:21.756 7fe995bfa700 10 monclient: tick > > -71> 2019-08-07 00:51:21.756 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:50:51.756982) > > -70> 2019-08-07 00:51:31.755 7fe995bfa700 10 monclient: tick > > -69> 2019-08-07 00:51:31.755 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:51:01.757206) > > -68> 2019-08-07 00:51:41.756 7fe995bfa700 10 monclient: tick > > -67> 2019-08-07 00:51:41.756 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:51:11.757364) > > -66> 2019-08-07 00:51:51.756 7fe995bfa700 10 monclient: tick > > -65> 2019-08-07 00:51:51.756 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:51:21.757535) > > -64> 2019-08-07 00:51:52.861 7fe987e49700 1 heartbeat_map > clear_timeout 'OSD::osd_op_tp thread 0x7fe987e49700' had timed out > after 15 > > -63> 2019-08-07 00:51:52.861 7fe987e49700 1 heartbeat_map > clear_timeout 'OSD::osd_op_tp thread 0x7fe987e49700' had suicide timed > out after 150 > > -62> 2019-08-07 00:51:52.948 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1721180160 mapped: 4297818112 > old cache_size: 1994018210 new cache size: 1992784572 > > -61> 2019-08-07 00:51:52.948 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 1992784572 kv_alloc: 763363328 kv_used: 749381098 meta_alloc: > 763363328 meta_used: 654593191 data_alloc: 452984832 data_used: 455929856 > > -60> 2019-08-07 00:51:57.923 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 1994110827 kv_alloc: 763363328 kv_used: 749381098 meta_alloc: > 763363328 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -59> 2019-08-07 00:51:57.973 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 1994110827 new cache size: 1994442069 > > -58> 2019-08-07 00:52:01.756 7fe995bfa700 10 monclient: tick > > -57> 2019-08-07 00:52:01.756 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:51:31.757684) > > -56> 2019-08-07 00:52:02.933 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 1995765747 kv_alloc: 763363328 kv_used: 749381098 meta_alloc: > 763363328 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -55> 2019-08-07 00:52:02.983 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 1995765747 new cache size: 1996096345 > > -54> 2019-08-07 00:52:07.943 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 1997417449 kv_alloc: 763363328 kv_used: 749381098 meta_alloc: > 763363328 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -53> 2019-08-07 00:52:07.993 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 1997417449 new cache size: 1997747404 > > -52> 2019-08-07 00:52:11.757 7fe995bfa700 10 monclient: tick > > -51> 2019-08-07 00:52:11.757 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:51:41.757855) > > -50> 2019-08-07 00:52:12.952 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 1999065941 kv_alloc: 763363328 kv_used: 749381098 meta_alloc: > 763363328 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -49> 2019-08-07 00:52:13.002 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 1999065941 new cache size: 1999395254 > > -48> 2019-08-07 00:52:17.962 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2000711226 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -47> 2019-08-07 00:52:18.012 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2000711226 new cache size: 2001039899 > > -46> 2019-08-07 00:52:21.756 7fe995bfa700 10 monclient: tick > > -45> 2019-08-07 00:52:21.756 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:51:51.758043) > > -44> 2019-08-07 00:52:22.971 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2002353314 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -43> 2019-08-07 00:52:23.022 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2002353314 new cache size: 2002681348 > > -42> 2019-08-07 00:52:27.982 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2003992210 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -41> 2019-08-07 00:52:28.031 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2003992210 new cache size: 2004319607 > > -40> 2019-08-07 00:52:31.756 7fe995bfa700 10 monclient: tick > > -39> 2019-08-07 00:52:31.756 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:52:01.758219) > > -38> 2019-08-07 00:52:32.991 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2005627920 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -37> 2019-08-07 00:52:33.041 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2005627920 new cache size: 2005954680 > > -36> 2019-08-07 00:52:38.001 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2007260450 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -35> 2019-08-07 00:52:38.051 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2007260450 new cache size: 2007586575 > > -34> 2019-08-07 00:52:41.757 7fe995bfa700 10 monclient: tick > > -33> 2019-08-07 00:52:41.757 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:52:11.758447) > > -32> 2019-08-07 00:52:43.011 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2008889806 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -31> 2019-08-07 00:52:43.061 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2008889806 new cache size: 2009215297 > > -30> 2019-08-07 00:52:48.021 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2010515995 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -29> 2019-08-07 00:52:48.071 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2010515995 new cache size: 2010840853 > > -28> 2019-08-07 00:52:51.757 7fe995bfa700 10 monclient: tick > > -27> 2019-08-07 00:52:51.757 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:52:21.758631) > > -26> 2019-08-07 00:52:53.031 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2012139023 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -25> 2019-08-07 00:52:53.081 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2012139023 new cache size: 2012463250 > > -24> 2019-08-07 00:52:58.042 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2013758896 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -23> 2019-08-07 00:52:58.092 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2013758896 new cache size: 2014082492 > > -22> 2019-08-07 00:53:01.758 7fe995bfa700 10 monclient: tick > > -21> 2019-08-07 00:53:01.758 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:52:31.758799) > > -20> 2019-08-07 00:53:03.052 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2015375620 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -19> 2019-08-07 00:53:03.102 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2015375620 new cache size: 2015698587 > > -18> 2019-08-07 00:53:08.062 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2016989201 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -17> 2019-08-07 00:53:08.112 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2016989201 new cache size: 2017311541 > > -16> 2019-08-07 00:53:11.758 7fe995bfa700 10 monclient: tick > > -15> 2019-08-07 00:53:11.758 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:52:41.759013) > > -14> 2019-08-07 00:53:13.071 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2018599645 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -13> 2019-08-07 00:53:13.121 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2018599645 new cache size: 2018921358 > > -12> 2019-08-07 00:53:18.081 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2020206960 kv_alloc: 771751936 kv_used: 749381098 meta_alloc: > 771751936 meta_used: 654590799 data_alloc: 452984832 data_used: 451538944 > > -11> 2019-08-07 00:53:18.130 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2020206960 new cache size: 2020528048 > > -10> 2019-08-07 00:53:21.757 7fe995bfa700 10 monclient: tick > > -9> 2019-08-07 00:53:21.757 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:52:51.759214) > > -8> 2019-08-07 00:53:23.090 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2021811150 kv_alloc: 780140544 kv_used: 749381098 meta_alloc: > 780140544 meta_used: 654590799 data_alloc: 461373440 data_used: 451538944 > > -7> 2019-08-07 00:53:23.140 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2021811150 new cache size: 2022131613 > > -6> 2019-08-07 00:53:28.100 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2023412220 kv_alloc: 780140544 kv_used: 749381098 meta_alloc: > 780140544 meta_used: 654590799 data_alloc: 461373440 data_used: 451538944 > > -5> 2019-08-07 00:53:28.150 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2023412220 new cache size: 2023732060 > > -4> 2019-08-07 00:53:31.758 7fe995bfa700 10 monclient: tick > > -3> 2019-08-07 00:53:31.758 7fe995bfa700 10 monclient: > _check_auth_rotating have uptodate secrets (they expire after > 2019-08-07 00:53:01.759334) > > -2> 2019-08-07 00:53:33.110 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _trim_shards cache_size: > 2025010178 kv_alloc: 780140544 kv_used: 749381098 meta_alloc: > 780140544 meta_used: 654590799 data_alloc: 461373440 data_used: 451538944 > > -1> 2019-08-07 00:53:33.160 7fe99966c700 5 > bluestore.MempoolThread(0x55ff04ad6a88) _tune_cache_size target: > 4294967296 heap: 6018998272 unmapped: 1725702144 mapped: 4293296128 > old cache_size: 2025010178 new cache size: 2025329397 > > 0> 2019-08-07 00:53:37.655 7fe987e49700 -1 *** Caught signal > (Aborted) ** > > in thread 7fe987e49700 thread_name:tp_osd_tp > > ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) > nautilus (stable) > > 1: (()+0xf5d0) [0x7fe9a7cba5d0] > > 2: (pthread_kill()+0x31) [0x7fe9a7cb79d1] > > 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char > const*, unsigned long)+0x466) [0x55fef8748176] > > 4: (ceph::HeartbeatMap::clear_timeout(ceph::heartbeat_handle_d*)+0x7b) > [0x55fef874878b] > > 5: > (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, > std::vector<ObjectStore::Transaction, > std::allocator<ObjectStore::Transaction> >&, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0xa9e) > [0x55fef86085de] > > 6: > (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, > ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, > ThreadPool::TPHandle*)+0x7f) [0x55fef81cd7ff] > > 7: (OSD::dispatch_context_transaction(PG::RecoveryCtx&, PG*, > ThreadPool::TPHandle*)+0x58) [0x55fef8118298] > > 8: (OSD::dequeue_peering_evt(OSDShard*, PG*, > std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x202) > [0x55fef81767c2] > > 9: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, > ThreadPool::TPHandle&)+0x50) [0x55fef83eb490] > > 10: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x9f4) [0x55fef816aef4] > > 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) > [0x55fef8769ce3] > > 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55fef876cd80] > > 13: (()+0x7dd5) [0x7fe9a7cb2dd5] > > 14: (clone()+0x6d) [0x7fe9a6b7202d] > > About server load: > > [root@CEPH008 ~]# top > > top - 00:57:30 up 186 days, 22 min, 1 user, load average: 11.65, > 13.42, 13.51 > > Tasks: 316 total, 1 running, 315 sleeping, 0 stopped, 0 zombie > > %Cpu(s): 2.3 us, 1.2 sy, 0.0 ni, 74.1 id, 22.4 wa, 0.0 hi, 0.1 > si, 0.0 st > > KiB Mem : 65737480 total, 431824 free, 49046608 used, 16259048 > buff/cache > > KiB Swap: 29241340 total, 19406504 free, 9834836 used. 15917556 avail Mem > > Currently the server is doing some deep-scrub that we got off during > the last two weeks due a node evict and a new node install. > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

2 2

Nautilus - Balancer is always on
by EDH - Manuel Rios Fernandez 07 Aug '19

07 Aug '19

Hi All, ceph mgr module disable balancer Error EINVAL: module 'balancer' cannot be disabled (always-on) Whats the way to restart balanacer? Restart MGR service? I wanna suggest to Balancer developer to setup a ceph-balancer.log for this module get more information about whats doing. Regards Manuel

1 0

Re: [ceph-users] Error Mounting CephFS
by Frank Schilder 07 Aug '19

07 Aug '19

On Centos7, the option "secretfile" requires installation of ceph-fuse. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of Yan, Zheng <ukernel(a)gmail.com> Sent: 07 August 2019 10:10:19 To: DHilsbos(a)performair.com Cc: ceph-users Subject: Re: [ceph-users] Error Mounting CephFS On Wed, Aug 7, 2019 at 3:46 PM <DHilsbos(a)performair.com> wrote: > > All; > > I have a server running CentOS 7.6 (1810), that I want to set up with CephFS (full disclosure, I'm going to be running samba on the CephFS). I can mount the CephFS fine when I use the option secret=, but when I switch to secretfile=, I get an error "No such process." I installed ceph-common. > > Is there a service that I'm not aware I should be starting? > Do I need to install another package? > mount.ceph is missing. check if it exists and is located in $PATH > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > DHilsbos(a)PerformAir.com > www.PerformAir.com > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Re: [ceph-users] OSD's keep crasching after clusterreboot
by Ansgar Jazdzewski 07 Aug '19

07 Aug '19

Hi, as a follow-up: * a full log of one OSD failing to start https://pastebin.com/T8UQ2rZ6 * our ec-pool cration in the fist place https://pastebin.com/20cC06Jn * ceph osd dump and ceph osd erasure-code-profile get cephfs https://pastebin.com/TRLPaWcH as we try to dig more into it, it looks like a bug in the cephfs or erasure-coding part of ceph. Ansgar Am Di., 6. Aug. 2019 um 14:50 Uhr schrieb Ansgar Jazdzewski <a.jazdzewski(a)googlemail.com>: > > hi folks, > > we had to move one of our clusters so we had to boot all servers, now > we found an Error on all OSD with the EC-Pool. > > do we miss some opitons, will an upgrade to 13.2.6 help? > > > Thanks, > Ansgar > > 2019-08-06 12:10:16.265 7fb337b83200 -1 > /build/ceph-13.2.4/src/osd/ECUtil.h: In function > 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread > 7fb337b83200 time 2019-08-06 12:10:16.263025 > /build/ceph-13.2.4/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % > stripe_size == 0) > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic > (stable) 1: (ceph::ceph_assert_fail(char const, char const, int, char > const)+0x102) [0x7fb32eeb83c2] 2: (()+0x2e5587) [0x7fb32eeb8587] 3: > (ECBackend::ECBackend(PGBackend::Listener, coll_t const&, > boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore, > CephContext, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned > long)+0x4de) [0xa4cbbe] 4: (PGBackend::build_pg_backend(pg_pool_t > const&, std::map<std::cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> >, > std::cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> >, std::less<std::cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, std > ::allocator<std::pair<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const, > std::cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > > > > const&, PGBackend::Listener, coll_t, > boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore, > CephContext)+0x2f9 ) [0x9474e9] 5: > (PrimaryLogPG::PrimaryLogPG(OSDService, std::shared_ptr<OSDMap const>, > PGPool const&, std::map<std::cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> >, > std::cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> >, std::less<std::cxx11::basic_string<char, > std::char_tra its<char>, std::allocator<char> > >, > std::allocator<std::pair<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const, > std::cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > > > > const&, spg_t)+0x138) [0x8f96e8] 6: > (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x11d3) > [0x753553] 7: (OSD::load_pgs()+0x4a9) [0x758339] 8: > (OSD::init()+0xcd3) [0x7619c3] 9: (main()+0x3678) [0x64d6a8] 10: > (libc_start_main()+0xf0) [0x7fb32ca68830] 11: (_start()+0x29) > [0x717389] NOTE: a copy of the executable, or objdump -rdS > <executable> is needed to interpret this.

1 0

Fwd: Re: [ceph-users] Ceph Scientific Computing User Group
by Jan Fajerski 07 Aug '19

07 Aug '19

Including new ceph-users list. ----- Forwarded message from Mike Perez <miperez(a)redhat.com> ----- Date: Fri, 2 Aug 2019 10:08:20 -0700 From: Mike Perez <miperez(a)redhat.com> To: Kevin Hrpcek <kevin.hrpcek(a)ssec.wisc.edu> CC: "ceph-users(a)lists.ceph.com" <ceph-users(a)lists.ceph.com> Subject: Re: [ceph-users] Ceph Scientific Computing User Group We have scheduled the next meeting on the community calendar for August 28 at 14:30 UTC. Each meeting will then take place on the last Wednesday of each month. Here's the pad to collect agenda/notes: [1]https://pad.ceph.com/p/Ceph_Science_User_Group_Index -- Mike Perez (thingee) On Tue, Jul 23, 2019 at 10:40 AM Kevin Hrpcek <[2]kevin.hrpcek(a)ssec.wisc.edu> wrote: Update We're going to hold off until August for this so we can promote it on the Ceph twitter with more notice. Sorry for the inconvenience if you were planning on the meeting tomorrow. Keep a watch on the list, twitter, or ceph calendar for updates. Kevin On 7/5/19 11:15 PM, Kevin Hrpcek wrote: We've had some positive feedback and will be moving forward with this user group. The first virtual user group meeting is planned for July 24th at 4:30pm central European time/10:30am American eastern time. We will keep it to an hour in length. The plan is to use the ceph bluejeans video conferencing and it will be put on the ceph community calendar. I will send out links when it is closer to the 24th. The goal of this user group is to promote conversations and sharing ideas for how ceph is used in the the scientific/hpc/htc communities. Please be willing to discuss your use cases, cluster configs, problems you've had, shortcomings in ceph, etc... Not everyone pays attention to the ceph lists so feel free to share the meeting information with others you know that may be interested in joining in. Contact me if you have questions, comments, suggestions, or want to volunteer a topic for meetings. I will be brainstorming some conversation starters but it would also be interesting to have people give a deep dive into their use of ceph and what they have built around it to support the science being done at their facility. Kevin On 6/17/19 10:43 AM, Kevin Hrpcek wrote: Hey all, At cephalocon some of us who work in scientific computing got together for a BoF and had a good conversation. There was some interest in finding a way to continue the conversation focused on ceph in scientific computing and htc/hpc environments. We are considering putting together monthly video conference user group meeting to facilitate sharing thoughts and ideas for this part of the ceph community. At cephalocon we mostly had teams present from the EU so I'm interested in hearing how much community interest there is in a ceph+science/HPC/HTC user group meeting. It will be impossible to pick a time that works well for everyone but initially we considered something later in the work day for EU countries. Reply to me if you're interested and please include your timezone. Kevin _______________________________________________ ceph-users mailing list [3]ceph-users(a)lists.ceph.com [4]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [5]ceph-users(a)lists.ceph.com [6]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [7]ceph-users(a)lists.ceph.com [8]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com References 1. https://pad.ceph.com/p/Ceph_Science_User_Group_Index 2. mailto:kevin.hrpcek@ssec.wisc.edu 3. mailto:ceph-users@lists.ceph.com 4. http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 5. mailto:ceph-users@lists.ceph.com 6. http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 7. mailto:ceph-users@lists.ceph.com 8. http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ----- End forwarded message ----- -- Jan Fajerski Engineer Enterprise Storage SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)

1 0

Can kstore be used as OSD objectstore backend when deploying a Ceph Storage Cluster? If can, how to?
by R.R.Yuan 07 Aug '19

07 Aug '19

Hi, All, When deploying a development cluster, there are three types of OSD objectstore backend: filestore, bluestore and kstore. But there is no "--kstore" option when using "ceph-deploy osd"command to deploy a real ceph cluster. Can kstore be used as OSD objectstore backend when deploy a real ceph cluster? If can, how to ? Thanks a lot R.R.Yuan

1 0

Can kstore be used as OSD objectstore backend when deploying a Ceph Storage Cluster? If can, how to?
by R_R_Yuan＠163.com 07 Aug '19

07 Aug '19

Hi, All, When deploying a development cluster, there are three types of OSD objectstore backend: filestore, bluestore and kstore. But there is no "--kestore" option when using "ceph-deploy osd" command to deploy a real ceph cluster. Can kstore be used as OSD objectstore backend when deploy a real ceph cluster? If can, how to ? Thanks a lot!

1 0

Multi zone syncing not working
by Freddy Andersen 06 Aug '19

06 Aug '19

Hi, I have tried to do this twice now and can’t get it to work 100%. In my last attempt I have secondary zone correctly syncing: sudo radosgw-admin sync status realm 2cc7226c-2cce-4b75-a719-97ba22756289 (prod) zonegroup 04702b5e-faa4-4110-9e98-6097f5e5e4b9 (us) zone ece9f510-e910-49be-a89f-7b1f47b3faf8 (zone2) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 8f0e7a41-f71c-45e8-869a-9bfbadbcbf2c (zone1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source but master zone can’t sync from secondary: sudo radosgw-admin sync status realm 2cc7226c-2cce-4b75-a719-97ba22756289 (prod) zonegroup 04702b5e-faa4-4110-9e98-6097f5e5e4b9 (us) zone 8f0e7a41-f71c-45e8-869a-9bfbadbcbf2c (zone1) metadata sync no sync (zone is master) 2019-08-06 16:43:36.228 7f059c7ab640 0 data sync zone:ece9f510 ERROR: failed to fetch datalog info data sync source: ece9f510-e910-49be-a89f-7b1f47b3faf8 (zone2) failed to retrieve sync info: (13) Permission denied I can see in our secondary rgw hosts that there is a 403 from master zone 2019-08-06 16:27:58.137 7ff3f3d30700 1 ====== req done req=0x7ff3f3d298d0 op status=0 http_status=403 latency=0s ====== 2019-08-06 16:27:58.137 7ff3f3d30700 1 civetweb: 0x563d52f64000: 10.130.30.210 - - [06/Aug/2019:16:27:58 -0700] "GET /admin/log?type=data&rgwx-zonegroup=04702b5e-faa4-4110-9e98-6097f5e5e4b9 HTTP/1.1" 403 320 - -

1 0

rados cache-flush-evict-all unable to flush all data into backend storage
by 化氢 06 Aug '19

06 Aug '19

hi， I created a cache layer using ceph osd tier. When I remove the cache layer, I use cache-flush-evict-all to synchronize the data, but the cache layer sometimes has several objects left, is this a bug? thanks

1 0