May 2020 - ceph-users - lists.ceph.io

octopus cluster deploy with cephadm failed on bootstrap
by Zhenshi Zhou 11 May '20

11 May '20

Hi all, I'm deploying a new octopus cluster using cephadm, follow docs <https://ceph.io/ceph-management/introducing-cephadm/>. However it failed on the bootstrap step. According to the logs, key generating failed because of the lack of directory and file. Did I miss something? here is the logs: [root@ceph-mon1 ~]# cephadm bootstrap --mon-ip 172.24.202.119 INFO:cephadm:Verifying podman|docker is present... INFO:cephadm:Verifying lvm2 is present... INFO:cephadm:Verifying time synchronization is in place... INFO:cephadm:Unit ntpd.service is enabled and running INFO:cephadm:Repeating the final host check... INFO:cephadm:podman|docker (/usr/bin/docker) is present INFO:cephadm:systemctl is present INFO:cephadm:lvcreate is present INFO:cephadm:Unit ntpd.service is enabled and running INFO:cephadm:Host looks OK INFO:root:Cluster fsid: 828e6b7a-91c2-11ea-8644-005056885573 INFO:cephadm:Verifying IP 172.24.202.119 port 3300 ... INFO:cephadm:Verifying IP 172.24.202.119 port 6789 ... INFO:cephadm:Mon IP 172.24.202.119 is in CIDR network 172.24.202.0/24 INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container... INFO:cephadm:Extracting ceph user uid/gid from container image... INFO:cephadm:Creating initial keys... INFO:cephadm:Creating initial monmap... INFO:cephadm:Creating mon... INFO:cephadm:Waiting for mon to start... INFO:cephadm:Waiting for mon... INFO:cephadm:Assimilating anything we can from ceph.conf... INFO:cephadm:Generating new minimal ceph.conf... INFO:cephadm:Restarting the monitor... INFO:cephadm:Setting mon public_network... INFO:cephadm:Creating mgr... INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring INFO:cephadm:Wrote config to /etc/ceph/ceph.conf INFO:cephadm:Waiting for mgr to start... INFO:cephadm:Waiting for mgr... INFO:cephadm:mgr not available, waiting (1/10)... INFO:cephadm:mgr not available, waiting (2/10)... INFO:cephadm:mgr not available, waiting (3/10)... INFO:cephadm:mgr not available, waiting (4/10)... INFO:cephadm:mgr not available, waiting (5/10)... INFO:cephadm:Enabling cephadm module... INFO:cephadm:Waiting for the mgr to restart... INFO:cephadm:Waiting for Mgr epoch 4... INFO:cephadm:Setting orchestrator backend to cephadm... INFO:cephadm:Generating ssh key... INFO:cephadm:Non-zero exit code 22 from /usr/bin/docker run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=ceph-mon1 -v /var/log/ceph/828e6b7a-91c2-11ea-8644-005056885573:/var/log/ceph:z -v /tmp/ceph-tmp8ydndiho:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp9yl7hx5u:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 cephadm generate-key INFO:cephadm:/usr/bin/ceph:stderr Error EINVAL: Traceback (most recent call last): INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/cephadm/module.py", line 1438, in _generate_key INFO:cephadm:/usr/bin/ceph:stderr with open(path, 'r') as f: INFO:cephadm:/usr/bin/ceph:stderr FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxvc18jh3/key' INFO:cephadm:/usr/bin/ceph:stderr INFO:cephadm:/usr/bin/ceph:stderr During handling of the above exception, another exception occurred: INFO:cephadm:/usr/bin/ceph:stderr INFO:cephadm:/usr/bin/ceph:stderr Traceback (most recent call last): INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command INFO:cephadm:/usr/bin/ceph:stderr return self.handle_command(inbuf, cmd) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command INFO:cephadm:/usr/bin/ceph:stderr return dispatch[cmd['prefix']].call(self, cmd, inbuf) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call INFO:cephadm:/usr/bin/ceph:stderr return self.func(mgr, **kwargs) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda> INFO:cephadm:/usr/bin/ceph:stderr wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper INFO:cephadm:/usr/bin/ceph:stderr return func(*args, **kwargs) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/cephadm/module.py", line 1443, in _generate_key INFO:cephadm:/usr/bin/ceph:stderr os.unlink(path) INFO:cephadm:/usr/bin/ceph:stderr FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxvc18jh3/key' INFO:cephadm:/usr/bin/ceph:stderr Traceback (most recent call last): File "./cephadm", line 4579, in <module> r = args.func() File "./cephadm", line 1122, in _default_image return func() File "./cephadm", line 2521, in command_bootstrap cli(['cephadm', 'generate-key']) File "./cephadm", line 2409, in cli ).run(timeout=timeout) File "./cephadm", line 2142, in run self.run_cmd(), desc=self.entrypoint, timeout=timeout) File "./cephadm", line 837, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=ceph-mon1 -v /var/log/ceph/828e6b7a-91c2-11ea-8644-005056885573:/var/log/ceph:z -v /tmp/ceph-tmp8ydndiho:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp9yl7hx5u:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 cephadm generate-key OS: CentOS Linux release 7.8.2003 (Core) Kernel: 3.10.0-514.6.1.el7.x86_64 Docker version: 19.03.8 cephadm: Using recent ceph image ceph/ceph:v15 ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) Thanks

1 1

Cluster rename procedure
by Anthony D'Atri 09 May '20

09 May '20

I’ve inherited a couple of clusters with non-default (ie, not “ceph”) internal names, and I want to rename them for the usual reasons. I had previously developed a full list of steps - which I no longer have access to. Anyone done this recently? Want to be sure I’m not missing something. * Nautilus, CentOS 7, RGW and RBD * Rename OSD mountpoints with mount —move * Rename systemd resources / mounts? * Rename /var/lib/ceph/{mon,osd} directories * Rename ceph*conf files on backend and client systems * Rename keyrings — just the filenames? * Rename log files * Ajust `ceph config` paths for admin socket, keyring, logs, mgr/mds/mon data, osd journal, rgw_data * Restart daemons * Ensure /var/run/ceph sockets are appropriately named Thanks — aad

2 1

Nautilus cluster rados gateway not sharding bucket indexes
by Marcel Kuiper 08 May '20

08 May '20

Hi Sorry for the repost, but I didn't get any respons on my first post so I will try to rephrase it We have several ceph clusters running nautilus (coming from mimic). On one of the clusters I got a health warning HEALTH_WARN 1 large omap objects * when I check on the rados gateway with radosgw-admin bucket limit check' I see 2 buckets over the limit * the config value for rgw_dynamic_resharding is true * on: 'radosgw-admin reshard stale-instances list' I get an error: Resharding disabled in a multisite env, stale instances unlikely from resharding. This is however a single site cluster. I have more clusters that are exactly the same and that do auto index resharding just fine. ANd above command does not give an error Can anyone help me tackle this problem. I've gone through the documentation and some blogs but did not find a solution yet Any help is much appreciated Thanks Marcel

2 2

Re: Data loss by adding 2OSD causing Long heartbeat ping times
by Frank Schilder 08 May '20

08 May '20

On all OSD nodes I'm using vm.min_free_kbytes = 4194304 (4GB). This was one of the first tunings on the cluster. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 08 May 2020 10:17 To: Frank Schilder Subject: Re: [ceph-users] Re: Data loss by adding 2OSD causing Long heartbeat ping times Just for grins, what is your vm.min_free_kbytes setting? Depending on the size and number of your OSDs and your workload, 2-4GB is a starting point. > > Hi XuYun and Martin, > > I checked that already. The OSDs in question have 8GB memory limit and the RAM of the servers is about 50% used. It could be memory fragmentation, which used to be a problem before bitmap allocator. However, my OSDs are configured to use bitmap, at least that is what they claim they are using. > > There might be a somewhat more fundamental issue, also related to my recent experience described in "Ceph meltdown, need help". The problems seem to have the same source, busy OSDs get behind with their internal cluster communication because (I suspect) client IO and admin IO (heartbeats, beacons, etc.) are handled in the same queue. If things get a bit busy, admin I/O slows down and avalanches happen. > > Also in the case here (long heartbeat) I first saw remapping and peering going on, then the heartbeats times of a few OSDs suddenly shot up. It is possible that some OSDs were already busy with client I/O. The additional peering seems to have the ability to add so much additional load to some OSDs that they start falling behind and getting marked out erroneously, with the consequence of even more peering, load, etc. > > I'm working on a new conversation "Cluster outage due to client IO" to have a clean focused thread. I need a bit more time to collect information though. For now, our cluster is up and running healthy. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Martin Verges <martin.verges(a)croit.io> > Sent: 07 May 2020 12:17:10 > To: XuYun > Cc: Frank Schilder; ceph-users > Subject: Re: [ceph-users] Re: Data loss by adding 2OSD causing Long heartbeat ping times > > Hello XuYun, > > In my experience, I would always disable swap, it won't do any good. > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges(a)croit.io<mailto:martin.verges@croit.io> > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > > Am Do., 7. Mai 2020 um 12:07 Uhr schrieb XuYun <yunxu(a)me.com<mailto:yunxu@me.com>>: > We had got some ping back/front problems after upgrading from filestore to bluestore. It turned out to be related to insufficient memory/swap usage. > >> 2020年5月6日下午10:08，Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> 写道： >> >> To answer some of my own questions: >> >> 1) Setting >> >> ceph osd set noout >> ceph osd set nodown >> ceph osd set norebalance >> >> before restart/re-deployment did not harm. I don't know if it helped, because I didn't retry the procedure that led to OSDs going down. See also point 3 below. >> >> 2) A peculiarity of this specific deployment of 2 OSDs was, that it was a mix of OSD deployment and restart after a reboot. I'm working on getting this sorted and this is a different story. For anyone who might find him-/herself in a situation where some OSDs are temporarily down/out with PGs remapped and objects degraded for whatever reason while new OSDs come up, the way to have ceph rescan the down/out OSDs after they come up is to >> >> - "ceph osd crush move" the new OSDs temporarily to a location outside the crush sub tree covering any pools (I have such a parking space in the crush hierarchy for easy draining and parking disks) >> - bring up the down/out OSDs >> - at this point, the cluster will fall back to the original crush map that was in place when the OSDs went down/out >> - the cluster will now find all shards that went orphan and health will be restored very quickly >> - once the cluster is healthy, "ceph osd crush move" the new OSDs back to their desired location >> - now you will see remapped PGs/misplaced objects, but no degraded objects >> >> 3) I still don't have an answer why long heartbeat ping times were observed. There seems to be a more serious issue and this will continue in its own thread "Cluster outage due to client IO" to be opened soon. >> >> Best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> >> ________________________________________ >> From: Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> >> Sent: 25 April 2020 15:34:25 >> To: ceph-users >> Subject: [ceph-users] Data loss by adding 2OSD causing Long heartbeat ping times >> >> Dear all, >> >> Two days ago I added very few disks to a ceph cluster and run into a problem I have never seen before when doing that. The entire cluster was deployed with mimic 13.2.2 and recently upgraded to 13.2.8. This is the first time I added OSDs under 13.2.8. >> >> I had a few hosts that I needed to add 1 or 2 OSDs to and I started with one that needed 1. Procedure was as usual: >> >> ceph osd set norebalance >> deploy additional OSD >> >> The OSD came up and PGs started peering, so far so good. To my surprise, however, I started seeing health-warnings about slow ping times: >> >> Long heartbeat ping times on back interface seen, longest is 1171.910 msec >> Long heartbeat ping times on front interface seen, longest is 1180.764 msec >> >> After peering it looked like it got better and I waited it out until the messages were gone. This took a really long time, at least 5-10 minutes. >> >> I went on to the next host and deployed 2 new OSDs this time. Same as above, but with much worse consequences. Apparently, the ping times exceeded a timeout for a very short moment and an OSD was marked out for ca. 2 seconds. Now all hell broke loose. I got health errors with the dreaded "backfill_toofull", undersized PGs and a large amount of degraded objects. I don't know what is causing what, but I ended up with data loss by just adding 2 disks. >> >> We have dedicated network hardware and each of the OSD hosts has 20GBit front and 40GBit back network capacity (LACP trunking). There are currently no more than 16 disks per server. The disks were added to an SSD pool. There was no traffic nor any other exceptional load on the system. I have ganglia resource monitoring on all nodes and cannot see a single curve going up. Network, CPU utilisation, load, everything below measurement accuracy. The hosts and network are quite overpowered and dimensioned to host many more OSDs (in future expansions). >> >> I have three questions, ordered by how urgently I need an answer: >> >> 1) I need to add more disks next week and need a workaround. Will something like this help avoiding the heartbeat time-out: >> >> ceph osd set noout >> ceph osd set nodown >> ceph osd set norebalance >> >> 2) The "lost" shards of the degraded objects were obviously still on the cluster somewhere. Is there any way to force the cluster to rescan OSDs for the shards that went orphan during the incident? >> >> 3) This smells a bit like a bug that requires attention. I was probably just lucky that I only lost 1 shard per PG. Has something similar reported before? Is this fixed in 13.2.10? Is it something new? Any settings that need to be looked at? If logs need to be collected, I can do so during my next attempt. However, I cannot risk data integrity of a production cluster and, therefore, probably not run the original procedure again. >> >> Many thanks for your help and best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph-mgr high CPU utilization
by Andras Pataki 08 May '20

08 May '20

I'm wondering if anyone still sees issues with ceph-mgr using CPU and being unresponsive even in recent Nautilus releases. We upgraded our largest cluster from Mimic to Nautilus (14.2.8) recently - it has about 3500 OSDs. Now ceph-mgr is constantly at 100-200% CPU (1-2 cores), and becomes unresponsive after a few minutes. The finisher-Mgr queue length grows (I've seen it at over 100k) - similar symptoms as seen with earlier Nautilus releases by many. This is what it looks like after an hour of running: "finisher-Mgr": { "queue_len": 66078, "complete_latency": { "avgcount": 21, "sum": 2098.408767721, "avgtime": 99.924227034 } }, We have a pretty vanilla manager config, only the balancer is enabled in upmap mode. Here are the enabled modules: "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator_cli", "progress", "rbd_support", "status", "volumes" ], "enabled_modules": [ "restful" ], Any ideas or outstanding issues in this area? Andras

3 6

Unit testing of CRUSH Algorithm
by Bobby 08 May '20

08 May '20

Hi, Are there any more resources of unit tests for CRUSH algorithm other than the test cases here: : https://github.com/ceph/ceph/tree/master/src/test/crush Or more unit testing of CRUSH apart from the these test cases would be an overkill? BR Bobby !

1 0

Cluster network and public network
by Nghia Viet Tran 08 May '20

08 May '20

Hi everyone, I have a question about the network setup. From the document, It’s recommended to have 2 NICs per hosts as described in below picture [Diagram] In the picture, OSD hosts will connect to the Cluster network for replicate and heartbeat between OSDs, therefore, we definitely need 2 NICs for it. But seems there are no connections between Ceph MON and Cluster network. Can we install 1 NIC on Ceph MON then? I appreciated any comments! Thank you! -- Nghia Viet Tran (Mr)

2 1

cephfs change/migrate default data pool
by Kenneth Waegeman 08 May '20

08 May '20

Hi all, I read in some release notes it is recommended to have your default data pool replicated and use erasure coded pools as additional pools through layouts. We have still a cephfs with +-1PB usage with a EC default pool. Is there a way to change the default pool or some other kind of migration without having to recreate the FS? Thanks! Kenneth

3 4

ceph octopus OSDs won't start with docker
by Sean Johnson 07 May '20

07 May '20

I have a seemingly strange situation. I have three OSDs that I created with Ceph Octopus using the `ceph orch daemon add <host>:device` command. All three were added and everything was great. Then I rebooted the host. Now the daemon’s won’t start via Docker. When I attempt to run the `docker` command directly it errors with: root@balin:/var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.12# /usr/bin/docker run --rm --net=host --privileged --group-add=disk --name ceph-c3d06c94-bb66-4f84-bf78-470a2364b667-osd.12 -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=balin -v /var/run/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667:/var/run/ceph:z -v /var/log/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667:/var/log/ceph:z -v /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.12:/var/lib/ceph/osd/ceph-12:z -v /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.12/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm --entrypoint /usr/bin/ceph-osd docker.io/ceph/ceph:v15 -n osd.12 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix="debug " debug 2020-05-07T22:58:06.258+0000 7f622a161ec0 0 set uid:gid to 167:167 (ceph:ceph) debug 2020-05-07T22:58:06.258+0000 7f622a161ec0 0 ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable), process ceph-osd, pid 1 debug 2020-05-07T22:58:06.258+0000 7f622a161ec0 0 pidfile_write: ignore empty --pid-file debug 2020-05-07T22:58:06.258+0000 7f622a161ec0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (13) Permission denied debug 2020-05-07T22:58:06.258+0000 7f622a161ec0 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-12: (2) No such file or directory The OSDs are able to come back online if I run `ceph-volume lvm activate —all`. Everything from a usage point of view is fine, even after a reboot, however I now have errors in the `ceph orch ps` list: osd.12 balin error 27s ago - <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> This is an Ubuntu 20.04 system, FWIW. I haven’t a clue where to go from here. While things are technically working since the OSDs are online and functioning, I’d really like to have them under the `ceph orch` management like the rest of the systems. ~Sean

1 0

How to apply ceph.conf changes using new tool cephadm
by Gencer W. Genç 07 May '20

07 May '20

Hi, === NOTE: I do not see my thread in ceph-list for some reason. I don't know if list received my question or not. So, sorry if this is duplicate. === I just deployed a new cluster with cephadm instead of ceph-deploy. In tyhe past, If i change ceph.conf for tweaking, i was able to copy them and apply to all servers. But i cannot find this on new cephadm tool. I did few changes on ceph.conf but ceph is unaware of those changes. How can i apply them? I've used it with docker. Thanks, Gencer.

3 3

Bluestore - How to review config?
by Dave Hall 07 May '20

07 May '20

Hello, Sorry if this has been asked before... A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance. While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware. I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration. Could anybody point me to a command or link to documentation on this? Thanks. -Dave -- Dave Hall Binghamton University

5 6

How many MDS servers
by Robert Ruge 07 May '20

07 May '20

Quick question Ceph guru's. For a 1.1PB raw cephfs system currently storing 191TB of data and 390 million objects (mostly small Python, ML training files etc.) how many MDS servers should I be running? System is Nautilus 14.2.8. I ask because up to know I have run one MDS with one standby-replay and occasionally it blows up with large memory consumption, 60Gb+ even though I have mds_cache_memory_limit = 32G and that was 16G until recently. It of course tries to restart on another MDS node fails again and after several attempts usually comes back up. Today I increased to two active MDS's but the question is what is the optimal number for a pretty active system? The single MDS seemed to regularly run around 1400 req/s and I often get up to six clients failing to respond to cache pressure. The current setup is: ceph fs status cephfs - 71 clients ====== +------+----------------+--------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+----------------+--------+---------------+-------+-------+ | 0 | active | a | Reqs: 447 /s | 12.0M | 11.9M | | 1 | active | b | Reqs: 154 /s | 1749k | 1686k | | 1-s | standby-replay | c | Evts: 136 /s | 1440k | 1423k | | 0-s | standby-replay | d | Evts: 402 /s | 16.8k | 298 | +------+----------------+--------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 160G | 169G | | cephfs_data | data | 574T | 140T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ | w | | x | | y | | z | +-------------+ MDS version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) Regards. Robert Ruge Systems & Network Manager Faculty of Science, Engineering & Built Environment [cid:image001.png@01D36789.04BE09A0] Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

4 4

Cluster blacklists MDS, can't start
by Robert LeBlanc 07 May '20

07 May '20

This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I try to unblack list them, but they still say this in the logs mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance) Looking at a pg dump, it looks like the epoch is passed that. $ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115] Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up? Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 4

4.14 kernel or greater recommendation for multiple active MDS
by Robert LeBlanc 07 May '20

07 May '20

In the Nautilus manual it recommends >= 4.14 kernel for multiple active MDSes. What are the potential issues for running the 4.4 kernel with multiple MDSes? We are in the process of upgrading the clients, but at times overrun the capacity of a single MDS server. MULTIPLE ACTIVE METADATA SERVERS <https://docs.ceph.com/docs/nautilus/cephfs/kernel-features/#multiple-active…> The feature has been supported since the Luminous release. It is recommended to use Linux kernel clients >= 4.14 when there are multiple active MDS. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 6

Question about bucket versions
by Katarzyna Myrek 07 May '20

07 May '20

Hi I have a few questions about bucket versioning. in the output of command "*radosgw-admin bucket stats --bucket=XXX"* there is info about versions: "ver": "0#521391,1#516042,2#518098,3#517681,4#518423", "master_ver": "0#0,1#0,2#0,3#0,4#0", Also "*metadata get"* returns info about versions: radosgw-admin metadata get bucket:XXX { "key": "bucket:XXX", "ver": { "tag": "_KrvQc6gBg1Zcrr8s8M5jXmk", "ver": 335 }, But I'm pretty sure that bucket versioning should be disabled, because "*aws s3api get-bucket-versioning*" returns nothing. How should I understand the current situation? The problem is that from the client side I can see that the bucket is very small. Less than 10GB while checking the bucket stats from radosgw-admins side shows the bucket is taking nearly 1TB. Kind regards / Pozdrawiam, Katarzyna Myrek

2 1

Re: rados buckets copy
by Andrei Mikhailovsky 07 May '20

07 May '20

Hi Manuel, My replica is 2, hence about 10TB of unaccounted usage. Andrei ----- Original Message ----- > From: "EDH - Manuel Rios" <mriosfer(a)easydatahost.com> > To: "Andrei Mikhailovsky" <andrei(a)arhont.com> > Sent: Tuesday, 28 April, 2020 23:57:20 > Subject: RE: rados buckets copy > Is your replica x3? 9x3 27... plus some overhead rounded.... > > Ceph df show including replicas , bucket stats just bucket usage no replicas. > > -----Mensaje original----- > De: Andrei Mikhailovsky <andrei(a)arhont.com> > Enviado el: miércoles, 29 de abril de 2020 0:55 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] rados buckets copy > > Hello, > > I have a problem with radosgw service where the actual disk usage (ceph df shows > 28TB usage) is way more than reported by the radosgw-admin bucket stats (9TB > usage). I have tried to get to the end of the problem, but no one seems to be > able to help. As a last resort I will attempt to copy the buckets, rename them > and remove the old buckets. > > What is the best way of doing this (probably on a high level) so that the copy > process doesn't carry on the wasted space to the new buckets? > > Cheers > > Andrei > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to > ceph-users-leave(a)ceph.io

2 3

Data loss by adding 2OSD causing Long heartbeat ping times
by Frank Schilder 07 May '20

07 May '20

Dear all, Two days ago I added very few disks to a ceph cluster and run into a problem I have never seen before when doing that. The entire cluster was deployed with mimic 13.2.2 and recently upgraded to 13.2.8. This is the first time I added OSDs under 13.2.8. I had a few hosts that I needed to add 1 or 2 OSDs to and I started with one that needed 1. Procedure was as usual: ceph osd set norebalance deploy additional OSD The OSD came up and PGs started peering, so far so good. To my surprise, however, I started seeing health-warnings about slow ping times: Long heartbeat ping times on back interface seen, longest is 1171.910 msec Long heartbeat ping times on front interface seen, longest is 1180.764 msec After peering it looked like it got better and I waited it out until the messages were gone. This took a really long time, at least 5-10 minutes. I went on to the next host and deployed 2 new OSDs this time. Same as above, but with much worse consequences. Apparently, the ping times exceeded a timeout for a very short moment and an OSD was marked out for ca. 2 seconds. Now all hell broke loose. I got health errors with the dreaded "backfill_toofull", undersized PGs and a large amount of degraded objects. I don't know what is causing what, but I ended up with data loss by just adding 2 disks. We have dedicated network hardware and each of the OSD hosts has 20GBit front and 40GBit back network capacity (LACP trunking). There are currently no more than 16 disks per server. The disks were added to an SSD pool. There was no traffic nor any other exceptional load on the system. I have ganglia resource monitoring on all nodes and cannot see a single curve going up. Network, CPU utilisation, load, everything below measurement accuracy. The hosts and network are quite overpowered and dimensioned to host many more OSDs (in future expansions). I have three questions, ordered by how urgently I need an answer: 1) I need to add more disks next week and need a workaround. Will something like this help avoiding the heartbeat time-out: ceph osd set noout ceph osd set nodown ceph osd set norebalance 2) The "lost" shards of the degraded objects were obviously still on the cluster somewhere. Is there any way to force the cluster to rescan OSDs for the shards that went orphan during the incident? 3) This smells a bit like a bug that requires attention. I was probably just lucky that I only lost 1 shard per PG. Has something similar reported before? Is this fixed in 13.2.10? Is it something new? Any settings that need to be looked at? If logs need to be collected, I can do so during my next attempt. However, I cannot risk data integrity of a production cluster and, therefore, probably not run the original procedure again. Many thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 4

Rados clone_range
by Ali Turan 07 May '20

07 May '20

Hello, I saw there was a clone_range function in librados earlier. But it removed in version 12 I beleive. I need excatly that function to avoid unnecessary network traffic. I need to combine many small objects into one. So clone range would be really useful for me. I can read from an object and write to another but this will cause unnecessary network traffic. How can I do this in new versions of librados ?

1 0

per rbd performance counters
by Void Star Nill 07 May '20

07 May '20

Hello, Is there a way to get read/write I/O statistics for each rbd device for each mapping? For example, when an application uses one of the volumes, I would like to find out what performance (avg read/write bandwidth, IOPS, etc) that application observed on a given volume. Is that possible? Thanks, Shridhar

3 2

changed caps not propagated to kernel cephfs mounts
by Andrej Filipcic 07 May '20

07 May '20

Hi, I have added a new fast_data pool to cephfs, fixed the auth caps, eg client.f9wn key: ........ caps: [mds] allow rw caps: [mon] allow r caps: [osd] allow rw pool=cephfs_data, allow rw pool=fast_data but the client with kernel mounted cephfs reports an error when trying to read or write, eg f9nd003 ~ # echo a > /ceph/grid/cache/a -bash: echo: write error: Operation not permitted mount is: cat /proc/mounts |grep ceph monitors...:/ /ceph ceph rw,relatime,name=f9wn,secret=<hidden>,acl 0 0 it seems that only umount + mount solves this issue, kernel is vanilla 4.19.60. Is there any way to force to propagate new auth capabilities without remounting the fs? Thanks, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

1 0

CephFS with active-active NFS Ganesha
by Michael Bisig 06 May '20

06 May '20

Hi all, I am trying to setup an active-active NFS Ganesha cluster (with two Ganeshas (v3.0) running in Docker containers). I could manage to get two Ganesha daemons running using the rados_cluster backend for active-active deployment. I have the grace db within the cephfs metadata pool in an own namespace which keeps track on the node status. Now, I can mount the exposed filesystem over NFS (v4.1, v4.2) with both daemons. So far so good. __ Testing high availability resulted in an unexpected behavior for that I am not sure whether it is intentional or whether it is a configuration problem. Problem: If both are running, no E or N flags are set within the grace db, as I expect. Once, one host goes down (or is taken down) ALL clients cannot read nor write to the mounted filesystem, even the clients which are not connected to dead ganesha. In the db, I see that the dead ganesha has state NE and the active has E. This state is what I expect from the Ganesha documentation. Nevertheless, I would assume that the clients connected to the active daemon are not blocked. This state is not cleaned up by itself (e.g. after the grace period). I can unlock this situation by 'lifting' the dead node with a direct db call (using ganesha-rados-grace tool). But within an active-active deployment this is not suitable. The ganesha config looks like: ------------ NFS_CORE_PARAM { Enable_NLM = false; Protocols = 4; } NFSv4 { RecoveryBackend = rados_cluster; Minor_Versions = 1,2; } RADOS_KV { pool = "cephfsmetadata"; nodeid = "a" ; namespace = "grace"; UserId = "ganesha"; Ceph_Conf = "/etc/ceph/ceph.conf"; } MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } EXPORT { Export_ID=101; Protocols = 4; Transports = TCP; Path = PATH; Pseudo = PSEUDO_PATH; Access_Type = RW; Attr_Expiration_Time = 0; Squash = no_root_squash; FSAL { Name = CEPH; User_Id = "ganesha"; Secret_Access_Key = CEPHXKEY; } } LOG { Default_Log_Level = "FULL_DEBUG"; } ------------ Does anyone have similar problems? Or if this behavior is by purpose, can you explain to me why this is the case? Thank you in advance for your time and thoughts. Kind regards, Michael

3 2

Re: Workload in Unit testing
by Bobby 06 May '20

06 May '20

Hi Sebastian, Thanks a lot for your reply. It was really helpful and it is clear that 'make check' don't start a ceph cluster. After your email I have figured it out. This brings me to another question :-) In my earlier email I should have defined what exactly I mean by 'workload' in my case. Given my current task/scenario, the definition of 'workload' is only the workload of client machine. Meaning, if there is a Ceph cluster, I am only concerned with the workload of a single ceph client node. And not the workload of other nodes that include OSDs, MONs, MDS etc. The question arises what exactly on ceph client? On client side, I would like to profile the workload of CRUSH. Because I am quite sure there are many computations in CRUSH that are compute intensive for CPU and can be offloaded. May be these compute intensive computations can be more parallelized. This is why I was profiling the binaries of unit tests (in particular CRUSH unit tests) on profiling tool Valgrind --tool=callgrind to see the function calls. May be this is not the right way? Please do comment on it :-). Considering my task, would you still recommend me to use Teuthology tests at this point? Please do comment on this also :-). Because integration tests (Teuthology framework) require multi-machine clusters to run. And according to my understanding that would be too complex for a single client workload or lets say if I am only interested in CRUSH workload. Thanks in advance :-) On Thu, May 7, 2020 at 12:20 AM Bobby <italienisch1987(a)gmail.com> wrote: > > > Hi Sebastian, > > Thanks a lot for your reply. It was really helpful and it is clear that > 'make check' don't start a ceph cluster. After your email I have figured it > out. This brings me to another question :-) > > In my earlier email I should have defined what exactly I mean by > 'workload' in my case. Given my current task/scenario, the definition of > 'workload' is only the workload of client machine. Meaning, if there is a > Ceph cluster, I am only concerned with the workload of a single ceph > client node. And not the workload of other nodes that include OSDs, MONs, > MDS etc. The question arises what exactly on ceph client? On client side, I > would like to profile the workload of CRUSH. Because I am quite sure there > are many computations in CRUSH that are compute intensive for CPU and can > be offloaded. May be these compute intensive computations can be more > parallelized. This is why I was profiling the binaries of unit tests (in > particular CRUSH unit tests) on profiling tool Valgrind --tool=callgrind to > see the function calls. May be this is not the right way? Please do comment > on it :-). > > Considering my task, would you still recommend me to use Teuthology tests > at this point? Please do comment on this also :-). Because integration > tests (Teuthology framework) require multi-machine clusters to run. And > according to my understanding that would be too complex for a single client > workload or lets say if I am only interested in CRUSH workload. > > Thanks in advance :-´) > > Bobby > > On Wed, May 6, 2020 at 5:37 PM Sebastian Wagner <sebastian.wagner(a)suse.com> > wrote: > >> Hi Bobby, >> >> `make check` aka unit tests don't start a ceph cluster. Instead they test >> individual functions. There is nothing similar to a "workload" involved >> here. >> >> Maybe, you're interested in the vstart_runner, which makes it possible to >> run Teuthology tests in a vstart cluster. >> >> Best, >> >> Sebastian >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> >

1 0

Re: What's the best practice for Erasure Coding
by Alex Gorbachev 06 May '20

06 May '20

Hi Frank, Reviving this old thread as to whether the performance on these raw NL-SAS drives is adequate? I was wondering if this is a deep archive with almost no retrieval, or how many drives are used? In my experience with large parallel writes, WAL/DB with bluestore, or journal drives on SSD with filestore have always been needed to sustain a reasonably consistent transfer rate. Very much appreciate any reference info as to your design. Best regards, Alex On Mon, Jul 8, 2019 at 4:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > Hi David, >> >> I'm running a cluster with bluestore on raw devices (no lvm) and all >> journals collocated on the same disk with the data. Disks are spinning >> NL-SAS. Our goal was to build storage at lowest cost, therefore all data on >> HDD only. I got a few SSDs that I'm using for FS and RBD meta data. All >> large pools are EC on spinning disk. >> >> I spent at least one month to run detailed benchmarks (rbd bench) >> depending on EC profile, object size, write size, etc. Results were varying >> a lot. My advice would be to run benchmarks with your hardware. If there >> was a single perfect choice, there wouldn't be so many options. For >> example, my tests will not be valid when using separate fast disks for WAL >> and DB. >> >> There are some results though that might be valid in general: >> >> 1) EC pools have high throughput but low IOP/s compared with replicated >> pools >> >> I see single-thread write speeds of up to 1.2GB (gigabyte) per second, >> which is probably the network limit and not the disk limit. IOP/s get >> better with more disks, but are way lower than what replicated pools can >> provide. On a cephfs with EC data pool, small-file IO will be comparably >> slow and eat a lot of resources. >> >> 2) I observe massive network traffic amplification on small IO sizes, >> which is due to the way EC overwrites are handled. This is one bottleneck >> for IOP/s. We have 10G infrastructure and use 2x10G client and 4x10G OSD >> network. OSD bandwidth at least 2x client network, better 4x or more. >> >> 3) k should only have small prime factors, power of 2 if possible >> >> I tested k=5,6,8,10,12. Best results in decreasing order: k=8, k=6. All >> other choices were poor. The value of m seems not relevant for performance. >> Larger k will require more failure domains (more hardware). >> >> 4) object size matters >> >> The best throughput (1M write size) I see with object sizes of 4MB or >> 8MB, with IOP/s getting somewhat better with slower object sizes but >> throughput dropping fast. I use the default of 4MB in production. Works >> well for us. >> >> 5) jerasure is quite good and seems most flexible >> >> jerasure is quite CPU efficient and can handle smaller chunk sizes than >> other plugins, which is preferrable for IOP/s. However, CPU usage can >> become a problem and a plugin optimized for specific values of k and m >> might help here. Under usual circumstances I see very low load on all OSD >> hosts, even under rebalancing. However, I remember that once I needed to >> rebuild something on all OSDs (I don't remember what it was, sorry). In >> this situation, CPU load went up to 30-50% (meaning up to half the cores >> were at 100%), which is really high considering that each server has only >> 16 disks at the moment and is sized to handle up to 100. CPU power could >> become a bottle for us neck in the future. >> >> These are some general observations and do not replace benchmarks for >> specific use cases. I was hunting for a specific performance pattern, which >> might not be what you want to optimize for. I would recommend to run >> extensive benchmarks if you have to live with a configuration for a long >> time - EC profiles cannot be changed. >> >> We settled on 8+2 and 6+2 pools with jerasure and object size 4M. We also >> use bluestore compression. All meta data pools are on SSD, only very little >> SSD space is required. This choice works well for the majority of our use >> cases. We can still build small expensive pools to accommodate special >> performance requests. >> >> Best regards, >> >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> >> ________________________________________ >> From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of David < >> xiaomajia.st(a)gmail.com> >> Sent: 07 July 2019 20:01:18 >> To: ceph-users(a)lists.ceph.com >> Subject: [ceph-users] What's the best practice for Erasure Coding >> >> Hi Ceph-Users, >> >> I'm working with a Ceph cluster (about 50TB, 28 OSDs, all Bluestore on >> lvm). >> Recently, I'm trying to use the Erasure Code pool. >> My question is "what's the best practice for using EC pools ?". >> More specifically, which plugin (jerasure, isa, lrc, shec or clay) >> should I adopt, and how to choose the combinations of (k,m) (e.g. >> (k=3,m=2), (k=6,m=3) ). >> >> Does anyone share some experience? >> >> Thanks for any help. >> >> Regards, >> David >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users(a)lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >

2 1

radosgw garbage collection error
by James, GleSYS 06 May '20

06 May '20

Hi, We’ve recently installed a new Ceph cluster running Octopus 15.2.1, and we’re using RGW with an erasure coded backed pool. I started to get a suspicion that deleted objects were not getting cleaned up properly, and I wanted to verify this by checking the garbage collector. That’s when I discovered that when I run “radosgw-admin gc list”, I get the following error: "ERROR: failed to list objs: (22) Invalid argument” When running the command with the debug-rgw=20 flag, I see a bit more information: 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=0 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=1 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=2 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=3 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=4 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=5 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=6 2020-05-05T18:39:19.455+0000 7f3312d82080 20 add_watcher() i=7 2020-05-05T18:39:19.455+0000 7f3312d82080 2 all 8 watchers are set, enabling cache 2020-05-05T18:39:19.455+0000 7f3312d82080 20 check_secure_mon_conn(): auth registy supported: methods=[2,1] modes=[2,1] 2020-05-05T18:39:19.455+0000 7f3312d82080 20 check_secure_mon_conn(): method 1 is insecure 2020-05-05T18:39:19.455+0000 7f32d4fd9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-05-05T18:39:19.519+0000 7f3246ffd700 20 reqs_thread_entry: start 2020-05-05T18:39:19.519+0000 7f3312d82080 20 init_complete bucket index max shards: 11 2020-05-05T18:39:19.519+0000 7f3244ff9700 20 reqs_thread_entry: start 2020-05-05T18:39:19.519+0000 7f323affd700 20 reqs_thread_entry: start ERROR: failed to list objs: (22) Invalid argument 2020-05-05T18:39:19.523+0000 7f32d4fd9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-05-05T18:39:19.523+0000 7f3312d82080 20 remove_watcher() i=0 2020-05-05T18:39:19.523+0000 7f3312d82080 2 removed watcher, disabling cache 2020-05-05T18:39:19.523+0000 7f3312d82080 20 remove_watcher() i=1 2020-05-05T18:39:19.523+0000 7f3312d82080 20 remove_watcher() i=2 2020-05-05T18:39:19.527+0000 7f3312d82080 20 remove_watcher() i=3 2020-05-05T18:39:19.527+0000 7f3312d82080 20 remove_watcher() i=4 2020-05-05T18:39:19.527+0000 7f3312d82080 20 remove_watcher() i=5 2020-05-05T18:39:19.527+0000 7f3312d82080 20 remove_watcher() i=6 2020-05-05T18:39:19.527+0000 7f3312d82080 20 remove_watcher() I=7 I find very little information regarding this error, so I wondered if someone here could help me troubleshoot the issue? Thanks, James.

2 4

Error with zabbix module on Ceph Octopus
by Matt Larson 06 May '20

06 May '20

Hi, I am trying to setup the Zabbix reporting module, but it is giving an error which looks like a Python error: ceph zabbix config-show Error EINVAL: TypeError: __init__() got an unexpected keyword argument 'index' I have configured the zabbix_host and identifier already at this point. The command: 'ceph zabbix send' also fails to run with 'Failed to send data to Zabbix' I am running: - CentOS 8.1 - ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) - Zabbix 4.4-1.el8 release (https://repo.zabbix.com/zabbix/4.4/rhel/8/x86_64/zabbix-release-4.4-1.el8.n…) - Python version 3.6.8 Any suggestions? I am wondering if this could be requiring Python 2.7 to run? -- Matt Larson, PhD Madison, WI 53705 U.S.A.

1 0

Re: ceph: Can't lookup inode 1 (err: -13)
by Patrick Donnelly 06 May '20

06 May '20

On Mon, Mar 9, 2020 at 3:19 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > For testing purposes I changed the kernel 3.10 for a 5.5, now I am > getting these messages. I assume the 3.10 was just never displaying > these. Could this be a problem with my caps of the fs id user? > > [Mon Mar 9 23:10:52 2020] ceph: Can't lookup inode 1 (err: -13) > [Mon Mar 9 23:12:03 2020] ceph: Can't lookup inode 1 (err: -13) > [Mon Mar 9 23:13:12 2020] ceph: Can't lookup inode 1 (err: -13) > [Mon Mar 9 23:14:19 2020] ceph: Can't lookup inode 1 (err: -13) For posterity, a tracker was opened for this bug: https://tracker.ceph.com/issues/44546 -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

RBD throughput/IOPS benchmarks
by Vincent KHERBACHE 06 May '20

06 May '20

Dear all, We currently run a small Ceph cluster on 2 machines and we wonder what are the theoretical max BW/IOPS we can achieve through RBD with our setup. Here are the environment details: - The Ceph release is an octopus 15.2.1 running on Centos 8, both machines have 180GB RAM, 72 cores, and 40 * 1.8TB SSD disks each - Regarding network we deployed two isolated 100Gb/s networks for front and back connectivity - Since all disks have the same performance, we created 1 OSD per SSD using bluestore (default setup with LVM) to reach a total of 80 OSDs (40 OSD per machine) - On top of that we have a single 2x replicated RBD pool with 2048 PGs in order to reach a global average of 50 PGs per OSD (our experiments with 100 PGs/OSD didn't provided perfomance improvement, only extra CPU consumption) - We kept default settings for all RBD images we created for benchmarks (4MB obj size, 4MB stripe width, 1 stripe) - The crush map and replication rules used are very simple (2 hosts, 40 OSDs per host with same device class and weight) - All tuning settings (caches sizing, op threads, bluestore, rocksdb options, etc.) are the default options provided with the Octopus release. Here are the best values observed so far using both rados bench and fio with many different setup (varying amount of clients, threads, RBD images, bloc sizes from 4k to 4m, random/sequential, iodepth, etc.): - Read BW: 24GB/s (looks like we reached the maximum network capacity of both machines here) - Read IOPS: 600k - Write BW: 7 GB/s - Write IOPS: 100k Those are simply the maximum numbers obtained regardless latency as we first want to stress the infrastructure to see what are the maximum thoughput & IOPS we can achieve. Latency care/measurements will come after. We also have the feelings that the 2x replication of the RBD pool is a big deal with only 2 nodes in the cluster, dividing maximum speeds by more than 2. This will probably have much less impact when scaling up the cluster with new nodes. We also noticed that at some point during recovery operations (eg. rebalancing PGs after new OSD was added into the pool) the total read/write throughput and IOPS are climbing to several GB/s and millions IOPS, so we wonder if we can achieve any better with legitimate RBD clients load. Do you guys would like to share numbers of your setup or have any hints for potential improvements? Thanks. Regards, -- Vincent Kherbache R&D Director Titan Datacenter

1 0

State of SMR support in Ceph?
by Oliver Freyermuth 06 May '20

06 May '20

Dear Cephalopodians, seeing the recent moves of major HDD vendors to sell SMR disks targeted for use in consumer NAS devices (including RAID systems), I got curious and wonder what the current status of SMR support in Bluestore is. Of course, I'd expect disk vendors to give us host-managed SMR disks for data center use cases (and to tell us when actually they do so...), but in that case, Bluestore surely needs some new intelligence for best performance in the shingled ages. I had a quick look at the repository and could only make out that libzbc has been added some years ago, but no activity after this (also no tickets in the issue tracker). Is this still something on the roadmap? It would be wonderful for backup / archiving / mostly data ingest / cold storage clusters to be able to use cheaper and larger disks once they become available :-). I'm still positively optimistic for such use cases even though my personal experience with SMR has not been so well up to now: I bought a (not cleanly labelled...) DM-SMR, used it for BTRFS archiving (btrbk, i.e. btrfs-send and -receive), and after it was filled once, it got excruciatinly slow (less than a few kiB/s even when changing only a few 100 MB after prolonged idle). But then, deleting btrfs snapshots of a desktop OS is pure random read/write access, and there are no optimizations for that use case in btrfs (and the drive did not even support TRIM/DISCARD), and I read on the btrfs list now that it can work well even in arrays mostly used for cold storage if the balancing is throttled ;-). Cheers, Oliver

4 5

Re: Cephfs snapshots in Nautilus
by Marc Roos 06 May '20

06 May '20

I have been using snapshots on cephfs since luminous, 1xfs and 1xactivemds and used an rsync on it for backup. Under luminious I did not encounter any problems with this setup. I think I was even snapshotting user dirs every 7 days having thousands of snapshots (which I later heard, is not recommend and one should stick below 400 or so?) When upgrading to nautilus, this snapshot feature was disabled (that is default in the upgrade). Did not notice nor expected this. When I enabled again snapshotting. I had problems with the rsync backup. So I reverted back to the slower ceph-fuse mount. I also brought down the snapshots to 36, but I am still stuck with "clients failing to respond to capability release", "clients failing to respond to cache pressure" and "MDSs report slow requests" Which is odd, since my use did not change since luminous. All in all is fine, but what I do not like is, that such a thing can happen between upgrades. -----Original Message----- From: Stolte, Felix [mailto:f.stolte@fz-juelich.de] Sent: 06 May 2020 09:09 To: ceph-users(a)ceph.io Subject: [ceph-users] Cephfs snapshots in Nautilus Hi Folks, I really like to use snapshots on cephfs, but even on octopus release snapshots are still marked as an experimental feature. Is anyone using snapshots in production environments? Which issues did you encounter? Do I risk a corrupted filesystem or just non-working snapshots? We run a single fs with one active mds. Best regards Felix ------------------------------------------------------------------------ ------------- ------------------------------------------------------------------------ ------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------ ------------- ------------------------------------------------------------------------ -------------

2 1

Fwd: Octopus on CentOS 7: lacking some packages
by Sam Huracan 06 May '20

06 May '20

Hi Cephers, I am trying to install Ceph Octopus using ceph-deploy on CentOS 7, as installing ceph-mgr-dashboard, it required these packages: - python3-cherrypy - python3-jwt - python3-routes https://pastebin.com/dSQPgGJD But, when I installed these packages, they were not available. https://pastebin.com/Hguf8kJe Is there anyone installed Octopus successfully on CentOS 7? Can you help me? Thanks in advance.

2 1

Re: Cephfs snapshots in Nautilus
by Yan, Zheng 06 May '20

06 May '20

On Wed, May 6, 2020 at 3:53 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I have been using snapshots on cephfs since luminous, 1xfs and > 1xactivemds and used an rsync on it for backup. > Under luminious I did not encounter any problems with this setup. I > think I was even snapshotting user dirs every 7 days having thousands of > snapshots (which I later heard, is not recommend and one should stick > below 400 or so?) > > When upgrading to nautilus, this snapshot feature was disabled (that is > default in the upgrade). Did not notice nor expected this. When I > enabled again snapshotting. I had problems with the rsync backup. So I > reverted back to the slower ceph-fuse mount. I also brought down the > snapshots to 36, but I am still stuck with "clients failing to respond > to capability release", "clients failing to respond to cache pressure" > and "MDSs report slow requests" > Which is odd, since my use did not change since luminous. > please open tracker tickets for these > All in all is fine, but what I do not like is, that such a thing can > happen between upgrades. > > > > > -----Original Message----- > From: Stolte, Felix [mailto:f.stolte@fz-juelich.de] > Sent: 06 May 2020 09:09 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Cephfs snapshots in Nautilus > > Hi Folks, > > > > I really like to use snapshots on cephfs, but even on octopus release > snapshots are still marked as an experimental feature. Is anyone using > snapshots in production environments? Which issues did you encounter? Do > I risk a corrupted filesystem or just non-working snapshots? > > > > We run a single fs with one active mds. > > > > Best regards > > Felix > > ------------------------------------------------------------------------ > ------------- > > ------------------------------------------------------------------------ > ------------- > > Forschungszentrum Juelich GmbH > > 52425 Juelich > > Sitz der Gesellschaft: Juelich > > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt > > ------------------------------------------------------------------------ > ------------- > > ------------------------------------------------------------------------ > ------------- > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Cephfs snapshots Nautilus
by Stolte, Felix 06 May '20

06 May '20

Hi Folks, I really like to use snapshots on cephfs, but even on octopus release snapshots are still marked as an experimental feature. Is anyone using snapshots in production environments? Which issues did you encounter? Do I risk a corrupted filesystem or just non-working snapshots? We run a single fs with one active mds. Best regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

1 0

nautilus cluster not dynamically resharding
by Marcel Ceph 06 May '20

06 May '20

Hi all, I have 3 nautilus clusters. They started out as mimic but were recently upgraded to nautilus. On two of them dynamic bucket index sharding seems to work automatically. However on one of the clusters it doesn;t and I have no clue why * If I execute radosgw-admin bucket limit check I find some buckets over the default limit { "bucket": "acc-dump", "tenant": "acc", "num_objects": 137290, "num_shards": 0, "objects_per_shard": 137290, "fill_status": "OVER 100.000000%" } * the config value for rgw_dynamic_resharding is true * on: 'radosgw-admin reshard stale-instances list' I get an error: Resharding disabled in a multisite env, stale instances unlikely from resharding These instances may not be safe to delete. Use --yes-i-really-mean-it to force displaying these instances. (This does not happen on the other two clusters) However this is a single site cluster radosgw-admin realm list { "default_info": "e724bd71-31eb-45c8-a456-151f6a5aa8b5", "realms": [ "intern" ] } radosgw-admin zone list { "default_info": "7f9bebd6-a9cf-4006-83b1-ff99391aacc0", "zones": [ "dc3-intern" ] } radosgw-admin zonegroup list { "default_info": "ce4329ae-2bc8-4117-9b82-271022b223fa", "zonegroups": [ "dc3" ] } * on: 'radosgw-admin reshard list' I get loads of errors [2020-05-06 10:08:49.261 7f1790b607c0 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000000 2020-05-06 10:08:49.265 7f1790b607c0 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000001 etc I've gone through the documentation and some blogs but I have no clue how to tackle this problem. Any help would be much appreciated Thanks Marcel

1 0

osd won't start
by Mazzystr 06 May '20

06 May '20

I've been using CentOS 7 and 5.6.10-1.el7.elrepo.x86_64 Linux kernel. After today's update and reboot osd's wont start. # podman run --privileged --pid=host --cpuset-cpus 0,1 --memory 2g --name ceph_osd0 --hostname ceph_osd0 --ip 172.30.0.10 -v /dev:/dev -v /etc/localtime:/etc/localtime:ro -v /etc/ceph:/etc/ceph/ -v /var/lib/ceph/osd/ceph-0:/var/lib/ceph/osd/ceph-0 -v /dev/:/dev/ -e OSD_DEVICE=/dev/mapper/ceph-0block -e OSD_BLUESTORE=1 -e CLUSTER=ceph -e OSD_ID=0 -e DEBUG=stayalive ceph/daemon:v5.0.1-stable-5.0-octopus-centos-8-x86_64 osd_directory_single 2020-05-05 20:19:54 /opt/ceph-container/bin/entrypoint.sh: STAYALIVE: container will not die if a command fails. 2020-05-05 20:19:54 /opt/ceph-container/bin/entrypoint.sh: static: does not generate config 2020-05-05 20:19:54 /opt/ceph-container/bin/entrypoint.sh: Looks like OSD: 0 is not started, starting it... 2020-05-05 20:19:54 /opt/ceph-container/bin/entrypoint.sh: SUCCESS exec: PID 45073: spawning ceph-osd --cluster ceph --setuser ceph --setgroup ceph --log-to-stderr=true --err-to-stderr=true --default-log-to-file=false --foreground -i 0 -k /var/lib/ceph/osd/ceph-0//keyring exec: Waiting 45073 to quit 2020-05-05T20:19:54.932-0400 7fb999290ec0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2020-05-05T20:19:54.934-0400 7fb999290ec0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2020-05-05T20:19:54.934-0400 7fb999290ec0 0 set uid:gid to 167:167 (ceph:ceph) 2020-05-05T20:19:54.934-0400 7fb999290ec0 0 ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable), process ceph-osd, pid 45073 2020-05-05T20:19:54.934-0400 7fb999290ec0 0 pidfile_write: ignore empty --pid-file 2020-05-05T20:19:54.934-0400 7fb999290ec0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2020-05-05T20:19:54.934-0400 7fb999290ec0 5 object store type is bluestore 2020-05-05T20:19:54.935-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2020-05-05T20:19:54.935-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2020-05-05T20:19:54.935-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported 2020-05-05T20:19:54.935-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 2020-05-05T20:19:54.935-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block.db type kernel 2020-05-05T20:19:54.935-0400 7fb999290ec0 1 bdev(0x561a009e0700 /var/lib/ceph/osd/ceph-0/block.db) open path /var/lib/ceph/osd/ceph-0/block.db 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bdev(0x561a009e0700 /var/lib/ceph/osd/ceph-0/block.db) open size 49998200832 (0xba4200000, 47 GiB) block_size 4096 (4 KiB) non-rotational discard not supported 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block.db size 47 GiB 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bdev(0x561a009e0a80 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bdev(0x561a009e0a80 /var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-0/block size 5.5 TiB 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block.wal type kernel 2020-05-05T20:19:54.936-0400 7fb999290ec0 1 bdev(0x561a009e0e00 /var/lib/ceph/osd/ceph-0/block.wal) open path /var/lib/ceph/osd/ceph-0/block.wal 2020-05-05T20:19:54.937-0400 7fb999290ec0 1 bdev(0x561a009e0e00 /var/lib/ceph/osd/ceph-0/block.wal) open size 997195776 (0x3b700000, 951 MiB) block_size 4096 (4 KiB) non-rotational discard not supported 2020-05-05T20:19:54.937-0400 7fb999290ec0 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-0/block.wal size 951 MiB 2020-05-05T20:19:54.937-0400 7fb999290ec0 1 bdev(0x561a009e0e00 /var/lib/ceph/osd/ceph-0/block.wal) close 2020-05-05T20:19:55.214-0400 7fb999290ec0 1 bdev(0x561a009e0700 /var/lib/ceph/osd/ceph-0/block.db) close 2020-05-05T20:19:55.453-0400 7fb999290ec0 1 bdev(0x561a009e0a80 /var/lib/ceph/osd/ceph-0/block) close 2020-05-05T20:19:55.714-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) close 2020-05-05T20:19:55.956-0400 7fb999290ec0 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2020-05-05T20:19:55.963-0400 7fb999290ec0 0 load: jerasure load: lrc load: isa 2020-05-05T20:19:55.963-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2020-05-05T20:19:55.963-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2020-05-05T20:19:55.963-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported 2020-05-05T20:19:55.963-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 2020-05-05T20:19:55.963-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) close 2020-05-05T20:19:56.235-0400 7fb999290ec0 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2020-05-05T20:19:56.235-0400 7fb999290ec0 0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2020-05-05T20:19:56.235-0400 7fb999290ec0 0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2020-05-05T20:19:56.235-0400 7fb999290ec0 0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2020-05-05T20:19:56.235-0400 7fb999290ec0 0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2020-05-05T20:19:56.235-0400 7fb999290ec0 2 osd.0 0 init /var/lib/ceph/osd/ceph-0 (looks like hdd) 2020-05-05T20:19:56.235-0400 7fb999290ec0 2 osd.0 0 journal /var/lib/ceph/osd/ceph-0/journal 2020-05-05T20:19:56.235-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 2020-05-05T20:19:56.235-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2020-05-05T20:19:56.235-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2020-05-05T20:19:56.236-0400 7fb999290ec0 1 bdev(0x561a009e0000 /var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported 2020-05-05T20:19:56.236-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 2020-05-05T20:19:56.236-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block.db type kernel 2020-05-05T20:19:56.236-0400 7fb999290ec0 1 bdev(0x561a009e0e00 /var/lib/ceph/osd/ceph-0/block.db) open path /var/lib/ceph/osd/ceph-0/block.db 2020-05-05T20:19:56.236-0400 7fb999290ec0 1 bdev(0x561a009e0e00 /var/lib/ceph/osd/ceph-0/block.db) open size 49998200832 (0xba4200000, 47 GiB) block_size 4096 (4 KiB) non-rotational discard not supported 2020-05-05T20:19:56.236-0400 7fb999290ec0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block.db size 47 GiB 2020-05-05T20:19:56.242-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2020-05-05T20:19:56.242-0400 7fb999290ec0 1 bdev(0x561a009e0700 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bdev(0x561a009e0700 /var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-0/block size 5.5 TiB 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bdev create path /var/lib/ceph/osd/ceph-0/block.wal type kernel 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bdev(0x561a009e0a80 /var/lib/ceph/osd/ceph-0/block.wal) open path /var/lib/ceph/osd/ceph-0/block.wal 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bdev(0x561a009e0a80 /var/lib/ceph/osd/ceph-0/block.wal) open size 997195776 (0x3b700000, 951 MiB) block_size 4096 (4 KiB) non-rotational discard not supported 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-0/block.wal size 951 MiB 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option compaction_readahead_size = 2097152 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option compression = kNoCompression 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option max_background_compactions = 2 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option max_write_buffer_number = 4 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option min_write_buffer_number_to_merge = 1 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option recycle_log_file_num = 4 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option writable_file_max_buffer_size = 0 2020-05-05T20:19:56.243-0400 7fb999290ec0 0 set rocksdb option write_buffer_size = 268435456 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bluefs mount 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bluefs _init_alloc id 0 alloc_size 0x100000 size 0x3b700000 2020-05-05T20:19:56.243-0400 7fb999290ec0 1 bluefs _init_alloc id 1 alloc_size 0x100000 size 0xba4200000 2020-05-05T20:19:56.244-0400 7fb999290ec0 1 bluefs _init_alloc id 2 alloc_size 0x10000 size 0x57541b00000 2020-05-05T20:19:56.343-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db set db_paths to db,47498290790 db.slow,5701112797593 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option compaction_readahead_size = 2097152 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option compression = kNoCompression 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option max_background_compactions = 2 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option max_write_buffer_number = 4 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option min_write_buffer_number_to_merge = 1 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option recycle_log_file_num = 4 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option writable_file_max_buffer_size = 0 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option write_buffer_size = 268435456 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option compaction_readahead_size = 2097152 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option compression = kNoCompression 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option max_background_compactions = 2 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option max_write_buffer_number = 4 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option min_write_buffer_number_to_merge = 1 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option recycle_log_file_num = 4 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option writable_file_max_buffer_size = 0 2020-05-05T20:19:56.343-0400 7fb999290ec0 0 set rocksdb option write_buffer_size = 268435456 2020-05-05T20:19:56.344-0400 7fb999290ec0 1 rocksdb: do_open column families: [default] 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: RocksDB version: 6.1.2 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Compile date Apr 8 2020 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: DB SUMMARY 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: CURRENT file: CURRENT 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: IDENTITY file: IDENTITY 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: MANIFEST file: MANIFEST-004159 size: 115013 Bytes 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: SST files in db dir, Total Num: 174, files: 001179.sst 002875.sst 002961.sst 003070.sst 003102.sst 003217.sst 003290.sst 003324.sst 003399.sst 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Write Ahead Log file in db.wal: 005071.log size: 254490391 ; 005073.log size: 254377461 ; 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.error_if_exists: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.create_if_missing: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.paranoid_checks: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.env: 0x5619ffd19b80 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.info_log: 0x5619ffd234e0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_file_opening_threads: 16 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.statistics: (nil) 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.use_fsync: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_log_file_size: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.log_file_time_to_roll: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.keep_log_file_num: 1000 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.recycle_log_file_num: 4 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.allow_fallocate: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.allow_mmap_reads: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.allow_mmap_writes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.use_direct_reads: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.create_missing_column_families: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.db_log_dir: 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.wal_dir: db.wal 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.table_cache_numshardbits: 6 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_subcompactions: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_background_flushes: -1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.WAL_ttl_seconds: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.WAL_size_limit_MB: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.is_fd_close_on_exec: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.advise_random_on_open: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.db_write_buffer_size: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.write_buffer_manager: 0x561a00a2b650 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.use_adaptive_mutex: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.rate_limiter: (nil) 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.wal_recovery_mode: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.enable_thread_tracking: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.enable_pipelined_write: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.row_cache: None 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.wal_filter: None 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.allow_ingest_behind: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.preserve_deletes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.two_write_queues: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.manual_wal_flush: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.atomic_flush: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_background_jobs: 2 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_background_compactions: 2 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.delayed_write_rate : 16777216 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_total_wal_size: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.stats_dump_period_sec: 600 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.stats_persist_period_sec: 600 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_open_files: -1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bytes_per_sync: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.wal_bytes_per_sync: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_readahead_size: 2097152 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Compression algorithms supported: 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kZSTD supported: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kXpressCompression supported: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kLZ4HCCompression supported: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kLZ4Compression supported: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kBZip2Compression supported: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kZlibCompression supported: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: kSnappyCompression supported: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Fast CRC32 supported: Supported on x86 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: [db/db_impl_readonly.cc:22] Opening the db in read only mode 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: db/MANIFEST-004159 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]: 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_filter: None 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_filter_factory: None 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.memtable_factory: SkipListFactory 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.table_factory: BlockBasedTable 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5619ffcb7360) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5619ffcfae10 block_cache_name: BinnedLRUCache block_cache_options: capacity : 429496729 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.write_buffer_size: 268435456 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_write_buffer_number: 4 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression: NoCompression 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression: Disabled 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.prefix_extractor: nullptr 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.num_levels: 7 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression_opts.window_bits: -14 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression_opts.level: 32767 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression_opts.strategy: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compression_opts.enabled: false 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.target_file_size_base: 67108864 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.target_file_size_multiplier: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.arena_block_size: 33554432 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.disable_auto_compactions: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.table_properties_collectors: 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.inplace_update_support: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.inplace_update_num_locks: 10000 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.memtable_huge_page_size: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.bloom_locality: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.max_successive_merges: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.optimize_filters_for_hits: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.paranoid_file_checks: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.force_consistency_checks: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.report_bg_io_stats: 0 2020-05-05T20:19:56.345-0400 7fb999290ec0 4 rocksdb: Options.ttl: 0 2020-05-05T20:19:56.352-0400 7fb999290ec0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:db/MANIFEST-004159 succeeded,manifest_file_number is 4159, next_file_number is 5076, last_sequence is 236178180, log_number is 5073,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2020-05-05T20:19:56.352-0400 7fb999290ec0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 5073 2020-05-05T20:19:56.352-0400 7fb999290ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1588724396353552, "job": 1, "event": "recovery_started", "log_files": [5071, 5073]} 2020-05-05T20:19:56.352-0400 7fb999290ec0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #5071 mode 0 2020-05-05T20:19:57.061-0400 7fb999290ec0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #5073 mode 0 2020-05-05T20:19:57.451-0400 7fb999290ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1588724397452521, "job": 1, "event": "recovery_finished"} 2020-05-05T20:19:57.451-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 2020-05-05T20:19:57.477-0400 7fb999290ec0 1 freelist init 2020-05-05T20:19:57.563-0400 7fb999290ec0 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc opening allocation metadata teardown: managing teardown after SIGTERM teardown: Sending SIGTERM to PID 51610 teardown: Waiting PID 51610 to terminate ./opt/ceph-container/bin/docker_exec.sh: line 32: 51610 Terminated "$@" teardown: Process 51610 is terminated ERRO[0006] Cannot get exit code: error getting exit file for container bbf03528bb8e9b7bd09e4b4215aaa3b24f9f46479f6181100994d9c7be080986: stat /var/run/libpod/exits/bbf03528bb8e9b7bd09e4b4215aaa3b24f9f46479f6181100994d9c7be080986-old: no such file or directory octapus-latest image displays the same behavior. Any help or guidance would be appreciated. /Chris

1 0

Workload in Unit testing
by Bobby 05 May '20

05 May '20

Hi all, Ceph documentation mentions it has two types of tests: *unit tests* (also called make check tests) and *integration tests*. Strictly speaking, the *make check tests* are not “unit tests”, but rather tests that can be run easily on a single build machine after compiling Ceph from source . unit tests: https://github.com/ceph/ceph/tree/master/src/test In order to develop on ceph, I am using a Ceph utility, *vstart.sh*, which allows me to deploy fake local cluster for development purpose. I am doing unit testing. And these tests are helping me. Thanks ! My question: How real and big is the workload of unit tests? Are these tests enough for profiling function calls count, loop counts, parallelism to a good extent? Thanks in advance ! BR Bobby !

1 0

How to apply ceph.conf changes using new tool cephadm
by Gencer W. Genç 05 May '20

05 May '20

Hi, I just deployed a new cluster with cephadm instead of ceph-deploy. In tyhe past, If i change ceph.conf for tweaking, i was able to copy them and apply to all servers. But i cannot find this on new cephadm tool. I did few changes on ceph.conf but ceph is unaware of those changes. How can i apply them? I've used it with docker. Thanks, Gencer.

3 2

Re: RGW and the orphans
by Katarzyna Myrek 05 May '20

05 May '20

Hi Eric, Would it be possible to use it with an older cluster version (like running new radosgw-admin in the container, connecting to the cluster on 14.2.X)? Kind regards / Pozdrawiam, Katarzyna Myrek czw., 16 kwi 2020 o 19:58 EDH - Manuel Rios <mriosfer(a)easydatahost.com> napisał(a): > > Hi Eric, > > > > Are there any ETA for get those script backported maybe in 14.2.10? > > > > Regards > > Manuel > > > > > > De: Eric Ivancich <ivancich(a)redhat.com> > Enviado el: jueves, 16 de abril de 2020 19:05 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl>; EDH - Manuel Rios <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] RGW and the orphans > > > > There is currently a PR for an “orphans list” capability. I’m currently working on the testing side to make sure it’s part of our teuthology suite. > > > > See: https://github.com/ceph/ceph/pull/34148 > > > > Eric > > > > > > On Apr 16, 2020, at 9:26 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi > > Thanks for the quick response. > > To be honest my cluster is getting full because of that trash and I am > at the point where I have to do the removal manually ;/. > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> > Enviado el: jueves, 16 de abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

4 9

asynchronous/non-sequential example read and write test codes Librados
by Bobby 05 May '20

05 May '20

Hi Cephers, I am working on Ceph librados. Currently I can test sequential/synchronous read and write tests both in C and C++. However I am struggling with asynchronous/non-sequential test codes. Are there any test repositories which contain asynchronous/non-sequential examples codes? Thanks in advance Bobby !

2 2

page cache flush before unmap?
by Void Star Nill 04 May '20

04 May '20

Hello, I wanted to know if rbd will flush any writes in the page cache when a volume is "unmap"ed on the host, of if we need to flush explicitly using "sync" before unmap? Thanks, Shridhar

2 2

mount issues with rbd running xfs - Structure needs cleaning
by Void Star Nill 04 May '20

04 May '20

Hello All, One of the use cases (e.g. machine learning workloads) for RBD volumes in our production environment is that, users could mount an RBD volume in RW mode in a container, write some data to it and later use the same volume in RO mode into a number of containers in parallel to consume the data. I am trying to test this scenario with different file systems (ext3/4 and xfs). I have an automated test code that creates a volume, maps it to a node, mounts in RW mode and write some data into it. Later the same volume is mounted in RO mode in a number of other nodes and a process reads from the file. I dont see any issues with ext3 or 4 filesystems, but with XFS, I notice that 1 or 2 (out of 6) parallel read-only mounts fail with "Structure needs cleaning" error. What is surprising is that, the rest of 4 or 5 mounts will be successful and I dont see any I/O issues on those - which suggests that there shouldn't be any corruptions on the volume itself. Also note that there is no other process writing to the volume at this time so no chance of corruption that way. I am doing xfs mounts with "ro,nouuid" mount options. Any inputs on why I may be seeing this issue randomly? Regards, Shridhar

6 7

pg_autoscaler on cache will not work
by Alex L 04 May '20

04 May '20

I activated autoscaler on all my pools but found this error msg when it came to the cache_pool. ceph-mgr.pve22.log.3:2020-05-01 23:59:24.014 7f5120eda700 0 mgr[pg_autoscaler] pg_num adjustment on cache_pool to 512 failed: (-1, '', 'splits in cache pools must be followed by scrubs and leave sufficient free space to avoid overfilling. use --yes-i-really-mean-it to force.') I ended up clearing the cache pool by limiting it to max 1 object and increasing the pg_num/pgp_num. Now I expect the scrub to being every 24h automatically. Should this be included in the docs? Is this a bug of pg_autoscaler ? BR, Alex

1 0

Upgrade Luminous to Nautilus on a Debian system
by Herve Ballans 04 May '20

04 May '20

Hi all, I'm planning to upgrade one on my Ceph Cluster currently on Luminous 12.2.13 / Debian Stretch (updated). On this cluster, Luminous is packaged from the official Ceph repo (deb https://download.ceph.com/debian-luminous/ stretch main) I would like to upgrade it with Debian Buster and Nautilus using the croit.io repository (deb https://mirror.croit.io/debian-nautilus/ buster main) I already prepared the steps procedure but I just want to verify one step regarding the upgrade of the ceph packages. Do I have to upgrade ceph in the same time than Debian or do i have to upgrade ceph after the Debian upgrade from Stretch to Buster ? 1) In the first case : * Replace stretch by buster in /etc/apt/sources.list * Modify the ceph.list repo by croit.io one * Upgrade the entire nodes 2) In the second case (upgrade Debian then Ceph) * Replace stretch by buster in /etc/apt/sources.list * keep the /etc/apt/sources.list.d/ceph.list as it is * Upgrade and reboot the nodes * replace the ceph.list file by croit.io * upgrade the ceph packages * restarting the Ceph services (in the right order MON -> MGR -> OSD -> MDS) Thanks a lot for your advices Regards, Hervé

3 6

pg incomplete blocked by destroyed osd
by Francois Legrand 04 May '20

04 May '20

Hi all, During a crash disaster we had destroyed and reinstalled with a different number a few osds. As an example osd 3 was destroyed and recreated with id 101 by command ceph osd purge 3 --yes-i-really-mean-it + ceph osd create (to block id 3) + ceph-deploy osd create --data /dev/sdxx <server> and finally ceph osd rm 3). Some of our pgs are now incomplet (which can be understood) but blocked by some of the removed osd : ex: here is an part of the ceph pg 30.3 query { "state": "incomplete", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 384075, "up": [ 103, 43, 29, 2, 66 ], "acting": [ 103, 43, 29, 2, 66 ], ........ "peer_info": [ { "peer": "2(3)", "pgid": "30.3s3", "last_update": "373570'105925965", "last_complete": "373570'105925965", ....... }, "up": [ 103, 43, 29, 2, 66 ], "acting": [ 103, 43, 29, 2, 66 ], "avail_no_missing": [], "object_location_counts": [], *"blocked_by": [** ** 3,** ** 49** ** ],* ............ "down_osds_we_would_probe": [ *3* ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { * "detail": "peering_blocked_by_history_les_bound"* } ] I don't understand why the removed osd are still considered and present in the pg infos. Is there a way to get rid of that ? Moreover, we have tons of slow ops (more than 15 000) but I guess that the two problems are linked. Thanks for your help. F.

2 1

Re: upmap balancer and consequences of osds briefly marked out
by Anthony D'Atri 04 May '20

04 May '20

Do I misunderstand this script, or does it not _quite_ do what’s desired here? I fully get the scenario of applying a full-cluster map to allow incremental topology changes. To be clear, if this is run to effectively freeze backfill during / following a traumatic event, it will freeze that adapted state, not strictly return to the pre-event state? And thus the pg-upmap balancer would still need to be run to revert to the prior state? And this would also hold true for a failed/replaced OSD? > On May 1, 2020, at 7:37 AM, Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Thanks Dan, that looks like a really neat method & script for a few use-cases. We've actually used several of the scripts in that repo over the years, so, many thanks for sharing. > > That method will definitely help in the scenario in which a set of unnecessary pg remaps have been triggered and can be caught early and reverted. I'm still a little concerned about the possibility of, for example, a brief network glitch occurring at night and then waking up to a full unbalanced cluster. Especially with NVMe clusters that can rapidly remap and rebalance (and for which we also have a greater impetus to squeeze out as much available capacity as possible with upmap due to cost per TB). It's just a risk I hadn't previously considered and was wondering if others have either run into it or felt any need to plan around it. > > Cheers, > Dylan > > >> From: Dan van der Ster <dan(a)vanderster.com> >> Sent: Friday, 1 May 2020 5:53 PM >> To: Dylan McCulloch <dmc(a)unimelb.edu.au> >> Cc: ceph-users <ceph-users(a)ceph.io> >> >> Subject: Re: [ceph-users] upmap balancer and consequences of osds briefly marked out >> >> Hi, >> >> You're correct that all the relevant upmap entries are removed when an >> OSD is marked out. >> You can try to use this script which will recreate them and get the >> cluster back to HEALTH_OK quickly: >> https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… >> >> Cheers, Dan >> >> >> On Fri, May 1, 2020 at 9:36 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: >>> >>> Hi all, >>> >>> We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. >>> >>> Currently on ceph version: 12.2.13 luminous >>> >>> We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite >> there being no end-user client I/O during that period. >>> >>> Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. >>> As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. >>> Is there any reasonable mitigation for that scenario? >>> >>> The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). >>> Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? >>> >>> I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full >> osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. >>> >>> Curious whether others have any thoughts or experience regarding this. >>> >>> Cheers, >>> Dylan >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: upmap balancer and consequences of osds briefly marked out
by Dan van der Ster 04 May '20

04 May '20

Hi Dylan, The backfillfull_ratio, which defaults to 0.9, prevents backfilling into an osd which is getting too full. So, worst case scenario is that your cluster will have some osds getting up to 90% full, after which case the upmap balancer should starting putting things back into place. Also, check that your "mon osd down out subtree limit" is set appropriately for your cluster. In our case, we set it to "host" -- we don't want to automatically "out" all the osds from an entire host, because this is normally something that we can quickly fix with a manual intervention. But I fear that wouldn't have helped in your case, because the firewall issue probably downed a random subset of osds from several hosts all at once. We've also had happen a couple times, and now set "mon osd down out interval = 3600" so that we have time to notice the network outage and set noout on the cluster to prevent lots of rebalancing carnage. Hope it helps, Dan On Fri, May 1, 2020 at 4:37 PM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Thanks Dan, that looks like a really neat method & script for a few use-cases. We've actually used several of the scripts in that repo over the years, so, many thanks for sharing. > > That method will definitely help in the scenario in which a set of unnecessary pg remaps have been triggered and can be caught early and reverted. I'm still a little concerned about the possibility of, for example, a brief network glitch occurring at night and then waking up to a full unbalanced cluster. Especially with NVMe clusters that can rapidly remap and rebalance (and for which we also have a greater impetus to squeeze out as much available capacity as possible with upmap due to cost per TB). It's just a risk I hadn't previously considered and was wondering if others have either run into it or felt any need to plan around it. > > Cheers, > Dylan > > > >From: Dan van der Ster <dan(a)vanderster.com> > >Sent: Friday, 1 May 2020 5:53 PM > >To: Dylan McCulloch <dmc(a)unimelb.edu.au> > >Cc: ceph-users <ceph-users(a)ceph.io> > > > >Subject: Re: [ceph-users] upmap balancer and consequences of osds briefly marked out > > > >Hi, > > > >You're correct that all the relevant upmap entries are removed when an > >OSD is marked out. > >You can try to use this script which will recreate them and get the > >cluster back to HEALTH_OK quickly: > >https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… > > > >Cheers, Dan > > > > > >On Fri, May 1, 2020 at 9:36 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > >> > >> Hi all, > >> > >> We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. > >> > >> Currently on ceph version: 12.2.13 luminous > >> > >> We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite > > there being no end-user client I/O during that period. > >> > >> Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. > >> As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. > >> Is there any reasonable mitigation for that scenario? > >> > >> The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). > >> Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? > >> > >> I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full > > osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. > >> > >> Curious whether others have any thoughts or experience regarding this. > >> > >> Cheers, > >> Dylan > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

large difference between "STORED" and "USED" size of ceph df
by Lee, H. (Hurng-Chun) 04 May '20

04 May '20

Hello, We use purely cephfs in out ceph cluster (version 14.2.7). The cephfs data is an EC pool (k=4, m=2) with hdd OSDs using bluestore. The default file layout (i.e. 4MB object size) is used. We see the following output of ceph df: --- RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 951 TiB 888 TiB 63 TiB 63 TiB 6.58 ssd 9.6 TiB 9.6 TiB 1.4 GiB 16 GiB 0.17 TOTAL 961 TiB 898 TiB 63 TiB 63 TiB 6.52 POOLS: POOL ID STORED OBJECTS USED %USE D MAX AVAIL cephfs-data 2 34 TiB 12.51M 52 TiB 5.93 553 TiB cephfs-metadata 4 994 MiB 98.61k 1.5 GiB 0.02 3.0 TiB --- What triggered my attention is the discrepency between the reported size of "USED" (52 TiB) and "STORED" (34 TiB) on the cephfs-data pool. From this document ( https://docs.ceph.com/docs/master/releases/nautilus/#upgrade-compatibility-… ), it says that - "USED" represents amount of space allocated purely for data by all OSD nodes in KB - "STORED" represents amount of data stored by the user. I seem to undersand that the "USED" size can be roughly taken as the number of objects (12.51M) times the object size (4MB) of the file layout; and since there are many files with size smaller than 4 MB in our system, the actual stored data is less. Is my interpretation correct? If so, does it mean that we will be wasting a lot of space when we have a lot files smaller than the object size of 4MB in the system? Thanks for the help! Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0) 243610977 web: http://www.ru.nl/donders/

2 1

14.2.9 MDS Failing
by Marco Pizzolo 03 May '20

03 May '20

Hello, Hoping you can help me. Ceph had been largely problem free for us for the better part of a year. We have a high file count in a single CephFS filesystem, and are seeing this error in the logs: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mds/OpenFileTable.cc: 777: FAILED ceph_assert(omap_num_objs == num_objs) The issued seemed to occur this morning, and restarting the MDS as well as rebooting the servers doesn't correct the problem. Not really sure where to look next as the MDS daemons crash. Appreciate any help you can provide Marco

4 9

repairing osd rocksdb
by Francois Legrand 02 May '20

02 May '20

Hi, We had a major crash which ended with ~1/3 of our osd downs. Trying to fix it we reinstalled a few down osd (that was a mistake, I agree) and destroy the datas on it. Finally, we could fix the problem (thanks to Igor Fedotov) and restart almost all of our osds except one for which the rocksdb seems corrupted (at least for one file). Unfortunately, we now have 4 pgs down (all involving the dead osd) and 8 pg incompletes (some of them also involving the down osd). Before considering data loss, we would like to try to restart the down osds hopping to recover the down pgs and maybe some of the incomplete ones. Does someone have an idea on how to do that (maybe by removing the file corrupting the rocksdb or forcing to ignore the data in error) ? If it's not possible, how can we fix (even with dataloss) the downs and incomplete pgs ? Thanks for your advices. F.

2 1

dashboard module missing dependencies in 15.2.1 Octopus
by Duncan Bellamy 01 May '20

01 May '20

Hi, I have installed ceph on Ubuntu Focal Fossa using the ubuntu repo, instead of ceph-deploy (as ceph-deploy install does not work for Focal Fossa yet) install I used: sudo apt-get install -y ceph ceph-mds radosgw ceph-mgr-dashboard The rest of the setup was the same as the quickstart on ceph.io <http://ceph.io/> with ceph-deploy. It installed ceph version 15.2.1 (octopus). If I do a 'ceph -s' I get the warning: health: HEALTH_WARN 2 mgr modules have failed dependencies If I run 'ceph mgr module ls', for enabled and active modules I get: "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator", "osd_support", "pg_autoscaler", "progress", "rbd_support", "status", "telemetry", "volumes" ], "enabled_modules": [ "iostat", "restful” Then when I run 'ceph mgr module enable dashboard’ I get the error: Error ENOENT: module 'dashboard' reports that it cannot run on the active manager daemon: No module named 'yaml' (pass --force to force enablement) I have tried searching, and searching with apt but cannot find any ‘yaml’ package that might be used by ceph. Duncan

2 1

Re: 回复: Re: OSDs continuously restarting under load
by David Turner 01 May '20

01 May '20

I have 2 filestore OSDs in a cluster facing "Caught signal (Bus error)" as well and can't find anything about it. Ceph 12.2.12. The disks are less than 50% full and basic writes have been successful. Both disks are on different nodes. The other 14 disks on each node are unaffected. Restarting the node doesn't change the behavior. The affected OSD still crashes and the other 14 start fine (which likely rules out the controller and other shared components along those lines). I've attempted [1] these commands on the OSDs to see how much of the disk I could access cleanly. The first is just to flush the journal to disk and it crashed out with the same error. The second command is to compact the DB which also crashed with the same error. On one of the OSDs I was able to make it a fair bit into compacting the DB before it crashed the first time and now it crashes instantly. That leads me to think that it might have gotten to a specific part of the disk and/or filesystem that is having problems. I'm currently running [2] xfs_repair on one of the disks to see if it might be the filesystem. On the other disk I'm running [3] badblocks to check for problems with underlying sectors. I'm assuming that if it's a bad block on the disk that is preventing the disk from starting that there's really nothing that I can do to recover the OSD and I'll just need to export any PGs on the disks that aren't active. Here's hoping I make it through this without data loss. Since I started this data migration I've already lost a couple disks (completely unreadable by the OS so I can't get copies of the PGs off of them). Luckily these ones seem like I might be able to access that part of the data at least. As well, I only have some unfound objects at the moment, but all of my PGs are active, which is an improvement. [1] sudo -u ceph ceph-osd -i 285 --flush-journal sudo -u ceph ceph-kvstore-tool leveldb /var/lib/ceph/osd/ceph-285/current/omap compact [2] xfs_repair -n /dev/sdi1 [3] badblocks -b 4096 -v /dev/sdn On Thu, Mar 19, 2020 at 9:04 AM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > Hi, Igor, > > thanks for the tip. Dmesg does not say any suspicious information. > > I will investigate whether hardware has any problem or not. > > best regards, > > samuel > > > > > > huxiaoyu(a)horebdata.cn > > 发件人： Igor Fedotov > 发送时间： 2020-03-19 12:07 > 收件人： huxiaoyu(a)horebdata.cn; ceph-users; ceph-users > 主题： Re: [ceph-users] OSDs continuously restarting under load > Hi, Samuel, > > I've never seen that sort of signal in the real life: > > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > > > I suppose this has some hardware roots. Have you checked dmesg output? > > > Just in case, here is some info on "Bus Error" signal, may be it will > provide some insight: https://en.wikipedia.org/wiki/Bus_error > > > Thanks, > > Igor > > > On 3/18/2020 5:06 PM, huxiaoyu(a)horebdata.cn wrote: > > Hello， folks, > > > > I am trying to add a ceph node into an existing ceph cluster. Once the > reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd > becomes unresponsive and restarting, eventually go down. > > > > What could be the problem? Any suggestion would be highly appreciated. > > > > best regards, > > > > samuel > > > > **************************************************** > > root@node81:/var/log/ceph# > > root@node81:/var/log/ceph# > > root@node81:/var/log/ceph# > > root@node81:/var/log/ceph# ceph osd df > > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > > 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 > > 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 > > 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 > > 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 > > 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 > > 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 > > 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 > > 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 > > 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 > > 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 > > 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 > > 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 > > 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 > > TOTAL 45.9TiB 351GiB 45.6TiB 0.75 > > > > > ------------------------------------------------------------------------------------ > 日志 > > > > root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs > > 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > > 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > > -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > > -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs > opened 0 pgs > > 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > > 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 > pgs > > -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > > -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs > opened 202 pgs > > 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > > 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 > pgs > > -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > > -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs > opened 177 pgs > > 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > > 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 > pgs > > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs > opened 170 pgs > > > > > > 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected > need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have > 3291'557 flags = none tried to add 3291'557 flags = none > > 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected > need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have > 4738'731 flags = none tried to add 4738'731 flags = none > > 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected > need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have > 1789'169 flags = delete tried to add 1789'169 flags = delete > > 2020-03-18 18:39:20.985370 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 > conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 > conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:21.495101 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 > conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 > conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) > ** > > in thread 201e35fdb40 thread_name:tp_osd_tp > > > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) > luminous (stable) > > 1: (()+0x145882c) [0x2000245882c] > > 2: (()+0x19890) [0x2000c54b890] > > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > std::vector<ObjectStore::Transaction, > std::allocator<ObjectStore::Transaction> >&, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) > [0x200022c7a6c] > > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, > ObjectStore::Transaction&&, Context*, Context*, Context*, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) > [0x20001c19ee0] > > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, > boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > > 8: > (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) > [0x2000202e970] > > 9: > (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) > [0x200020442bc] > > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) > [0x20001ecea74] > > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, > ThreadPool::TPHandle&)+0x814) [0x20001de1384] > > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, > boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) > [0x20001b817d4] > > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> > const&)+0xb8) [0x20001f98968] > > 14: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) > [0x200024d60a4] > > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) > [0x200024da278] > > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > > 19: (()+0x80fc) [0x2000c53a0fc] > > 20: (()+0x119854) [0x2000f2ad854] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > > > --- begin dump of recent events --- > > -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) > register_command perfcounters_dump hook 0x2000ce09e40 > > -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) > register_command 1 hook 0x2000ce09e40 > > -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) > register_command perf dump hook 0x2000ce09e40 > > -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) > register_command perfcounters_schema hook 0x2000ce09e40 > > -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) > register_command perf histogram dump hook 0x2000ce09e40 > > -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) > register_command 2 hook 0x2000ce09e40 > > -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) > register_command perf schema hook 0x2000ce09e40 > > -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) > register_command perf histogram schema hook 0x2000ce09e40 > > -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) > register_command perf reset hook 0x2000ce09e40 > > -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) > register_command config show hook 0x2000ce09e40 > > -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) > register_command config help hook 0x2000ce09e40 > > -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) > register_command config set hook 0x2000ce09e40 > > -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) > register_command config get hook 0x2000ce09e40 > > -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) > register_command config diff hook 0x2000ce09e40 > > -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) > register_command config diff get hook 0x2000ce09e40 > > -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) > register_command log flush hook 0x2000ce09e40 > > -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) > register_command log dump hook 0x2000ce09e40 > > -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) > register_command log reopen hook 0x2000ce09e40 > > -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) > register_command dump_mempools hook 0x2000ce0d038 > > -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the > following dangerous and experimental features are enabled: bluestore,rocksdb > > -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the > following dangerous and experimental features are enabled: bluestore,rocksdb > > -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 > (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process > ceph-osd, pid 1159687 > > -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore > empty --pid-file > > -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the > following dangerous and experimental features are enabled: bluestore,rocksdb > > -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: > lrc > > -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block type kernel > > -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > > -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize > 8192, using bdev_block_size 4096 anyway > > -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, > 3500 GB) block_size 4096 (4096 B) rotational > > -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) close > > -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block type kernel > > -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 > /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > > -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 > /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize > 8192, using bdev_block_size 4096 anyway > > -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 > /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, > 3500 GB) block_size 4096 (4096 B) rotational > > -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block.db type kernel > > -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 > /var/lib/ceph/osd/ceph-6/block.db) open path > /var/lib/ceph/osd/ceph-6/block.db > > -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 > /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports > st_blksize 8192, using bdev_block_size 4096 anyway > > -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 > /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, > 38146 MB) block_size 4096 (4096 B) rotational > > -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block type kernel > > -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 > /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > > -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 > /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize > 8192, using bdev_block_size 4096 anyway > > -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 > /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, > 3500 GB) block_size 4096 (4096 B) rotational > > -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option > compaction_readahead_size = 2MB > > -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option > compaction_style = kCompactionStyleLevel > > -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option > compaction_threads = 32 > > -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option > compression = kNoCompression > > -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option > flusher_threads = 8 > > -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option > level0_file_num_compaction_trigger = 64 > > -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option > level0_slowdown_writes_trigger = 128 > > -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option > level0_stop_writes_trigger = 256 > > -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option > max_background_compactions = 64 > > -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option > max_bytes_for_level_base = 2GB > > -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option > max_write_buffer_number = 64 > > -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option > min_write_buffer_number_to_merge = 32 > > -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option > recycle_log_file_num = 64 > > -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option > target_file_size_base = 4MB > > -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option > write_buffer_size = 4MB > > -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option > compaction_readahead_size = 2MB > > -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option > compaction_style = kCompactionStyleLevel > > -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option > compaction_threads = 32 > > -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option > compression = kNoCompression > > -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option > flusher_threads = 8 > > -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option > level0_file_num_compaction_trigger = 64 > > -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option > level0_slowdown_writes_trigger = 128 > > -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option > level0_stop_writes_trigger = 256 > > -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option > max_background_compactions = 64 > > -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option > max_bytes_for_level_base = 2GB > > -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option > max_write_buffer_number = 64 > > -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option > min_write_buffer_number_to_merge = 32 > > -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option > recycle_log_file_num = 64 > > -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option > target_file_size_base = 4MB > > -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option > write_buffer_size = 4MB > > -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> > /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: > loading cls_hello > > -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not > permitted to load lua > > -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not > permitted to load sdk > > -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not > permitted to load kvs > > -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> > /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: > loading cephfs > > -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map > has features 432629239337189376, adjusting msgr requires for clients > > -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map > has features 432629239337189376 was 8705, adjusting msgr requires for mons > > -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map > has features 1009089991640629248, adjusting msgr requires for osds > > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs > opened 170 pgs > > -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using > weightedpriority op queue with priority op cut off at 196. > > -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 > log_to_monitors {default=true} > > -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with > init, starting boot process > > -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map > Got map version 34 > > -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map > Active mgr is now 192.168.230.120:6808/44007 > > -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect > Starting new session with 192.168.230.120:6808/44007 > > -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc > handle_mgr_configure stats_period=5 > > -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc > handle_mgr_configure updated stats threshold: 5 > > -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 > conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 > conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 > conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 > conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 > conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 > conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 > conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 > conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 > conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 > conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 > conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 > flags = none tried to add 3097'452 flags = none > > -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 > flags = none tried to add 1915'212 flags = none > > -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 > flags = none tried to add 1915'213 flags = none > > -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 > flags = none tried to add 5165'734 flags = none > > -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 > flags = none tried to add 2855'272 flags = none > > -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 > flags = none tried to add 1915'214 flags = none > > -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 > flags = none tried to add 3798'560 flags = none > > -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 > flags = none tried to add 2197'242 flags = none > > -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 > flags = delete tried to add 3099'543 flags = delete > > -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 > flags = delete tried to add 5169'1101 flags = delete > > -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 > flags = none tried to add 1915'215 flags = none > > -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 > flags = none tried to add 5175'1280 flags = none > > -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 > flags = delete tried to add 5166'844 flags = delete > > -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 > flags = delete tried to add 1805'174 flags = delete > > -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 > flags = none tried to add 1915'216 flags = none > > -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 > flags = none tried to add 5175'1283 flags = none > > -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 > flags = none tried to add 1915'217 flags = none > > -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 > flags = none tried to add 5175'1284 flags = none > > -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 > flags = delete tried to add 1805'179 flags = delete > > -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 > flags = delete tried to add 5166'785 flags = delete > > -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 > flags = delete tried to add 1805'200 flags = delete > > -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 > flags = none tried to add 3894'562 flags = none > > -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 > flags = none tried to add 5172'1141 flags = none > > -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 > flags = none tried to add 1915'218 flags = none > > -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 > flags = delete tried to add 1805'198 flags = delete > > -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 > flags = delete tried to add 3908'593 flags = delete > > -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 > flags = none tried to add 5176'1523 flags = none > > -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 > flags = none tried to add 5165'735 flags = none > > -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 > flags = none tried to add 3908'563 flags = none > > -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 > flags = none tried to add 4427'730 flags = none > > -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 > flags = none tried to add 4743'732 flags = none > > -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 > flags = none tried to add 5175'1287 flags = none > > -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 > flags = none tried to add 5172'1144 flags = none > > -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 > flags = none tried to add 1915'219 flags = none > > -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 > flags = delete tried to add 3092'344 flags = delete > > -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 > flags = delete tried to add 3092'381 flags = delete > > -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 > flags = delete tried to add 1805'197 flags = delete > > -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 > flags = delete tried to add 1805'199 flags = delete > > -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 > flags = none tried to add 2197'271 flags = none > > -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 > flags = none tried to add 3291'557 flags = none > > -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 > flags = none tried to add 4738'731 flags = none > > -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 > flags = delete tried to add 1789'169 flags = delete > > -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 > conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 > conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 > conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 > conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal > (Bus error) ** > > in thread 201e35fdb40 thread_name:tp_osd_tp > > > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) > luminous (stable) > > 1: (()+0x145882c) [0x2000245882c] > > 2: (()+0x19890) [0x2000c54b890] > > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > std::vector<ObjectStore::Transaction, > std::allocator<ObjectStore::Transaction> >&, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) > [0x200022c7a6c] > > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, > ObjectStore::Transaction&&, Context*, Context*, Context*, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) > [0x20001c19ee0] > > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, > boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > > 8: > (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) > [0x2000202e970] > > 9: > (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) > [0x200020442bc] > > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) > [0x20001ecea74] > > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, > ThreadPool::TPHandle&)+0x814) [0x20001de1384] > > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, > boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) > [0x20001b817d4] > > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> > const&)+0xb8) [0x20001f98968] > > 14: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) > [0x200024d60a4] > > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) > [0x200024da278] > > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > > 19: (()+0x80fc) [0x2000c53a0fc] > > 20: (()+0x119854) [0x2000f2ad854] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > > > > > > > huxiaoyu(a)horebdata.cn > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 1

Re: upmap balancer and consequences of osds briefly marked out
by Dan van der Ster 01 May '20

01 May '20

Hi, You're correct that all the relevant upmap entries are removed when an OSD is marked out. You can try to use this script which will recreate them and get the cluster back to HEALTH_OK quickly: https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… Cheers, Dan On Fri, May 1, 2020 at 9:36 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Hi all, > > We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. > > Currently on ceph version: 12.2.13 luminous > > We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite there being no end-user client I/O during that period. > > Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. > As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. > Is there any reasonable mitigation for that scenario? > > The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). > Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? > > I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. > > Curious whether others have any thoughts or experience regarding this. > > Cheers, > Dylan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

upmap balancer and consequences of osds briefly marked out
by Dylan McCulloch 01 May '20

01 May '20

Hi all, We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. Currently on ceph version: 12.2.13 luminous We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite there being no end-user client I/O during that period. Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. Is there any reasonable mitigation for that scenario? The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. Curious whether others have any thoughts or experience regarding this. Cheers, Dylan

1 0

test -- please ignore
by Tim Serong 01 May '20

01 May '20

Just testing, sorry for the noise. Regards, Tim -- Tim Serong Senior Clustering Engineer SUSE tserong(a)suse.com

1 0