July 2020 - ceph-users - lists.ceph.io

Can't add osd with cephadm as non-root user
by Michael Preuss 22 Jul '20

22 Jul '20

Hi all, I tried to use Cephadm as non-root, it works, until I tried to install a new osd. I get this Error Message: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1167, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 113, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 311, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 75, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 66, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 715, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 633, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Failed to execute command: sudo /usr/bin/cephadm --image docker.io/ceph/ceph:v15.2.4 ceph-volume --fsid 1234abcd --config-json - -- lvm prepare --bluestore --data /dev/sdb --no-systemd Has someone a hint how I can solve this problem? Thanks, Michael

2 2

Module 'cephadm' has failed: auth get failed: failed to find client.crash.ceph0-ote in keyring retval:
by biohazd＠yahoo.com 22 Jul '20

22 Jul '20

anyone seen this error on the new Ceph 15.2.4 cluster using cpehadm to manage it ? Module 'cephadm' has failed: auth get failed: failed to find client.crash.ceph0-ote in keyring retval:

6 9

ceph crush reweight #osd 0 strange redistribution
by Marcel Kuiper 22 Jul '20

22 Jul '20

Hi List This is a rephrase of an earlier question that puzzels me. I took out a disk on nautilus 14.2.8 with 'ceph crush reweight osd.111 0'. I expected that PGs would mainly go to osds on other nodes but to my surprise most PG's ended up on osds on the same node. Here is an overview of the number of PG's that got send to an other osd. The node that had osd 111 has osds 108-116 as well NR PG's OSD_ID 1 61 1 83 1 86 3 108 4 109 5 110 2 112 5 113 7 114 5 115 2 116 As you can see only 3 PG's were send to other node's osd's. The rest was send to osds on the same node :-O. There are 81 osds in the same room as osd.111. I would have expected (certainly since node weight decreases with crush reweight) setting to 0 that PG's would mostly go to other nodes. Can someone explain this behaviour? All pools make use of crush_rule 1, min_size = 2, size = 3 with the following rules: "rule_id": 1, "rule_name": "hdd", "ruleset": 1, "type": 1, "min_size": 2, "max_size": 3, "steps": [ { "op": "take", "item": -31, "item_name": "DC3" }, { "op": "choose_firstn", "num": 0, "type": "room" }, { "op": "chooseleaf_firstn", "num": 1, "type": "host" }, ] I cannot attache the complete osd tree but the structure is: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -31 1275.35522 root DC3 -32 600.69037 room az-a -39 133.14589 rack rack_W26 -27 66.57289 host st3g3psm2 108 hdd 7.39699 osd.108 up 1.00000 1.00000 109 hdd 7.39699 osd.109 up 1.00000 1.00000 110 hdd 7.39699 osd.110 up 1.00000 1.00000 111 hdd 7.39699 osd.111 up 1.00000 1.00000 112 hdd 7.39699 osd.112 up 1.00000 1.00000 113 hdd 7.39699 osd.113 up 1.00000 1.00000 114 hdd 7.39699 osd.114 up 1.00000 1.00000 115 hdd 7.39699 osd.115 up 1.00000 1.00000 116 hdd 7.39699 osd.116 up 1.00000 1.00000 with 3 rooms, 13 racks, 19 hosts, 172 osds, 3968 PG's, 10 pools. As you see there are racks in the tree but these are not taken into consideration in the crush rule 1. The cluster is not yet fully in use hence the PG to OSD ratio is still low. We expect more pools to be added in the near future Marcel

1 0

Re: Ceph Dashboard and Firefox
by biohazd＠yahoo.com 22 Jul '20

22 Jul '20

The issue is on the "Firefox Extended release 68.10.0" I switched over to normal Firefox 78.0.2 and it works fine now.

1 0

Help add node to cluster using cephadm
by davidthuong2424＠gmail.com 22 Jul '20

22 Jul '20

I need help about add node when install ceph with cephadm . When i run cpeh orch add host ceph2 error enoent: new host ceph2 (ceph2) failed check: ['traceback (most recent call last):', Please help me fix it. Thanks & Best Regards David

4 6

Help add node to cluster using cephadm
by Hoài Thương 22 Jul '20

22 Jul '20

Dear Support, I need help about add node when install ceph with cephadm . When i run cpeh orch add host ceph2 error enoent: new host ceph2 (ceph2) failed check: ['traceback (most recent call last):', Please help me fix it. Thanks & Best Regards David

1 0

script for compiling and running the Ceph source code
by Bobby 22 Jul '20

22 Jul '20

Hi, I am trying to profile the number of invocations to a particular function in Ceph source code. I have instrumented the code with time functions. Can someone please share the script for compiling and running the Ceph source code? I am struggling with it. That would be great help ! BR Bobby !

2 2

Re: osd out vs crush reweight]
by Marcel Kuiper 21 Jul '20

21 Jul '20

Hi Dominic, This cluster is running 14.2.8 (nautilus) There's 172 osds divided over 19 nodes. There are currently 10 pools. All pools have 3 replica's of data There are 3968 PG's (the cluster is not yet fully in use. The number of PGs is expected to grow) Marcel > Marcel; > > Short answer; yes, it might be expected behavior. > > PG placement is highly dependent on the cluster layout, and CRUSH rules. > So... Some clarifying questions. > > What version of Ceph are you running? > How many nodes do you have? > How many pools do you have, and what are their failure domains? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International, Inc. > DHilsbos(a)PerformAir.com > www.PerformAir.com > > > -----Original Message----- > From: Marcel Kuiper [mailto:ceph@mknet.nl] > Sent: Tuesday, July 21, 2020 6:52 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] osd out vs crush reweight > > Hi list, > > I ran a test with marking an osd out versus setting its crush weight to 0. > I compared to what osds pages were send. The crush map has 3 rooms. This > is what happened. > > On ceph osd out 111 (first room; this node has osds 108 - 116) pg's were > send to the following osds > > NR PG's OSD > 2 1 > 1 4 > 1 5 > 1 6 > 1 7 > 2 8 > 1 31 > 1 34 > 1 35 > 1 56 > 2 57 > 1 58 > 1 61 > 1 83 > 1 84 > 1 88 > 1 99 > 1 100 > 2 107 > 1 114 > 2 117 > 1 118 > 1 119 > 1 121 > > All PG's were send to osds on other nodes in the same room, except for 1 > PG on osd 114. I think this works as expected > > Now I marked the osd in and wait until all stabilized. Then I set the > crush weight to 0. ceph osd crush reweight osd.111 0. I thought this > lowers the crush weight of the node so even less chances that PG's end up > on an osd of the same node. However the result are > > NR PG's OSD > 1 61 > 1 83 > 1 86 > 3 108 > 4 109 > 5 110 > 2 112 > 5 113 > 7 114 > 5 115 > 2 116 > > except for 3 PG's all other PG's ended up on an osd belonging to the same > node :-O. Is this expected behaviour? Can someone explain?? This is on > nautilus 14.2.8. > > Thanks > > Marcel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 6

Problem with OSD::osd_op_tp thread had timed out and other connected issues
by Jan Pekař - Imatic 21 Jul '20

21 Jul '20

Hello, I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network Cluster was empty, fresh install. We filled cluster with data (small blocks) using RGW. Cluster is now used for testing so no client was using it during my admin operations mentioned below After a while (7TB of data / 40M objects uploaded) we decided, that we increase pg_num from 128 to 256 to better spread data and to speedup this operation, I've set ceph config set mgr target_max_misplaced_ratio 1 so that whole cluster rebalance as quickly as it can. I have 3 issues/questions below: 1) I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to restart with logged heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide timed out after 150 after a while OSD's were back so I continued after a while with my tests. My question - increasing number of PG with maximal target_max_misplaced_ratio was too much for that OSDs? It is not recommended to do it this way? I had no problem with this increase before, but configuration of cluster was slightly different and it was luminous version. 2) Rebuild was still slow so I increased number of backfills ceph tell osd.* injectargs "--osd-max-backfills 10" and reduced recovery sleep time ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01" and after few hours I noticed, that some of my OSD's were restarted during recovery, in log I can see ... |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy | I observed network graph usage and network utilization was low during recovery (10Gbit was not saturated). So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that OSD is using threads and HDD timeouts are not influencing heartbeats to other OSD's and MON. It looks like it is not true. 3) After OSD was wrongly marked down I can see that cluster has object degraded. There were no degraded object before that. Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs degraded, 8 pgs undersized It means that this OSD disconnection causes data degraded? How is it possible, when no OSD was lost. Data should be on that OSD and after peering should be everything OK. With luminous I had no problem, after OSD up degraded objects where recovered/found during few seconds and cluster was healthy within seconds. Thank you very much for additional info. I can perform additional tests you recommend because cluster is used for testing purpose now. With regards Jan Pekar -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

4 7

osd out vs crush reweight
by Marcel Kuiper 21 Jul '20

21 Jul '20

Hi list, I ran a test with marking an osd out versus setting its crush weight to 0. I compared to what osds pages were send. The crush map has 3 rooms. This is what happened. On ceph osd out 111 (first room; this node has osds 108 - 116) pg's were send to the following osds NR PG's OSD 2 1 1 4 1 5 1 6 1 7 2 8 1 31 1 34 1 35 1 56 2 57 1 58 1 61 1 83 1 84 1 88 1 99 1 100 2 107 1 114 2 117 1 118 1 119 1 121 All PG's were send to osds on other nodes in the same room, except for 1 PG on osd 114. I think this works as expected Now I marked the osd in and wait until all stabilized. Then I set the crush weight to 0. ceph osd crush reweight osd.111 0. I thought this lowers the crush weight of the node so even less chances that PG's end up on an osd of the same node. However the result are NR PG's OSD 1 61 1 83 1 86 3 108 4 109 5 110 2 112 5 113 7 114 5 115 2 116 except for 3 PG's all other PG's ended up on an osd belonging to the same node :-O. Is this expected behaviour? Can someone explain?? This is on nautilus 14.2.8. Thanks Marcel

2 1

Re: Thank you!
by Marc Roos 21 Jul '20

21 Jul '20

>> I'm happy user since 2014 and I never lost any data. When I remember >> how painfull was the firmware upgrade of emc, netapp, hp storage and the >> time passed to recover lost data ..... Ceph is just amazing ! Interesting I always wondered how ceph compares to propriatary solutions. I am getting the impression that closed source environments will not survive in the long run. If you see how eg CERN is handling this 'bug of the year'. It just shows the value of large support base, and having access to detailed info like this lz4 patch.

2 1

[ceph] [nautilus][ceph-ansible] - Dynamic bucket resharding problem
by Erik Johansson 21 Jul '20

21 Jul '20

Hello! I've run into a bit of an issue with one of our radosgw production clusters.. Setup is two radosgw nodes behind haproxy loadbalancing, which in turn are connected to the ceph cluster. Everything running 14.2.2 so Nautilus. It's tied to a openstack cluster, so keystone as authentication backend (should really matter though). Today both rgw backends crashed. Checking logs it seems to be related to dynamic resharding of a bucket, causing Lock errors: Logs snippet: https://pastebin.com/uBCnhinF Checking http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021368.html (old), I performed a manual reshard of affected bucket with success (radosgw-admin bucket reshard --bucket="XXX/YYY" --num-shards=256) Checking the metadata for bucket, it now correctly shows 256, up from 128. HOWEVER, the dynamic resharding still kept happening and bringing down the backeds. I suspect it is because of the old reshard op hanging around when checking a `reshard list`: https://pastebin.com/dPChwBCT As the resharding seems to have been successful when running manually, I now want to remove that reshard op, but can't, getting this https://pastebin.com/071kfAsa error when trying.. Right now I had to resort to setting rgw_dynamic_resharding = false in ceph.conf to stop the problem from occuring. Ideas? Cheers Erik

1 0

EC profile datastore usage - question
by Mateusz Skała 21 Jul '20

21 Jul '20

Hello Community, I would like to ask about help in explanation situation. There is Rados gateway with EC pool profile k=6 m=4. So it shoud take something about 1.4 - 2.0 data usage more from raw data if I’m correct. Rados df shows me: 116 TiB used and WR 26 TiB Can You explain this? It is about 4.5*WR used data. Why? Regards Mateusz Skała

4 4

Ceph Dashboard and Firefox
by biohazd＠yahoo.com 21 Jul '20

21 Jul '20

Hi I'm using Ceph Dashboard on Firefox on Macbook, and lately it has been hanging with "A web page is loading slowly - stop it or wait". Some pages load load, but some show that warning and stop loading. I'm on "Firefox Extended Support release 68.10.0esr (64-bit)" Anyone else seen this issue?

2 1

ceph (rhel) packages rebuilt without release change ?
by SCHAER Frederic 21 Jul '20

21 Jul '20

Hi, On a previously installed machine I get : # rpm -qi ceph-selinux-14.2.10-0.el7.x86_64 |grep Build Build Date : Thu 25 Jun 2020 08:08:52 PM CEST Build Host : braggi01.front.sepia.ceph.com # rpm -q --requires ceph-selinux-14.2.10-0.el7.x86_64 |grep selinux libselinux-utils selinux-policy-base >= 3.13.1-252.el7_7.6 On a VM setup the same way, but now failing the ceph install, this is what I get with the exact same repos (wich I sync from ceph ones) : # repoquery -q --requires ceph-selinux-14.2.10-0.el7.x86_64 |grep selinux libselinux-utils selinux-policy-base >= 3.13.1-266.el7_8.1 # After OS update and then ceph install: # rpm -qi ceph-selinux-14.2.10-0.el7.x86_64 |grep Build Build Date : Thu 09 Jul 2020 08:09:46 PM CEST Build Host : braggi03.front.sepia.ceph.com You'll notice the requirements changed for selinux-policy-base from 3.13.1-252.el7_7.6 to 3.13.1-266.el7_8.1 And the build info changed too. But the RPM package version and release did not change and I don't get it. I would have assumed an rpm change would at least infer a minor release bump ? Are the RPMs being rebuilt with the same versions/release but with different OSes version (frequently ??) ? Thanks && Regards

1 0

Thank you!
by DHilsbos＠performair.com 21 Jul '20

21 Jul '20

I just want to thank the Ceph community, and the Ceph developers for such a wonderful product. We had a power outage on Saturday, and both Ceph clusters went offline, along with all of our other servers. Bringing Ceph back to full functionality was an absolute breeze, no problems, no hiccups, no nothing. Just start the servers, and watch everything sort itself out. Again; Thank you! Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

6 5

Fw:Re: "ceph daemon osd.x ops" shows different number from "ceph osd status <bucket>"
by rainning 21 Jul '20

21 Jul '20

aha, thanks very much for pointing out, Anthony! Just a summary for the screenshot pasted in my previous email. Based on my understanding, "ceph daemon osd.x ops" or "ceph daemon osd.x dump_ops_in_flight" shows the ops currently being processed in the osd.x. I also noticed that there is another ceph cli that shows the number of ops with particular osds, i.e. "ceph osd status {<bucket>}", and the number doesn't match with the daemon one, nor "ceph daemon osd.x dump_historic_ops". I haven't looked into the code yet, and just try to get a quick help from the community about how the ops number from "ceph osd status {<bucket>}" is calculated? Is it a statistical value? Thanks! ------------------ Original ------------------ From: "Anthony D'Atri" <anthony.datri(a)gmail.com>; Date: Tue, Jul 21, 2020 02:01 AM To: "rainning"<tweetypie(a)qq.com>; Subject: Re: [ceph-users] "ceph daemon osd.x ops" shows different number from "ceph osd status <bucket>" Your messages are very hard to read. On Jul 20, 2020, at 12:58 AM, rainning <tweetypie(a)qq.com> wrote: "ceph daemon osd.x ops" shows ops currently in flight, the number is different from "ceph osd status <bucket&gt;". Is the number of ops in "ceph osd status <bucket&gt;" also the ops currently in flight? root@osd021:~#&nbsp;ceph&nbsp;daemon&nbsp;osd.636&nbsp;ops { &nbsp;&nbsp;&nbsp;&nbsp;"ops":&nbsp;[], &nbsp;&nbsp;&nbsp;&nbsp;"num_ops":&nbsp;0 } root@osd021:~# ceph osd status osd021 +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ |&nbsp; id |&nbsp; host&nbsp; |&nbsp; used | avail | wr ops | wr data | rd ops | rd data |&nbsp;&nbsp; state&nbsp;&nbsp; | +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ |&nbsp; 37 | osd021 |&nbsp; 542G | 1134G |&nbsp;&nbsp; 22&nbsp;&nbsp; |&nbsp;&nbsp; 225k&nbsp; |&nbsp;&nbsp; 12&nbsp;&nbsp; |&nbsp;&nbsp; 274k&nbsp; | exists,up | |&nbsp; 91 | osd021 |&nbsp; 567G | 1109G |&nbsp;&nbsp; 38&nbsp;&nbsp; |&nbsp;&nbsp; 281k&nbsp; |&nbsp;&nbsp; 12&nbsp;&nbsp; |&nbsp; 1006k&nbsp; | exists,up | | 147 | osd021 |&nbsp; 551G | 1125G |&nbsp;&nbsp; 32&nbsp;&nbsp; |&nbsp;&nbsp; 307k&nbsp; |&nbsp;&nbsp;&nbsp; 4&nbsp;&nbsp; |&nbsp;&nbsp; 247k&nbsp; | exists,up | | 201 | osd021 |&nbsp; 549G | 1127G |&nbsp;&nbsp; 82&nbsp;&nbsp; |&nbsp; 1505k&nbsp; |&nbsp;&nbsp;&nbsp; 5&nbsp;&nbsp; |&nbsp; 42.4k&nbsp; | exists,up | | 264 | osd021 | 1503G | 2222G |&nbsp;&nbsp;&nbsp; 3&nbsp;&nbsp; |&nbsp; 55.2k&nbsp; |&nbsp;&nbsp;&nbsp; 4&nbsp;&nbsp; |&nbsp; 12.2k&nbsp; | exists,up | | 317 | osd021 | 1537G | 2188G |&nbsp;&nbsp; 13&nbsp;&nbsp; |&nbsp;&nbsp; 138k&nbsp; |&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; |&nbsp; 6190&nbsp;&nbsp; | exists,up | | 375 | osd021 | 1369G | 2356G |&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; |&nbsp;&nbsp; 834k&nbsp; |&nbsp;&nbsp;&nbsp; 5&nbsp;&nbsp; |&nbsp; 67.4k&nbsp; | exists,up | | 430 | osd021 | 1424G | 2301G |&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; |&nbsp; 8192&nbsp;&nbsp; |&nbsp;&nbsp;&nbsp; 6&nbsp;&nbsp; |&nbsp; 1661k&nbsp; | exists,up | | 480 | osd021 | 1435G | 2289G |&nbsp;&nbsp;&nbsp; 3&nbsp;&nbsp; |&nbsp; 16.6k&nbsp; |&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp; |&nbsp; 42.4k&nbsp; | exists,up | | 535 | osd021 | 1494G | 2231G |&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp; |&nbsp; 12.0k&nbsp; |&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; |&nbsp; 27.8k&nbsp; | exists,up | | 584 | osd021 | 1484G | 2241G |&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; |&nbsp; 5529&nbsp;&nbsp; |&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; |&nbsp; 12.7k&nbsp; | exists,up | | 636 | osd021 | 1394G | 2331G |&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp; |&nbsp;&nbsp; 850k&nbsp; |&nbsp; 406&nbsp;&nbsp; |&nbsp; 78.1M&nbsp; | exists,up | +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Single Server Ceph OSD Recovery
by Daniel Da Cunha 21 Jul '20

21 Jul '20

Hello, As a hobbies, I have been using Ceph Nautilus as a single server with 8 OSDs. Part of the setup I set the crush map to fail at OSD level: step chooseleaf firstn 0 type osd. Sadly, I didn’t take the necessary precaution for my boot disk and the OS failed. I have backups of /etc/ceph/ but I am not able to recover the OS. Can you think of a way for me to recreate the OS and adopt the 8 OSD without loosing the data? Thanks & regards, Daniel

3 4

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

ceph osd log -> set_numa_affinity unable to identify public interface
by EDH - Manuel Rios 20 Jul '20

20 Jul '20

Hi , Today checking the osd logs at boot after upgrade to 14.2.10 we found that: set_numa_affinity unable to identify public interface 'p3p1.4094' numa node: (2) No such file or directory "2020-07-20 20:41:41.134 7f2cd15ca700 -1 osd.12 1120769 set_numa_affinity unable to identify public interface 'p3p1.4094' numa node: (2) No such file or directory" Sounds weird it can find numa based in the p3p1 but no in p3p1.<vlan-id> Can we ignore that "warning" or just need to tune something more? Regards Manuel

1 0

Degradation of write-performance after upgrading to Octopus
by Thomas Gradisnik 20 Jul '20

20 Jul '20

We have deployed a small test cluster consisting of three nodes. Each node is running a mon/mgr and two osds (Samsung PM983 3,84TB NVMe split into two partitions), so six osds in total. We started with Ceph 14.2.7 some weeks ago (upgraded to 14.2.9 later) and ran different tests using fio against some rbd volumes in order to get an overview what performance we could expect. The configuration is unchanged compared to the defaults, we only set several debugging options to 0/0. Yesterday we upgraded the whole cluster following the upgrade guidelines to Ceph 15.2.3, which worked without any problems so far. Nevertheless when running the same tests as before with Ceph 14.2.9, we are seeing some clear degradations in write-performance (beside some performance improvements, which shall also be mentioned). Here the results of concern (each with the relevant fio settings used): Test "read-latency-max" (rw=randread, iodepth=64, bs=4k) read_iops: 32500 -> 87000 Test "write-latency-max" (rw=randwrite, iodepth=64, bs=4k) write_iops: 22500 -> 11500 Test "write-throughput-iops-max" (rw=write, iodepth=64, bs=4k) write_iops: 7000 -> 14000 Test "usecase1" (rw=randrw, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/,4k/50:8k/20:16k/20:32k/5:64k/2:128k/:256k/, rwmixread=1, rate_process=poisson, iodepth=64) write_iops: 21000 -> 8500 Test "usecase1-readonly" (rw=randread, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/, rate_process=poisson, iodepth=64) read_iops: 28000 -> 58000 The last two tests represent a typical use case on our systems. Therefore we are especially concerned by the drop in performance from 21000 w/ops to 8500 w/ops (about 60%) after upgrading to Ceph 15.2.3. We ran all tests several times, the values are averaged over all iterations and fairly consistent and reproducible. We even tried wiping the whole cluster, downgrading to Ceph 14.2.9 again, setting up a new cluster/pool, running the tests and upgrading to Ceph 15.2.3 again. The tests have been performed on one of the three cluster nodes using a 50G rbd volume, which had been prefilled with random data before each test-run. Have any changes been introduced with Octopus that could explain the observed changes in performance? What we already tried: - Disabling rbd cache - Reverting rbc cache policy to writeback (default in 14.2) - Setting rbd io scheduler to none - Deploying a fresh cluster starting with Ceph 15.2.3 Kernel is 5.4.38 … I don't know if some other system specs would be helpful besides the already mentioned (since we are talking about a relative change in performance after upgrading Ceph without any further changes) - if so, please let us know.

6 10

ceph/rados performace sync vs async
by Daniel Mezentsev 20 Jul '20

20 Jul '20

Hi All, I started a small project related to metrics collection and processing, Ceph was chosen as a storage backend. Decided to use rados directly, to avoid any additional layers. I got a very simple client - it works fine, but performance is very low. Can't get more than 30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be mentioned about the client - I'm using sbcl (yep lisp). Call to rados API is just cffi call. Did try in async mode. Wow ! Saturated network bandwidth for large objects (4Mb and bigger), for small objects - saturated OSD IOPS ~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty reasonable. Bottom line - issue not with the lisp client - im getting close to C performance, difference is sync vs async IO. Why it's so big - sync operations are approx 2-3 times slower then async . Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

4 6

HP printer offline
by masonlava77＠gmail.com 20 Jul '20

20 Jul '20

When I make my hard efforts to print the documents through my HP printer, suddenly my HP printer goes offline mode. I am applying an appropriate command to my HP printer, I am facing HP printer offline problem. This offline error is an annoying issue for me, so I am not able to work on my HP printer. I am trying to find out the major reasons of HP printer offline error, but I don’t get the success to find out actual reasons. As per my technical skills, I have applied my technical knowledge to sort out HP printer offline issue. So anyone can suggest the smart methods to rectify this offline error. https://www.hpprintersupportpro.com/blog/stop-my-hp-printer-from-going-offl…

1 0

HP printer offline
by masonlava77＠gmail.com 20 Jul '20

20 Jul '20

When I make my hard efforts to print the documents through my HP printer, suddenly my HP printer goes offline mode. I am applying an appropriate command to my HP printer, I am facing HP printer offline problem. This offline error is an annoying issue for me, so I am not able to work on my HP printer. I am trying to find out the major reasons of HP printer offline error, but I don’t get the success to find out actual reasons. As per my technical skills, I have applied my technical knowledge to sort out HP printer offline issue. So anyone can suggest the smart methods to rectify this offline error. https://www.hpprintersupportpro.com/blog/stop-my-hp-printer-from-going-offl…

1 0

Cache Tier OSDs full and near full - not flushing and evicting
by Priya Sehgal 20 Jul '20

20 Jul '20

Hi, We have a large production cluster with Writeback Cache Tier. Recently, we observed that some of the OSDs were near full and got full from the Cache Tier and the cluster is in error state. The target_max_bytes was not set correctly and hence I think the flush and eviction never happened. I wish to disable caching and remove the cache tier, but before that wish to flush all the contents of the cache. I have set the cache mode to "forward" as suggested here: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/ht…. But, rados df still shows same usage of cache tier. It's not automatically flushing and evicting. I tried "cache-try-flush", but this command gets blocked for some reason for many objects (although this is supposed to be non-blocking). How do I make sure all the objects are flushed to backing tier and objects evicted so that I can safely deprecate/remove the cache tier which is currently having many full and near full OSDs (in cache tier). -- Regards, Priya

1 0

ceph OSD node optimised sysctl configuration
by Jeremi Avenant 20 Jul '20

20 Jul '20

Hi All Is there perhaps any updated documentation about ceph OSD node optimised sysctl configuration? I'm seeing a lot of these: $ netstat -s ... 4955341 packets pruned from receive queue because of socket buffer overrun ... 5866 times the listen queue of a socket overflowed ... TCPBacklogDrop: 14601 OS: Ubuntu 16 HWE Ceph Luminous : 12.2.13 Ceph-ansible: 3.2.20 + containerised Thanks in advance. -- *Jeremi-Ernst Avenant, Mr.* Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

1 0

"ceph daemon osd.x ops" shows different number from "ceph osd status <bucket>"
by rainning 20 Jul '20

20 Jul '20

"ceph daemon osd.x ops" shows ops currently in flight, the number is different from "ceph osd status <bucket>". Is the number of ops in "ceph osd status <bucket>" also the ops currently in flight? root@osd021:~# ceph daemon osd.636 ops {     "ops": [],     "num_ops": 0 } root@osd021:~# ceph osd status osd021 +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ |  id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   | +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ |  37 | osd021 |  542G | 1134G |   22   |   225k  |   12   |   274k  | exists,up | |  91 | osd021 |  567G | 1109G |   38   |   281k  |   12   |  1006k  | exists,up | | 147 | osd021 |  551G | 1125G |   32   |   307k  |    4   |   247k  | exists,up | | 201 | osd021 |  549G | 1127G |   82   |  1505k  |    5   |  42.4k  | exists,up | | 264 | osd021 | 1503G | 2222G |    3   |  55.2k  |    4   |  12.2k  | exists,up | | 317 | osd021 | 1537G | 2188G |   13   |   138k  |    1   |  6190   | exists,up | | 375 | osd021 | 1369G | 2356G |    1   |   834k  |    5   |  67.4k  | exists,up | | 430 | osd021 | 1424G | 2301G |    1   |  8192   |    6   |  1661k  | exists,up | | 480 | osd021 | 1435G | 2289G |    3   |  16.6k  |    2   |  42.4k  | exists,up | | 535 | osd021 | 1494G | 2231G |    2   |  12.0k  |    1   |  27.8k  | exists,up | | 584 | osd021 | 1484G | 2241G |    1   |  5529   |    1   |  12.7k  | exists,up | | 636 | osd021 | 1394G | 2331G |    2   |   850k  |  406   |  78.1M  | exists,up | +-----+--------+-------+-------+--------+---------+--------+---------+-----------+

1 0

Ceph config changed on ansible deployed cluster
by Szabo, Istvan (Agoda) 19 Jul '20

19 Jul '20

Hi, How is it possible to change config on an ansible deployed cluster due to the missing ceph-deploy. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Choose Best PPC Services for successful your Business.
by jamesmacle9＠gmail.com 18 Jul '20

18 Jul '20

PPC Campaign is meaningless without a proper goal in mind. As the provider of best ppc services in Delhi, we follow a tested set of steps for a successful campaign. It is very important to lay a set of rules and then place all your efforts playing by them so that you are able to achieve what you want for your business. Without a proper framework, your ad ends up being futile. It is very important to align your campaign with the business goal that you have in mind. Make sure you know what you want your ad to do like generate sales, create awareness, collect relevant user data or just increase the number of visitors. You should know whom to target, agenda of the campaign and the budget. It is as follows- 1. Decide the objective 2. Set the target 3. Lead Generation 4. Site Traffic https://www.pnjsharptech.com/ppc-services-india

1 0

Ceph Science meeting
by Khan, Babar 17 Jul '20

17 Jul '20

Hi Ceph users, Can someone please tell me when is Ceph Science user group meeting? Best regards Babar

2 2

RadosGw swift / S3
by St-Germain, Sylvain (SSC/SPC) 17 Jul '20

17 Jul '20

Hi everyone, I try to configure a simple rados gateway with three nodes and I add a virtual IP address for the load balancer. I make calls to the gateway using S3 and SWIFT. In the case of S3 everything works, I can make requests directly on the node where there is a gateway or use the virtual address and it responds well. In the case of SWIFT, I can query the nodes and it's fine, but when I use the virtual IP address it doesn't respond. The call sequence is the same for the nodes and the virtual IP address. Do you have any leads why it doesn't work? Version Info ceph version : 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) OS : CentOS 8 Host file (ceph-ansible) ... [rgws] node-01 node-02 node-03 [rgwloadbalancers] node-01 node-02 node-03 ... rgwloadbalancers.yml (ceph-ansible) ########### # GENERAL # ########### haproxy_frontend_port: 80 haproxy_frontend_ssl_port: 443 haproxy_frontend_ssl_certificate: haproxy_ssl_dh_param: 4096 haproxy_ssl_ciphers: - EECDH+AESGCM - EDH+AESGCM haproxy_ssl_options: - no-sslv3 - no-tlsv10 - no-tlsv11 - no-tls-tickets # virtual_ips: - XXX.XXX.XXX.189 # virtual_ip_netmask: 24 virtual_ip_interface: eno5 /etc/ceph/ceph.conf (node1 example) [client.node1.rgw0] host = node1 keyring = /var/lib/ceph/radosgw/ceph-rgw. node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw- node1.rgw0.log rgw frontends = beast endpoint=xxx.xxx.xxx.190:8080 rgw thread pool size = 512 [client.rgw.node2.rgw0] host = node2 keyring = /var/lib/ceph/radosgw/ceph-rgw.node2.rgw0/keyring log file = /var/log/ceph/ceph-rgw- node2.rgw0.log rgw frontends = beast endpoint=xxx.xxx.xxx.191:8080 rgw thread pool size = 512 [client.rgw.node3.rgw0] host = node3 keyring = /var/lib/ceph/radosgw/ceph-rgw.node3.rgw0/keyring log file = /var/log/ceph/ceph-rgw-node3.rgw0.log rgw frontends = beast endpoint=xxx.xxx.xxx.192:8080 rgw thread pool size = 512 Thank you ! Sylvain

1 0

Re: 1 pg inconsistent
by Abhimnyu Dhobale 17 Jul '20

17 Jul '20

Thanks For your reply, Please find the below output and suggest. [root@vpsapohmcs01 ~]# rados -p vpsacephcl01 list-inconsistent-obj 1.3c9 --format=json-pretty { "epoch": 845, "inconsistents": [ { "object": { "name": "rbd_data.515c96b8b4567.000000000000c377", "nspace": "", "locator": "", "snap": "head", "version": 21101 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.515c96b8b4567.000000000000c377", "key": "", "snapid": -2, "hash": 867656649, "max": 0, "pool": 1, "namespace": "" }, "version": "853'21101", "prior_version": "853'21100", "last_reqid": "client.2317742.0:24909022", "user_version": 21101, "size": 4194304, "mtime": "2020-07-16 21:02:20.564245", "local_mtime": "2020-07-16 21:02:20.572003", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 5, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 19, "primary": true, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 24, "primary": false, "errors": [ "read_error" ], "size": 4194304 } ] } ] } On Tue, Jul 14, 2020 at 6:40 PM Eric Smith <Eric.Smith(a)vecima.com> wrote: > If you run (Substitute your pool name for <pool>): > > rados -p <pool> list-inconsistent-obj 1.574 --format=json-pretty > > You should get some detailed information about which piece of data > actually has the error and you can determine what to do with it from there. > > -----Original Message----- > From: Abhimnyu Dhobale <adhobale8(a)gmail.com> > Sent: Tuesday, July 14, 2020 5:13 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] 1 pg inconsistent > > Good Day, > > Ceph is showing below error frequently. every time after pg repair it is > resolved. > > [root@vpsapohmcs01 ~]# ceph health detail HEALTH_ERR 1 scrub errors; > Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors > PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.574 is > active+clean+inconsistent, acting [19,25,2] > > [root@vpsapohmcs02 ~]# cat /var/log/ceph/ceph-osd.19.log | grep error > 2020-07-12 11:42:11.824 7f864e0b2700 -1 log_channel(cluster) log [ERR] : > 1.574 shard 25 soid > 1:2ea0a7a3:::rbd_data.515c96b8b4567.0000000000007a7c:head : candidate had > a read error > 2020-07-12 11:42:15.035 7f86520ba700 -1 log_channel(cluster) log [ERR] : > 1.574 deep-scrub 1 errors > > [root@vpsapohmcs01 ~]# ceph --version > ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic > (stable) > > Request you to please suggest. > > -- > Thanks & Regards > Abhimnyu Dhobale > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > -- Thanks & Regards Abhimnyu Dhobale 8149311170

2 1

Repo sync
by Szabo, Istvan (Agoda) 17 Jul '20

17 Jul '20

Hi, Earlier I've synced the repos the following way: rsync -avSH rsync://hk.ceph.com/rpm-luminous/el7 . Today I've tried to sync from us-west and hk, but I got an error: rsync -avSH rsync://us-west.ceph.com/rpm-nautilus/el7/noarch . @ERROR: Unknown module 'rpm-nautilus' rsync error: error starting client-server protocol (code 5) at main.c(1503) [receiver=3.0.6] rsync -avSH rsync://us-west.ceph.com/rpm-nautilus/el7/noarch . @ERROR: Unknown module 'rpm-nautilus' rsync error: error starting client-server protocol (code 5) at main.c(1503) [receiver=3.0.6] rsync -avSH rsync://download.ceph.com/rpm-nautilus/el7/noarch . @ERROR: Unknown module 'rpm-nautilus' rsync error: error starting client-server protocol (code 5) at main.c(1503) [receiver=3.0.6] Also the HK repos doesn't have the newest nautilus, just 14.2.9. Any issue? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

Technician’s help to solve canon printer offline error
by michaels1990johnson＠gmail.com 17 Jul '20

17 Jul '20

When you give a right command to your canon printer for printing the documents, your canon printer is showing offline status. Your canon printer is displaying canon printer offline error. If you’re receiving this offline error again and again, you can get connected with our live printer technicians to get full technical guidance for solving canon printer offline error. As a technically recognized third party support guide, we recognize the exact reasons of this error and try to use the effective troubleshooting solutions to resolve this offline problem. Our technician’s help is available round the clock to provide instant support for any issue. https://www.canonprintersupportpro.us/blog/fix-canon-printer-offline-issue/

1 0

Technician’s help to solve canon printer offline error
by michaels1990johnson＠gmail.com 17 Jul '20

17 Jul '20

When you give a right command to your canon printer for printing the documents, your canon printer is showing offline status. Your canon printer is displaying canon printer offline error. If you’re receiving this offline error again and again, you can get connected with our live printer technicians to get full technical guidance for solving canon printer offline error. As a technically recognized third party support guide, we recognize the exact reasons of this error and try to use the effective troubleshooting solutions to resolve this offline problem. Our technician’s help is available round the clock to provide instant support for any issue. https://www.canonprintersupportpro.us/blog/fix-canon-printer-offline-issue/

1 0

Re: Cephfs multiple active-active MDS stability and optimization
by Dan van der Ster 16 Jul '20

16 Jul '20

Hi, We are using 10 active MDS's with v12.2.12 -- so it is "stable" but we have several measures and lots of experience to make it like that. If I were starting a new cluster now, I would use the latest nautilus or octopus and test the hell out of it before going into prod. Don't start with mimic now, it's end-of-line. First, are you really sure you need multi-active MDS? We only use it where the metadata workload clearly exceeds the abilities of a single active MDS. Evidence of this would be a high, flat-lined CPU usage on the active mds, or better would be to track the "hcr" or "handle_client_request" metric with your monitoring or locally on an MDS with "ceph daemonperf mds.`hostname -s`". A single MDS can normally achieve a few thousand hcr/second at best. Otherwise, here are some relatively advanced things to try to validate the setup... understanding and succeeding in these things should help with your nerves: - Start the cluster, run some workloads, try increasing and decreasing max_mds on the fly and make sure this is working well - is the metadata balancing working well with your common workloads? run your test workloads for hours or days and check that the RSS of each MDS is not growing unexpectedly - does mds balancing make sense for your workload, or are there some places where pinning to subdirs to a rank is worthwhile? - with fully active mds, fully loaded metadata caches, test the failover to standby several times. Try "nice" failovers (e.g. systemctl stop ceph-mds.target on an active) as well as "not-so-nice" failovers (e.g. killall -9 ceph-mds) - Try the cephfs scrub features. Maybe even intentionally corrupt a file or direntry object then check if cephfs scrub behaves as expected Hope that helps! Dan On Thu, Jul 16, 2020 at 3:01 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Cepher， > > I am planning a cephfs cluster with ca. 100 OSD nodes, each of which has 12 disks, and 2 NVMe (for db wal and cephfs metadata pool). Fpr performance and scalability reasons, i would like to try multi MDS working ative-active. From what i learned in the past, i am not sure about the following questions. > > 1 Which Ceph version should i run? I had a good experience with Luminous 12.2.13, and not familiar yet with Mimic and Nautilus. Is Lumious 12.2.13 stable enouth to run multiple active-active MDS servers for CephFS? > > 2 If i had to go Mimic or Nautilus for CephFS, which one is perferable? > > 3 I did has some experience with Ceph RBD, but not CephFS, So my question is, what should i pay attention to whening running CephFS? I am somehow nervous...... > > best regards, > > Samuel > > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

CEPH performance issues running as Spark storage layer
by arabtura＠gmail.com 16 Jul '20

16 Jul '20

Hi All, We’re using CEPH cluster (Nautulis 14.2.10) as a S3 object storage layer for Spark 3 with Yarn running in distributed environment. The issue we see however is slow performance when running even simple spark query on data stored on large number of objects, for example 50.000 objects. We’re aware of slow object listing in S3, but should that really kill the performance while using spark for reading\analyzing\writing the data on S3? Running the same query on the same dataset content but stored in 100 files is multiple times faster. Bucket and bucket index we use for spark are stored on OSDs with SSDs (we’re having 150 of them) , we’re using 12 RGW instances, each limited to 32 concurrent connection by custom app to prevent RGW queue blowing up). When running spark queries, RGW queues rises – depending on number of spark executors – to around 30 per instance providing the number of executors per RGW instance is also in similar. We don’t see any other bottlenecks on infra side than RGW queues. We applied various tuning options for CEPH regarding RGW\OSD\Bluestore performance (ie: objecter_inflight_op_bytes, objecter_inflight_ops, rgw_bucket_index_max_aio, rgw_cache_lru_size ) but the spark works still really slow in above mentioned scenario. The other problem we observed is that when using 4 RGW instead of 12 we see performance degradation only in about 40-50%. We would expect that RGW scaling will behave more efficiently. Does anyone using CEPH in similar way as a storage layer for Spark? Do you observer similar behavior and maybe have some workarounds\solutions for slowness when working with high number of objects?

1 0

Cephfs multiple active-active MDS stability and optimization
by huxiaoyu＠horebdata.cn 16 Jul '20

16 Jul '20

Dear Cepher， I am planning a cephfs cluster with ca. 100 OSD nodes, each of which has 12 disks, and 2 NVMe (for db wal and cephfs metadata pool). Fpr performance and scalability reasons, i would like to try multi MDS working ative-active. From what i learned in the past, i am not sure about the following questions. 1 Which Ceph version should i run? I had a good experience with Luminous 12.2.13, and not familiar yet with Mimic and Nautilus. Is Lumious 12.2.13 stable enouth to run multiple active-active MDS servers for CephFS? 2 If i had to go Mimic or Nautilus for CephFS, which one is perferable? 3 I did has some experience with Ceph RBD, but not CephFS, So my question is, what should i pay attention to whening running CephFS? I am somehow nervous...... best regards, Samuel huxiaoyu(a)horebdata.cn

1 0

AdminSocket occurs segment fault with samba vfs ceph plugin
by 380562518＠qq.com 16 Jul '20

16 Jul '20

Ceph Version: 14.2.5 Samba Version:4.10.4 OS: Centos 7.6.1810 Procedure: 1，Setup a ceph cluster and create a file system. 2，Setup a samba share with samba vfs ceph plugin. 3，Use 'ceph daemon /var/run/ceph/ceph-client.admin.*****.asok help' to get debug command. 4，It will occurs a segment fault when exec the 3rd command. The call trace list belows: #0 0x00007f35425cb387 in raise () from /lib64/libc.so.6 #1 0x00007f35425cca78 in abort () from /lib64/libc.so.6 #2 0x00007f3545a8fff3 in dump_core () at ../source3/lib/dumpcore.c:338 #3 0x00007f3545a8109b in smb_panic_s3 (why=<optimized out>) at ../source3/lib/util.c:839 #4 0x00007f354627297f in smb_panic (why=why@entry=0x7f35462bb55b "internal error") at ../lib/util/fault.c:174 #5 0x00007f3546272bb6 in fault_report (sig=<optimized out>) at ../lib/util/fault.c:88 #6 sig_fault (sig=<optimized out>) at ../lib/util/fault.c:99 #7 <signal handler called> #8 0x00007f35226d026f in acquire (this=0x564a0a014050) at /usr/src/debug/ceph-14.2.5-1.0.8/build/boost/include/boost/smart_ptr/detail/shared_count.hpp:426 #9 acquire_object_id (this=0x7f350fffe220) at /usr/src/debug/ceph-14.2.5-1.0.8/build/boost/include/boost/spirit/home/classic/core/non_terminal/impl/object_with_id.ipp:157 #10 object_with_id (this=0x7f350fffe220) at /usr/src/debug/ceph-14.2.5-1.0.8/build/boost/include/boost/spirit/home/classic/core/non_terminal/impl/object_with_id.ipp:79 #11 grammar (this=0x7f350fffe220) at /usr/src/debug/ceph-14.2.5-1.0.8/build/boost/include/boost/spirit/home/classic/core/non_terminal/grammar.hpp:51 #12 Json_grammer (semantic_actions=..., this=0x7f350fffe220) at /usr/src/debug/ceph-14.2.5-1.0.8/src/json_spirit/json_spirit_reader_template.h:401 #13 json_spirit::read_range_or_throw<__gnu_cxx::__normal_iterator<char const*, std::string>, json_spirit::Value_impl<json_spirit::Config_map<std::string> > > (begin=123 '{', end=0 '\000', value=...) at /usr/src/debug/ceph-14.2.5-1.0.8/src/json_spirit/json_spirit_reader_template.h:585 #14 0x00007f35226d0c3c in json_spirit::read_range<__gnu_cxx::__normal_iterator<char const*, std::string>, json_spirit::Value_impl<json_spirit::Config_map<std::string> > > ( begin=123 '{', end=..., value=...) at /usr/src/debug/ceph-14.2.5-1.0.8/src/json_spirit/json_spirit_reader_template.h:607 #15 0x00007f35226bb07d in read_string<std::basic_string<char>, json_spirit::Value_impl<json_spirit::Config_map<std::basic_string<char> > > > (value=..., s=<error reading variable: Cannot access memory at address 0x7869666572702263>) at /usr/src/debug/ceph-14.2.5-1.0.8/src/json_spirit/json_spirit_reader.cpp:78 #16 json_spirit::read (s="{\"prefix\": \"get_command_descriptions\"}", value=...) at /usr/src/debug/ceph-14.2.5-1.0.8/src/json_spirit/json_spirit_reader.cpp:78 #17 0x00007f35222953a7 in cmdmap_from_json(std::vector<std::string, std::allocator<std::string> >, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > >*, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&) () at /usr/src/debug/ceph-14.2.5-1.0.8/src/common/cmdparse.cc:294 #18 0x00007f3522255492 in AdminSocket::execute_command(std::string const&, ceph::buffer::v14_2_0::list&) () at /opt/rh/devtoolset-8/root/usr/include/c++/8/new:169 #19 0x00007f35222567ef in AdminSocket::do_accept() () at /usr/src/debug/ceph-14.2.5-1.0.8/src/common/admin_socket.cc:341 #20 0x00007f3522258cc8 in AdminSocket::entry (this=0x564a09f3a950) at /usr/src/debug/ceph-14.2.5-1.0.8/src/common/admin_socket.cc:241 #21 0x00007f352270f68f in execute_native_thread_routine () from /usr/lib64/ceph/libceph-common.so.0 #22 0x00007f35466d1ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f35426938dd in clone () from /lib64/libc.so.6

2 2

client - monitor communication.
by Budai Laszlo 16 Jul '20

16 Jul '20

Hello everybody, I'm trying to figure out how often the ceph client is contacting the monitors for updating its own information about the cluster map. Can anyone point me to a document describing this client <-> monitor communication? Thank you, Laszlo

4 6

Monitor IPs
by Will Payne 16 Jul '20

16 Jul '20

I need to change the network my monitors are on. It seems this is not a trivial thing to do. Are there any up-to-date instructions for doing so on a cephadm-deployed cluster? I’ve found some steps in older versions of the docs but not sure if these are still correct - they mention using the ceph-mon command which I don’t have. Will

5 4

about replica size
by Zhenshi Zhou 16 Jul '20

16 Jul '20

Hi, As we all know, the default replica setting of 'size' is 3 which means there are 3 copies of an object. What is the disadvantages if I set it to 2, except I get fewer copies? Thanks

6 6

high commit_latency and apply_latency
by rainning 16 Jul '20

16 Jul '20

we have a cluster with very low load, however, "ceph osd perf" shows high commit_latency and apply_latency. root@stor-mgt01:~# ceph -s   cluster:     id:     3d1ec789-829d-4e0f-b707-9363356a68f1     health: HEALTH_WARN             application not enabled on 3 pool(s)     services:     mon: 3 daemons, quorum a,b,c     mgr: a(active)     mds: rook-ceph-filesystem-1/1/1 up  {0=rook-ceph-filesystem-a=up:active}, 1 up:standby-replay     osd: 55 osds: 55 up, 55 in     rgw: 3 daemons active     data:     pools:   18 pools, 3328 pgs     objects: 1.06M objects, 1.37TiB     usage:   4.18TiB used, 141TiB / 145TiB avail     pgs:     3328 active+clean     io:     client:   46.2KiB/s rd, 2.58MiB/s wr, 197op/s rd, 273op/s wr ---------------------------------------------------------------------------------------- osd commit_latency(ms) apply_latency(ms)  54                  0                 0  53                  0                 0  52                  0                 0  51                  0                 0  50                  0                 0  49                 11                11  48                  9                 9  47                  3                 3  46                 39                39  21                 28                28  20                 26                26  19                 28                28  18                  1                 1  17                 17                17  16                  6                 6  15                 11                11  14                  9                 9  13                  8                 8  12                 12                12  11                 14                14  10                 21                21   0                 10                10   1                 12                12   2                 30                30   3                  3                 3   4                  3                 3   5                 23                23   6                 56                56   7                  2                 2   8                 36                36   9                  5                 5  22                 26                26  23                  3                 3  24                  2                 2  25                  3                 3  26                 19                19  27                  3                 3  28                  4                 4  29                  8                 8  30                 16                16  31                 21                21  32                 13                13  33                  7                 7  34                 10                10  35                 12                12  36                  5                 5  37                  5                 5  38                  4                 4  39                 29                29  40                 26                26  41                 35                35  42                  5                 5  43                  4                 4  44                  8                 8  45                  6                 6

1 0

How to deal with the incomplete records in rocksdb
by zhouli_2000＠163.com 16 Jul '20

16 Jul '20

Hi all, rocksdb failed to open when the ceph-osd process was restarted after unplugging the OSD data disk with Ceph 14.2.5 on Centos 7.6. 1) After unplugging the OSD data disk, the ceph-osd process exist. -3> 2020-07-13 15:25:35.912 7f1ad7254700 -1 bdev(0x559d1134f880 /var/lib/ceph/osd/ceph-10/block) _sync_write sync_file_range error: (5) Input/output error -2> 2020-07-13 15:25:35.912 7f1ad9c5f700 -1 bdev(0x559d1134f880 /var/lib/ceph/osd/ceph-10/block) _aio_thread got r=-5 ((5) Input/output error) -1> 2020-07-13 15:25:35.917 7f1ad9c5f700 -1 /root/rpmbuild/BUILD/ceph-14.2.5-1.0.9/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f1ad9c5f700 time 2020-07-13 15:25:35.913821 /root/rpmbuild/BUILD/ceph-14.2.5-1.0.9/src/os/bluestore/KernelDevice.cc: 534: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") ceph version 14.2.5-93-g9a4f93e (9a4f93e7143bcdd5fadc88eb58bb730ae97b89c5) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x559d05b6069a] 2: (KernelDevice::_aio_thread()+0xebe) [0x559d061a54ee] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559d061a7add] 4: (()+0x7dd5) [0x7f1ae66aedd5] 5: (clone()+0x6d) [0x7f1ae5572ead] 2) Plug the disk back in and restart the ceph-osd process, rocksdb found that incomplete records existed and stop to work. 2020-07-13 15:51:38.305 7f9801ef5a80 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #9 mode 0 2020-07-13 15:51:38.748 7f9801ef5a80 3 rocksdb: [db/db_impl_open.cc:518] db.wal/000009.log: dropping 2922 bytes; Corruption: missing start of fragmented record(2) 2020-07-13 15:51:38.748 7f9801ef5a80 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work 2020-07-13 15:51:38.748 7f9801ef5a80 4 rocksdb: [db/db_impl.cc:563] Shutdown complete 2020-07-13 15:51:38.748 7f9801ef5a80 -1 rocksdb: Corruption: missing start of fragmented record(2) 2020-07-13 15:51:38.748 7f9801ef5a80 -1 bluestore(/var/lib/ceph/osd/ceph-10) _open_db erroring opening db: 2020-07-13 15:51:38.748 7f9801ef5a80 1 bluefs umount 2020-07-13 15:51:38.776 7f9801ef5a80 1 fbmap_alloc 0x55c897e0a900 shutdown 2020-07-13 15:51:38.776 7f9801ef5a80 1 bdev(0x55c898a6ce00 /var/lib/ceph/osd/ceph-10/block) close Why does rocksdb not automatically delete these incomplete records and continue work? In addition, after the occurrence of this situation, what method should be used to recover.

1 0

回复：Re: osd bench with or without a separate WAL device deployed
by rainning 16 Jul '20

16 Jul '20

I tested osd bench with different block size: 1MB, 512KB, 256KB, 128KB, 64KB, and 32KB. osd.2 is one from the cluster where osds have better 4KB osd bench, and osd.30 is from the cluster where osds have lower 4KB osd bench.  Before 32KB, osd.30 was better than osd.2, however, there was a big drop on osd.30 with 32KB block size. root@cmn01:~# ceph tell osd.2 bench 1073741824 1048576 {     "bytes_written": 1073741824,     "blocksize": 1048576,     "bytes_per_sec": 188747963 } root@cmn01:~# ceph tell osd.2 bench 1073741824 524288 {     "bytes_written": 1073741824,     "blocksize": 524288,     "bytes_per_sec": 181071543 } root@cmn01:~# ceph tell osd.2 bench 786432000 262144 {     "bytes_written": 786432000,     "blocksize": 262144,     "bytes_per_sec": 159007035 } root@cmn01:~# ceph tell osd.2 bench 393216000 131072 {     "bytes_written": 393216000,     "blocksize": 131072,     "bytes_per_sec": 127179122 } root@cmn01:~# ceph tell osd.2 bench 196608000 65536 {     "bytes_written": 196608000,     "blocksize": 65536,     "bytes_per_sec": 83365482 } root@cmn01:~# ceph tell osd.2 bench 98304000 32768 {     "bytes_written": 98304000,     "blocksize": 32768,     "bytes_per_sec": 48351258 } root@cmn01:~# ceph tell osd.2 bench 49152000 16384 {     "bytes_written": 49152000,     "blocksize": 16384,     "bytes_per_sec": 31725841 } ------------------------------------------------------------------------------------------------------------------ root@stor-mgt01:~# ceph tell osd.30 bench 1073741824 1048576 {     "bytes_written": 1073741824,     "blocksize": 1048576,     "elapsed_sec": 5.344805,     "bytes_per_sec": 200894474.890259,     "iops": 191.587901 } root@stor-mgt01:~# ceph tell osd.30 bench 1073741824 524288 {     "bytes_written": 1073741824,     "blocksize": 524288,     "elapsed_sec": 5.303052,     "bytes_per_sec": 202476205.680661,     "iops": 386.192714 } root@stor-mgt01:~# ceph tell osd.30 bench 786432000 262144 {     "bytes_written": 786432000,     "blocksize": 262144,     "elapsed_sec": 3.878248,     "bytes_per_sec": 202780204.655892,     "iops": 773.545092 } root@stor-mgt01:~# ceph tell osd.30 bench 393216000 131072 {     "bytes_written": 393216000,     "blocksize": 131072,     "elapsed_sec": 1.939532,     "bytes_per_sec": 202737591.242988,     "iops": 1546.765070 } root@stor-mgt01:~# ceph tell osd.30 bench 196608000 65536 {     "bytes_written": 196608000,     "blocksize": 65536,     "elapsed_sec": 1.081617,     "bytes_per_sec": 181772360.338257,     "iops": 2773.626104 } root@stor-mgt01:~# ceph tell osd.30 bench 98304000 32768 {     "bytes_written": 98304000,     "blocksize": 32768,     "elapsed_sec": 2.908703,     "bytes_per_sec": 33796507.598640,     "iops": 1031.387561 } root@stor-mgt01:~# ceph tell osd.30 bench 49152000 16384 {     "bytes_written": 49152000,     "blocksize": 16384,     "elapsed_sec": 3.907744,     "bytes_per_sec": 12578102.861185,     "iops": 767.706473 } ------------------ 原始邮件 ------------------ 发件人: "rainning" <tweetypie(a)qq.com>; 发送时间: 2020年7月16日(星期四) 上午9:42 收件人: "Zhenshi Zhou"<deaderzzs(a)gmail.com>; 抄送: "ceph-users"<ceph-users(a)ceph.io>; 主题: 回复：[ceph-users] Re: osd bench with or without a separate WAL device deployed Hi Zhenshi, I did try with bigger block size. Interestingly, the one whose 4KB osd bench was lower performed slightly better in 4MB osd bench. Let me try some other bigger block sizes, e.g. 16K, 64K, 128K, 1M etc, to see if there is any pattern. Moreover, I did compare two SSDs, they respectively are INTEL SSDSC2KB480G8 and INTEL SSDSC2KB960G8. Performance wise, there is no much difference. Thanks, Ning ------------------ 原始邮件 ------------------ 发件人: "Zhenshi Zhou" <deaderzzs(a)gmail.com>; 发送时间: 2020年7月16日(星期四) 上午9:24 收件人: "rainning"<tweetypie(a)qq.com>; 抄送: "ceph-users"<ceph-users(a)ceph.io>; 主题: [ceph-users] Re: osd bench with or without a separate WAL device deployed Maybe you can try writing with bigger block size and compare the results. For bluestore, the write operations contain two modes. One is COW, the other is RMW. AFAIK only RMW uses wal in order to prevent data from being interrupted. rainning <tweetypie(a)qq.com> 于2020年7月15日周三下午11:04写道： > Hi Zhenshi, thanks very much for the reply. > > Yes I know it is ood that the bluestore is deployed only with a separate > db device  but no a WAL device. The cluster was deployed in k8s using rook. > I was told it was because the rook we used didn't support that. > > Moreover, the comparison was made on osd bench, so the network should not > be the case. As far as the storage node hardware, although two clusters are > indeed different, their CPUs and HDDs do have almost same performance > numbers. I haven't compared SSDs that are used as db/WAL devices, it might > cause difference, but I am not sure if it can make two times difference. > > ---Original--- > *From:* "Zhenshi Zhou"<deaderzzs(a)gmail.com> > *Date:* Wed, Jul 15, 2020 18:39 PM > *To:* "rainning"<tweetypie(a)qq.com>; > *Cc:* "ceph-users"<ceph-users(a)ceph.io>; > *Subject:* [ceph-users] Re: osd bench with or without a separate WAL > device deployed > > I deployed the cluster either with separate db/wal or put db/wal/data > together. Never tried to have only a seperate db. > AFAIK wal does have an effect on writing but I'm not sure if it could be > two times of the bench value. Hardware and > network environment are also important factors. > > rainning <tweetypie(a)qq.com> 于2020年7月15日周三下午4:35写道： > > > Hi all, > > > > > > I am wondering if there is any performance comparison done on osd bench > > with and without a separate WAL device deployed given that there is > always > > a separate db device deployed on SSD in both cases. > > > > > > The reason I am asking this question is that we have two clusters and > osds > > in one have separate db and WAL device deployed on SSD but osds in > another > > only have a separate db device deployed. And we found 4KB osd bench (i.e. > > ceph tell osd.X bench 12288000 4096) for the ones having a separate WAL > > device was two times of the ones without a separate WAL device. Is the > > performance difference caused by the separate WAL device? > > > > > > Thanks, > > Ning > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

osd bench with or without a separate WAL device deployed
by rainning 16 Jul '20

16 Jul '20

Hi all, I am wondering if there is any performance comparison done on osd bench with and without a separate WAL device deployed given that there is always a separate db device deployed on SSD in both cases. The reason I am asking this question is that we have two clusters and osds in one have separate db and WAL device deployed on SSD but osds in another only have a separate db device deployed. And we found 4KB osd bench (i.e. ceph tell osd.X bench 12288000 4096) for the ones having a separate WAL device was two times of the ones without a separate WAL device. Is the performance difference caused by the separate WAL device? Thanks, Ning

2 5

Re: [RGW] Space usage vastly overestimated since Octopus upgrade
by David Monschein 16 Jul '20

16 Jul '20

Hi Liam, All, We have also run into this bug: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PCYY2MKRPCP… Like you, we are also running Octopus 15.2.3 Downgrading the RGWs at this point is not ideal, but if a fix isn't found soon we might have to. Has a bug report been filed for this yet? - Dave

1 0

crimson/seastor
by Frank Ritchie 15 Jul '20

15 Jul '20

Hi all, Does anyone know when we can expect Crimson/Seastor to be "Production Ready" and/or what level of performance increase can be expected? thx Frank

1 0

Rados Gateway sync requests are not balance between nodes
by Nghia Viet Tran 15 Jul '20

15 Jul '20

Hello everyone, Below is our current setup, the master zone is on the master datacenter and the backup zone is on the standby datacenter which has very slow bandwidth internet connection. The client application connects to the Master zone load balancer for uploading objects to cluster. Later on, the Gate way nodes from the Backup zone will synchronize new objects into the Backup cluster by requesting object data to the load balancer of the master zone. [A screenshot of a cell phone Description automatically generated] The current connection inside Gateway nodes of the backup zone [A screenshot of a computer Description automatically generated] From the above image, you can see that the first Gateway node create a lot of connection to the master zone balancer (cephmm-03) for checking objects status. That means this gateway mostly will take the role for downloading new objects from the master zone to the backup zone (very high load on this node). On the other hand, the second Gateway node just has only 2 connections to the master node balancer (mostly free all the time). This second node only increase connections if the first node (or may be the third node) is down… Can you explain this behavior of Ceph in this case and how to let all the gateway node in active mode? I appreciate any comments from you! -- Nghia Viet Tran (Mr)

1 0

Ceph and Red Hat Summit 2020
by Bobby 15 Jul '20

15 Jul '20

Hi, Any Ceph related event in Red Hat Summit 2020 happening today? BR Bobby !

1 0

Re: mon_osd_down_out_subtree_limit not working?
by Frank Schilder 15 Jul '20

15 Jul '20

Yes. After the time-out of 600 secs the OSDs got marked down, all PGs got remapped and recovery/rebalancing started as usual. In the past, I did service on servers with the flag noout set and would expect that mon_osd_down_out_subtree_limit=host has the same effect when shutting down an entire host. Unfortunately, in my case these two settings behave differently. If I understand the documentation correctly, the OSDs should not get marked out automatically. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 14 July 2020 04:32:05 To: Frank Schilder Subject: Re: [ceph-users] mon_osd_down_out_subtree_limit not working? Did it start rebalancing? > On Jul 13, 2020, at 4:29 AM, Frank Schilder <frans(a)dtu.dk> wrote: > > if I shut down all OSDs on this host, these OSDs should not be marked out automatically after mon_osd_down_out_interval(=600) seconds. I did a test today and, unfortunately, the OSDs do get marked as out. Ceph status was showing 1 host down as expected.

2 7

Re: Poor Windows performance on ceph RBD.
by Frank Schilder 15 Jul '20

15 Jul '20

> If I may ask, which version of the virtio drivers do you use? https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-vir… Looks like virtio-win-0.1.185.* > And do you use caching on libvirt driver level? In the ONE interface, we use DISK = [ driver = "raw" , cache = "none"] which translates to <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> in the XML. We have no qemu settings in the ceph.conf. Looks like caching is disabled. Not sure if this is the recommended way though and why caching is disabled by default. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: André Gemünd <andre.gemuend(a)scai.fraunhofer.de> Sent: 13 July 2020 11:18 To: Frank Schilder Subject: Re: [ceph-users] Re: Poor Windows performance on ceph RBD. If I may ask, which version of the virtio drivers do you use? And do you use caching on libvirt driver level? Greetings André ----- Am 13. Jul 2020 um 10:43 schrieb Frank Schilder frans(a)dtu.dk: >> > To anyone who is following this thread, we found a possible explanation for >> > (some of) our observations. > >> If someone is following this, they probably want the possible >> explanation and not the knowledge of you having the possible >> explanation. > >> So you are saying if you do eg. a core installation (without gui) of >> 2016/2019 disable all services. The fio test results are signficantly >> different to eg. a centos 7 vm doing the same fio test? Are you sure >> this is not related to other processes writing to disk? > > Right, its not an explanation but rather a further observation. We don't really > have an explanation yet. > > Its an identical installation of both server versions, same services configured. > Our operators are not really into debugging Windows, that's why we were asking > here. Their hypothesis is, that the VD driver for accessing RBD images has > problems with Windows servers newer than 2016. I'm not a Windows guy, so can't > really comment on this. > > The test we do is a simple copy-test of a single 10g file and we monitor the > transfer speed. This info was cut out of this e-mail, the original report for > reference is: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT4… > . > > We are very sure that it is not related to other processes writing to disk, we > monitor that too. There is also no competition on the RBD pool at the time of > testing. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Sent: 13 July 2020 10:24 > To: ceph-users; Frank Schilder > Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD. > >>> To anyone who is following this thread, we found a possible > explanation for >>> (some of) our observations. > > If someone is following this, they probably want the possible > explanation and not the knowledge of you having the possible > explanation. > > So you are saying if you do eg. a core installation (without gui) of > 2016/2019 disable all services. The fio test results are signficantly > different to eg. a centos 7 vm doing the same fio test? Are you sure > this is not related to other processes writing to disk? > > > > -----Original Message----- > From: Frank Schilder [mailto:frans@dtu.dk] > Sent: maandag 13 juli 2020 9:28 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Poor Windows performance on ceph RBD. > > To anyone who is following this thread, we found a possible explanation > for (some of) our observations. > > We are running Windows servers version 2016 and 2019 as storage servers > exporting data on an rbd image/disk. We recently found that Windows > server 2016 runs fine. It is still not as fast as Linux + SAMBA share on > an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth. > With Windows server 2019, however, we observe near-complete stall of > file transfers and time-outs using standard copy tools (robocopy). We > don't have an explanation yet and are downgrading Windows servers where > possible. > > If anyone has a hint what we can do, please let us know. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Dipl.-Inf. André Gemünd, Leiter IT / Head of IT Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemuend(a)scai.fraunhofer.de Tel: +49 2241 14-2193 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

1 1

how to configure cephfs-shell correctly
by Zhenshi Zhou 15 Jul '20

15 Jul '20

Hi all, I want to use cephfs-shell dealing with operations like directory creation, instead of mounting the root directory and create manually. But I get errors when I execute the command 'cephfs-shell'. Traceback (most recent call last): File "./cephfs-shell", line 9, in <module> import cephfs as libcephfs ModuleNotFoundError: No module named 'cephfs' CentOS7 use python2 and I've already installed python3 in the system. What else should I do, reinstall libcephfs? Thanks

2 9

YUM doesn't find older release version of nautilus
by Lee, H. (Hurng-Chun) 14 Jul '20

14 Jul '20

Hi, We use the official Ceph RPM repository (http://download.ceph.com/rpm- nautilus/el7) for installing packages on the client nodes running CentOS7. But we noticed today that the repo only provides the latest version (2:14.2.10-0.el7) of nautilus so that we couldn't install an older (2:14.2.7-0.el7) version of the ceph-common package. The ceph.repo file contains: --- [Ceph] name=Ceph packages for $basearch baseurl=http://download.ceph.com/rpm-nautilus/el7/$basearch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc --- and --- $ yum clean all $ yum makecache $ yum list --disablerepo=* --enablerepo=Ceph --showduplicates ceph- common Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Available Packages ceph-common.x86_64 2:14.2.10- 0.el7 Ceph --- Does it mean that the official repo no longer provides RPM packages for older versions? Thanks! Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0) 243610977 web: http://www.ru.nl/donders/

3 3

Web UI errors
by Will Payne 14 Jul '20

14 Jul '20

Hi, I’ve been getting errors in the NFS section of the web interface. I’ve just tried upgrading to 15.2.4 to see if that helped but no joy. The initial NFS page loads OK and when I click Add, a form loads. However, when this form attempts to update its values, I get a red box informing me that the server returned an error 500. it makes four HTTP calls - to daemon, clients, filesystems and fsals - these all fail. Any suggestions on what might be wrong or where some useful logs might be? Ta, Will

2 1

User stats - Object count wrong in Octopus?
by David Monschein 14 Jul '20

14 Jul '20

Hi All, Sorry for the double email, I accidentally sent the previous e-mail with an accidental KB shortcut before it was finished :) I'm investigating what appears to be a bug in RGW stats. This is a brand new cluster running 15.2.3 One of our customers reached out, saying they were hitting their quota (S3 error: 403 (QuotaExceeded)). The user-wide max_objects quota we set is 50 million objects, so this would be impossible since the entire cluster isn't even close to 50 million objects yet: [root@os1 ~]# ceph status | grep objects objects: 7.58M objects, 6.8 TiB The customer in question has three buckets, and if I query the bucket stats, the total number of objects for all 3 buckets comes to about ~372k: [root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 | grep num_objects "num_objects": 324880 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo | grep num_objects "num_objects": 47476 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-test | grep num_objects "num_objects": 1 But things get interesting when I query the user stats: [root@os1 ~]# radosgw-admin user stats --uid=user-in-question | grep num_objects "num_objects": 52543794 How is Ceph arriving at 52+ million objects? Here is the full output from the bucket stats and user stats if it's any help. Thanks for any assistance. [root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 { "bucket": "df-fs1", "num_shards": 491, "tenant": "", "zonegroup": "a1b72aa0-eb06-4a96-8af4-db39d4dc3e09", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9736264.34", "marker": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9736264.21", "index_type": "Normal", "owner": "01c235fd-fc9b-4c54-a2a5-b38054608b9e", "ver": "0#57,1#58,2#59,3#56,4#63,5#62,6#52,7#61,8#48,9#58,10#56,11#58,12#64,13#45,14#57,15#66,16#66,17#55,18#52,19#60,20#74,21#59,22#57,23#47,24#60,25#52,26#62,27#58,28#60,29#47,30#57,31#45,32#48,33#56,34#64,35#68,36#47,37#49,38#62,39#48,40#64,41#44,42#60,43#60,44#60,45#59,46#65,47#47,48#57,49#57,50#63,51#68,52#61,53#57,54#46,55#59,56#57,57#41,58#59,59#58,60#57,61#53,62#52,63#45,64#55,65#58,66#54,67#43,68#60,69#51,70#62,71#55,72#52,73#55,74#60,75#77,76#59,77#54,78#57,79#65,80#60,81#64,82#62,83#48,84#57,85#51,86#53,87#61,88#62,89#63,90#59,91#53,92#55,93#59,94#50,95#51,96#59,97#58,98#55,99#44,100#47,101#48,102#61,103#47,104#54,105#57,106#50,107#72,108#66,109#53,110#48,111#49,112#51,113#69,114#61,115#59,116#56,117#52,118#61,119#64,120#60,121#56,122#62,123#46,124#50,125#57,126#59,127#58,128#62,129#62,130#54,131#47,132#79,133#52,134#53,135#54,136#52,137#40,138#45,139#54,140#43,141#55,142#56,143#70,144#52,145#58,146#55,147#46,148#54,149#64,150#52,151#56,152#55,153#70,154#45,155#66,156#65,157#58,158#61,159#58,160#57,161#62,162#49,163#76,164#70,165#59,166#68,167#64,168#51,169#68,170#62,171#54,172#51,173#64,174#47,175#59,176#53,177#51,178#44,179#51,180#71,181#60,182#57,183#65,184#68,185#58,186#63,187#51,188#65,189#66,190#47,191#63,192#55,193#56,194#59,195#59,196#53,197#38,198#53,199#47,200#64,201#59,202#56,203#51,204#70,205#50,206#55,207#63,208#42,209#44,210#56,211#50,212#56,213#55,214#56,215#55,216#51,217#57,218#58,219#60,220#47,221#49,222#51,223#42,224#52,225#54,226#56,227#58,228#57,229#53,230#55,231#53,232#47,233#59,234#59,235#58,236#50,237#51,238#54,239#51,240#60,241#53,242#65,243#63,244#60,245#57,246#41,247#64,248#54,249#48,250#49,251#60,252#53,253#53,254#52,255#57,256#62,257#74,258#62,259#57,260#42,261#63,262#52,263#55,264#63,265#47,266#51,267#61,268#49,269#62,270#61,271#63,272#45,273#62,274#53,275#53,276#58,277#58,278#67,279#58,280#59,281#60,282#52,283#43,284#66,285#63,286#50,287#65,288#62,289#55,290#60,291#53,292#50,293#61,294#67,295#69,296#50,297#59,298#50,299#61,300#55,301#61,302#61,303#59,304#51,305#57,306#50,307#60,308#66,309#64,310#66,311#59,312#56,313#51,314#41,315#53,316#54,317#63,318#59,319#54,320#47,321#62,322#57,323#53,324#55,325#54,326#58,327#59,328#42,329#55,330#57,331#53,332#56,333#63,334#55,335#56,336#56,337#52,338#55,339#48,340#50,341#57,342#58,343#47,344#65,345#44,346#70,347#63,348#48,349#59,350#48,351#58,352#57,353#49,354#55,355#55,356#63,357#63,358#59,359#52,360#55,361#60,362#60,363#63,364#62,365#63,366#58,367#68,368#58,369#61,370#64,371#49,372#60,373#62,374#55,375#55,376#52,377#53,378#65,379#58,380#69,381#54,382#53,383#65,384#68,385#61,386#36,387#60,388#55,389#53,390#58,391#53,392#65,393#49,394#52,395#56,396#62,397#50,398#72,399#56,400#52,401#54,402#54,403#63,404#58,405#62,406#56,407#49,408#54,409#61,410#65,411#48,412#60,413#57,414#49,415#61,416#54,417#62,418#53,419#53,420#52,421#51,422#63,423#59,424#48,425#55,426#66,427#56,428#44,429#54,430#61,431#52,432#43,433#44,434#56,435#51,436#60,437#49,438#57,439#54,440#59,441#56,442#54,443#55,444#60,445#58,446#55,447#67,448#53,449#58,450#59,451#58,452#64,453#64,454#62,455#63,456#65,457#60,458#57,459#54,460#47,461#44,462#56,463#58,464#47,465#55,466#44,467#60,468#74,469#68,470#70,471#55,472#60,473#64,474#54,475#56,476#60,477#47,478#63,479#54,480#54,481#55,482#51,483#58,484#47,485#49,486#60,487#52,488#56,489#46,490#43", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0", "mtime": "2020-07-08T23:46:44.705864Z", "creation_time": "2020-07-04T19:41:11.084899Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#", "usage": { "rgw.main": { "size": 69374944756, "size_actual": 70013927424, "size_utilized": 69374944756, "size_kb": 67748970, "size_kb_actual": 68372976, "size_kb_utilized": 67748970, "num_objects": 324880 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } [root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo { "bucket": "df-oldrepo", "num_shards": 967, "tenant": "", "zonegroup": "a1b72aa0-eb06-4a96-8af4-db39d4dc3e09", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732904.1", "marker": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732660.31", "index_type": "Normal", "owner": "01c235fd-fc9b-4c54-a2a5-b38054608b9e", "ver": "0#2,1#2,2#2,3#2,4#2,5#2,6#2,7#2,8#2,9#2,10#2,11#2,12#2,13#2,14#2,15#3,16#2,17#2,18#2,19#2,20#2,21#2,22#3,23#2,24#2,25#2,26#2,27#2,28#2,29#2,30#2,31#2,32#2,33#2,34#2,35#2,36#2,37#2,38#3,39#2,40#2,41#2,42#2,43#2,44#2,45#3,46#2,47#2,48#2,49#2,50#2,51#2,52#2,53#2,54#2,55#2,56#2,57#2,58#2,59#2,60#2,61#2,62#2,63#2,64#2,65#2,66#2,67#2,68#3,69#2,70#2,71#2,72#2,73#2,74#2,75#2,76#2,77#2,78#2,79#2,80#2,81#2,82#2,83#2,84#2,85#2,86#2,87#2,88#2,89#2,90#2,91#2,92#3,93#2,94#2,95#2,96#2,97#2,98#2,99#2,100#2,101#2,102#2,103#2,104#3,105#2,106#2,107#2,108#2,109#2,110#2,111#2,112#2,113#2,114#2,115#2,116#2,117#3,118#2,119#3,120#3,121#2,122#2,123#2,124#2,125#2,126#3,127#2,128#2,129#2,130#2,131#2,132#2,133#2,134#2,135#2,136#2,137#2,138#2,139#2,140#2,141#2,142#2,143#2,144#2,145#2,146#2,147#2,148#2,149#2,150#2,151#2,152#2,153#2,154#2,155#2,156#2,157#2,158#2,159#2,160#2,161#2,162#2,163#2,164#2,165#2,166#2,167#2,168#2,169#2,170#2,171#2,172#2,173#2,174#2,175#2,176#2,177#2,178#2,179#2,180#3,181#2,182#2,183#2,184#2,185#2,186#2,187#2,188#2,189#2,190#2,191#2,192#2,193#2,194#2,195#2,196#2,197#2,198#2,199#2,200#2,201#2,202#2,203#2,204#2,205#2,206#2,207#2,208#2,209#2,210#2,211#2,212#2,213#2,214#2,215#2,216#2,217#2,218#2,219#2,220#2,221#2,222#2,223#2,224#2,225#2,226#2,227#2,228#2,229#2,230#2,231#2,232#2,233#2,234#2,235#2,236#2,237#2,238#2,239#2,240#2,241#2,242#2,243#2,244#2,245#2,246#2,247#2,248#2,249#2,250#2,251#2,252#2,253#2,254#2,255#2,256#2,257#2,258#2,259#2,260#2,261#2,262#2,263#2,264#2,265#2,266#2,267#2,268#2,269#2,270#2,271#2,272#2,273#2,274#2,275#2,276#2,277#2,278#2,279#2,280#2,281#2,282#2,283#2,284#2,285#2,286#2,287#2,288#2,289#2,290#2,291#2,292#2,293#2,294#2,295#2,296#2,297#2,298#2,299#3,300#2,301#2,302#2,303#2,304#2,305#2,306#2,307#2,308#2,309#2,310#2,311#2,312#2,313#2,314#2,315#2,316#2,317#2,318#2,319#2,320#2,321#2,322#2,323#2,324#2,325#2,326#2,327#2,328#2,329#2,330#2,331#2,332#2,333#2,334#2,335#2,336#2,337#2,338#2,339#2,340#2,341#2,342#2,343#2,344#2,345#2,346#2,347#2,348#2,349#2,350#2,351#2,352#2,353#2,354#2,355#2,356#2,357#2,358#2,359#2,360#2,361#2,362#2,363#2,364#2,365#2,366#2,367#2,368#2,369#2,370#2,371#2,372#2,373#2,374#2,375#2,376#2,377#2,378#2,379#2,380#2,381#2,382#2,383#2,384#2,385#2,386#2,387#2,388#2,389#2,390#2,391#2,392#2,393#2,394#2,395#2,396#2,397#2,398#2,399#2,400#2,401#2,402#2,403#2,404#2,405#2,406#2,407#2,408#2,409#2,410#2,411#2,412#2,413#2,414#2,415#2,416#2,417#2,418#2,419#2,420#2,421#2,422#2,423#2,424#2,425#2,426#2,427#2,428#2,429#3,430#2,431#2,432#2,433#2,434#2,435#2,436#2,437#2,438#2,439#2,440#2,441#2,442#2,443#2,444#2,445#2,446#2,447#2,448#2,449#2,450#2,451#2,452#2,453#2,454#2,455#2,456#2,457#2,458#2,459#2,460#2,461#2,462#2,463#2,464#2,465#2,466#2,467#2,468#2,469#2,470#2,471#2,472#2,473#2,474#2,475#2,476#2,477#2,478#2,479#2,480#2,481#2,482#2,483#2,484#2,485#2,486#2,487#2,488#2,489#2,490#2,491#2,492#2,493#2,494#2,495#2,496#2,497#2,498#2,499#2,500#2,501#2,502#2,503#2,504#2,505#2,506#2,507#2,508#3,509#2,510#2,511#2,512#2,513#2,514#2,515#2,516#2,517#2,518#2,519#2,520#2,521#2,522#2,523#2,524#2,525#2,526#2,527#2,528#2,529#2,530#2,531#2,532#2,533#2,534#2,535#2,536#2,537#2,538#2,539#2,540#2,541#2,542#2,543#2,544#2,545#2,546#2,547#2,548#2,549#2,550#2,551#2,552#2,553#2,554#2,555#2,556#2,557#2,558#2,559#2,560#2,561#2,562#2,563#2,564#2,565#2,566#2,567#2,568#2,569#2,570#2,571#2,572#2,573#2,574#2,575#2,576#2,577#2,578#2,579#2,580#2,581#2,582#2,583#2,584#2,585#2,586#2,587#2,588#2,589#2,590#2,591#2,592#2,593#2,594#2,595#2,596#2,597#2,598#2,599#2,600#2,601#2,602#2,603#2,604#2,605#2,606#2,607#2,608#2,609#2,610#2,611#2,612#2,613#2,614#2,615#2,616#2,617#2,618#2,619#2,620#2,621#2,622#2,623#2,624#2,625#2,626#2,627#2,628#2,629#2,630#2,631#2,632#2,633#2,634#2,635#2,636#2,637#2,638#2,639#2,640#2,641#2,642#2,643#2,644#2,645#2,646#2,647#2,648#2,649#2,650#2,651#2,652#2,653#2,654#2,655#2,656#2,657#2,658#2,659#2,660#2,661#2,662#2,663#2,664#2,665#2,666#2,667#2,668#2,669#2,670#2,671#2,672#2,673#2,674#2,675#2,676#2,677#2,678#2,679#2,680#2,681#2,682#2,683#2,684#2,685#2,686#2,687#2,688#2,689#2,690#2,691#2,692#2,693#2,694#2,695#2,696#2,697#2,698#2,699#2,700#2,701#2,702#2,703#2,704#2,705#2,706#2,707#2,708#2,709#2,710#2,711#2,712#2,713#2,714#2,715#2,716#2,717#2,718#2,719#2,720#2,721#2,722#2,723#2,724#2,725#2,726#2,727#2,728#2,729#2,730#2,731#2,732#2,733#2,734#2,735#2,736#2,737#2,738#2,739#2,740#2,741#2,742#2,743#2,744#2,745#2,746#2,747#2,748#2,749#2,750#2,751#2,752#2,753#2,754#2,755#2,756#2,757#2,758#2,759#2,760#2,761#2,762#2,763#2,764#2,765#2,766#2,767#2,768#2,769#2,770#2,771#2,772#2,773#2,774#2,775#2,776#2,777#2,778#2,779#2,780#2,781#2,782#2,783#2,784#2,785#2,786#2,787#2,788#2,789#2,790#2,791#2,792#2,793#2,794#2,795#2,796#2,797#2,798#2,799#2,800#2,801#2,802#2,803#2,804#2,805#2,806#2,807#2,808#2,809#2,810#2,811#2,812#2,813#2,814#2,815#2,816#2,817#2,818#2,819#2,820#2,821#2,822#2,823#2,824#2,825#2,826#2,827#2,828#2,829#2,830#2,831#2,832#2,833#2,834#2,835#2,836#2,837#2,838#2,839#2,840#2,841#2,842#2,843#2,844#2,845#2,846#2,847#2,848#2,849#2,850#2,851#2,852#2,853#2,854#2,855#2,856#2,857#2,858#2,859#2,860#2,861#2,862#2,863#2,864#2,865#2,866#2,867#2,868#2,869#2,870#2,871#2,872#3,873#2,874#2,875#2,876#2,877#2,878#2,879#2,880#2,881#2,882#2,883#2,884#2,885#2,886#2,887#2,888#2,889#2,890#2,891#2,892#2,893#2,894#2,895#2,896#2,897#2,898#2,899#2,900#2,901#2,902#2,903#2,904#2,905#2,906#2,907#2,908#2,909#2,910#2,911#2,912#2,913#2,914#2,915#2,916#2,917#2,918#2,919#2,920#2,921#2,922#2,923#2,924#2,925#2,926#2,927#2,928#2,929#2,930#2,931#2,932#2,933#2,934#2,935#2,936#2,937#2,938#2,939#2,940#2,941#2,942#2,943#2,944#2,945#2,946#2,947#2,948#2,949#2,950#2,951#2,952#2,953#2,954#2,955#2,956#2,957#2,958#2,959#2,960#2,961#2,962#2,963#2,964#2,965#2,966#2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0", "mtime": "2020-07-10T03:28:55.524523Z", "creation_time": "2020-07-08T14:45:49.265385Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#", "usage": { "rgw.main": { "size": 10075673528, "size_actual": 10174205952, "size_utilized": 10075673528, "size_kb": 9839525, "size_kb_actual": 9935748, "size_kb_utilized": 9839525, "num_objects": 47476 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } [root@os1 ~]# radosgw-admin bucket stats --bucket=df-test { "bucket": "df-test", "num_shards": 11, "tenant": "", "zonegroup": "a1b72aa0-eb06-4a96-8af4-db39d4dc3e09", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732580.22", "marker": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732580.22", "index_type": "Normal", "owner": "01c235fd-fc9b-4c54-a2a5-b38054608b9e", "ver": "0#1,1#3,2#1,3#1,4#1,5#1,6#1,7#1,8#3,9#1,10#2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "2020-07-08T14:33:11.589606Z", "creation_time": "2020-07-08T14:33:11.575423Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 511, "size_actual": 4096, "size_utilized": 511, "size_kb": 1, "size_kb_actual": 4, "size_kb_utilized": 1, "num_objects": 1 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } [root@os1 ~]# radosgw-admin user stats --uid=user-in-question { "stats": { "size": 11154608110526, "size_actual": 11263376089088, "size_utilized": 0, "size_kb": 10893171983, "size_kb_actual": 10999390712, "size_kb_utilized": 0, "num_objects": 52543794 }, "last_stats_sync": "2020-07-14T12:08:58.888165Z", "last_stats_update": "2020-07-14T12:08:58.887088Z" } Cheers, Dave

1 0

User stats - Object count wrong in Octopus?
by David Monschein 14 Jul '20

14 Jul '20

Hi All, I'm investigating what appears to be a bug in RGW stats. This is a brand new cluster running 15.2.3 One of our customers reached out, saying they were hitting their quota (S3 error: 403 (QuotaExceeded)). The user-wide max_objects quota we set is 50 million objects, so this would be impossible since the entire cluster isn't even close to 50 million objects yet: [root@os1 ~]# ceph status | grep objects objects: 7.58M objects, 6.8 TiB The customer in question has three buckets, and if I query the bucket stats, the total number of objects for all 3 buckets comes to about ~372k: [root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 | grep num_objects "num_objects": 324880 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo | grep num_objects "num_objects": 47476 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-test | grep num_objects "num_objects": 1 But things get interesting when I query the user stats:

1 0

cephadm adoption failed
by Tobias Gall 14 Jul '20

14 Jul '20

Hello, I'm trying to adopt an existing cluster. The cluster consists of 5 converged (mon, mgr, osd, mds on same host) servers running Octopus 15.2.4. I've followed the guide: https://docs.ceph.com/docs/octopus/cephadm/adoption/ Adopting the first mon I've got the following problem: root@mulberry:/home/toga# cephadm adopt --style legacy --name mon.mulberry INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container... INFO:cephadm:Stopping old systemd unit ceph-mon@mulberry... INFO:cephadm:Disabling old systemd unit ceph-mon@mulberry... INFO:cephadm:Moving data... INFO:cephadm:Chowning content... Traceback (most recent call last): File "/usr/sbin/cephadm", line 4761, in <module> r = args.func() File "/usr/sbin/cephadm", line 1162, in _default_image return func() File "/usr/sbin/cephadm", line 3241, in command_adopt command_adopt_ceph(daemon_type, daemon_id, fsid); File "/usr/sbin/cephadm", line 3387, in command_adopt_ceph call_throws(['chown', '-c', '-R', '%d.%d' % (uid, gid), data_dir_dst]) File "/usr/sbin/cephadm", line 844, in call_throws out, err, ret = call(command, **kwargs) File "/usr/sbin/cephadm", line 784, in call message = message_b.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 1023: unexpected end of data In `cephadm ls` the old mon is gone and the new is present: { "style": "cephadm:v1", "name": "mon.mulberry", "fsid": "74307e84-e1fe-4706-8312-fe47703928a1", "systemd_unit": "ceph-74307e84-e1fe-4706-8312-fe47703928a1(a)mon.mulberry", "enabled": false, "state": "stopped", "container_id": null, "container_image_name": null, "container_image_id": null, "version": null, "started": null, "created": null, "deployed": null, "configured": null } But there is no container running. How can I resolve this? Regards, Tobias

2 4

Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
by Dan van der Ster 14 Jul '20

14 Jul '20

Thanks for this info -- adding it to our list of reasons never to use FileStore again. In your case, are you able to migrate? On Tue, Jul 14, 2020 at 3:13 PM Eric Smith <Eric.Smith(a)vecima.com> wrote: > > FWIW Bluestore is not affected by this problem! > > -----Original Message----- > From: Eric Smith <Eric.Smith(a)vecima.com> > Sent: Saturday, July 11, 2020 6:40 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot > > It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues. > > Eric > > -----Original Message----- > From: Eric Smith <Eric.Smith(a)vecima.com> > Sent: Friday, July 10, 2020 1:46 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot > > For what it's worth - all of our objects are generating LONG named object files like so... > > \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long > > Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad? > > -----Original Message----- > From: Eric Smith <Eric.Smith(a)vecima.com> > Sent: Friday, July 10, 2020 6:59 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot > > I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: > > 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 > 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 > 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool > 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 > 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 > 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] > 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# > 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 > 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D > 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA > 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 > 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 > 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 > 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 > 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 > 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 > 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 > 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 > 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# > 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A > > Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
by Eric Smith 14 Jul '20

14 Jul '20

FWIW Bluestore is not affected by this problem! -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Saturday, July 11, 2020 6:40 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues. Eric -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 1:46 PM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot For what it's worth - all of our objects are generating LONG named object files like so... \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad? -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 6:59 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

1 pg inconsistent
by Abhimnyu Dhobale 14 Jul '20

14 Jul '20

Good Day, Ceph is showing below error frequently. every time after pg repair it is resolved. [root@vpsapohmcs01 ~]# ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.574 is active+clean+inconsistent, acting [19,25,2] [root@vpsapohmcs02 ~]# cat /var/log/ceph/ceph-osd.19.log | grep error 2020-07-12 11:42:11.824 7f864e0b2700 -1 log_channel(cluster) log [ERR] : 1.574 shard 25 soid 1:2ea0a7a3:::rbd_data.515c96b8b4567.0000000000007a7c:head : candidate had a read error 2020-07-12 11:42:15.035 7f86520ba700 -1 log_channel(cluster) log [ERR] : 1.574 deep-scrub 1 errors [root@vpsapohmcs01 ~]# ceph --version ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable) Request you to please suggest. -- Thanks & Regards Abhimnyu Dhobale

4 3

v14.2.10 Nautilus crash
by Markus Binz 14 Jul '20

14 Jul '20

Hello, yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok. There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster). no restart possible, always the same crash. 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. This is a mixed cluster of ubuntu xenial and bionic, it happens on both. It look's like, it happens when the new monmap arrived at the osd. The only fix i was able to come up with, downgrade ceph-osd to v14.2.9. Should i open a bug report? Regards Markus

3 7

missing ceph-mgr-dashboard and ceph-grafana-dashboards rpms for el7 and 14.2.10
by Joel Davidow 13 Jul '20

13 Jul '20

https://download.ceph.com/rpm-nautilus/el8/noarch/ contains ceph-mgr-dashboard-14.2.10-0.el8.noarch.rpm and ceph-grafana-dashboards-14.2.10-0.el8.noarch.rpm but there is no 14.2.10.0-el7.noarch.rpm for either ceph-mgr-dashboard or ceph-grafana-dashboards in https://download.ceph.com/rpm-nautilus/el7/noarch/. I tried using 14.2.9 rpms but there is a dependency of ceph-mgr 14.2.9 and I just installed 14.2.10 which I want to continue running. # rpm -Uvh https://download.ceph.com/rpm-nautilus/el7/noarch/ceph-mgr-dashboard-14.2.9… Retrieving https://download.ceph.com/rpm-nautilus/el7/noarch/ceph-mgr-dashboard-14.2.9… error: Failed dependencies: ceph-grafana-dashboards = 2:14.2.9-0.el7 is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch ceph-mgr = 2:14.2.9-0.el7 is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch python-jwt is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch python-routes is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch Is this the appropriate forum to raise this issue or should I use a different process to get the missing rpms addressed?

1 0

multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD) and CephFS
by Bobby 13 Jul '20

13 Jul '20

Hi, I have a question regarding support for multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD). The below given link says that the driver has been using the BLK-MQ interface for a while but not actually multiple queues until now with having a queue per-CPU. A change to not hold onto caps that aren't actually needed. These improvements and more can be found as part of the Ceph changes for Linux 5.7, which should be released as stable in early June. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.7-Ceph-Performa… My question is: Is it possible that through Ceph FS (Filesystem in User Space) I can develop a multi-queue driver for Ceph? Asking because this way I can avoid kernel space. ( https://docs.ceph.com/docs/nautilus/start/quick-cephfs/) Looking forward for some help BR Bobby

1 0

Ceph stuck at: objects misplaced (0.064%)
by Ml Ml 13 Jul '20

13 Jul '20

Hello, ceph is stuck since 4 days with 0.064% misplaced and i dunno why. Can anyone help me to get it fixed? I did restart some OSDs and reweight them again to get some data moving but that did not help. root@node01:~ # ceph -s cluster: id: 251c937e-0b55-48c1-8f34-96e84e4023d4 health: HEALTH_WARN 1803/2799972 objects misplaced (0.064%) mon node02 is low on available space services: mon: 3 daemons, quorum node01,node02,node03 mgr: node03(active), standbys: node01, node02 osd: 16 osds: 16 up, 16 in; 1 remapped pgs data: pools: 1 pools, 512 pgs objects: 933.32k objects, 2.68TiB usage: 9.54TiB used, 5.34TiB / 14.9TiB avail pgs: 1803/2799972 objects misplaced (0.064%) 511 active+clean 1 active+clean+remapped io: client: 131KiB/s rd, 8.57MiB/s wr, 28op/s rd, 847op/s wr root@node01:~ # ceph health detail HEALTH_WARN 1803/2800179 objects misplaced (0.064%); mon node02 is low on available space OBJECT_MISPLACED 1803/2800179 objects misplaced (0.064%) MON_DISK_LOW mon node02 is low on available space mon.node02 has 28% avail root@node01:~ # ceph versions { "mon": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 3 }, "mgr": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 3 }, "osd": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 16 }, "mds": {}, "overall": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 22 } } root@node02:~ # df -h Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs 13G 1.3G 12G 11% /run /dev/sda3 46G 31G 14G 70% / tmpfs 63G 57M 63G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda1 922M 206M 653M 24% /boot /dev/fuse 30M 144K 30M 1% /etc/pve /dev/sde1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-11 /dev/sdf1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-14 /dev/sdc1 889G 676G 214G 77% /var/lib/ceph/osd/ceph-3 /dev/sdb1 889G 667G 222G 76% /var/lib/ceph/osd/ceph-2 /dev/sdd1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-7 tmpfs 13G 0 13G 0% /run/user/0 root@node02:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 14.34781 root default -2 4.25287 host node01 0 hdd 0.85999 osd.0 up 0.80005 1.00000 1 hdd 0.86749 osd.1 up 0.85004 1.00000 6 hdd 0.87270 osd.6 up 0.90002 1.00000 12 hdd 0.78000 osd.12 up 0.95001 1.00000 13 hdd 0.87270 osd.13 up 0.95001 1.00000 -3 3.91808 host node02 2 hdd 0.70000 osd.2 up 0.80005 1.00000 3 hdd 0.59999 osd.3 up 0.85004 1.00000 7 hdd 0.87270 osd.7 up 0.85004 1.00000 11 hdd 0.87270 osd.11 up 0.75006 1.00000 14 hdd 0.87270 osd.14 up 0.85004 1.00000 -4 6.17686 host node03 4 hdd 0.87000 osd.4 up 1.00000 1.00000 5 hdd 0.87000 osd.5 up 1.00000 1.00000 8 hdd 0.87270 osd.8 up 1.00000 1.00000 10 hdd 0.87270 osd.10 up 1.00000 1.00000 15 hdd 0.87270 osd.15 up 1.00000 1.00000 16 hdd 1.81879 osd.16 up 1.00000 1.00000 root@node01:~ # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 14.55780 - 14.9TiB 9.45TiB 7.46TiB 1.47GiB 23.2GiB 5.43TiB 63.52 1.00 - root default -2 4.27286 - 4.35TiB 3.15TiB 2.41TiB 486MiB 7.62GiB 1.21TiB 72.32 1.14 - host node01 0 hdd 0.85999 0.80005 888GiB 619GiB 269GiB 92.3MiB 0B 269GiB 69.72 1.10 89 osd.0 1 hdd 0.86749 0.85004 888GiB 641GiB 248GiB 109MiB 0B 248GiB 72.12 1.14 92 osd.1 6 hdd 0.87270 0.90002 894GiB 634GiB 632GiB 98.9MiB 2.65GiB 259GiB 70.99 1.12 107 osd.6 12 hdd 0.79999 0.95001 894GiB 664GiB 661GiB 94.4MiB 2.52GiB 230GiB 74.31 1.17 112 osd.12 13 hdd 0.87270 0.95001 894GiB 665GiB 663GiB 91.7MiB 2.46GiB 229GiB 74.43 1.17 112 osd.13 -3 4.10808 - 4.35TiB 3.17TiB 2.18TiB 479MiB 6.99GiB 1.18TiB 72.86 1.15 - host node02 2 hdd 0.78999 0.75006 888GiB 654GiB 235GiB 95.6MiB 0B 235GiB 73.57 1.16 94 osd.2 3 hdd 0.70000 0.80005 888GiB 737GiB 151GiB 114MiB 0B 151GiB 82.98 1.31 105 osd.3 7 hdd 0.87270 0.85004 894GiB 612GiB 610GiB 88.9MiB 2.43GiB 281GiB 68.50 1.08 103 osd.7 11 hdd 0.87270 0.75006 894GiB 576GiB 574GiB 81.8MiB 2.19GiB 317GiB 64.47 1.01 97 osd.11 14 hdd 0.87270 0.85004 894GiB 669GiB 666GiB 98.8MiB 2.37GiB 225GiB 74.85 1.18 112 osd.14 -4 6.17686 - 6.17TiB 3.13TiB 2.86TiB 541MiB 8.58GiB 3.04TiB 50.73 0.80 - host node03 4 hdd 0.87000 1.00000 888GiB 504GiB 384GiB 124MiB 0B 384GiB 56.72 0.89 72 osd.4 5 hdd 0.87000 1.00000 888GiB 520GiB 368GiB 96.2MiB 0B 368GiB 58.57 0.92 75 osd.5 8 hdd 0.87270 1.00000 894GiB 508GiB 505GiB 80.2MiB 2.07GiB 386GiB 56.80 0.89 85 osd.8 10 hdd 0.87270 1.00000 894GiB 374GiB 373GiB 51.9MiB 1.73GiB 519GiB 41.88 0.66 63 osd.10 15 hdd 0.87270 1.00000 894GiB 504GiB 502GiB 60.1MiB 1.99GiB 390GiB 56.37 0.89 84 osd.15 16 hdd 1.81879 1.00000 1.82TiB 797GiB 794GiB 129MiB 2.79GiB 1.04TiB 42.77 0.67 134 osd.16 TOTAL 14.9TiB 9.45TiB 7.46TiB 1.47GiB 23.2GiB 5.43TiB 63.52 MIN/MAX VAR: 0.66/1.31 STDDEV: 11.59 root@node02:~ # df -h Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs 13G 1.4G 12G 11% /run /dev/sda3 46G 31G 14G 70% / tmpfs 63G 63M 63G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda1 922M 206M 653M 24% /boot /dev/fuse 30M 140K 30M 1% /etc/pve /dev/sde1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-11 /dev/sdf1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-14 /dev/sdc1 889G 738G 152G 83% /var/lib/ceph/osd/ceph-3 /dev/sdb1 889G 654G 235G 74% /var/lib/ceph/osd/ceph-2 /dev/sdd1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-7 tmpfs 13G 0 13G 0% /run/user/0 Any idea? Thanks, Michael

4 3

cephfs: creating two subvolumegroups with dedicated data pool...
by Christoph Ackermann 13 Jul '20

13 Jul '20

Hello list, i have a strange issue creating cephfs subvolumegroup with dedicated pools.... We have four fs (volumes); one is "backup" I would like to set up two subvolumegroup "veeampool" and "bareospool" each with dedicated data pools "cephfs.backup.veeam.data" and "cephfs.backup.bareos.data". Both EC42 with overwrites enabled and application cephfs. Pls see below. isceph@ceph-deploy:~$ sudo ceph fs subvolumegroup create backup veeampool --pool_layout cephfs.backup.veeam.data ...First one is ok. isceph@ceph-deploy:~$ sudo ceph fs subvolumegroup create backup bareospool --pool_layout cephfs.backup.bareos.data Error EINVAL: Invalid pool layout 'cephfs.backup.bareos.data'. It must be a valid data pool Second one throws this error... Is there only one data pool allowed per volume? Many thanks for clarification Christoph ...pool ls detail [..] pool 44 'cephfs.backup.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1132068 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 45 'cephfs.backup.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1132068 flags hashpspool stripe_width 0 application cephfs pool 46 'cephfs.backup.veeam.data' erasure profile ec-42-profile size 6 min_size 5 crush_rule 9 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 1174754 lfor 0/0/1174708 flags hashpspool,ec_overwrites stripe_width 16384 application cephfs pool 48 'cephfs.backup.bareos.data' erasure profile ec-42-profile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 1176205 flags hashpspool,ec_overwrites stripe_width 16384 application cephfs -- Christoph Ackermann | System Engineer INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | mailto:C.Ackermann@infoserve.de | https://www.infoserve.de INFOSERVE Datenschutzhinweise: https://infoserve.de/datenschutz Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: Saarbrücken Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599

2 2

"task status" section in ceph -s output new?
by Rainer Krienke 13 Jul '20

13 Jul '20

Hello, today I upgraded from 14.2.9 to 14.2.10. Everything worked just fine. I think at the time I upgraded the last of my three MDS servers something new appeard in the output of ceph -s: # ceph -s cluster: id: xyz health: HEALTH_OK services: ... ... task status: scrub status: mds.myceph3: idle The "task status" section is new to me. Is it there to stay and what kind of task status messages might appear there? At the moment it displays information about ceph fs scrub status telling me nothig is going on here. Since I recently started using cephfs reading the message I started asking myself if I should scrub cephfs regularly or does ceph do this on its own or is scrubbing only needed in case cephs has been damaged? Does anyone know more about this? Thanks for your help Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312

2 1

ceph fs resize
by hw 13 Jul '20

13 Jul '20

Hi! My ceph version is ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) I have ceph fs and I add new osd to my cluster. ceph pg stat: 289 pgs: 1 active+clean+scrubbing+deep, 288 active+clean; 873 GiB data, 2.7 TiB used, 3.5 TiB / 6.2 TiB avail How I can extend my ceph fs from 3.5 TiB to 6.2 TiB avail Detail information: ceph fs status static - 2 clients ====== RANK STATE MDS ACTIVITY DNS INOS 0 active static.ceph02.sgpdiv Reqs: 0 /s 136k 128k POOL TYPE USED AVAIL static_metadata metadata 10.1G 973G static data 2754G 973G STANDBY MDS static.ceph05.aylgvy static.ceph04.wsljnw MDS version: ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE device_health_metrics 896.1k 3.0 6399G 0.0000 1.0 1 on static 874.0G 3.0 6399G 0.4097 1.0 256 on static_metadata 3472M 3.0 6399G 0.0016 4.0 32 on ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 6.2 TiB 3.5 TiB 2.7 TiB 2.7 TiB 43.55 TOTAL 6.2 TiB 3.5 TiB 2.7 TiB 2.7 TiB 43.55 --- POOLS --- POOL ID STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 896 KiB 20 2.6 MiB 0 973 GiB static 14 874 GiB 1.44M 2.7 TiB 48.54 973 GiB static_metadata 15 3.4 GiB 2.53M 10 GiB 0.35 973 GiB ceph fs volume ls [ { "name": "static" } ] ceph fs subvolume ls static []

2 1

Re: Ceph `realm pull` permission denied error
by Alex Hussein-Kershaw 13 Jul '20

13 Jul '20

I got to the bottom of this – was caused by NTP server issues and a 1 hour time discrepancy between the two clusters. This was pretty painful to get to the bottom of (didn’t find any useful logs, most description Ceph gave me was the “permission denied” error) – hopefully some future engineer can save some time from my troubles! Thanks, Alex From: Alex Hussein-Kershaw Sent: 13 July 2020 12:22 To: Zhenshi Zhou <deaderzzs(a)gmail.com> Cc: ceph-users(a)ceph.io Subject: RE: [ceph-users] Ceph `realm pull` permission denied error Hi Zhenshi, Thanks for the suggestion, unfortunately I have tried this already and had no luck ☹ Best wishes, Alex From: Zhenshi Zhou <deaderzzs(a)gmail.com<mailto:deaderzzs@gmail.com>> Sent: 13 July 2020 10:58 To: Alex Hussein-Kershaw <Alex.Hussein-Kershaw(a)metaswitch.com<mailto:Alex.Hussein-Kershaw@metaswitch.com>> Cc: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] Ceph `realm pull` permission denied error NOTE: Message is from an external sender Hi Alex, I didn't deploy this in containers/vms, as well as ansible or other tools. However I deployed multisite once and I remember that I restarted the rgw on the master site before I sync realm on the secondary site. I'm not sure if this can help. Alex Hussein-Kershaw <Alex.Hussein-Kershaw(a)metaswitch.com<mailto:Alex.Hussein-Kershaw@metaswitch.com>> 于2020年7月13日周一下午5:48写道： Hi Ceph Users, I'm struggling with an issue that I'm hoping someone can point me towards a solution. We are using Nautilus (14.2.9) deploying Ceph in containers, in VMs. The setup that I'm working with has 3 VMs, but of-course our design expects this to be scaled by a user as appropriate. I have a cluster deployed and it's functioning happily as storage for our product, the error occurs when I go to setup a second cluster and pair it with the first. I'm using ceph-ansible to deploy. I get the following error about 20 minutes into running the site-container playbook. 2020-07-09 14:21:10,966 p=2134 u=qs-admin | TASK [ceph-rgw : fetch the realm] *********************************************************************************************** ************************************************************************************ 2020-07-09 14:21:10,966 p=2134 u=qs-admin | Thursday 09 July 2020 14:21:10 +0000 (0:00:00.410) 0:16:18.245 ********* 2020-07-09 14:21:11,901 p=2134 u=qs-admin | fatal: [10.225.21.213 -> 10.225.21.213]: FAILED! => changed=true cmd: - docker - exec - ceph-mon-albamons_sc2 - radosgw-admin - realm - pull - --url=https://10.225.36.197:7480<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2F10.225.36…> - --access-key=2CQ006Lereqpysbr0l0s - --secret=JM3S5Hd49Nz03eIbTTNnEyqcXJkIOXbp0gWIUEbp delta: '0:00:00.545895' end: '2020-07-09 14:21:11.516539' msg: non-zero return code rc: 13 start: '2020-07-09 14:21:10.970644' stderr: |- request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> Re-running the command manually reproduces the error. I understand that the permission denied error appears to indicate the keys are not valid, suggested by https://tracker.ceph.com/issues/36619<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.c…>. However, I've triple checked the keys are correct on the other site. I'm at a loss of where to look for debugging, I've turned up logs on both the local and remote site for RGW and MON processes but neither seem to yield anything related. I've tried restarting everything as suggested in the error text from all the processes to a full reboot of all the VMs. I've no idea why the keys are being declined either, as they are correct (or atleast `radosgw-admin period get` on the primary site thinks so). Thanks for your help, Alex _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

mon_osd_down_out_subtree_limit not working?
by Frank Schilder 13 Jul '20

13 Jul '20

Hi all, on a mimic 13.2.8 cluster I have set mon advanced mon_osd_down_out_subtree_limit host According to the documentation: ======================= mon osd down out subtree limit Description The smallest CRUSH unit type that Ceph will not automatically mark out. For instance, if set to host and if all OSDs of a host are down, Ceph will not automatically mark out these OSDs. ======================= if I shut down all OSDs on this host, these OSDs should not be marked out automatically after mon_osd_down_out_interval(=600) seconds. I did a test today and, unfortunately, the OSDs do get marked as out. Ceph status was showing 1 host down as expected. Am I doing something wrong or misreading the documentation? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Re: Ceph `realm pull` permission denied error
by Zhenshi Zhou 13 Jul '20

13 Jul '20

Hi Alex, I didn't deploy this in containers/vms, as well as ansible or other tools. However I deployed multisite once and I remember that I restarted the rgw on the master site before I sync realm on the secondary site. I'm not sure if this can help. Alex Hussein-Kershaw <Alex.Hussein-Kershaw(a)metaswitch.com> 于2020年7月13日周一下午5:48写道： > Hi Ceph Users, > > I'm struggling with an issue that I'm hoping someone can point me towards > a solution. > > We are using Nautilus (14.2.9) deploying Ceph in containers, in VMs. The > setup that I'm working with has 3 VMs, but of-course our design expects > this to be scaled by a user as appropriate. I have a cluster deployed and > it's functioning happily as storage for our product, the error occurs when > I go to setup a second cluster and pair it with the first. I'm using > ceph-ansible to deploy. I get the following error about 20 minutes into > running the site-container playbook. > > 2020-07-09 14:21:10,966 p=2134 u=qs-admin | TASK [ceph-rgw : fetch the > realm] > *********************************************************************************************** > > ************************************************************************************ > 2020-07-09 14:21:10,966 p=2134 u=qs-admin | Thursday 09 July 2020 > 14:21:10 +0000 (0:00:00.410) 0:16:18.245 ********* > 2020-07-09 14:21:11,901 p=2134 u=qs-admin | fatal: [10.225.21.213 -> > 10.225.21.213]: FAILED! => changed=true > cmd: > - docker > - exec > - ceph-mon-albamons_sc2 > - radosgw-admin > - realm > - pull > - --url=https://10.225.36.197:7480 > - --access-key=2CQ006Lereqpysbr0l0s > - --secret=JM3S5Hd49Nz03eIbTTNnEyqcXJkIOXbp0gWIUEbp > delta: '0:00:00.545895' > end: '2020-07-09 14:21:11.516539' > msg: non-zero return code > rc: 13 > start: '2020-07-09 14:21:10.970644' > stderr: |- > request failed: (13) Permission denied > If the realm has been changed on the master zone, the master zone's > gateway may need to be restarted to recognize this user. > stderr_lines: <omitted> > stdout: '' > stdout_lines: <omitted> > > Re-running the command manually reproduces the error. I understand that > the permission denied error appears to indicate the keys are not valid, > suggested by https://tracker.ceph.com/issues/36619. However, I've triple > checked the keys are correct on the other site. I'm at a loss of where to > look for debugging, I've turned up logs on both the local and remote site > for RGW and MON processes but neither seem to yield anything related. I've > tried restarting everything as suggested in the error text from all the > processes to a full reboot of all the VMs. I've no idea why the keys are > being declined either, as they are correct (or atleast `radosgw-admin > period get` on the primary site thinks so). > > Thanks for your help, > Alex > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Adding OpenStack Keystone integrated radosGWs to an existing radosGW cluster
by Thomas Byrne - UKRI STFC 13 Jul '20

13 Jul '20

Hi all, We run a simple single zone nautilus radosGW instance with a few gateway machines for some of our users. I've got some more gateway machines earmarked for the purpose of adding some OpenStack Keystone integrated RadosGW gateways to the cluster. I'm not sure how best to add them alongside the exiting radosGWs gateways/infrastructure. The options I think I have are: 1) Add keystone integration to all radosGWs gateways. Simplest, but I have (possibly unfounded) concerns about issues with keystone causing problems for non OpenStack users (added authentication latency), and I'm not sure I fully understand how the OpenStack users/buckets will interact with our existing users. 2) Add keystone integration to separate gateways. This keeps the radosGW servers separate, and deals with one of my concerns above. 3) Add a separate radosGW zone/instance (not sure what the correct term is), and have separate gateways for this instance. Seems very heavyweight for what I'm trying to achieve, but that may be my inexperience talking. 4) Something else entirely? Any advice would be greatly appreciated! Cheers, Tom This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

1 0

Poor Windows performance on ceph RBD.
by Frank Schilder 13 Jul '20

13 Jul '20

Dear all, maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM. If anyone has pointers what to look for, we would be very grateful. The OpenNebula installation is more or less default. The current OS and libvirt versions we use are: Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0-23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0-33.1.el7 @centos-qemu-ev Some benchmark results from good to worse workloads: rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB. We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes. If anyone has seen and solved this before, please let us know. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 9

Error on upgrading to 15.2.4 / invalid service name using containers
by Mario J. Barchéin Molina 13 Jul '20

13 Jul '20

Hello. I'm trying to upgrade to ceph 15.2.4 from 15.2.3. The upgrade is almost finished, but it has entered in a service start/stop loop. I'm using a container deployment over Debian 10 with 4 nodes. The problem is with a service named literally "mds.label:mds". It has the colon character, which is of special use in docker. This character can't appear in the name of the container and also breaks the volumen binding syntax. I have seen in the /var/lib/ceph/UUID/ the files for this service: root@ceph-admin:/var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca# ls -la total 48 drwx------ 12 167 167 4096 jul 10 02:54 . drwxr-x--- 3 ceph ceph 4096 jun 24 16:36 .. drwx------ 3 nobody nogroup 4096 jun 24 16:37 alertmanager.ceph-admin drwx------ 3 167 167 4096 jun 24 16:36 crash drwx------ 2 167 167 4096 jul 10 01:35 crash.ceph-admin drwx------ 4 998 996 4096 jun 24 16:38 grafana.ceph-admin drwx------ 2 167 167 4096 jul 10 02:55 mds.label:mds.ceph-admin.rwmtkr drwx------ 2 167 167 4096 jul 10 01:33 mgr.ceph-admin.doljkl drwx------ 3 167 167 4096 jul 10 01:34 mon.ceph-admin drwx------ 2 nobody nogroup 4096 jun 24 16:38 node-exporter.ceph-admin drwx------ 4 nobody nogroup 4096 jun 24 16:38 prometheus.ceph-admin drwx------ 4 root root 4096 jul 3 02:43 removed root@ceph-admin:/var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca/mds.label:mds.ceph-admin.rwmtkr# ls -la total 32 drwx------ 2 167 167 4096 jul 10 02:55 . drwx------ 12 167 167 4096 jul 10 02:54 .. -rw------- 1 167 167 295 jul 10 02:55 config -rw------- 1 167 167 152 jul 10 02:55 keyring -rw------- 1 167 167 38 jul 10 02:55 unit.configured -rw------- 1 167 167 48 jul 10 02:54 unit.created -rw------- 1 root root 24 jul 10 02:55 unit.image -rw------- 1 root root 0 jul 10 02:55 unit.poststop -rw------- 1 root root 981 jul 10 02:55 unit.run root@ceph-admin:/var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca/mds.label:mds.ceph-admin.rwmtkr# cat unit.run /usr/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/0ce93550-b628-11ea-9484-f6dc192416ca /usr/bin/docker run --rm --net=host --ipc=host --name ceph-0ce93550-b628-11ea-9484-f6dc192416ca-mds.label:mds.ceph-admin.rwmtkr -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=ceph-admin -v /var/ru n/ceph/0ce93550-b628-11ea-9484-f6dc192416ca:/var/run/ceph:z -v /var/log/ceph/0ce93550-b628-11ea-9484-f6dc192416ca:/var/log/ceph:z -v /var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca/crash:/var/lib/ceph/c rash:z -v /var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca/mds.label:mds.ceph-admin.rwmtkr:/var/lib/ceph/mds/ceph-label:mds.ceph-admin.rwmtkr:z -v /var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca/mds.l abel:mds.ceph-admin.rwmtkr/config:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph-mds docker.io/ceph/ceph:v15 -n mds.label:mds.ceph-admin.rwmtkr -f --setuser ceph --setgroup ceph --default-log-to-file=fal se --default-log-to-stderr=true --default-log-stderr-prefix="debug " If I try to manually run the docker command, this is the error: docker: Error response from daemon: Invalid container name (ceph-0ce93550-b628-11ea-9484-f6dc192416ca-mds.label:mds.ceph-admin.rwmtkr), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed. If I try with a different container name, then the volume binding error rises: docker: Error response from daemon: invalid volume specification: '/var/lib/ceph/0ce93550-b628-11ea-9484-f6dc192416ca/mds.label:mds.ceph-admin.rwmtkr:/var/lib/ceph/mds/ceph-label:mds.ceph-admin.rwmtkr:z'. This mds is not needed and I would be happy simply removing it, but I don't know how to do it. The documentation says how to do it for "normal" services, but my installation is a container deployment. I have tried to remove the directory and restart the upgrading process but then the directory with this service appears again. Please, how can I remove or rename this service so I can complete the upgrading? Also, I think it's a bug to allow docker-forbidden characters in the service names when using container deployment and it should be checked. Thank you very much. -- *Mario J. Barchéin Molina* *Departamento de I+D+i* mario(a)intelligenia.com Madrid: +34 911 86 35 46 US: +1 (918) 856 - 3838 Granada: +34 958 07 70 70 ―― intelligenia · Intelligent Engineering · Web & APP & Intranet www.intelligenia.com · @intelligenia <http://twitter.com/intelligenia> · fb.com/intelligenia · blog.intelligenia.com Madrid · C/ de la Alameda 22, 28014, Madrid, Spain <https://maps.google.com/?q=C/+de+la+Alameda+22,+28014,+Madrid,+Spain&entry=…> Miami · 2153 Coral Way #400, Miami, FL, US, 33145 <https://www.google.es/maps/place/2153+Coral+Way+%23400,+Miami,+FL+33145/@25…> Granada‎ · C/ Luis Amador nº 24, 18014, Granada, Spain <https://www.google.es/maps/place/intelligenia/@37.1947393,-3.6170297,17z/da…> *PROTECCIÓN DE DATOS*: le informamos que los datos personales y dirección de correo electrónico, recabados del propio interesado, serán tratados bajo la responsabilidad de Intelligenia Soluciones Informáticas, S.L. para el envío de comunicaciones sobre nuestros servicios y se conservarán mientras exista un interés mutuo para ello. Los datos no serán comunicados a terceros, salvo obligación legal. Le informamos que puede ejercer los derechos de acceso, rectificación, portabilidad y supresión de sus datos y los de limitación y oposición a su tratamiento dirigiéndose a C/ Luis Amador (Centro de negocios Cámara) 24 , 18014 Granada. Si considera que el tratamiento no se ajusta a la normativa vigente, podrá presentar una reclamación ante la autoridad de control en aepd.es. En cumplimiento de lo previsto en el artículo 21 de la Ley 34/2002 de Servicios de la Sociedad de la Información y Comercio Electrónico (LSSICE), si usted no desea recibir más información sobre nuestros servicios, puede darse de baja enviando un correo electrónico a info(a)intelligenia.com indicando en el *Asunto *"*BAJA*" o "*NO ENVIAR*".

2 2

Ceph `realm pull` permission denied error
by Alex Hussein-Kershaw 13 Jul '20

13 Jul '20

Hi Ceph Users, I'm struggling with an issue that I'm hoping someone can point me towards a solution. We are using Nautilus (14.2.9) deploying Ceph in containers, in VMs. The setup that I'm working with has 3 VMs, but of-course our design expects this to be scaled by a user as appropriate. I have a cluster deployed and it's functioning happily as storage for our product, the error occurs when I go to setup a second cluster and pair it with the first. I'm using ceph-ansible to deploy. I get the following error about 20 minutes into running the site-container playbook. 2020-07-09 14:21:10,966 p=2134 u=qs-admin | TASK [ceph-rgw : fetch the realm] *********************************************************************************************** ************************************************************************************ 2020-07-09 14:21:10,966 p=2134 u=qs-admin | Thursday 09 July 2020 14:21:10 +0000 (0:00:00.410) 0:16:18.245 ********* 2020-07-09 14:21:11,901 p=2134 u=qs-admin | fatal: [10.225.21.213 -> 10.225.21.213]: FAILED! => changed=true cmd: - docker - exec - ceph-mon-albamons_sc2 - radosgw-admin - realm - pull - --url=https://10.225.36.197:7480 - --access-key=2CQ006Lereqpysbr0l0s - --secret=JM3S5Hd49Nz03eIbTTNnEyqcXJkIOXbp0gWIUEbp delta: '0:00:00.545895' end: '2020-07-09 14:21:11.516539' msg: non-zero return code rc: 13 start: '2020-07-09 14:21:10.970644' stderr: |- request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> Re-running the command manually reproduces the error. I understand that the permission denied error appears to indicate the keys are not valid, suggested by https://tracker.ceph.com/issues/36619. However, I've triple checked the keys are correct on the other site. I'm at a loss of where to look for debugging, I've turned up logs on both the local and remote site for RGW and MON processes but neither seem to yield anything related. I've tried restarting everything as suggested in the error text from all the processes to a full reboot of all the VMs. I've no idea why the keys are being declined either, as they are correct (or atleast `radosgw-admin period get` on the primary site thinks so). Thanks for your help, Alex

1 0

compaction_threads and flusher_threads can not used
by 精灵王 13 Jul '20

13 Jul '20

Hello!    For ceph nautilus v14.2.10, I cannot used "compaction_threads and flusher_threads" 。     Why restrict bluestore_rocksdb_options to setting parameters?

1 0

Re: about replica size
by Zhenshi Zhou 13 Jul '20

13 Jul '20

Thank you all for helping me understand clearly about the 'size'. Ml Ml <mliebherr99(a)googlemail.com> 于2020年7月10日周五下午11:08写道： > If size is 2 and one disks fails you are already going to be in error > state with read only. > > Let's say you reboot one node, you will instantly get into trouble. > > If you are going to reboot one node and at the same time the other disk > fails, then you very like loose data. > > Just never ever use size 2. Not even temporary :) > > > Zhenshi Zhou <deaderzzs(a)gmail.com> schrieb am Fr., 10. Juli 2020, 04:11: > >> Hi, >> >> As we all know, the default replica setting of 'size' is 3 which means >> there >> are 3 copies of an object. What is the disadvantages if I set it to 2, >> except >> I get fewer copies? >> >> Thanks >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

ceph install with Ansible
by Hauke Homburg 13 Jul '20

13 Jul '20

Hello, Ich need to install in CentOS 7 Ceph with Ansible. I searched at Ansible Galaxy and some Websites for a good Howto and Playbook. Does anyone have a good Howto to do this? Thanks for help. Regards Hauke -- www.compi-creative.net

2 2

Spillover warning log file?
by Lindsay Mathieson 13 Jul '20

13 Jul '20

Dumb question - in what log file are the rocks db spillover warnings posted? Thanks. -- Lindsay

2 2

Research and Industrial conferences for Ceph research results
by Bobby 12 Jul '20

12 Jul '20

Hi Cephers, Can someone please share about the research and industrial conferences where one can publish Ceph related new research results? Additionally, are there any conferences which are particularly interested in Ceph results? I would like to know all suitable conferences. Thanks :-) Looking forward to hearing from you. BR Bobby !!

1 0

MON store.db keeps growing with Octopus
by Michael Fladischer 12 Jul '20

12 Jul '20

Hi, our cluster is on Octopus 15.2.4. We noticed that our MON all ran out of space yesterday because the store.db folder kept growing until it filled up the filesystem. We added more space to the MON nodes but store.db keeps growing. Right now it's ~220GiB on the two MON nodes that are active. We shut down on MON node when it hit ~98GiB and it seems that it trimmed its local store.db down to 102MiB and now also keeps growing again. Checking the keys in store.db while the MON is offline shows a lot of "logm" and "osdmap" keys: ceph-monstore-tool <path> dump-keys|awk '{print $1}'|uniq -c 86 auth 2 config 11 health 275929 logm 55 mds_health 1 mds_metadata 602 mdsmap 599 mgr 1 mgr_command_descs 3 mgr_metadata 209 mgrstat 461 mon_config_key 1 mon_sync 7 monitor 1 monitor_store 7 monmap 454 osd_metadata 1 osd_pg_creating 4804 osd_snap 138366 osdmap 538 paxos 5 pgmap I already tried compacting it with "ceph tell ..." and "ceph-monstore-tool <path> compact" but it stayed the same size. Also copying it with "ceph-monstore-tool <path> store-copy <new-path>" just created a copy of the same size. Out cluster is currently in WARN status because we are low on space and several OSDs are in a backfill_full state. Could this be related? Regards, Michael

2 3

incomplete PG
by Wladimir Mutel 12 Jul '20

12 Jul '20

Dear everyone, during a very risky process or rebuilding my OSDs to relocate their WAL&DBs onto NVMe SSD partitions, I created a situation where 2 of 3 OSDs of jerasure 2+1 pool had become destroyed, without waiting for complete rebuild of their respective PGs on newly-created OSDs. And now I have an incomplete PG, looking like this : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete pg 1.90 is incomplete, acting [2,3,5] (reducing pool jerasure21 min_size from 2 may help; search ceph.com/docs for 'incomplete') PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 1.90 0 0 0 0 0 0 0 0 incomplete 47m 0'0 223998:104494 [2,3,5]p2 [2,3,5]p2 2020-04-01T13:22:08.983764+0300 2020-03-29T18:36:21.037198+0300 This corrupted some (or maybe all) of my iSCSI RBD targets, and now I would like to determine what of them will have to be dropped & recreated so as not to refer to these incomplete objects. Is there a known recipe or scenario to do that ? Also I would like to know if anything special needs to be done to the incomplete PG to make it complete and usable again. Thanks in advance for your help.

1 0

RGW multi-object delete failing with 403 denied
by Chris Palmer 11 Jul '20

11 Jul '20

Hi An RGW access denied problem that I can't get anywhere with... * Bucket mybucket owned by user "c" * Bucket policy grants s3:listBucket on mybucket, and s3:putObject & s3:deleteObject on mybucket/* to user "j", and s3:getObject to * (I even granted s3:* on mybucket/* to "j" with no effect) * User "j" can create objects in mybucket, and can delete individual objects (using DELETE) * User "j" get 403 when trying to do a multi-object-delete (POST /mybucket/?delete with a list of 4 object keys) Code is a Java servlet running in Wildfly, loading its credentials from the default ~/.aws/credentials file. It enables path-style access. If I change the credentials in there to those of the bucket owner "c" it works... What's different about permissioning for multi-object-delete? Log file shows access has been granted, but further down there is a suspicious "Permissions for user not found" (don't know if that is expected or not). Thanks, Chris ------- Extract from RGW log with debugging at level 20: 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::LocalEngine granted access 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::AWSAuthStrategy granted access 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete normalizing buckets and tenants 2020-07-11T17:55:54.038+0100 7f45adad7700 10 s->object=<NULL> s->bucket=mybucket 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init permissions 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc288 obj=default.rgw.meta:root:mybucket state=0x5628b912e9a0 s->prefetch_data=0 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x16, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x11, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>c</ID><DisplayName>C</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>c</ID><DisplayName>C</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc668 obj=default.rgw.meta:users.uid:j state=0x5628b912e9a0 s->prefetch_data=0 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x6, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty 2020-07-11T17:55:54.038+0100 7f45adad7700 20 Read xattr: user.rgw.idtag 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x3, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete recalculating target 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete reading permissions 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init op 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op mask 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete required_mask= 4 user.op_mask=7 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op permissions 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete -- Getting permissions begin with perm_mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for uid=j 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for user not found 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=1 mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=2 mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found 2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0), owner=c, perm=0 2020-07-11T17:55:54.038+0100 7f45adad7700 10 req 15 0.004000002s s3:multi_object_delete identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=0, user_perm_mask=2, acl perm=0 2020-07-11T17:55:54.038+0100 7f45adad7700 1 op->ERRORHANDLER: err_no=-13 new_err_no=-13 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete op status=0 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete http status=403 2020-07-11T17:55:54.038+0100 7f45adad7700 1 ====== req done req=0x7f45adaced50 op status=0 http_status=403 latency=0.004000002s ====== 2020-07-11T17:55:54.038+0100 7f45adad7700 20 process_request() returned -13 2020-07-11T17:55:54.038+0100 7f45adad7700 1 civetweb: 0x5628b9424000: 192.168.80.135 - - [11/Jul/2020:17:55:54 +0100] "POST /mybucket/?delete HTTP/1.1" 403 464 - aws-sdk-java/1.11.820 Linux/5.7.7-200.fc32.x86_64 OpenJDK_64-Bit_Server_VM/14.0.1+7 java/14.0.1 vendor/Red_Hat,_Inc.

2 1

[errno 2] RADOS object not found (error connecting to the cluster)
by Cem Zafer 11 Jul '20

11 Jul '20

Hi, I have executed the ceph -n osd.0 --show-config command but replied with that error message. [errno 2] RADOS object not found (error connecting to the cluster) Could someone prompt me to the right direction what could be the problem? Thanks. Regards. ceph version 15.2.4 I copied the client.admin key to the hosts and here is my ceph.conf file. # minimal ceph.conf for 4372945a-b43d-11ea-b1b7-49709def22d4 [global] fsid = 4372945a-b43d-11ea-b1b7-49709def22d4 mon_host = 192.168.1.10,192.168.1.11,192.168.1.12 mon_initial_members = 192.168.1.10,192.168.1.11 public network = 192.168.1.0/24 cluster network = 10.10.10.0/24 auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx

1 0

Radosgw activity in cephadmin
by 7vik.sathvik＠gmail.com 11 Jul '20

11 Jul '20

Hi , I have been trying to get S3 compliant up and running on Ceph using Cephadm, but unable to proceed with radosgw -admin

1 0

Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
by Eric Smith 11 Jul '20

11 Jul '20

It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues. Eric -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 1:46 PM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot For what it's worth - all of our objects are generating LONG named object files like so... \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad? -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 6:59 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Podman 2 + cephadm bootstrap == mon won't start
by David Orman 11 Jul '20

11 Jul '20

Hi, Using the repo suggested for Ubuntu 18 ( https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/sta… ) podman 2.0.2~1 is installed. However, when attempting to use cephadm to bootstrap a cluster, we see an error when attempting to start the mon container: "Error: invalid config provided: AppArmorProfile and privileged are mutually exclusive options" From the bit of reading we've done, this looks to be an issue with Podman v2 compatibility, and it appears to break with Ceph 15.2.4. Has anybody else run into this/been able to workaround it? We'll have to downgrade podman, but unfortunately, that repo does not keep previous versions. Thanks!

2 2

Convert RBD Export-Diff to RAW without a Ceph Cluster?
by Van Alstyne, Kenneth 10 Jul '20

10 Jul '20

Apologies if this is a resend, as my external email address changed and I had to update my account. Hello all, strange question: I am working with several RBD export-diffs and need them in some other format, usable by something else, such as just a "RAW" image. The obvious answer to do that is rbd import-diff the files and then rbd export to a file, but is there a way to get there directly without requiring a supplementary Ceph cluster? Thanks, -- Kenneth Van Alstyne Systems Architect M: 228.547.8045 15052 Conference Center Dr, Chantilly, VA 20151 perspecta

1 0

Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
by Eric Smith 10 Jul '20

10 Jul '20

For what it's worth - all of our objects are generating LONG named object files like so... \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad? -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 6:59 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

A MON doesn't start after Octopus update
by Hervé Ballans 10 Jul '20

10 Jul '20

Hi, We encounter a strange issue on one of our Ceph cluster. Cluster has been running with no concerns since many years. But recently, after an upgrade from Octopus 15.2.1 to 15.2.4, all is OK but one of the 3 MON refuses to start ! Journalctl shows some problems regarding the RocksDBStore and a problem of Bad table magic number. On the other nodes, MONs start correctly, and all other services too, MDS, MGR and OSD (just we have 2 OSDs stopped due to hardware disk problem). Some OSD are nearfull but I don't think there is a link with the MON issue ?.. Could someone help me on this issue please ? Thnks in advance, Hervé Here are few logs : # ceph versions { "mon": { "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 3 }, "mgr": { "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 3 }, "osd": { "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 131 }, "mds": { "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 3 }, "overall": { "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 140 } } # ceph -s cluster: id: 00ff6861-f95c-4740-a570-da6ad6a261cd health: HEALTH_WARN 1/3 mons down, quorum inf-ceph-mds,cluster-r730-k80-1 8 backfillfull osd(s) 18 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull Degraded data redundancy: 5570/50534523 objects degraded (0.011%), 2 pgs degraded, 2 pgs undersized 2 pool(s) backfillfull 1 pool(s) nearfull 25 daemons have recently crashed services: mon: 3 daemons, quorum inf-ceph-mds,cluster-r730-k80-1 (age 44h), out of quorum: cluster-r820-1 mgr: cluster-r730-k80-1(active, since 44h), standbys: inf-ceph-mds, cluster-r820-1 mds: cephfs_cluster:2 {0=cluster-r730-k80-1=up:active,1=inf-ceph-mds=up:active} 1 up:standby osd: 133 osds: 131 up (since 42h), 131 in (since 42h); 21 remapped pgs task status: scrub status: mds.cluster-r730-k80-1: idle mds.inf-ceph-mds: idle data: pools: 3 pools, 4129 pgs objects: 16.84M objects, 47 TiB usage: 143 TiB used, 33 TiB / 176 TiB avail pgs: 5570/50534523 objects degraded (0.011%) 75039/50534523 objects misplaced (0.148%) 4104 active+clean 19 active+remapped+backfill_toofull 3 active+clean+scrubbing+deep 2 active+undersized+degraded+remapped+backfill_toofull 1 active+clean+scrubbing io: client: 19 MiB/s rd, 26 MiB/s wr, 5 op/s rd, 11 op/s wr # systemctl status ceph-mon(a)cluster-r820-1.service ● ceph-mon(a)cluster-r820-1.service - Ceph cluster monitor daemon Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled) Active: failed (Result: signal) since Fri 2020-07-10 11:44:33 CEST; 24min ago Process: 201002 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id cluster-r820-1 --setuser ceph --setgroup ceph (code=killed, signal=ABRT) Main PID: 201002 (code=killed, signal=ABRT) Jul 10 11:44:33 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Service RestartSec=10s expired, scheduling restart. Jul 10 11:44:33 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Scheduled restart job, restart counter is at 5. Jul 10 11:44:33 cluster-r820-1 systemd[1]: Stopped Ceph cluster monitor daemon. Jul 10 11:44:33 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Start request repeated too quickly. Jul 10 11:44:33 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Failed with result 'signal'. Jul 10 11:44:33 cluster-r820-1 systemd[1]: Failed to start Ceph cluster monitor daemon. # journalctl --follow Jul 10 11:43:38 cluster-r820-1 systemd[1]: Started Ceph cluster monitor daemon. Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: /build/ceph-15.2.4/src/kv/RocksDBStore.cc: In function 'virtual int RocksDBStore::get(const string&, const string&, ceph::bufferlist*)' thread 7fe5331a65c0 time 2020-07-10T11:43:39.192838+0200 Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: /build/ceph-15.2.4/src/kv/RocksDBStore.cc: 1152: ceph_abort_msg("Bad table magic number: expected 9863518390377041911, found 5836668660487291005 in /var/lib/ceph/mon/ceph-cluster-r820-1/store.db/716151.sst") Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fe534aea7c6] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: *** Caught signal (Aborted) ** Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: in thread 7fe5331a65c0 thread_name:ceph-mon Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2020-07-10T11:43:39.192+0200 7fe5331a65c0 -1 /build/ceph-15.2.4/src/kv/RocksDBStore.cc: In function 'virtual int RocksDBStore::get(const string&, const string&, ceph::bufferlist*)' thread 7fe5331a65c0 time 2020-07-10T11:43:39.192838+0200 Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: /build/ceph-15.2.4/src/kv/RocksDBStore.cc: 1152: ceph_abort_msg("Bad table magic number: expected 9863518390377041911, found 5836668660487291005 in /var/lib/ceph/mon/ceph-cluster-r820-1/store.db/716151.sst") Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fe534aea7c6] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (()+0x12730) [0x7fe533bea730] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (gsignal()+0x10b) [0x7fe5336cd7bb] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (abort()+0x121) [0x7fe5336b8535] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x7fe534aea897] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 6: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 7: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 8: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2020-07-10T11:43:39.196+0200 7fe5331a65c0 -1 *** Caught signal (Aborted) ** Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: in thread 7fe5331a65c0 thread_name:ceph-mon Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (()+0x12730) [0x7fe533bea730] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (gsignal()+0x10b) [0x7fe5336cd7bb] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (abort()+0x121) [0x7fe5336b8535] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x7fe534aea897] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 6: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 7: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 8: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: -1> 2020-07-10T11:43:39.192+0200 7fe5331a65c0 -1 /build/ceph-15.2.4/src/kv/RocksDBStore.cc: In function 'virtual int RocksDBStore::get(const string&, const string&, ceph::bufferlist*)' thread 7fe5331a65c0 time 2020-07-10T11:43:39.192838+0200 Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: /build/ceph-15.2.4/src/kv/RocksDBStore.cc: 1152: ceph_abort_msg("Bad table magic number: expected 9863518390377041911, found 5836668660487291005 in /var/lib/ceph/mon/ceph-cluster-r820-1/store.db/716151.sst") Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fe534aea7c6] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 0> 2020-07-10T11:43:39.196+0200 7fe5331a65c0 -1 *** Caught signal (Aborted) ** Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: in thread 7fe5331a65c0 thread_name:ceph-mon Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (()+0x12730) [0x7fe533bea730] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (gsignal()+0x10b) [0x7fe5336cd7bb] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (abort()+0x121) [0x7fe5336b8535] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x7fe534aea897] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 6: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 7: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 8: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: -1> 2020-07-10T11:43:39.192+0200 7fe5331a65c0 -1 /build/ceph-15.2.4/src/kv/RocksDBStore.cc: In function 'virtual int RocksDBStore::get(const string&, const string&, ceph::bufferlist*)' thread 7fe5331a65c0 time 2020-07-10T11:43:39.192838+0200 Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: /build/ceph-15.2.4/src/kv/RocksDBStore.cc: 1152: ceph_abort_msg("Bad table magic number: expected 9863518390377041911, found 5836668660487291005 in /var/lib/ceph/mon/ceph-cluster-r820-1/store.db/716151.sst") Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fe534aea7c6] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 0> 2020-07-10T11:43:39.196+0200 7fe5331a65c0 -1 *** Caught signal (Aborted) ** Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: in thread 7fe5331a65c0 thread_name:ceph-mon Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 1: (()+0x12730) [0x7fe533bea730] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 2: (gsignal()+0x10b) [0x7fe5336cd7bb] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 3: (abort()+0x121) [0x7fe5336b8535] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x7fe534aea897] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 5: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ce) [0x559f6081b2ae] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 6: (main()+0x108d) [0x559f605a85fd] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 7: (__libc_start_main()+0xeb) [0x7fe5336ba09b] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: 8: (_start()+0x2a) [0x559f605b93da] Jul 10 11:43:39 cluster-r820-1 ceph-mon[200827]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 10 11:43:39 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Main process exited, code=killed, status=6/ABRT Jul 10 11:43:39 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Failed with result 'signal'. Jul 10 11:43:49 cluster-r820-1 systemd[1]: ceph-mon(a)cluster-r820-1.service: Service RestartSec=10s expired, scheduling restart. ...

2 2

Luminous 12.2.12 - filestore OSDs take an hour to boot
by Eric Smith 10 Jul '20

10 Jul '20

I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long.

1 0

Lost Journals for XFS OSDs
by Mike Dawson 10 Jul '20

10 Jul '20

Tonight an old Ceph cluster we run suffered a hardware failure that resulted in the loss of Ceph journal SSDs on 7 nodes out of 36. Overview of this old setup: - Super-old Ceph Dumpling v0.67 - 3x replication for RBD w/ 3 failure domains in replication hierarchy - OSDs on XFS on spinning disks with Journals on SSD In total we lost 7 SSDs hosting Journals for 21 OSDs (3 each). The lost nodes span all three failure domains which makes me nervous that there are likely missing Placement Groups in the pool. Due to how Ceph shards data across the Placement Groups, I'm concerned I may have lost all the RBD volumes in this pool. The obvious solution is to attempt to bring the OSDs back online (for at least one failure domain) to ensure there is at least one complete copy of the data then rebuild everything else. The issue is I lost the journals when the SSDs died. I don't see much published about recovering OSDs in the event of a lost journal except: https://ceph.io/geen-categorie/ceph-recover-osds-after-ssd-journal-failure/ And that doesn't mention if the data is valid afterwards. I think I recall Inktank used to deal with this situation and may have had a potential solution. At this point, I'll take any constructive advice. Thanks you in advance, Mike

1 0

Re: RBD thin provisioning and time to format a volume
by Void Star Nill 09 Jul '20

09 Jul '20

On Thu, Jul 9, 2020 at 10:33 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > What about ntfs? You have there a not quick option. Maybe it writes to > the whole disk some random pattern. Why do you ask? > I am writing an API layer to plug into our platform, so I want to know if the format times can be deterministic or unbounded. From what I saw with ext3, ext4 and xfs volumes, the format time is actually not dependent on the size of the volume, so I just wanted to confirm if we can assume that, or if I am missing something. Thanks, Shridhar > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: RBD thin provisioning and time to format a > volume > > Thanks Jason. > > Do you mean to say some filesystems will initialize the entire disk > during format? Does that mean we will see the entire size of the volume > getting allocated during formatting? > Or do you mean to say, some filesystem formatting just takes longer than > others, as it does more initialization? > > I am just trying to understand if there are cases where Ceph will > allocate all the blocks for a filesystem during formation operations, OR > if they continue to be thin provisioned (allocate as you go based on > real data). So far I have tried with ext3, ext4 and xfs and none of them > dont allocate all the blocks during format. > > -Shridhar > > > On Thu, 9 Jul 2020 at 06:58, Jason Dillaman <jdillama(a)redhat.com> wrote: > > > On Thu, Jul 9, 2020 at 12:02 AM Void Star Nill > > <void.star.nill(a)gmail.com> > > wrote: > > > > > > > > > > > > On Wed, Jul 8, 2020 at 4:56 PM Jason Dillaman <jdillama(a)redhat.com> > > wrote: > > >> > > >> On Wed, Jul 8, 2020 at 3:28 PM Void Star Nill > > >> <void.star.nill(a)gmail.com> > > wrote: > > >> > > > >> > Hello, > > >> > > > >> > My understanding is that the time to format an RBD volume is not > > dependent > > >> > on its size as the RBD volumes are thin provisioned. Is this > correct? > > >> > > > >> > For example, formatting a 1G volume should take almost the same > > >> > time > > as > > >> > formatting a 1TB volume - although accounting for differences in > > latencies > > >> > due to load on the Ceph cluster. Is that a fair assumption? > > >> > > >> Yes, that is a fair comparison when creating the RBD image. > > >> However, a format operation might initialize and discard extents on > > > >> the disk, so a larger disk will take longer to format. > > > > > > > > > Thanks for the response Jason. Could you please explain a bit more > > > on > > the the format operation? > > > > I'm not sure what else there is to explain. When you create a file > > system on top of any block device, it needs to initialize the block > > device. Depending on the file system, it might take more time for > > larger block devices because it's doing more work. > > > > > Is there a relative time that we can determine based on the volume > size? > > > > > > Thanks > > > Shridhar > > > > > > > > >> > > >> > > >> > Thanks, > > >> > Shridhar > > >> > _______________________________________________ > > >> > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send > > > >> > an email to ceph-users-leave(a)ceph.io > > >> > > > >> > > >> > > >> -- > > >> Jason > > >> > > > > > > -- > > Jason > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > >

1 0

post - bluestore default vs tuned performance comparison
by Frank Ritchie 09 Jul '20

09 Jul '20

Hi, For this post: https://ceph.io/community/bluestore-default-vs-tuned-performance-comparison/ I don't see a way to contact the authors so I thought I would try here. Does anyone know how the rocksdb tuning parameters of: " bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB " were chosen? Some of the settings seem to not be in line with the rocksdb tuning guide: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide thx Frank

2 1

RBD thin provisioning and time to format a volume
by Void Star Nill 09 Jul '20

09 Jul '20

Hello, My understanding is that the time to format an RBD volume is not dependent on its size as the RBD volumes are thin provisioned. Is this correct? For example, formatting a 1G volume should take almost the same time as formatting a 1TB volume - although accounting for differences in latencies due to load on the Ceph cluster. Is that a fair assumption? Thanks, Shridhar

3 5

Bucket index logs (bilogs) not being trimmed automatically (multisite, ceph nautilus 14.2.9)
by david.piper＠metaswitch.com 09 Jul '20

09 Jul '20

Hi all, We're seeing a problem in our multisite Ceph deployment, where bilogs aren't being trimmed for several buckets. This is causing bilogs to accumulate over time, leading to large OMAP object warnings for the indexes on these buckets. In every case, Ceph reports that the bucket is in sync and the data is consistent across both sites, so we're perplexed as to why the logs aren't being trimmed. It's not affecting all of our buckets, we're not sure what's 'different' in the affected cases causing them to accumulate. We're seeing this in both unsharded and sharded buckets. Some buckets with heavy activity (lots of object updates) have accumulated millions of bilogs, but this does not affect all of our very active buckets. I've tried running 'radosgw-admin bilog autotrim' against an affected bucket, and it doesn't appear to do anything. I've used 'radosgw-admin bilog trim' with a suitable 'end-marker' to trim all of the bilogs, but the implications of doing this aren't clear to me, and the logs continue to accumulate afterwards. We're running Ceph Nautilus 14.2.9 and running the services in containers, we have 3 hosts on each site with 1 OSD on each host. We're adjusting a fairly minimal set of config and I don't think it includes anything that would affect bilog trimming. Checking the running config on the mon service, I think these defaulted parameters are relevant: "rgw_sync_log_trim_concurrent_buckets": "4", "rgw_sync_log_trim_interval": "1200", "rgw_sync_log_trim_max_buckets": "16", "rgw_sync_log_trim_min_cold_buckets": "4", I can't find any documentation on these parameters but we have more than 16 buckets, so is it possible that some buckets are just never being selected for trimming? Any other ideas as to what might be causing this, or anything else we could try to help diagnose or fix this? Thanks in advance! I've included an example below for one such affected bucket, showing its current state. Zone details (as per 'radosgw-admin zonegroup get') are at the bottom. $ radosgw-admin bucket sync status --bucket=edin2z6-sharedconfig realm b7f31089-0879-4fa2-9cbc-cfdf5f866a35 (geored_realm) zonegroup 5d74eb0e-5d99-481f-ae33-43483f6cebc0 (geored_zg) zone c48f33ad-6d79-4b9f-a22f-78589f67526e (siteA) bucket edin2z6-sharedconfig[033709fc-924a-4582-b00d-97c90e9e61b6.3634407.1] source zone 0a3c29b7-1a2c-432d-979b-d324a05cc831 (siteApubsub) full sync: 0/1 shards incremental sync: 0/1 shards bucket is caught up with source source zone 9f5fba56-4a32-46a6-8695-89253be81614 (siteB) full sync: 0/1 shards incremental sync: 1/1 shards bucket is caught up with source source zone c72b3aa8-a051-4665-9421-909510702412 (siteBpubsub) full sync: 0/1 shards incremental sync: 0/1 shards bucket is caught up with source $ radosgw-admin bilog list --bucket edin2z6-sharedconfig --max-entries 600000000 | grep op_id | wc -l 1299392 $ rados -p siteA.rgw.buckets.index listomapkeys .dir.033709fc-924a-4582-b00d-97c90e9e61b6.3634407.1 | wc -l 1299083 $ radosgw-admin bucket stats --bucket=edin2z6-sharedconfig { "bucket": "edin2z6-sharedconfig", "num_shards": 0, "tenant": "", "zonegroup": "5d74eb0e-5d99-481f-ae33-43483f6cebc0", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "033709fc-924a-4582-b00d-97c90e9e61b6.3634407.1", "marker": "033709fc-924a-4582-b00d-97c90e9e61b6.3634407.1", "index_type": "Normal", "owner": "edin2z6", "ver": "0#1622676", "master_ver": "0#0", "mtime": "2020-01-14 14:30:18.606142Z", "max_marker": "0#00001622675.2115836.5", "usage": { "rgw.main": { "size": 15209, "size_actual": 40960, "size_utilized": 15209, "size_kb": 15, "size_kb_actual": 40, "size_kb_utilized": 15, "num_objects": 7 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } $ radosgw-admin bucket limit check ... { "bucket": "edin2z6-sharedconfig", "tenant": "", "num_objects": 7, "num_shards": 0, "objects_per_shard": 7, "fill_status": "OK" }, ... $ radosgw-admin zonegroup get ++ sudo docker ps --filter name=ceph-rgw-.*rgw -q ++ sudo docker exec d2c999b1f3f8 radosgw-admin { "id": "5d74eb0e-5d99-481f-ae33-43483f6cebc0", "name": "geored_zg", "api_name": "geored_zg", "is_master": "true", "endpoints": [ "https://10.254.2.93:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "c48f33ad-6d79-4b9f-a22f-78589f67526e", "zones": [ { "id": "0a3c29b7-1a2c-432d-979b-d324a05cc831", "name": "siteApubsub", "endpoints": [ "https://10.254.2.93:7481", "https://10.254.2.94:7481", "https://10.254.2.95:7481" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "pubsub", "sync_from_all": "false", "sync_from": [ "siteA" ], "redirect_zone": "" }, { "id": "9f5fba56-4a32-46a6-8695-89253be81614", "name": "siteB", "endpoints": [ "https://10.254.2.224:7480", "https://10.254.2.225:7480", "https://10.254.2.226:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "c48f33ad-6d79-4b9f-a22f-78589f67526e", "name": "siteA", "endpoints": [ "https://10.254.2.93:7480", "https://10.254.2.94:7480", "https://10.254.2.95:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "c72b3aa8-a051-4665-9421-909510702412", "name": "siteBpubsub", "endpoints": [ "https://10.254.2.224:7481", "https://10.254.2.225:7481", "https://10.254.2.226:7481" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "pubsub", "sync_from_all": "false", "sync_from": [ "siteB" ], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "b7f31089-0879-4fa2-9cbc-cfdf5f866a35" }

1 0

default.rgw.data.root pool
by Seena Fallah 09 Jul '20

09 Jul '20

Hi all. Is there any docs related to default.rgw.data.root pool? I have this pool and there are no objects in default.rgw.meta pool. Thanks for your help.

1 0

Ceph df Vs Dashboard pool usage mismatch
by Richard Kearsley 09 Jul '20

09 Jul '20

Hi My ceph dashboard reports 64% usage for rgw.buckets.data: [image: cephdashboard.png] But "ceph df" command shows 56.81% used: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 611 TiB 282 TiB 328 TiB 329 TiB 53.81 TOTAL 611 TiB 282 TiB 328 TiB 329 TiB 53.81 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 12 KiB 25 7.5 MiB 0 50 TiB rgw.control 5 0 B 15 0 B 0 50 TiB rgw.meta 6 13 KiB 51 14 MiB 0 50 TiB rgw.log 7 7.5 MiB 783 69 MiB 0 50 TiB rgw.buckets.index 8 57 MiB 16 57 MiB 0 50 TiB buckets.non-ec 10 0 B 0 0 B 0 50 TiB rgw.buckets.data 11 217 TiB 57.48M 328 TiB 56.81 187 TiB Does ceph df have the correct one? 🤔 ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) Thanks Richard

3 2

Questions on Ceph on ARM
by norman 09 Jul '20

09 Jul '20

Hi all, I'm using Ceph on X86 and ARM, is it safe make x86 and arm64 in the same cluster?

5 10

bluestore: osd bluestore_allocated is much larger than bluestore_stored
by Jerry Pu 09 Jul '20

09 Jul '20

Hi: We have a cluster (v13.2.4), and we do some tests on a EC k=2, m=1 pool "VMPool0". We deploy some VMs (Windows, CentOS7) on the pool and then use IOMeter to write data to these VMs. After a period of time, we observe a strange thing that pool actual usage is much larger than stored data * 1.5 (stored_raw). [root@Sim-str-R6-4 ~]# ceph df GLOBAL: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB 48.74 TOTAL 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB 48.74 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 1 29 GiB 100.00 0 B 2597 cephfs_md 2 831 MiB 100.00 0 B 133 erasure_meta_hdd 3 22 MiB 100.00 0 B 170 VMPool0 4 1.2 TiB 56.77 644 GiB 116011 stresspool 5 2.6 MiB 100.00 0 B 32 [root@Sim-str-R6-4 ~]# ceph df detail -f json-pretty -----snippet----- { "name": "VMPool0", "id": 4, "stats": { "kb_used": 1328888832, "bytes_used": 1360782163968, <---------------- "percent_used": 0.567110, "max_avail": 692481687552, "objects": 116011, "quota_objects": 0, "quota_bytes": 0, "dirty": 116011, "rd": 27449034, "rd_bytes": 126572760064, "wr": 20675381, "wr_bytes": 1006460652544, "comp_ratio": 1.000000, "stored": 497657610240, "stored_raw": 746486431744, <---------------- } }, The perf counters of all osds (all hdd) used by VMPool0 also show that bluestore_allocated is much larger than bluestore_stored. [root@Sim-str-R6-4 ~]# for i in {0..3}; do echo $i; ceph daemon osd.$i perf dump | grep bluestore | head -6; done 0 "bluestore": { "bluestore_allocated": 175032369152, <---------------- "bluestore_stored": 83557936482, <---------------- "bluestore_compressed": 958795770, "bluestore_compressed_allocated": 6431965184, "bluestore_compressed_original": 18576584704, 1 "bluestore": { "bluestore_allocated": 119943593984, <---------------- "bluestore_stored": 53325238866, <---------------- "bluestore_compressed": 670158436, "bluestore_compressed_allocated": 4751818752, "bluestore_compressed_original": 13752328192, 2 "bluestore": { "bluestore_allocated": 155444707328, <---------------- "bluestore_stored": 69067116553, <---------------- "bluestore_compressed": 565170876, "bluestore_compressed_allocated": 4614324224, "bluestore_compressed_original": 13469696000, 3 "bluestore": { "bluestore_allocated": 128179240960, <---------------- "bluestore_stored": 60884752114, <---------------- "bluestore_compressed": 1653455847, "bluestore_compressed_allocated": 9741795328, "bluestore_compressed_original": 27878768640, [root@Sim-str-R6-5 osd]# for i in {4..7}; do echo $i; sh -c "ceph daemon osd.$i perf dump | grep bluestore | head -6"; done 4 "bluestore": { "bluestore_allocated": 165950652416, <---------------- "bluestore_stored": 80255191687, <---------------- "bluestore_compressed": 1526871060, "bluestore_compressed_allocated": 8900378624, "bluestore_compressed_original": 25324142592, 5 admin_socket: exception getting command descriptions: [Errno 111] Connection refused 6 "bluestore": { "bluestore_allocated": 166022152192, <---------------- "bluestore_stored": 84645390708, <---------------- "bluestore_compressed": 1169055606, "bluestore_compressed_allocated": 8647278592, "bluestore_compressed_original": 25135091712, 7 "bluestore": { "bluestore_allocated": 204633604096, <---------------- "bluestore_stored": 100116382041, <---------------- "bluestore_compressed": 1081260422, "bluestore_compressed_allocated": 6510018560, "bluestore_compressed_original": 18654052352, [root@Sim-str-R6-6 osd]# for i in {8..12}; do echo $i; ceph daemon osd.$i perf dump | grep bluestore | head -6; done 8 "bluestore": { "bluestore_allocated": 106330193920, <---------------- "bluestore_stored": 45282848089, <---------------- "bluestore_compressed": 1136610231, "bluestore_compressed_allocated": 7248609280, "bluestore_compressed_original": 20882960384, 9 "bluestore": { "bluestore_allocated": 120657412096, <---------------- "bluestore_stored": 52550745942, <---------------- "bluestore_compressed": 1321632665, "bluestore_compressed_allocated": 7401504768, "bluestore_compressed_original": 21073027072, 10 "bluestore": { "bluestore_allocated": 155985772544, <---------------- "bluestore_stored": 73236910054, <---------------- "bluestore_compressed": 98351920, "bluestore_compressed_allocated": 772210688, "bluestore_compressed_original": 2242043904, 11 "bluestore": { "bluestore_allocated": 106040524800, <---------------- "bluestore_stored": 45353612134, <---------------- "bluestore_compressed": 874216443, "bluestore_compressed_allocated": 4962844672, "bluestore_compressed_original": 14160310272, 12 "bluestore": { "bluestore_allocated": 118751363072, <---------------- "bluestore_stored": 52194408691, <---------------- "bluestore_compressed": 782919969, "bluestore_compressed_allocated": 5546311680, "bluestore_compressed_original": 16043233280, The config min_alloc_size_hdd of all osds are 64K and rados objects of VMPool0 are all 4M rbd_data.x.xxxxxxx.xxxxxxx objects. It's kind of strange that allocation space is much larger than stored data. Can anyone explains this?

2 7