January 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1

Radosgw PubSub Traffic
by Dustin Guerrero 02 Jun '20

02 Jun '20

Hey all, We’ve been running some benchmarks against Ceph which we deployed using the Rook operator in Kubernetes. Everything seemed to scale linearly until a point where I see a single OSD receiving much higher CPU load than the other OSDs (nearly 100% saturation). After some investigation we noticed a ton of pubsub traffic in the strace coming from the RGW pods like so: [pid 22561] sendmsg(77, {msg_name(0)=NULL, msg_iov(3)=[{"\21\2)\0\0\0\10\0:\1\0\0\10\0\0\0\0\0\10\0\0\0\0\0\0\20\0\0-\321\211K"..., 73}, {"\200\0\0\0pubsub.user.ceph-user-wwITOk"..., 314}, {"\0\303\34[\360\314\233\2138\377\377\377\377\377\377\377\377", 17}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL|MSG_MORE <unfinished …> I’ve checked other OSDs and only a single OSD receives these messages. I suspect its creating a bottleneck. Does anyone have an idea on why these are being generated or how to stop them? The pubsub sync module doesn’t appear to be enabled, and our benchmark is doing simple gets/puts/deletes. We’re running Ceph 14.2.5 nautilus Thank you!

2 2

General question CephFS or RBD
by Willi Schiegel 29 May '20

29 May '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

2 1

Ceph and iSCSI
by Bobby 29 May '20

29 May '20

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

2 1

ceph with rdma can not mount with kernel
by 李亚锋 29 May '20

29 May '20

2 1

Ceph on CentOS 8?
by Jan Kasprzak 29 May '20

29 May '20

Hello, Ceph users, does anybody use Ceph on recently released CentOS 8? Apparently there are no el8 packages neither at download.ceph.com, nor in the native CentOS package tree. I am thinking about upgrading my cluster to C8 (because of other software running on it apart from Ceph). Do el7 packages simply work? Can they be rebuilt using rpmbuild --rebuild? Or is running Ceph on C8 more complicated than that? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | sir_clive> I hope you don't mind if I steal some of your ideas? laryross> As far as stealing... we call it sharing here. --from rcgroups

9 8

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

upmap balancer
by Frank R 16 Mar '20

16 Mar '20

Hi all, Does using the Upmap balancer require that all OSDs be the same size (per device class)? thx Frank

4 7

Re: Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)
by Ilya Dryomov 15 Mar '20

15 Mar '20

On Tue, Jan 14, 2020 at 10:31 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I think this is new since I upgraded to 14.2.6. kworker/7:3 100% > > [@~]# echo l > /proc/sysrq-trigger > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192e7e>] > [<ffffffffb0192e7e>] generic_swap+0x1e/0x30 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a20 EFLAGS: 00000206 > [Tue Jan 14 10:05:08 2020] RAX: 0000000000000072 RBX: 0000000000000060 > RCX: 0000000000000072 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000006 RSI: ffffa0c788914a7a > RDI: ffffa0c788914a42 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a20 R08: 0000000000000028 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe94984224500 > R12: ffffa0c788914a40 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0c788914a18 > R15: ffffa0c788914a78 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > > > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192200>] > [<ffffffffb0192200>] __x86_indirect_thunk_rax+0x0/0x20 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a28 EFLAGS: 00000286 > [Tue Jan 14 10:05:08 2020] RAX: ffffffffb0192e60 RBX: 0000000000000010 > RCX: 0000000000000010 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000008 RSI: ffffa0d0e108b828 > RDI: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a98 R08: 0000000000000000 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe949a9842280 > R12: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0d0e108b818 > R15: ffffa0d0e108b828 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] ? sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 Hi Marc, It's busy processing snapshot contexts, I think it's more or less the same issue that you reported in "ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)". Did "ceph: build_snap_context 100020c9287 ffff911a9a26bd00 fail -12" errors go away? How many snapshots do you have now? Did you file a tracker ticket? It's better to have all details in one place rather than scattered across different threads on the list. Adding Jeff and Zheng, who can take a closer look. Thanks, Ilya

2 2

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

High CPU usage by ceph-mgr in 14.2.6
by jbardgett＠godaddy.com 07 Mar '20

07 Mar '20

After upgrading one of our clusters from Luminous 12.2.12 to Nautilus 14.2.6, I am seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). The way we found this was due to Prometheus being unable to report out certain pieces of data, specifically OSD Usage, OSD Apply and Commit Latency. Which are all similar issues people were having in previous versions of Nautilus. Bryan Stillwell reported this previously on a separate cluster, 14.2.5, we have here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VW3GNVJGOOW… That issue was resolved with the upgrade to 14.2.6. We are seeing a similar issue on this other cluster with a couple differences. This cluster has 1900+ OSD in it, the previous one had 300+ The top user is libceph-common, instead of mmap 4.86% libceph-common.so.0 [.] EventCenter::create_time_event 2.78% [kernel] [k] nmi 2.64% libstdc++.so.6.0.19 [.] __dynamic_cast On all our other clusters that have been upgraded to 14.2.6 we are not experiencing this issue, the next largest being 800+ OSD. We feel this is related to the size of the cluster, similarly to the previous report. Anyone else experiencing this and/or can provide some direction on how to go about resolving this? Thanks, Joe

6 7

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

ceph-mon using 100% CPU after upgrade to 14.2.5
by Bryan Stillwell 24 Feb '20

24 Feb '20

On our test cluster after upgrading to 14.2.5 I'm having problems with the mons pegging a CPU core while moving data around. I'm currently converting the OSDs from FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process into a state where it pegs a CPU core on one of the mons: 1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the cluster was running 14.2.4. Thanks, Bryan

5 8

slow using ISCSI - Help-me
by Gesiel Galvão Bernardes 19 Feb '20

19 Feb '20

Hi, I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs: Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Can anyone help-me please? Gesiel

2 14

ceph status reports: slow ops - this is related to long running process /usr/bin/ceph-osd
by Thomas 18 Feb '20

18 Feb '20

Hi, ceph status reports: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available 4 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 59 pgs inactive, 16 pgs peering Degraded data redundancy: 597/153910758 objects degraded (0.000%), 2 pgs degraded, 1 pg undersized Degraded data redundancy (low space): 23 pgs backfill_toofull 1 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 3 pools have too many placement groups 164 slow requests are blocked > 32 sec 1082 stuck requests are blocked > 4096 sec 1490 slow ops, oldest one blocked for 19711 sec, daemons [osd,0,osd,175,osd,186,osd,5,osd,6,osd,63,osd,68,osd,9,mon,ld5505,mon,ld5506]... have slow ops. services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 5h) mgr: ld5507(active, since 5h), standbys: ld5506, ld5505 mds: pve_cephfs:0/1, 1 failed osd: 419 osds: 416 up, 416 in; 6024 remapped pgs data: pools: 6 pools, 8864 pgs objects: 51.30M objects, 196 TiB usage: 594 TiB used, 907 TiB / 1.5 PiB avail pgs: 0.666% pgs not active 597/153910758 objects degraded (0.000%) 52964415/153910758 objects misplaced (34.412%) 5954 active+remapped+backfill_wait 2786 active+clean 40 active+remapped+backfilling 35 activating 23 active+remapped+backfill_wait+backfill_toofull 16 peering 7 activating+remapped 1 activating+undersized+degraded 1 active+clean+scrubbing 1 active+recovering+degraded io: client: 3.5 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 551 MiB/s, 137 objects/s I'm concerned about the slow ops on osd.0 and osd.9. On the relevant OSD node I can see 2 relevant services running for hours: ceph 14795 1 99 09:58 ? 08:49:22 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph ceph 15394 1 99 09:58 ? 07:10:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph In the relevant osd log I can find similar messages: root@ld5505:~# tail -f /var/log/ceph/ceph-osd.0.log 2019-10-08 15:35:32.830 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:33.806 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:34.842 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:35.862 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) root@ld5505:~# tail -f /var/log/ceph/ceph-osd.9.log 2019-10-08 15:35:38.822 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:39.854 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:40.850 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:41.862 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) Question: How can I analyse and solve the issue with slow ops? THX

3 2

RBD-mirror instabilities
by Oliver Freyermuth 15 Feb '20

15 Feb '20

Dear Cephalopodians, running 13.2.6 on the source cluster and 14.2.5 on the rbd mirror nodes and the target cluster, I observe regular failures of rbd-mirror processes. With failures, I mean that traffic stops, but the daemons are still listed as active rbd-mirror daemons in "ceph -s", and the daemons are still running. This comes in sync with a hefty load of below messages in the mirror logs. This happens "sometimes" when some OSDs go down and up in the target cluster (which happens each night since the disks in that cluster shortly go offline during "online" smart self-tests - that's a problem in itself, but it's a cluster built from hardware that would have been trashed otherwise). The rbd daemons keep running in any case, but synchronization stops. If not all rbd mirror daemons have failed (we have three running, and it usually does not hit all of them), the "surviving" seem(s) not to take care of the images the other daemons had locked. Right now, I am eyeing with a "quick solution" of regularly restarting the rbd-mirror daemons, but if there are any good ideas on which debug info I could collect to get this analyzed and fixed, that would of course be appreciated :-). Cheers, Oliver ----------------------------------------------- 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_process_entry_safe: failed to commit journal event: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_replay_complete: replay encountered an error: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_process_entry_safe: failed to commit journal event: (125) Operation canceled 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_replay_complete: replay encountered an error: (125) Operation canceled 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dce2e05b0 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dcf47ee70 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:55.716 7f31f5b6f700 -1 rbd::mirror::ImageReplayer: 0x559dcb997680 [2/f8218221-6608-4a2b-8831-84ca0c2cb418] operator(): start failed: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0577bd16-acc4-4e9a-81f0-c698a24f8771: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=05bd4cca-a561-4a5c-ad83-9905ad5ce34e: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0e614ece-65b1-4b4a-99bd-44dd6235eb70: blacklisted detected during image replay -----------------------------------------------

2 6

Write i/o in CephFS metadata pool
by Samy Ascha 10 Feb '20

10 Feb '20

Hi! I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool. The filesystem is otherwise stable and I'm seeing no usage issues. I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool. For example: # ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live. I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment. Some of it may be expected in when considering details of my environment and CephFS implementation specifics, so please ask away if more details are needed. With my experience using NFS, I would start by looking at client io stats, like `nfsstat` and tuning e.g. mount options, but I haven't been able to find such statistics for CephFS clients. Is there anything of the sort for CephFS? Are similar stats obtainable in some other way? This might be a somewhat broad question and shallow description, so yeah, let me know if there's anything you would like more details on. Thanks a lot, Samy

5 8

getting rid of incomplete pg errors
by Hartwig Hauschild 07 Feb '20

07 Feb '20

Hi. before I descend into what happened and why it happened: I'm talking about a test-cluster so I don't really care about the data in this case. We've recently started upgrading from luminous to nautilus, and for us that means we're retiring ceph-disk in favour of ceph-volume with lvm and dmcrypt. Our setup is in containers and we've got DBs separated from Data. When testing our upgrade-path we discovered that running the host on ubuntu-xenial and the containers on centos-7.7 leads to lvm inside the containers not using lvmetad because it's too old. That in turn means that not running `vgscan --cache` on the host before adding a LV to a VG essentially zeros the metadata for all LVs in that VG. That happened on two out of three hosts for a bunch of OSDs and those OSDs are gone. I have no way of getting them back, they've been overwritten multiple times trying to figure out what went wrong. So now I have a cluster that's got 16 pgs in 'incomplete', 14 of them with 0 objects, 2 with about 150 objects each. I have found a couple of howtos that tell me to use ceph-objectstore-tool to find the pgs on the active osds and I've given that a try, but ceph-objectstore-tool always tells me it can't find the pg I am looking for. Can I tell ceph to re-init the pgs? Do I have to delete the pools and recreate them? There's no data I can't get back in there, I just don't feel like scrapping and redeploying the whole cluster. -- Cheers, Hardy

3 4

Ubuntu 18.04.4 Ceph 12.2.12
by Atherion 06 Feb '20

06 Feb '20

Hi Ceph Community. We currently have a luminous cluster running and some machines still on Ubuntu 14.04 We are looking to upgrade these machines to 18.04 but the only upgrade path for luminous with the ceph repo is through 16.04. It is doable to get to Mimic but then we have to upgrade all those machines to 16.04 but then we have to upgrade again to 18.04 when we get to Mimic, it is becoming a huge time sink. I did notice in the Ubuntu repos they have added 12.2.12 in 18.04.4 release. Is this a reliable build we can use? https://ubuntu.pkgs.org/18.04/ubuntu-proposed-main-amd64/ceph_12.2.12-0ubun… If so then we can go straight to 18.04.4 and not waste so much time. Best

3 3

data loss on full file system?
by Håkan T Johansson 05 Feb '20

05 Feb '20

Hi, for test purposes, I have set up two 100 GB OSDs, one taking a data pool and the other metadata pool for cephfs. Am running 14.2.6-1-gffd69200ad-1 with packages from https://mirror.croit.io/debian-nautilus Am then running a program that creates a lot of 1 MiB files by calling fopen() fwrite() fclose() for each of them. Error codes are checked. This works successfully for ~100 GB of data, and then strangely also succeeds for many more 100 GB of data... ?? All written files have size 1 MiB with 'ls', and thus should contain the data written. However, on inspection, the files written after the first ~100 GiB, are full of just 0s. (hexdump -C) To further test this, I use the standard tool 'cp' to copy a few random-content files into the full cephfs filessystem. cp reports no complaints, and after the copy operations, content is seen with hexdump -C. However, after forcing the data out of cache on the client by reading other earlier created files, hexdump -C show all-0 content for the files copied with 'cp'. Data that was there is suddenly gone...? I am new to ceph. Is there an option I have missed to avoid this behaviour? (I could not find one in https://docs.ceph.com/docs/master/man/8/mount.ceph/ ) Is this behaviour related to https://docs.ceph.com/docs/mimic/cephfs/full/ ? (That page states 'sometime after a write call has already returned 0'. But if write returns 0, then no data has been written, so the user program would not assume any kind of success.) Best regards, Håkan

2 4

ceph fs dir-layouts and sub-directory mounts
by Frank Schilder 03 Feb '20

03 Feb '20

I would like to (in this order) - set the data pool for the root "/" of a ceph-fs to a custom value, say "P" (not the initial data pool used in fs new) - create a sub-directory of "/", for example "/a" - mount the sub-directory "/a" with a client key with access restricted to "/a" The client will not be able to see the dir layout attribute set at "/", its not mounted. Will the data of this client still go to the pool "P", that is, does "/a" inherit the dir layout transparently to the client when following the steps above? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

TR: Understand ceph df details
by CUZA Frédéric 03 Feb '20

03 Feb '20

Turns out it is probably orphans. We are running ceph luminous : 12.2.12 And the orphans find is stuck in the stage : "iterate_bucket_index" on shard "0" for 2 days now. Anyone is facing this issue ? Regards, De : ceph-users <ceph-users-bounces(a)lists.ceph.com<mailto:ceph-users-bounces@lists.ceph.com>> Envoyé : 21 January 2020 10:10 À : ceph-users(a)lists.ceph.com<mailto:ceph-users@lists.ceph.com> Objet : [ceph-users] Understand ceph df details Hi everyone, I'm trying to understand where is the difference between the command : ceph df details And the result I'm getting when I run this script : total_bytes=0 while read user; do echo $user bytes=$(radosgw-admin user stats --uid=${user} | grep total_bytes_rounded | tr -dc "0-9") if [ ! -z ${bytes} ]; then total_bytes=$((total_bytes + bytes)) pretty_bytes=$(echo "scale=2; $bytes / 1000^4" | bc) echo " ($bytes B) $pretty_bytes TiB" fi pretty_total_bytes=$(echo "scale=2; $total_bytes / 1000^4" | bc) done <<< "$(radosgw-admin user list | jq -r .[])" echo "" echo "Total : ($total_bytes B) $pretty_total_bytes TiB" When I run df I get this : default.rgw.buckets.data 70 N/A N/A 226TiB 89.23 27.2TiB 61676992 61.68M 2.05GiB 726MiB 677TiB And when I use my script I don't have the same result : Total : (207579728699392 B) 207.57 TiB It means that I have 20 TiB somewhere but I can't find and must of all understand where this 20 TiB. Does anyone have an explanation ? Fi : [root@ceph_monitor01 ~]# radosgw-admin gc list -include-all | grep oid | wc -l 23

2 1

small cluster HW upgrade
by Philipp Schwaha 02 Feb '20

02 Feb '20

We have a small ceph cluster running built from components that were phased out from compute applications. the current cluster consists of i7-860s. 6 disks (5TB, 7200RPM) per node and 8 nodes totaling 48 OSDs. A compute cluster will be discontinued, which will make Ryzen 5-1600 hardware available (8 nodes with 16GB RAM each) with which to replace the CPUs of the current setup. How could we best distribute the OSDs (keeping existing disks for storage) to the Ryzen systems to get a good performance improvement? Unfortunately the interconnect is still only 1Gb/s so expected to be a limiting factor. Would it make sense to create fewer bigger nodes, e.g., to use 6 nodes with 8 disks each or even more condensed? We would like to move the Luminous cluster to Nautilus/Bluestore and can get SSDs for each of the nodes, as it appears to be essential to get performance. Can actually benefit from improvements in the OSDs if the network is so limited? Would bonding of network interfaces be a workaround until we can get a network update or are we overestimating the power of the upgraded OSD nodes? What strategy would you suggest with these resources? Any comments and suggestions would be highly welcome :) Thanks in advance Philipp

4 4

CephFS - objects in default data pool
by CASS Philip 02 Feb '20

02 Feb '20

I have a query about https://docs.ceph.com/docs/master/cephfs/createfs/: "The data pool used to create the file system is the "default" data pool and the location for storing all inode backtrace information, used for hard link management and disaster recovery. For this reason, all inodes created in CephFS have at least one object in the default data pool." This does not match my experience (nautilus servers, nautlius FUSE client or Centos 7 kernel client). I have a cephfs with a replicated top-level pool and a directory set to use erasure coding with setfattr, though I also did the same test using the subvolume commands with the same result. "Ceph df detail" shows no objects used in the top level pool, as shown in https://gist.github.com/pcass-epcc/af24081cf014a66809e801f33bcb535b (also displayed in-line below) It would be useful if indeed clients didn't have to write to the top-level pool, since that would mean we could give different clients permission only to pool-associated subdirectories without giving everyone write access to a pool with data structures shared between all users of the filesystem. [root@hdr-admon01 ec]# ceph df detail; ceph fs ls; ceph fs status RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.3 PiB 3.3 PiB 32 TiB 32 TiB 0.95 nvme 2.9 TiB 2.9 TiB 504 MiB 2.5 GiB 0.08 TOTAL 3.3 PiB 3.3 PiB 32 TiB 32 TiB 0.95 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs.fs1.metadata 5 162 MiB 63 324 MiB 0.01 1.4 TiB N/A N/A 63 0 B 0 B cephfs.fs1-replicated.data 6 0 B 0 0 B 0 1.0 PiB N/A N/A 0 0 B 0 B cephfs.fs1-ec.data 7 8.0 GiB 2.05k 11 GiB 0 2.4 PiB N/A N/A 2.05k 0 B 0 B name: fs1, metadata pool: cephfs.fs1.metadata, data pools: [cephfs.fs1-replicated.data cephfs.fs1-ec.data ] fs1 - 4 clients === +------+--------+------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+------------+---------------+-------+-------+ | 0 | active | hdr-meta02 | Reqs: 0 /s | 29 | 16 | +------+--------+------------+---------------+-------+-------+ +----------------------------+----------+-------+-------+ | Pool | type | used | avail | +----------------------------+----------+-------+-------+ | cephfs.fs1.metadata | metadata | 324M | 1414G | | cephfs.fs1-replicated.data | data | 0 | 1063T | | cephfs.fs1-ec.data | data | 11.4G | 2505T | +----------------------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ | hdr-meta01 | +-------------+ MDS version: ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) [root@hdr-admon01 ec]# ll /test-fs/ec/ total 12582912 -rw-r--r--. 1 root root 4294967296 Jan 27 22:26 new-file -rw-r--r--. 2 root root 4294967296 Jan 28 14:06 new-file2 -rw-r--r--. 2 root root 4294967296 Jan 28 14:06 new-file-same-inode-as-newfile2 Regards, Phil _________________________________________ Philip Cass HPC Systems Specialist - Senior Systems Administrator EPCC [cid:image002.png@01D5D5EF.2E463230] Advanced Computing Facility Bush Estate Penicuik Tel: +44 (0)131 4457815 Email: p.cass(a)epcc.ed.ac.uk<mailto:p.cass@epcc.ed.ac.uk> _________________________________________ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. The information contained in this e-mail (including any attachments) is confidential and is intended for the use of the addressee only. If you have received this message in error, please delete it and notify the originator immediately. Please consider the environment before printing this email.

4 6

Changing failure domain
by Francois Legrand 01 Feb '20

01 Feb '20

Hi, I have a cephfs in production based on 2 pools (data+metadata). Data is in erasure coding with the profile : crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 Metadata is in replicated mode with k=3 The crush rules are as follow : [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "ec_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 5, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] When we installed it, everything was in the same room, but know we splitted our cluster (6 servers but soon 8) in 2 rooms. Thus we updated the crushmap by adding a room layer (with ceph osd crush add-bucket room1 room etc) and move all our servers in the tree to the correct place (ceph osd crush move server1 room=room1 etc...). Now, we would like to change the rules to set a failure domain to room instead of host (to be sure that in case of disaster in one of the rooms we will still have a copy in the other). What is the best strategy to do this ? F.

4 10

General question CephFS or RBD
by Willi Schiegel 01 Feb '20

01 Feb '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

3 2

Getting rid of trim_object Snap .... not in clones
by Andreas John 01 Feb '20

01 Feb '20

Hello, in my cluster one after the other OSD dies until I recognized that it was simply an "abort" in the daemon caused probably by 2020-01-31 15:54:42.535930 7faf8f716700 -1 log_channel(cluster) log [ERR] : trim_object Snap 29c44 not in clones Close to this msg I get a stracktrace: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af) 1: /usr/bin/ceph-osd() [0xb35f7d] 2: (()+0x11390) [0x7f0fec74b390] 3: (gsignal()+0x38) [0x7f0feab43428] 4: (abort()+0x16a) [0x7f0feab4502a] 5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f0feb48684d] 6: (()+0x8d6b6) [0x7f0feb4846b6] 7: (()+0x8d701) [0x7f0feb484701] 8: (()+0x8d919) [0x7f0feb484919] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27e) [0xc3776e] 10: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x10dd) [0x868cfd] 11: (ReplicatedPG::repop_all_committed(ReplicatedPG::RepGather*)+0x80) [0x8690e0] 12: (Context::complete(int)+0x9) [0x6c8799] 13: (void ReplicatedBackend::sub_op_modify_reply<MOSDRepOpReply, 113>(std::tr1::shared_ptr<OpRequest>)+0x21b) [0xa5ae0b] 14: (ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x15b) [0xa53edb] 15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x1cb) [0x84c78b] 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3ef) [0x6966ff] 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x4e4) [0x696e14] 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x71e) [0xc264fe] 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xc29950] 20: (()+0x76ba) [0x7f0fec7416ba] 21: (clone()+0x6d) [0x7f0feac1541d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Yes, I know it's still hammer, I want to upgrade soon, but I want to resolve that issue first. If I lose that PG, I don't worry. So: What it the best approach? Can I use something like ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> ? I assume 29c44 is my Object, but what's the clone od? Best regards, derjohn

1 3

v14.2.7 Nautilus released
by David Galloway 31 Jan '20

31 Jan '20

This is the seventh update to the Ceph Nautilus release series. This is a hotfix release primarily fixing a couple of security issues. We recommend that all users upgrade to this release. Notable Changes --------------- * CVE-2020-1699: Fixed a path traversal flaw in Ceph dashboard that could allow for potential information disclosure (Ernesto Puerta) * CVE-2020-1700: Fixed a flaw in RGW beast frontend that could lead to denial of service from an unauthenticated client (Or Friedmann) -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

1 0

Inactive pgs preventing osd from starting
by Ragan, Tj (Dr.) 31 Jan '20

31 Jan '20

Hi All, Long story-short, we’re doing disaster recovery on a cephfs cluster, and are at a point where we have 8 pgs stuck incomplete. Just before the disaster, I increased the pg_count on two of the pools, and they had not completed increasing the pgp_num yet. I’ve since forced pgp_num to the current values. So far, I’ve tried mark_unfound_lost but they don’t report any unfound objects, and I’ve tried force-create-pg but that has no effect, except on one of the pgs, which went to creating+incomplete. During the disaster recovery, I had to re-create several OSDs (due to unreadable superblocks,) and now one of the new osds, as well as one of the existing osds won’t start. The log from the startup of osd.29 is here: https://pastebin.com/PX9AAj8m, which seems to indicate that it won’t start because it’s supposed to have copies of the incomplete placement groups. ceph pg 5.38 query (one of the incomplete) gives: https://pastebin.com/Jf4GnZTc I have hunted around in the osds listed for all the placement groups for any sign of a pg that I could mark as complete with ceph-objectstore-tool, but can’t find any. I don’t care about the data in the pgs, but I can’t abandon the filesystem. Any help would be greatly appreciated. -TJ Ragan

2 2

Upgrading mimic 13.2.2 to mimic 13.2.8
by Frank Schilder 31 Jan '20

31 Jan '20

Dear all, is it possible to upgrade from 13.2.2 directly to 13.2.8 after setting "ceph osd set pglog_hardlimit" (mimic 13.2.5 release notes), or do I need to follow this path: 13.2. 2 -> 5 -> 6 -> 8 ? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

Micron SSD/Basic Config
by Adam Boyhan 31 Jan '20

31 Jan '20

Looking to role out a all flash Ceph cluster. Wanted to see if anyone else was using Micron drives along with some basic input on my design so far? Basic Config Ceph OSD Nodes 8x Supermicro A+ Server 2113S-WTRT - AMD EPYC 7601 32 Core 2.2Ghz - 256G Ram - AOC-S3008L-L8e HBA - 10GB SFP+ for client network - 40GB QSFP+ for ceph cluster network OSD 10x Micron 5300 PRO 7.68TB in each ceph node - 80 total drives across the 8 nodes WAL/DB 5x Micron 7300 MAX NVMe 800GB per Ceph Node - Plan on dedicating 1 for each 2 OSD's Still thinking out a external monitor node as I have a lot of options, but this is a pretty good start. Open to suggestions as well!

4 3

Re: Micron SSD/Basic Config
by vitalif＠yourcmc.ru 31 Jan '20

31 Jan '20

I think 800 GB NVMe per 2 SSDs is an overkill. 1 OSD usually only requires 30 GB block.db, so 400 GB per an OSD is a lot. On the other hand, does 7300 have twice the iops of 5300? In fact, I'm not sure if a 7300 + 5300 OSD will perform better than just a 5300 OSD at all. It would be interesting if you could benchmark & compare it though :) > Hmm change 40Gbps to 100Gbps networking. > > 40Gbps technology its just a bond of 4x10 Links with some latency due > link aggregation. > 100 Gbps and 25Gbps got less latency and Good performance. In ceph a > 50% of the latency comes from Network commits and the other 50% from > disk commits. > > A fast graph : > https://blog.mellanox.com/wp-content/uploads/John-Kim-030416-Fig-3a-1024x74… > Article: > https://blog.mellanox.com/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-… > > Micron got their own Whitepaper for CEPH and looks like performs fine. > https://www.micron.com/-/media/client/global/documents/products/other-docum… > > > AS your Budget is high, please buy 3 x 1.5K $ nodes for your monitors > and you Will sleep better. They just need 4 cores / 16GB RAM and > 2x128GB SSD or NVME M2 . > > -----Mensaje original----- > De: Adam Boyhan <adamb(a)medent.com> > Enviado el: viernes, 31 de enero de 2020 13:59 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Micron SSD/Basic Config > > Looking to role out a all flash Ceph cluster. Wanted to see if anyone > else was using Micron drives along with some basic input on my design > so far? > > Basic Config > Ceph OSD Nodes > 8x Supermicro A+ Server 2113S-WTRT > - AMD EPYC 7601 32 Core 2.2Ghz > - 256G Ram > - AOC-S3008L-L8e HBA > - 10GB SFP+ for client network > - 40GB QSFP+ for ceph cluster network > > OSD > 10x Micron 5300 PRO 7.68TB in each ceph node > - 80 total drives across the 8 nodes > > WAL/DB > 5x Micron 7300 MAX NVMe 800GB per Ceph Node > - Plan on dedicating 1 for each 2 OSD's > > Still thinking out a external monitor node as I have a lot of options, > but this is a pretty good start. Open to suggestions as well! > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Re: Micron SSD/Basic Config
by Paul Emmerich 31 Jan '20

31 Jan '20

On Fri, Jan 31, 2020 at 2:06 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Hmm change 40Gbps to 100Gbps networking. > > 40Gbps technology its just a bond of 4x10 Links with some latency due link aggregation. > 100 Gbps and 25Gbps got less latency and Good performance. In ceph a 50% of the latency comes from Network commits and the other 50% from disk commits. 40G ethernet is not the same as 4x 10G bond. A bond load balances on a per-packet (or well, per flow usually) basis. A 40G link uses all four links even for a single packet. 100G is "just" 4x 25G I also wouldn't agree that network and disk latency is a 50/50 split in Ceph unless you have some NVRAM disks or something. Even for the network speed the processing and queuing in the network stack dominates over the serialization delay from a 40G/100G difference (4kb at 100G is 320ns, and 800ns at 40G for the serialization; I don't have any figures for processing times on 40/100G ethernet, but 10G fiber is at 300ns, 10G base-t at 2300 nanoseconds) Paul > > A fast graph : https://blog.mellanox.com/wp-content/uploads/John-Kim-030416-Fig-3a-1024x74… > Article: https://blog.mellanox.com/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-… > > Micron got their own Whitepaper for CEPH and looks like performs fine. > https://www.micron.com/-/media/client/global/documents/products/other-docum… > > > AS your Budget is high, please buy 3 x 1.5K $ nodes for your monitors and you Will sleep better. They just need 4 cores / 16GB RAM and 2x128GB SSD or NVME M2 . > > -----Mensaje original----- > De: Adam Boyhan <adamb(a)medent.com> > Enviado el: viernes, 31 de enero de 2020 13:59 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Micron SSD/Basic Config > > Looking to role out a all flash Ceph cluster. Wanted to see if anyone else was using Micron drives along with some basic input on my design so far? > > Basic Config > Ceph OSD Nodes > 8x Supermicro A+ Server 2113S-WTRT > - AMD EPYC 7601 32 Core 2.2Ghz > - 256G Ram > - AOC-S3008L-L8e HBA > - 10GB SFP+ for client network > - 40GB QSFP+ for ceph cluster network > > OSD > 10x Micron 5300 PRO 7.68TB in each ceph node > - 80 total drives across the 8 nodes > > WAL/DB > 5x Micron 7300 MAX NVMe 800GB per Ceph Node > - Plan on dedicating 1 for each 2 OSD's > > Still thinking out a external monitor node as I have a lot of options, but this is a pretty good start. Open to suggestions as well! > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Micron SSD/Basic Config
by Adam Boyhan 31 Jan '20

31 Jan '20

Appreciate the input. Looking at those articles they make me feel like the 40G they are talking about is 4x Bonded 10G connections. Im looking at 40Gbps without bonding for throughput. Is that still the same? [ https://www.fs.com/products/29126.html | https://www.fs.com/products/29126.html ] Yep most of this is based on the white paper with a few changes here and there. From: "EDH - Manuel Rios" <mriosfer(a)easydatahost.com> To: "adamb" <adamb(a)medent.com>, "ceph-users" <ceph-users(a)ceph.io> Sent: Friday, January 31, 2020 8:05:52 AM Subject: RE: Micron SSD/Basic Config Hmm change 40Gbps to 100Gbps networking. 40Gbps technology its just a bond of 4x10 Links with some latency due link aggregation. 100 Gbps and 25Gbps got less latency and Good performance. In ceph a 50% of the latency comes from Network commits and the other 50% from disk commits. A fast graph : https://blog.mellanox.com/wp-content/uploads/John-Kim-030416-Fig-3a-1024x74… Article: https://blog.mellanox.com/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-… Micron got their own Whitepaper for CEPH and looks like performs fine. https://www.micron.com/-/media/client/global/documents/products/other-docum… AS your Budget is high, please buy 3 x 1.5K $ nodes for your monitors and you Will sleep better. They just need 4 cores / 16GB RAM and 2x128GB SSD or NVME M2 .

2 1

Network performance checks
by Massimo Sgaravatto 31 Jan '20

31 Jan '20

After having upgraded my ceph cluster from Luminous to Nautilus 14.2.6 , from time to time "ceph health detail" claims about some"Long heartbeat ping times on front/back interface seen". As far as I can understand (after having read https://docs.ceph.com/docs/nautilus/rados/operations/monitoring/) this means that the ping from one OSD to another one exceeded 1 s. I have some questions on these network performance checks 1) What is meant exactly with front and back interface ? 2) I can see the involved OSDs only in the output of "ceph health detail" (when there is the problem) but I can't find this information in the log files. In the mon log file I can only see messages such as: 2020-01-28 11:14:07.641 7f618e644700 0 log_channel(cluster) log [WRN] : Health check failed: Long heartbeat ping times on back interface seen, longest is 1416.618 msec (OSD_SLOW_PING_TIME_BACK) but the involved OSDs are not reported in this log. Do I just need to increase the verbosity of the mon log ? 3) Is 1 s a reasonable value for this threshold ? How could this value be changed ? What is the relevant configuration variable ? 4) https://docs.ceph.com/docs/nautilus/rados/operations/monitoring/ suggests to use the dump_osd_network command. I think there is an error in that page: it says that the command should be issued on ceph-mgr.x.asok, while I think that instead the ceph-osd-x.asok should be used I have an other ceph cluster (running nautilus 14.2.6 as well) where there aren't OSD_SLOW_PING_* error messages in the mon logs, but: ceph daemon /var/run/ceph/ceph-osd..asok dump_osd_network 1 reports a lot of entries (i.e. pings exceeded 1 s). How can this be explained ? Thanks, Massimo

2 4

Re: moving small production cluster to different datacenter
by Wido den Hollander 31 Jan '20

31 Jan '20

On 1/28/20 6:58 PM, Anthony D'Atri wrote: > > >> I did this ones. This cluster was running IPv6-only (still is) and thus >> I had the flexibility of new IPs. > > Dumb question — how was IPv6 a factor in that flexibility? Was it just that you had unused addresses within an existing block? > There are no dumb questions :-) Usually Ceph is put into RFC1918 IPv4 space (10.x, 172.X) and those are usually more difficult to route in networks. IPv6 address space is globally routed in most networks thus making this easier. As long as the hosts can talk IP(4/6) with each other you can perform such a migration. Wido

3 3

recovering monitor failure
by vishal＠denovogroup.org 31 Jan '20

31 Jan '20

Iam testing failure scenarios for my cluster. I have 3 monitors. Lets say if mons 1 and 2 go down and so monitors can't form a quorum, how can I recover? Are the instructions at followling link valid for deleting mons 1 and 2 from monmap, https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/ht… One more question - lets say I delete mons 1 and 2 from monmap. And the cluster has only mon 3 remaining so mon 3 has quorum. Now what happens if mon 1 and 2 come up? Do they join mon 3 and so there will again be 3 monitors in the cluster? Thanks

3 4

ceph-iscsi create RBDs on erasure coded data pools
by Wesley Dillingham 30 Jan '20

30 Jan '20

Is it possible to create an EC backed RBD via ceph-iscsi tools (gwcli, rbd-target-api)? It appears that a pre-existing RBD created with the rbd command can be imported, but there is no means to directly create an EC backed RBD. The API seems to expect a single pool field in the body to work with. Perhaps there is a lower level construct where you can set the metadata of a particular RADOS pool to always use Pool X for data-pool when using Pool Y for RBD header and metadata. This way the clients, in our case ceph-iscsi needn't be modified or concerned with the dual-pool situation unless explicitly specified. For out particular use case we expose limited functionality of rbd-target-api to clients and it would be helpful for them to keep track of a single pool and not be concerned with two pools but if a data-pool and a "main" pool could be passed via the API that would be okay too. Thanks a lot. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

1 0

moving small production cluster to different datacenter
by Marc Roos 30 Jan '20

30 Jan '20

Say one is forced to move a production cluster (4 nodes) to a different datacenter. What options do I have, other than just turning it off at the old location and on on the new location? Maybe buying some extra nodes, and move one node at a time?

5 5

Can Ceph Do The Job?
by Adam Boyhan 30 Jan '20

30 Jan '20

We are looking to role out a all flash Ceph cluster as storage for our cloud solution. The OSD's will be on slightly slower Micron 5300 PRO's, with WAL/DB on Micron 7300 MAX NVMe's. My main concern with Ceph being able to fit the bill is its snapshot abilities. For each RBD we would like the following snapshots 8x 30 minute snapshots (latest 4 hours) With our current solution (HPE Nimble) we simply pause all write IO on the 10 minute mark for roughly 2 seconds and then we take a snapshot of the entire Nimble volume. Each VM within the Nimble volume is sitting on a Linux Logical Volume so its easy for us to take one big snapshot and only get access to a specific clients data. Are there any options for automating managing/retention of snapshots within Ceph besides some bash scripts? Is there anyway to take snapshots of all RBD's within a pool at a given time? Is there anyone successfully running with this many snapshots? If anyone is running a similar setup, would love to hear how your doing it.

4 4

Servicing multiple OpenStack clusters from the same Ceph cluster
by Paul Browne 30 Jan '20

30 Jan '20

Hello, We have a medium-sized Ceph Luminous cluster that, up til now, has been the RBD image backend solely for an OpenStack Newton cluster that's marked for upgrade to Stein later this year. Recently we deployed a brand new Stein cluster however, and I'm curious whether the idea of pointing the new OpenStack cluster at the same RBD pools for Cinder/Glance/Nova as the Luminous cluster would be considered bad practice, or even potentially dangerous. One argument for doing it may be that multiple CInder/Glance/Nova pools serving disparate groups of clients would come at a PG cost to the cluster, though the separation of multiple, distinct pools also has its advantages. The UUIDs generated for RBD images in the pools by OpenStack services *should* be unique and collision-less between the 2 OpenStack clusters, in theory. One other point I was curious about was RBD image feature sets; Stein Ceph clients will be running later versions of Ceph libraries than Newton clients. If the 2 sets of clients were to share pools, would that itself cause problems (in the case that neither set needed to share RBD images within pools, only the pool itself) with some images in the pool having different feature lists? -- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk Tel: 0044-1223-746548 *******************

5 5

health_warn: slow_ops 4 slow ops
by Ignacio Ocampo 30 Jan '20

30 Jan '20

Hi Ceph Community (I'm new here :), I'm learning Ceph in a Virtual Environment Vagrant/Virtualbox (I understand this is far from a real environment in several ways, mainly performance, but I'm ok with that at this point :) I've 3 nodes, and after few *vagrant halt/up*, when I do *ceph -s*, I got the following message: [vagrant@ceph-node1 ~]$ sudo ceph -s cluster: id: 7f8cb5f0-1989-4ab1-8fb9-d5c08aa96658 health: *HEALTH_WARN* Reduced data availability: 512 pgs inactive 4 slow ops, oldest one blocked for 1576 sec, daemons [osd.6,osd.7,osd.8] have slow ops. services: mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 7m) mgr: ceph-node1(active, since 26m), standbys: ceph-node2, ceph-node3 osd: 9 osds: 9 up (since 25m), 9 in (since 2d) data: pools: 1 pools, 512 pgs objects: 0 objects, 0 B usage: 9.1 GiB used, 162 GiB / 171 GiB avail pgs: 100.000% pgs unknown 512 unknown Here the output of *ceph health detail*: [vagrant@ceph-node1 ~]$ sudo ceph health detail HEALTH_WARN Reduced data availability: 512 pgs inactive; 4 slow ops, oldest one blocked for 1810 sec, daemons [osd.6,osd.7,osd.8] have slow ops. PG_AVAILABILITY Reduced data availability: 512 pgs inactive pg 2.1cd is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ce is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1cf is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d0 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d1 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d2 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d3 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d4 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d5 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d6 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d7 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d8 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1d9 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1da is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1db is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1dc is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1dd is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1de is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1df is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e0 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e1 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e2 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e3 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e4 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e5 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e6 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e7 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e8 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1e9 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ea is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1eb is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ec is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ed is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ee is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ef is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f0 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f1 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f2 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f3 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f4 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f5 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f6 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f7 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f8 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1f9 is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1fa is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1fb is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1fc is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1fd is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1fe is stuck inactive for 1815.881027, current state unknown, last acting [] pg 2.1ff is stuck inactive for 1815.881027, current state unknown, last acting [] SLOW_OPS 4 slow ops, oldest one blocked for 1810 sec, daemons [osd.6,osd.7,osd.8] have slow ops. Do you have any guidance on how to proceed with this? I'm trying to understand why the cluster is HEALTH_WARN and what I need to do in order to make it health again. Thanks! -- Ignacio Ocampo

2 1

Nautilus 14.2.6 ceph-volume bluestore _read_fsid unparsable uuid
by Dave Hall 30 Jan '20

30 Jan '20

All, I've just spent a significant amount of time unsuccessfully chasing the _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6. Since this is a brand new cluster, last night I gave up and moved back to Debian 9 / Luminous 12.2.11. In both cases I'm using the packages from Debian Backports with ceph-ansible as my deployment tool. Note that above I said 'the _read_fsid unparsable uuid' error. I've searched around a bit and found some previously reported issues, but I did not see any conclusive resolutions. I would like to get to Nautilus as quickly as possible, so I'd gladly provide additional information to help track down the cause of this symptom. I can confirm that, looking at the ceph-volume.log on the OSD host I see no difference between the ceph-volume lvm batch command generated by the ceph-ansible versions associated with these two Ceph releases: ceph-volume --cluster ceph lvm batch --bluestore --yes --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1 Note that I'm using --block-db-size to divide my NVMe into 12 segments as I have 4 empty drive bays on my OSD servers that I may eventually be able to fill. My OSD hardware is: Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors I'd send the output of ceph-volume inventory on Luminous, but I'm getting -->: KeyError: 'human_readable_size'. Please let me know if I can provide any further information. Thanks. -Dave -- Dave Hall Binghamton University

3 7

Re: Servicing multiple OpenStack clusters from the same Ceph cluster
by Anastasios Dados 29 Jan '20

29 Jan '20

Yes but we are offering our rbd volumes in another cloud product which can enable them migrate their volumes to openstack when they want. Sent from my iPhone On 29 Jan 2020, at 18:38, Matthew H <matthew.heler(a)hotmail.com> wrote: You should have used separate pool name scemes for each OpenStack cluster.. ________________________________ From: tdados(a)hotmail.com <tdados(a)hotmail.com> Sent: Wednesday, January 29, 2020 12:29 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Servicing multiple OpenStack clusters from the same Ceph cluster Hello, We have recently deployed that and it's working fine. We have deployed different keys for the different openstack clusters ofcourse and they are using the same cinder/nova/glance pools. The only risk is if a client from one openstack cluster creates a volume and the id that will be generated ends up being the same on an existing volume from the other openstack cluster. But that's like possibility of 1 in 5 billion or something. We took the risk. Regards _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

cephfs : write error: Operation not permitted
by Yoann Moulin 29 Jan '20

29 Jan '20

Hello, On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool. > $ ceph -s > cluster: > id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 > health: HEALTH_OK > > services: > mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) > mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 > mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} > osd: 24 osds: 24 up (since 22h), 24 in (since 22h) > rgw: 1 daemon active (iccluster043.rgw0) > > data: > pools: 9 pools, 568 pgs > objects: 800 objects, 225 KiB > usage: 24 GiB used, 87 TiB / 87 TiB avail > pgs: 568 active+clean The 2 cephfs pools: > $ ceph osd pool ls detail | grep cephfs > pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs > pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs The status of the cephfs filesystem: > $ ceph fs status > cephfs - 1 clients > ====== > +------+--------+--------------+---------------+-------+-------+ > | Rank | State | MDS | Activity | dns | inos | > +------+--------+--------------+---------------+-------+-------+ > | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | > | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | > | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | > +------+--------+--------------+---------------+-------+-------+ > +-----------------+----------+-------+-------+ > | Pool | type | used | avail | > +-----------------+----------+-------+-------+ > | cephfs_metadata | metadata | 4608k | 27.6T | > | cephfs_data | data | 0 | 27.6T | > +-----------------+----------+-------+-------+ > +-------------+ > | Standby MDS | > +-------------+ > +-------------+ > MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) > # mkdir folder > # echo "foo" > bar > -bash: echo: write error: Operation not permitted > # ls -al > total 4 > drwxrwxrwx 1 root root 2 Jan 22 07:30 . > drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. > -rw-r--r-- 1 root root 0 Jan 22 07:30 bar > drwxrwxrwx 1 root root 1 Jan 21 16:49 folder > # df -hT . > Filesystem Type Size Used Avail Use% Mounted on > 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs I try 2 client config : > $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw > [snip] > $ ceph auth get client.fsadmin > exported keyring for client.fsadmin > [client.fsadmin] > key = [snip] > caps mds = "allow rw" > caps mon = "allow r" > caps osd = "allow rw tag cephfs data=cephfs" > $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw > [snip] > $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data " > [snip] > ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin > $ ceph auth get client.cephfsadmin > exported keyring for client.cephfsadmin > [client.cephfsadmin] > key = [snip] > caps mds = "allow rw" > caps mon = "allow r" > caps osd = "allow rw tag cephfs pool=cephfs_data " I don't where to look to get more information about that issue. Anyone can help me? Thanks Best regards, -- Yoann Moulin EPFL IC-IT

5 15

Re: Luminous Bluestore OSDs crashing with ASSERT
by Stefan Priebe - Profihost AG 29 Jan '20

29 Jan '20

Hello Igor, i updated all servers to latest 4.19.97 kernel but this doesn't fix the situation. I can provide you with all those logs - any idea where to upload / how to sent them to you? Greets, Stefan Am 20.01.20 um 13:12 schrieb Igor Fedotov: > Hi Stefan, > > these lines are result of transaction dump performed on a failure during > transaction submission (which is shown as > > "submit_transaction error: Corruption: block checksum mismatch code = 2" > > Most probably they are out of interest (checksum errors are unlikely to > be caused by transaction content) and hence we need earlier stuff to > learn what caused that > > checksum mismatch. > > It's hard to give any formal overview of what you should look for, from > my troubleshooting experience generally one may try to find: > > - some previous error/warning indications (e.g. allocation, disk access, > etc) > > - prior OSD crashes (sometimes they might have different causes/stack > traces/assertion messages) > > - any timeout or retry indications > > - any uncommon log patterns which aren't present during regular running > but happen each time before the crash/failure. > > Anyway I think the inspection depth should be much(?) deeper than > presumably it is (from what I can see from your log snippets). > > Ceph keeps last 10000 log events with an increased log level and dumps > them on crash with negative index starting at -9999 up to -1 as a prefix. > > -1> 2020-01-16 01:10:13.404090 7f3350a14700 -1 rocksdb: > > > It would be great If you share several log snippets for different > crashes containing these last 10000 lines. > > > Thanks, > > Igor > > > On 1/19/2020 9:42 PM, Stefan Priebe - Profihost AG wrote: >> Hello Igor, >> >> there's absolutely nothing in the logs before. >> >> What do those lines mean: >> Put( Prefix = O key = >> 0x7f8000000000000001cc45c881217262'd_data.4303206b8b4567.0000000000009632!='0xfffffffffffffffeffffffffffffffff6f00120000'x' >> >> Value size = 480) >> Put( Prefix = O key = >> 0x7f8000000000000001cc45c881217262'd_data.4303206b8b4567.0000000000009632!='0xfffffffffffffffeffffffffffffffff'o' >> >> Value size = 510) >> >> on the right size i always see 0xfffffffffffffffeffffffffffffffff on all >> failed OSDs. >> >> greets, >> Stefan >> Am 19.01.20 um 14:07 schrieb Stefan Priebe - Profihost AG: >>> Yes, except that this happens on 8 different clusters with different >>> hw but same ceph version and same kernel version. >>> >>> Greets, >>> Stefan >>> >>>> Am 19.01.2020 um 11:53 schrieb Igor Fedotov <ifedotov(a)suse.de>: >>>> >>>> So the intermediate summary is: >>>> >>>> Any OSD in the cluster can experience interim RocksDB checksum >>>> failure. Which isn't present after OSD restart. >>>> >>>> No HW issues observed, no persistent artifacts (except OSD log) >>>> afterwards. >>>> >>>> And looks like the issue is rather specific to the cluster as no >>>> similar reports from other users seem to be present. >>>> >>>> >>>> Sorry, I'm out of ideas other then collect all the failure logs and >>>> try to find something common in them. May be this will shed some >>>> light.. >>>> >>>> BTW from my experience it might make sense to inspect OSD log prior >>>> to failure (any error messages and/or prior restarts, etc) sometimes >>>> this might provide some hints. >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> >>>>> On 1/17/2020 2:30 PM, Stefan Priebe - Profihost AG wrote: >>>>> HI Igor, >>>>> >>>>>> Am 17.01.20 um 12:10 schrieb Igor Fedotov: >>>>>> hmmm.. >>>>>> >>>>>> Just in case - suggest to check H/W errors with dmesg. >>>>> this happens on around 80 nodes - i don't expect all of those have not >>>>> identified hw errors. Also all of them are monitored - no dmesg >>>>> outpout >>>>> contains any errors. >>>>> >>>>>> Also there are some (not very much though) chances this is another >>>>>> incarnation of the following bug: >>>>>> https://tracker.ceph.com/issues/22464 >>>>>> https://github.com/ceph/ceph/pull/24649 >>>>>> >>>>>> The corresponding PR works around it for main device reads (user data >>>>>> only!) but theoretically it might still happen >>>>>> >>>>>> either for DB device or DB data at main device. >>>>>> >>>>>> Can you observe any bluefs spillovers? Are there any correlation >>>>>> between >>>>>> failing OSDs and spillover presence if any, e.g. failing OSDs always >>>>>> have a spillover. While OSDs without spillovers never face the >>>>>> issue... >>>>>> >>>>>> To validate this hypothesis one can try to monitor/check (e.g. once a >>>>>> day for a week or something) "bluestore_reads_with_retries" >>>>>> counter over >>>>>> OSDs to learn if the issue is happening >>>>>> >>>>>> in the system. Non-zero values mean it's there for user data/main >>>>>> device and hence is likely to happen for DB ones as well (which >>>>>> doesn't >>>>>> have any workaround yet). >>>>> OK i checked bluestore_reads_with_retries on 360 osds but all of >>>>> them say 0. >>>>> >>>>> >>>>>> Additionally you might want to monitor memory usage as the above >>>>>> mentioned PR denotes high memory pressure as potential trigger for >>>>>> these >>>>>> read errors. So if such pressure happens the hypothesis becomes >>>>>> more valid. >>>>> we already do this heavily and have around 10GB of memory per OSD. >>>>> Also >>>>> no of those machines show any io pressure at all. >>>>> >>>>> All hosts show a constant rate of around 38GB to 45GB mem available in >>>>> /proc/meminfo. >>>>> >>>>> Stefan >>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> PS. Everything above is rather a speculation for now. Available >>>>>> information is definitely not enough for extensive troubleshooting >>>>>> the >>>>>> cases which happens that rarely. >>>>>> >>>>>> You might want to start collecting failure-related information >>>>>> (including but not limited to failure logs, perf counter dumps, >>>>>> system >>>>>> resource reports etc) for future analysis. >>>>>> >>>>>> >>>>>> >>>>>> On 1/16/2020 11:58 PM, Stefan Priebe - Profihost AG wrote: >>>>>>> Hi Igor, >>>>>>> >>>>>>> answers inline. >>>>>>> >>>>>>> Am 16.01.20 um 21:34 schrieb Igor Fedotov: >>>>>>>> you may want to run fsck against failing OSDs. Hopefully it will >>>>>>>> shed >>>>>>>> some light. >>>>>>> fsck just says everything fine: >>>>>>> >>>>>>> # ceph-bluestore-tool --command fsck --path >>>>>>> /var/lib/ceph/osd/ceph-27/ >>>>>>> fsck success >>>>>>> >>>>>>> >>>>>>>> Also wondering if OSD is able to recover (startup and proceed >>>>>>>> working) >>>>>>>> after facing the issue? >>>>>>> no recover needed. It just runs forever after restarting. >>>>>>> >>>>>>>> If so do you have any one which failed multiple times? Do you >>>>>>>> have logs >>>>>>>> for these occurrences? >>>>>>> may be but there are most probably weeks or month between those >>>>>>> failures >>>>>>> - most probably logs are already deleted. >>>>>>> >>>>>>>> Also please note that patch you mentioned doesn't fix previous >>>>>>>> issues >>>>>>>> (i.e. duplicate allocations), it prevents from new ones only. >>>>>>>> >>>>>>>> But fsck should show them if any... >>>>>>> None showed. >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Igor >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 1/16/2020 10:04 PM, Stefan Priebe - Profihost AG wrote: >>>>>>>>> Hi Igor, >>>>>>>>> >>>>>>>>> ouch sorry. Here we go: >>>>>>>>> >>>>>>>>> -1> 2020-01-16 01:10:13.404090 7f3350a14700 -1 rocksdb: >>>>>>>>> submit_transaction error: Corruption: block checksum mismatch >>>>>>>>> code = 2 >>>>>>>>> Rocksdb transaction: >>>>>>>>> Put( Prefix = M key = >>>>>>>>> 0x0000000000000402'.OBJ_0000000000000002.953BFD0A.bb85c.rbd%udata%e3e8eac6b8b4567%e0000000000001f2e..' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 97) >>>>>>>>> Put( Prefix = M key = >>>>>>>>> 0x0000000000000402'.MAP_00000000000BB85C_0000000000000002.953BFD0A.bb85c.rbd%udata%e3e8eac6b8b4567%e0000000000001f2e..' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 93) >>>>>>>>> Put( Prefix = M key = >>>>>>>>> 0x0000000000000916'.0000823257.00000000000073922044' Value size >>>>>>>>> = 196) >>>>>>>>> Put( Prefix = M key = >>>>>>>>> 0x0000000000000916'.0000823257.00000000000073922045' Value size >>>>>>>>> = 184) >>>>>>>>> Put( Prefix = M key = 0x0000000000000916'._info' Value size = 899) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00000000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 418) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00030000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 474) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f0007c000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 392) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00090000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 317) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f000a0000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 521) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f000f4000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 558) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00130000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 649) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00194000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 449) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f001cc000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 580) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00200000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 435) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00240000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 569) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00290000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 465) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f002e0000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 710) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00300000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 599) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f0036c000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 372) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f003a6000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 130) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f003b4000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 540) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f003fc000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 47) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff'o' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 1731) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0xfffffffffffffffeffffffffffffffff6f00040000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 675) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0xfffffffffffffffeffffffffffffffff6f00080000'x' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 395) >>>>>>>>> Put( Prefix = O key = >>>>>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0xfffffffffffffffeffffffffffffffff'o' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Value size = 1328) >>>>>>>>> Put( Prefix = X key = 0x0000000018a38deb Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x0000000018a38dea Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a035b Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a035c Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0355 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0356 Value size = 17) >>>>>>>>> Put( Prefix = X key = 0x000000001a54f6e4 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000001b1c061e Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a038f Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0389 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0358 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a035f Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0357 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0387 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a038a Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0388 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x00000000134c3fbe Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x00000000134c3fb5 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a036e Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a036d Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x00000000134c3fb8 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a0371 Value size = 14) >>>>>>>>> Put( Prefix = X key = 0x000000000d7a036a Value size = 14) >>>>>>>>> 0> 2020-01-16 01:10:13.413759 7f3350a14700 -1 >>>>>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void >>>>>>>>> BlueStore::_kv_sync_thread()' thread 7f3350a14700 time 2020-01-16 >>>>>>>>> 01:10:13.404113 >>>>>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED >>>>>>>>> assert(r == 0) >>>>>>>>> >>>>>>>>> ceph version 12.2.12-11-gd3eae83543 >>>>>>>>> (d3eae83543bffc0fc6c43823feb637fa851b6213) luminous (stable) >>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>> char >>>>>>>>> const*)+0x102) [0x55c9a712d232] >>>>>>>>> 2: (BlueStore::_kv_sync_thread()+0x24c5) [0x55c9a6fb54b5] >>>>>>>>> 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55c9a6ff608d] >>>>>>>>> 4: (()+0x7494) [0x7f33615f9494] >>>>>>>>> 5: (clone()+0x3f) [0x7f3360680acf] >>>>>>>>> >>>>>>>>> I already picked those: >>>>>>>>> https://github.com/ceph/ceph/pull/28644 >>>>>>>>> >>>>>>>>> Greets, >>>>>>>>> Stefan >>>>>>>>> Am 16.01.20 um 17:00 schrieb Igor Fedotov: >>>>>>>>>> Hi Stefan, >>>>>>>>>> >>>>>>>>>> would you please share log snippet prior the assertions? Looks >>>>>>>>>> like >>>>>>>>>> RocksDB is failing during transaction submission... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Igor >>>>>>>>>> >>>>>>>>>> On 1/16/2020 11:56 AM, Stefan Priebe - Profihost AG wrote: >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> does anybody know a fix for this ASSERT / crash? >>>>>>>>>>> >>>>>>>>>>> 2020-01-16 02:02:31.316394 7f8c3f5ab700 -1 >>>>>>>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void >>>>>>>>>>> BlueStore::_kv_sync_thread()' thread 7f8c3f5ab700 time >>>>>>>>>>> 2020-01-16 >>>>>>>>>>> 02:02:31.304993 >>>>>>>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r >>>>>>>>>>> == 0) >>>>>>>>>>> >>>>>>>>>>> ceph version 12.2.12-11-gd3eae83543 >>>>>>>>>>> (d3eae83543bffc0fc6c43823feb637fa851b6213) luminous (stable) >>>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, >>>>>>>>>>> int, char >>>>>>>>>>> const*)+0x102) [0x55e6df9d9232] >>>>>>>>>>> 2: (BlueStore::_kv_sync_thread()+0x24c5) [0x55e6df8614b5] >>>>>>>>>>> 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55e6df8a208d] >>>>>>>>>>> 4: (()+0x7494) [0x7f8c50190494] >>>>>>>>>>> 5: (clone()+0x3f) [0x7f8c4f217acf] >>>>>>>>>>> >>>>>>>>>>> all bluestore OSDs are randomly crashing sometimes (once a >>>>>>>>>>> week). >>>>>>>>>>> >>>>>>>>>>> Greets, >>>>>>>>>>> Stefan >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list >>>>>>>>>>> ceph-users(a)lists.ceph.com >>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users(a)lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>

2 1

Concurrent append operations
by David Bell 29 Jan '20

29 Jan '20

Hello, I am currently evaluating Ceph for our needs and I have a question about the 'object append' feature. I note that the rados core API supports an 'append' operation, and the S3-compatible interface has too. My question is: does Ceph support concurrent append? I would like to use Ceph as a temporary store, a "buffer" if you will, for incoming data from a variety of sources. Each object would hold data for a particular identifier. I'd like to know if two or more different clients can 'append' to the same object, and the data doesn't overwrite each other, and each 'append' is added to the end of the object? Performance wise we'd likely be performing 15-20 thousand writes per second, so we'd be building a pretty big cluster on very fast flash disk. Data would only reside on the system for about an hour at most before being read and deleted. Cheers, David Bell

2 1

Ceph MDS specific perf info disappeared in Nautilus
by Stefan Kooman 29 Jan '20

29 Jan '20

Hi, The command "ceph daemon mds.$mds perf dump" does not give the collection with MDS specific data anymore. In Mimic I get the following MDS specific collections: - mds - mds_cache - mds_log - mds_mem - mds_server - mds_sessions But those are not available in Nautilus anymore (14.2.4). Also not listed in a "perf schema". Where did these metrics go? Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 3

No Activity?
by DHilsbos＠performair.com 28 Jan '20

28 Jan '20

All; I haven't had a single email come in from the ceph-users list at ceph.io since 01/22/2020. Is there just that little traffic right now? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 2

unable to obtain rotating service keys
by Raymond Clotfelter 28 Jan '20

28 Jan '20

I have a server with 12 OSDs on it. Five of them are unable to start, and give the following error message in the their logs: 2020-01-28 13:00:41.760 7f61fb490c80 0 monclient: wait_auth_rotating timed out after 30 2020-01-28 13:00:41.760 7f61fb490c80 -1 osd.178 411005 unable to obtain rotating service keys; retrying These OSDs were up and running when they initially just died on me. I tried to restart them and they failed to come up. I rebooted the node and they did not recover. All 5 died within a few hours and were all 5 down by time I started poking them. I previously had this happen with 2 other OSDs, one each on 2 servers each with 12 OSDs. I ended up just purging and recreating those OSDs. I would really like to find a solution to fix this problem that does not involve purging the OSDs. I have tried stopping and starting all monitors and managers, one at a time, and all at the same time. Additionally, all servers in the cluster have been restarted over the past couple of days for various other reasons. I am on Ceph 14.2.6, Debian buster and am using the Debian packages. All of my servers are kept in the time sync via ntp, and this has been verified multiple times that everything remains in time sync. I have googled the error message and tried all of the solutions offered from that, but nothing makes any difference. I would appreciate any constructive advice. Thanks. -- ray

1 0

librados behavior when some OSDs are unreachables
by David DELON 28 Jan '20

28 Jan '20

Hi, i had a problem with one application (seafile) which uses CEPH backend with librados. The corresponding pools are defined with size=3 and each object copy is on a different host. The cluster health is OK: all the monitors see all the hosts. Now, a network problem just happens between my RADOS client and a single host. Then, when my application/client tries to access an object which is situed on the unreachable host (primary for the corresponding PG), it does not failover to another copy/host (and my application crashes later because after a while, with many requests, too many files are opened on Linux). Is it the normal behavior? My storage is resilient (great!) but not its access... If on the host, i stop the OSDs or change the affinity to zero, it solves, so it seems like the librados just check and trust the osdmap And doing a tcpdump show the client tries to access the same OSD without timeout. It can be easily reproduced with defining a netfilter rule on a host to drop packets coming from the client. Note: i am still on Luminous (both on lient and cluster sides). Thanks for reading. D.

2 1

Question about erasure code
by Zorg 28 Jan '20

28 Jan '20

Hi, we are planning to use EC I have 3 questions about it 1 / what is the advantage of having more machines than (k + m)? We are planning to have 11 nodes and use k=8 and m=3. does it improved performance to have more node than K+M? of how many ? what ratio? 2 / what behavior should we expect if we lose 1,2 or 3 nodes: perfs / recovery. how many nodes could we loose before the cluster state change to read only 3 / what is the maximum occupancy rate on the EC? Thans for your help Zorg

2 1

Renaming LVM Groups of OSDs
by Stolte, Felix 28 Jan '20

28 Jan '20

Hi all, I would like to rename the logical volumes / volume groups used by my OSDs. Do I need to change anything else than the block and block.db links under /var/lib/ceph/osd/<osdid>? IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

3 2

Re: EC pool creation results in incorrect M value?
by Geoffrey Rhodes 28 Jan '20

28 Jan '20

Hi Eric, With regards to "From the output of “ceph osd pool ls detail” you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. Am I missing something?" For your EC profile you need to set the pool min_size=3 to still read/write to the pool with two host failures. RUN: sudo ceph osd pool set ec32pool min_size 3 Kind regards Geoffrey Rhodes On Mon, 27 Jan 2020 at 22:11, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: EC pool creation results in incorrect M value? (Paul Emmerich) > 2. Re: EC pool creation results in incorrect M value? (Smith, Eric) > 3. Re: EC pool creation results in incorrect M value? (Smith, Eric) > 4. data loss on full file system? (Håkan T Johansson) > > > ---------------------------------------------------------------------- > > Date: Mon, 27 Jan 2020 17:14:55 +0100 > From: Paul Emmerich <paul.emmerich(a)croit.io> > Subject: [ceph-users] Re: EC pool creation results in incorrect M > value? > To: "Smith, Eric" <Eric.Smith(a)ccur.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > < > CAD9yTbFb28FX_XaNXRUUxQ1UqAYa_ouhO_fkE+Vbts1VXGkuDw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > min_size in the crush rule and min_size in the pool are completely > different things that happen to share the same name. > > Ignore min_size in the crush rule, it has virtually no meaning in > almost all cases (like this one). > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to > create an EC3+2 pool with the following commands: > > > > Create the EC profile: > > > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > > > Verify profile creation: > > > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > > > crush-device-class= > > > > crush-failure-domain=host > > > > crush-root=sgshared > > > > jerasure-per-chunk-alignment=false > > > > k=3 > > > > m=2 > > > > plugin=jerasure > > > > technique=reed_sol_van > > > > w=8 > > > > Create a pool using this profile: > > > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > > > List pool detail: > > > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > stripe_width 12288 application ES > > > > Here’s the crush rule that’s created: > > { > > > > "rule_id": 11, > > > > "rule_name": "es32", > > > > "ruleset": 11, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 5, > > > > "steps": [ > > > > { > > > > "op": "set_chooseleaf_tries", > > > > "num": 5 > > > > }, > > > > { > > > > "op": "set_choose_tries", > > > > "num": 100 > > > > }, > > > > { > > > > "op": "take", > > > > "item": -2, > > > > "item_name": "sgshared" > > > > }, > > > > { > > > > "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host" > > > > }, > > > > { > > > > "op": "emit" > > > > } > > > > ] > > > > }, > > > > > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the > crush rule says min_size=3 however the pool does NOT survive 2 hosts > failing. > > > > > > > > Am I missing something? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 27 Jan 2020 16:22:06 +0000 > From: "Smith, Eric" <Eric.Smith(a)ccur.com> > Subject: [ceph-users] Re: EC pool creation results in incorrect M > value? > To: Paul Emmerich <paul.emmerich(a)croit.io> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <BN8PR14MB28206D6D29507FD9499DEE91EA0B0(a)BN8PR14MB2820.nam > prd14.prod.outlook.com> > Content-Type: text/plain; charset="utf-8" > > Thanks for the info regarding min_size in the crush rule - does this seem > like a bug to you then? Is anyone else able to reproduce this? > > -----Original Message----- > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Monday, January 27, 2020 11:15 AM > To: Smith, Eric <Eric.Smith(a)ccur.com> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] EC pool creation results in incorrect M value? > > min_size in the crush rule and min_size in the pool are completely > different things that happen to share the same name. > > Ignore min_size in the crush rule, it has virtually no meaning in almost > all cases (like this one). > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to > create an EC3+2 pool with the following commands: > > > > Create the EC profile: > > > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > > > Verify profile creation: > > > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > > > crush-device-class= > > > > crush-failure-domain=host > > > > crush-root=sgshared > > > > jerasure-per-chunk-alignment=false > > > > k=3 > > > > m=2 > > > > plugin=jerasure > > > > technique=reed_sol_van > > > > w=8 > > > > Create a pool using this profile: > > > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > > > List pool detail: > > > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > > stripe_width 12288 application ES > > > > Here’s the crush rule that’s created: > > { > > > > "rule_id": 11, > > > > "rule_name": "es32", > > > > "ruleset": 11, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 5, > > > > "steps": [ > > > > { > > > > "op": "set_chooseleaf_tries", > > > > "num": 5 > > > > }, > > > > { > > > > "op": "set_choose_tries", > > > > "num": 100 > > > > }, > > > > { > > > > "op": "take", > > > > "item": -2, > > > > "item_name": "sgshared" > > > > }, > > > > { > > > > "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host" > > > > }, > > > > { > > > > "op": "emit" > > > > } > > > > ] > > > > }, > > > > > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the > crush rule says min_size=3 however the pool does NOT survive 2 hosts > failing. > > > > > > > > Am I missing something? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 27 Jan 2020 16:45:52 +0000 > From: "Smith, Eric" <Eric.Smith(a)ccur.com> > Subject: [ceph-users] Re: EC pool creation results in incorrect M > value? > To: "Smith, Eric" <Eric.Smith(a)ccur.com>, Paul Emmerich > <paul.emmerich(a)croit.io> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <BN8PR14MB2820B1DF71DE395B7B5AA1ECEA0B0(a)BN8PR14MB2820.nam > prd14.prod.outlook.com> > Content-Type: text/plain; charset="utf-8" > > OK I see this: https://github.com/ceph/ceph/pull/8008 > > Perhaps it's just to be safe... > > -----Original Message----- > From: Smith, Eric <Eric.Smith(a)ccur.com> > Sent: Monday, January 27, 2020 11:22 AM > To: Paul Emmerich <paul.emmerich(a)croit.io> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: EC pool creation results in incorrect M value? > > Thanks for the info regarding min_size in the crush rule - does this seem > like a bug to you then? Is anyone else able to reproduce this? > > -----Original Message----- > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Monday, January 27, 2020 11:15 AM > To: Smith, Eric <Eric.Smith(a)ccur.com> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] EC pool creation results in incorrect M value? > > min_size in the crush rule and min_size in the pool are completely > different things that happen to share the same name. > > Ignore min_size in the crush rule, it has virtually no meaning in almost > all cases (like this one). > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to > create an EC3+2 pool with the following commands: > > > > Create the EC profile: > > > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > > > Verify profile creation: > > > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > > > crush-device-class= > > > > crush-failure-domain=host > > > > crush-root=sgshared > > > > jerasure-per-chunk-alignment=false > > > > k=3 > > > > m=2 > > > > plugin=jerasure > > > > technique=reed_sol_van > > > > w=8 > > > > Create a pool using this profile: > > > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > > > List pool detail: > > > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > > stripe_width 12288 application ES > > > > Here’s the crush rule that’s created: > > { > > > > "rule_id": 11, > > > > "rule_name": "es32", > > > > "ruleset": 11, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 5, > > > > "steps": [ > > > > { > > > > "op": "set_chooseleaf_tries", > > > > "num": 5 > > > > }, > > > > { > > > > "op": "set_choose_tries", > > > > "num": 100 > > > > }, > > > > { > > > > "op": "take", > > > > "item": -2, > > > > "item_name": "sgshared" > > > > }, > > > > { > > > > "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host" > > > > }, > > > > { > > > > "op": "emit" > > > > } > > > > ] > > > > }, > > > > > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the > crush rule says min_size=3 however the pool does NOT survive 2 hosts > failing. > > > > > > > > Am I missing something? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 27 Jan 2020 21:10:10 +0100 > From: Håkan T Johansson <f96hajo(a)chalmers.se> > Subject: [ceph-users] data loss on full file system? > To: <ceph-users(a)ceph.io> > Message-ID: > <alpine.DEB.2.20.2001272106210.7767(a)planck-o.fy.chalmers.se> > Content-Type: text/plain; format=flowed; charset="UTF-8" > > > Hi, > > for test purposes, I have set up two 100 GB OSDs, one > taking a data pool and the other metadata pool for cephfs. > > Am running 14.2.6-1-gffd69200ad-1 with packages from > https://mirror.croit.io/debian-nautilus > > Am then running a program that creates a lot of 1 MiB files by calling > fopen() > fwrite() > fclose() > for each of them. Error codes are checked. > > This works successfully for ~100 GB of data, and then strangely also > succeeds > for many more 100 GB of data... ?? > > All written files have size 1 MiB with 'ls', and thus should contain the > data > written. However, on inspection, the files written after the first ~100 > GiB, > are full of just 0s. (hexdump -C) > > > To further test this, I use the standard tool 'cp' to copy a few > random-content > files into the full cephfs filessystem. cp reports no complaints, and > after > the copy operations, content is seen with hexdump -C. However, after > forcing > the data out of cache on the client by reading other earlier created > files, > hexdump -C show all-0 content for the files copied with 'cp'. Data that > was > there is suddenly gone...? > > > I am new to ceph. Is there an option I have missed to avoid this > behaviour? > (I could not find one in > https://docs.ceph.com/docs/master/man/8/mount.ceph/ ) > > Is this behaviour related to > https://docs.ceph.com/docs/mimic/cephfs/full/ > ? > > (That page states 'sometime after a write call has already returned 0'. > But if > write returns 0, then no data has been written, so the user program would > not > assume any kind of success.) > > Best regards, > > Håkan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 84, Issue 44 > ****************************************** >

1 0

Re: EC pool creation results in incorrect M value?
by Smith, Eric 27 Jan '20

27 Jan '20

OK I see this: https://github.com/ceph/ceph/pull/8008 Perhaps it's just to be safe... -----Original Message----- From: Smith, Eric <Eric.Smith(a)ccur.com> Sent: Monday, January 27, 2020 11:22 AM To: Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: EC pool creation results in incorrect M value? Thanks for the info regarding min_size in the crush rule - does this seem like a bug to you then? Is anyone else able to reproduce this? -----Original Message----- From: Paul Emmerich <paul.emmerich(a)croit.io> Sent: Monday, January 27, 2020 11:15 AM To: Smith, Eric <Eric.Smith(a)ccur.com> Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] EC pool creation results in incorrect M value? min_size in the crush rule and min_size in the pool are completely different things that happen to share the same name. Ignore min_size in the crush rule, it has virtually no meaning in almost all cases (like this one). Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to create an EC3+2 pool with the following commands: > > Create the EC profile: > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > Verify profile creation: > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > crush-device-class= > > crush-failure-domain=host > > crush-root=sgshared > > jerasure-per-chunk-alignment=false > > k=3 > > m=2 > > plugin=jerasure > > technique=reed_sol_van > > w=8 > > Create a pool using this profile: > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > List pool detail: > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > stripe_width 12288 application ES > > Here’s the crush rule that’s created: > { > > "rule_id": 11, > > "rule_name": "es32", > > "ruleset": 11, > > "type": 3, > > "min_size": 3, > > "max_size": 5, > > "steps": [ > > { > > "op": "set_chooseleaf_tries", > > "num": 5 > > }, > > { > > "op": "set_choose_tries", > > "num": 100 > > }, > > { > > "op": "take", > > "item": -2, > > "item_name": "sgshared" > > }, > > { > > "op": "chooseleaf_indep", > > "num": 0, > > "type": "host" > > }, > > { > > "op": "emit" > > } > > ] > > }, > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. > > > > Am I missing something? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: EC pool creation results in incorrect M value?
by Paul Emmerich 27 Jan '20

27 Jan '20

min_size in the crush rule and min_size in the pool are completely different things that happen to share the same name. Ignore min_size in the crush rule, it has virtually no meaning in almost all cases (like this one). Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to create an EC3+2 pool with the following commands: > > Create the EC profile: > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > Verify profile creation: > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > crush-device-class= > > crush-failure-domain=host > > crush-root=sgshared > > jerasure-per-chunk-alignment=false > > k=3 > > m=2 > > plugin=jerasure > > technique=reed_sol_van > > w=8 > > Create a pool using this profile: > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > List pool detail: > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool stripe_width 12288 application ES > > Here’s the crush rule that’s created: > { > > "rule_id": 11, > > "rule_name": "es32", > > "ruleset": 11, > > "type": 3, > > "min_size": 3, > > "max_size": 5, > > "steps": [ > > { > > "op": "set_chooseleaf_tries", > > "num": 5 > > }, > > { > > "op": "set_choose_tries", > > "num": 100 > > }, > > { > > "op": "take", > > "item": -2, > > "item_name": "sgshared" > > }, > > { > > "op": "chooseleaf_indep", > > "num": 0, > > "type": "host" > > }, > > { > > "op": "emit" > > } > > ] > > }, > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. > > > > Am I missing something? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

EC pool creation results in incorrect M value?
by Smith, Eric 27 Jan '20

27 Jan '20

I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I'm attempting to create an EC3+2 pool with the following commands: 1. Create the EC profile: * ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 technique=reed_sol_van crush-failure-domain=host crush-root=sgshared 2. Verify profile creation: [root@mon-1 ~]# ceph osd erasure-code-profile get es32 crush-device-class= crush-failure-domain=host crush-root=sgshared jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 1. Create a pool using this profile: * ceph osd pool create ec32pool 1024 1024 erasure es32 2. List pool detail: * pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool stripe_width 12288 application ES 3. Here's the crush rule that's created: { "rule_id": 11, "rule_name": "es32", "ruleset": 11, "type": 3, "min_size": 3, "max_size": 5, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "sgshared" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, From the output of "ceph osd pool ls detail" you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. Am I missing something?

1 0

Re: Ceph MDS randomly hangs with no useful error message
by janek.bevendorff＠uni-weimar.de 26 Jan '20

26 Jan '20

1 0

How to accelerate deep scrub effectively?
by 徐蕴 25 Jan '20

25 Jan '20

Hi, I’ve just upgrade my cluster from Jewel to Nautilus (still running filestore). Since I'v stopped deepscrub for several days for this upgrade, now I got warning of "3 pgs not deep-scrubbed in time”. I tried to increase osd_max_scrubs to 3, osd_scrub_load_threshold to 5.0 and osd_deep_scrub_stride to 1048576 in order to accelerate the deepscrub processes. But deep scrubs seem still very slow. Is there any other way to improve the deep scrub performance? The cluster has 8 nodes, 52 HDD OSDs, 168 TiB / 313 TiB avail data, cluster traffic is low at present. BR, Xu Yun

1 0

Re: Unable to track different ceph client version connections
by Konstantin Shalygin 25 Jan '20

25 Jan '20

> We upgraded our Ceph cluster from Hammer to Luminous and it is running > fine. Post upgrade we live migrated all our Openstack instances (not 100% > sure). Currently we see 1658 clients still on Hammer version. To track the > clients we increased the debugging of debug_mon=10/10, debug_ms=1/5, > debug_monc=5/20 on all three monitors and looking at all three monitor logs > at /var/log/ceph/mon.<id>.log and grepping for hammer and 0x81dff8eeacfffb but > not seeing anything in logs even after hours of waiting. > > Earlier in our other clusters it used to show in logs from which Openstack > compute node it is originating from. Am I missing something or do I need > to add more logging or need to check a different log on all three ceph > monitor nodes? Look your clients at mon sessions: `ceph daemon /var/run/ceph/ceph-mon.ceph-mon0.asok sessions | grep hammer | awk '{print $2}'` k

2 1

CephFS writes cause system reboot
by Ragan, Tj (Dr.) 24 Jan '20

24 Jan '20

Hello all, I have a problem with my CephFS that I’m stumped on. I recently had to rebuild a node who’s system disk failed. Once I did that, I re-created the osd directory structure in /var/lib/ceph/osd and the osds came back into the cluster, then had to backfill. However, I now have the problem that writing more than a few MB into the filesystem causes the node (I’ve tried it on 6 different nodes now,) to reboot. Note that this did not happen until I brought the osds back into the cluster. Shutting them down (or that entire node,) doesn’t fix the problem. Updating to the latest (Centos7) kernel didn’t fix the problem. This happens with both the kernel and fuse mounts, across several data pools, including a newly created data pool. I’m running 14.2.6. Sometimes I get the following message just before the system reboots (while running an rsync job): Message from syslogd@admin2 at Jan 24 17:21:07 ... kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [rsync:4399] after failing during write to kernel mounted FS, dmesg has (in the abrt crash report): [ 243.911318] ------------[ cut here ]------------ [ 243.911350] kernel BUG at mm/slub.c:3772! [ 243.911368] invalid opcode: 0000 [#1] SMP [ 243.911390] Modules linked in: fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 ceph nfs libceph lockd dns_resolver grace fscache nf_nat_ftp nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt rpcrdma sunrpc rdma_ucm ib_iser rdma_cm ib_umad iw_cm ib_ipoib libiscsi scsi_transport_iscsi [ 243.911780] ib_cm iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass mlx4_ib crc32_pclmul ghash_clmulni_intel ib_uverbs aesni_intel lrw gf128mul ib_core glue_helper ablk_helper cryptd pcspkr joydev lpc_ich i2c_i801 sg mei_me mei ioatdma wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx4_core drm ahci igb libahci crct10dif_pclmul crct10dif_common libata crc32c_intel ptp pps_core devlink drm_panel_orientation_quirks dca i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod [ 243.912143] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Not tainted 3.10.0-1062.4.3.el7.x86_64 #1 [ 243.912177] Hardware name: Supermicro SYS-6018R-WTR/X10DRW-i, BIOS 2.0b 04/13/2017 [ 243.912207] task: ffff96b7e95a20e0 ti: ffff96b7e95b0000 task.ti: ffff96b7e95b0000 [ 243.912236] RIP: 0010:[<ffffffffb322355c>] [<ffffffffb322355c>] kfree+0x13c/0x140 [ 243.912271] RSP: 0018:ffff96c6bf843d60 EFLAGS: 00010246 [ 243.912293] RAX: 002fffff00000000 RBX: ffff96c6aaa60000 RCX: 0000000000000ec0 [ 243.912321] RDX: 002fffff00000000 RSI: 0000000000000001 RDI: ffff96c6aaa60000 [ 243.912349] RBP: ffff96c6bf843d78 R08: ffff96c6aaa60000 R09: ffff96b7e95b3e60 [ 243.912377] R10: 7fffffffffffffff R11: ffffe81100aa9800 R12: ffff96c6aaa60ec0 [ 243.912405] R13: ffffffffb36376c5 R14: 000000000000003d R15: ffff96c7691186a8 [ 243.912433] FS: 0000000000000000(0000) GS:ffff96c6bf840000(0000) knlGS:0000000000000000 [ 243.912465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 243.912488] CR2: 000055ba0b02bfb8 CR3: 000000203972e000 CR4: 00000000003607e0 [ 243.912516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 243.912544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 243.912572] Call Trace: [ 243.912584] <IRQ> [ 243.912598] [<ffffffffb36376c5>] skb_release_data+0xf5/0x140 [ 243.912625] [<ffffffffb3637734>] skb_release_all+0x24/0x30 [ 243.912650] [<ffffffffb3637c1c>] consume_skb+0x2c/0x90 [ 243.912674] [<ffffffffb364cb9d>] __dev_kfree_skb_any+0x3d/0x50 [ 243.912705] [<ffffffffc0ad8bd5>] ipoib_ib_handle_tx_wc+0x75/0x1d0 [ib_ipoib] [ 243.912737] [<ffffffffc0ad8f3a>] ipoib_tx_poll+0x9a/0x100 [ib_ipoib] [ 243.912765] [<ffffffffb364f57f>] net_rx_action+0x26f/0x390 [ 243.912790] [<ffffffffb30a5305>] __do_softirq+0xf5/0x280 [ 243.912815] [<ffffffffb379042c>] call_softirq+0x1c/0x30 [ 243.912840] [<ffffffffb302f715>] do_softirq+0x65/0xa0 [ 243.912862] [<ffffffffb30a5685>] irq_exit+0x105/0x110 [ 243.912884] [<ffffffffb3791886>] do_IRQ+0x56/0xf0 [ 243.913860] [<ffffffffb378336a>] common_interrupt+0x16a/0x16a [ 243.914841] <EOI> [ 243.914853] [<ffffffffb35c10f7>] ? cpuidle_enter_state+0x57/0xd0 [ 243.916792] [<ffffffffb35c124e>] cpuidle_idle_call+0xde/0x230 [ 243.917768] [<ffffffffb3037c6e>] arch_cpu_idle+0xe/0xc0 [ 243.918731] [<ffffffffb3100d3a>] cpu_startup_entry+0x14a/0x1e0 [ 243.919701] [<ffffffffb305a0b7>] start_secondary+0x1f7/0x270 [ 243.920650] [<ffffffffb30000d5>] start_cpu+0x5/0x14 [ 243.921567] Code: 49 8b 03 31 f6 f6 c4 40 74 04 41 8b 73 68 4c 89 df e8 89 2f fa ff eb 84 4c 8b 58 30 48 8b 10 80 e6 80 4c 0f 44 d8 e9 28 ff ff ff <0f> 0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 [ 243.923534] RIP [<ffffffffb322355c>] kfree+0x13c/0x140 [ 243.924435] RSP <ffff96c6bf843d60> Just before the crash, I get the following in my mds log (level 10): 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server straydn is [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server witnesses , have 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker acquire_locks request(client.77200959:238582 cr=0x55d019672580) 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=0 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x1 [...2,head] / auth v34216266 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000001 [...2,head] /home/ auth v98722180 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=0 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=0 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=0 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json) auth_pin by 0x55d01a129400 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x607) auth_pin by 0x55d01a129400 on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x1 [...2,head] / auth v34216266 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x1) auth_pin by 0x55d01a129400 on [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000001 [...2,head] /home/ auth v98722180 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000001) auth_pin by 0x55d01a129400 on [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=0 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000002) auth_pin by 0x55d01a129400 on [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000a52) auth_pin by 0x55d01a129400 on [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x1000268e7cb) auth_pin by 0x55d01a129400 on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x607 10003178634) auth_pin by 0x55d01a129400 on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 authpin=1 0x55d0053208c0] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d01a129400 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d01a129400 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=0 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json.2465308719) auth_pin by 0x55d01a129400 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=1 0x55d018f10000] now 1 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn lock) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x607 10003178634) auth_pin by 0x55d005320980 on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn lock) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=2 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=2 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=2 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json) auth_pin by 0x55cfeeef3e80 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json.2465308719) auth_pin by 0x55d018f100c0 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=1 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (ifile sync) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ifile sync) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (ifile lock w=1) [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=1 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (inest lock) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=1 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (inest lock w=1) [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock w=1) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=2 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (ifile excl) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (ifile excl w=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (inest lock) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (inest lock w=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=3 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (iversion lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (iversion lock w=1 last_client=77200959) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (isnap sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (isnap sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (isnap lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d006c3a5e0 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (isnap xlock x=1 by 0x55d01a129400) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (ilink sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ilink sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (ilink lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d006c3a530 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 3 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (ilink xlock x=1 by 0x55d01a129400) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink xlock x=1 by 0x55d01a129400) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (iversion lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (iversion lock w=1 last_client=77200959) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (isnap sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (isnap sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (isnap lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d006c3bae0 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (isnap xlock x=1 by 0x55d01a129400) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (ilink sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ilink sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (ilink lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d006c3ba30 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 3 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (ilink xlock x=1 by 0x55d01a129400) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink xlock x=1 by 0x55d01a129400) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore owner 1000:1003 mode 040700) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *] 2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore owner 1000:1003 mode 040700) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *] 2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 owner 1000:1000 mode 0100600) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.snaprealm(0x1 seq 1 0x55cfeff4a280) have_past_parents_open [1,head] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.snaprealm(0x1 seq 1 0x55cfeff4a280) have_past_parents_open [1,head] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server _rename_prepare request(client.77200959:238582 cr=0x55d019672580) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 authpin=1 0x55d018f10000] [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] I have no idea if any of these are red herrings (sorry.) Please help, at this point I’m baffled! Thank you. -TJ

1 0

Ceph-volume lvm batch: strategy changed after filtering
by Stolte, Felix 24 Jan '20

24 Jan '20

Hey guys, I’m struggling with the ceph-volume command in nautilus 14.2.6. I have 12 disks on each server, 3 of them ssds (sda,sdb,sdc) and 9 spinning disks (sdd .. sdl). Initial deploy witch ceph-volume batch works fine, one ssd is used for wal and db for 3 spinning disks. But running the ‘ceph-volume lvm batch --report /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl’ after the initial deploy I get 'Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering' Looking at /var/log/ceph/ceph-volume.log I see: [ceph_volume.devices.lvm.batch][INFO ] Ignoring devices already used by ceph: /dev/sdd, /dev/sde, /dev/sdf, /dev/sdg, /dev/sdh, /dev/sdi, /dev/sdj, /dev/sdk, /dev/sdl I do understand that before filtering I have MixedType (ssd + sata) and after filtering (only ssds left) it is single type. So the error message is correct. But why does ceph-volume filter out the spinning disks with data partitions but not the ssds with wal/db? Is this intended? Regards Felix IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

2 1

Map osd to physical disk in a containerized RHCS
by John Molefe 24 Jan '20

24 Jan '20

Hi all, I've been trying out RHCS 3.3 using Bluestore for a few months now, however I came across the challenge below... Can someone please assist with a guide on how to map and replace failed bluestore OSD disks in a containerized Rehat Ceph environment? Thanks in advance. Regards John. Vrywaringsklousule / Disclaimer: http://www.nwu.ac.za/it/gov-man/disclaimer.html

1 0

Re: Problem : "1 pools have many more objects per pg than average"
by St-Germain, Sylvain (SSC/SPC) 23 Jan '20

23 Jan '20

Ok case close, I found the solution : 1- Connect on the active mgr 2- Show config #ceph daemon mgr.`hostname -s` config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "10.000000", 3- Change the value #ceph config set mgr.`hostname -s` mon_pg_warn_max_object_skew 20 4- Confirm the change #ceph daemon mgr.`hostname -s` config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "20.000000", 5- Result #ceph health detail HEALTH_OK Sylvain -----Message d'origine----- De : St-Germain, Sylvain (SSC/SPC) Envoyé : 23 janvier 2020 07:55 À : 'ceph-users(a)ceph.io' <ceph-users(a)ceph.io> Objet : Problem : "1 pools have many more objects per pg than average" ///////////////////////// Problem /////////////////////////// I've got a Warning on my cluster that I cannot remove : "1 pools have many more objects per pg than average" Does somebody has some insight ? I think it's normal to have this warning because I have just one pool in use, but how can I remove this warning ? Thx ! ///////////////////////// INFORMATION /////////////////////////// *** Here's some information about the cluster # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool default.rgw.buckets.data objects per pg (50567) is more than 14.1249 times cluster average (3580) # sudo ceph-conf -D | grep mon_pg_warn_max_object_skew mon_pg_warn_max_object_skew = 10.000000 # ceph daemon mon.dao-wkr-04 config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "10.000000" # ceph -v ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 873 TiB 823 TiB 50 TiB 51 TiB 5.80 TOTAL 873 TiB 823 TiB 50 TiB 51 TiB 5.80 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 11 3.5 KiB 8 1.5 MiB 0 249 TiB default.rgw.control 12 0 B 8 0 B 0 249 TiB default.rgw.meta 13 52 KiB 186 34 MiB 0 249 TiB default.rgw.log 14 0 B 207 0 B 0 249 TiB default.rgw.buckets.index 15 1.2 GiB 131 1.2 GiB 0 249 TiB cephfs_data 29 915 MiB 202 1.5 GiB 0 467 TiB cephfs_metadata 30 145 KiB 23 2.1 MiB 0 249 TiB default.rgw.buckets.data 31 30 TiB 12.95M 50 TiB 6.32 467 TiB # ceph osd dump | grep default.rgw.buckets.data pool 31 'default.rgw.buckets.data' erasure size 8 min_size 6 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 9502 lfor 0/2191/7577 flags hashpspool stripe_width 20480 target_size_ratio 0.4 application rgw /////////////////////////// SOLUTION TRIED /////////////////////////// 1- I try to increase the value of mon_pg_warn_max_object_skew parameter #sudo ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20' #sudo ceph tell osd.* injectargs '--mon_pg_warn_max_object_skew 20' + And reboot the monitor The parameter didn't change.

1 0

ceph-volume lvm filestore OSDs fail to start on reboot. Permission denied on journal partition
by Wesley Dillingham 23 Jan '20

23 Jan '20

After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing the following behavior on OSDs which were created with "ceph-volume lvm create --filestore --osd-id <osd> --data <device> --journal <journal>" Upon restart of the server containing these OSDs they fail to start with the following error in the logs: 2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199) mount(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) Permission denied /var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case and inspecting the ownership on /dev/sdc5 it is root:root, chowning that to ceph:ceph causes the osd to start and come back up and in near instantly. As a note these OSDs we experience this with are OSDs which have previously failed and been replaced using the above ceph-volume, longer running OSDs in the same server created with ceph-disk or ceph-volume simple (that have a corresponding .json in /etc/ceph/osd) start up fine and get ceph:ceph on their journal partition. Bluestore OSDs also do not have any issue. My hope is that I can preemptively fix these OSDs before shutting them down so that reboots happen seamlessly. Thanks for any insight. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

4 4

Re: ceph 14.2.6 problem with default args to rbd (--name)
by Jason Dillaman 23 Jan '20

23 Jan '20

CCing correct mailing list ... On Thu, Jan 23, 2020 at 3:19 PM Jason Dillaman <jdillama(a)redhat.com> wrote: > > On Mon, Jan 20, 2020 at 8:26 AM Rainer Krienke <krienke(a)uni-koblenz.de> wrote: > > > > Hello, > > > > I am fighting with rbd and CEPH_ARGS in order to make typing easier on a > > client. First I created a keyring on one of the ceph nodes: > > > > # ceph auth add client.rainer mon 'profile rbd' osd 'profile rbd' > > added key for client.rainer > > > > Then I added this keyring to /etc/ceph/ceph.keyring on a client machine. > > # cat /etc/ceph/ceph.keyring: > > [client.rainer] > > key = xxyxyxyxyxyxyxyxyxyxyxyxyxyxy== > > > > Now on the client I try: > > > > # rbd ls > > 2020-01-20 14:15:49.124 7fa9191f0700 -1 monclient(hunting): > > handle_auth_bad_method server allowed_methods [2] but i only support [2] > > 2020-01-20 14:15:49.124 7fa9189ef700 -1 monclient(hunting): > > handle_auth_bad_method server allowed_methods [2] but i only support [2] > > rbd: couldn't connect to the cluster! > > rbd: listing images failed: (1) Operation not permitted > > > > Using the --name arg things work good: > > > > # rbd ls --name client.rainer > > <list of rbds> > > ... > > > > Now to make typing easier I tried: > > # export CEPH_ARGS="--name client.rainer --keyring=/etc/ceph/ceph.keyring" > > # rbd ls > > 2020-01-20 14:18:33.367 7f691177c700 -1 monclient(hunting): > > handle_auth_bad_method server allowed_methods [2] but i only support [2] > > rbd: couldn't connect to the cluster! > > rbd: listing images failed: (1) Operation not permitted > > > > According to the documentation this should work, but it seems it > > doesn't. Something I am doing wrong or is this a bug? > > Hmm, it looks like this was unintentionally broken globally throughout > Ceph in commit 7f23142f5ccc5ac8153d32b2c9a8353593831967 back in the > mimic release. Previously, the CEPH_ARGS environment overrides were > combined with the command-line arguments near startup, but now they > are not combined until after numerous early arguments are parsed (e.g. > "name", "id", "cluster", "conf", etc). I'll open a tracker ticket > against this issue. > > > Thanks > > Rainer > > -- > > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 > > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 > > Web: http://userpages.uni-koblenz.de/~krienke > > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > Jason -- Jason

1 0

Re: OSD crash after change of osd_memory_target
by Martin Mlynář 23 Jan '20

23 Jan '20

Hi Igor, Dne 23. 01. 20 v 15:37 Igor Fedotov napsal(a): > > Martin, > > suggest a couple more checks: > > 1) Try different value(s) for memory target. Including one that is > equal to default 4Gb > I've tried many random numbers and even original value acquired by: # ceph daemon osd.0 config get osd_memory_target { "osd_memory_target": "4294967296" } > 2) Make sure all the daemons (MON specifically) are updated and has > the same build as OSDs one. > # ceph versions { "mon": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 4 }, "mgr": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 4 }, "osd": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 24 }, "mds": {}, "overall": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 32 } } BTW I've opened offical bug: https://tracker.ceph.com/issues/43766 Regards, Martin > > Thanks, > > Igor > >>>

1 0

Radosgw/Objecter behaviour for homeless session
by Biswajeet Patra 23 Jan '20

23 Jan '20

Hi All, I have a query regarding objecter behaviour for homeless session. In situations when all OSDs containing copies (*let say replication 3*) of an object are down, the objecter assigns a homeless session (OSD=-1) to a client request. This request makes radosgw thread hang indefinitely as the data could never be served because all required OSDs are down. With multiple similar requests, all the radosgw threads gets exhausted and hanged indefinitely waiting for the OSDs to come up. This creates complete service unavailability as no rgw threads are present to process valid requests which could have been directed towards active PGs/OSDs. I think we should have behaviour in objecter or radosgw to terminate request and return early in case of a homeless session. Let me know your thoughts on this. Regards, Biswajeet -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

CephFS ghost usage/inodes
by Florian Pritz 23 Jan '20

23 Jan '20

Hi, When we tried putting some load on our test cephfs setup by restoring a backup in artifactory, we eventually ran out of space (around 95% used in `df` = 3.5TB) which caused artifactory to abort the restore and clean up. However, while a simple `find` no longer shows the files, `df` still claims that we have around 2.1TB of data on the cephfs. `df -i` also shows 2.4M used inodes. When using `du -sh` on a top-level mountpoint, I get 31G used, which is data that is still really here and which is expected to be here. Consequently, we also get the following warning: > MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average > pool cephfs_data objects per pg (38711) is more than 231.802 times cluster average (167) We are running ceph 14.2.5. We have snapshots enabled on cephfs, but there are currently no active snapshots listed by `ceph daemon mds.$hostname dump snaps --server` (see below). I can't say for sure if we created snapshots during the backup restore. > { > "last_snap": 39, > "last_created": 38, > "last_destroyed": 39, > "pending_noop": [], > "snaps": [], > "need_to_purge": {}, > "pending_update": [], > "pending_destroy": [] > } We only have a single CephFS. We use the pool_namespace xattr for our various directory trees on the cephfs. `ceph df` shows: > POOL ID STORED OBJECTS USED %USED MAX AVAIL > cephfs_data 6 2.1 TiB 2.48M 2.1 TiB 24.97 3.1 TiB `ceph daemon mds.$hostname perf dump | grep stray` shows: > "num_strays": 0, > "num_strays_delayed": 0, > "num_strays_enqueuing": 0, > "strays_created": 5097138, > "strays_enqueued": 5097138, > "strays_reintegrated": 0, > "strays_migrated": 0, `rados -p cephfs_data df` shows: > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR > cephfs_data 2.1 TiB 2477540 0 4955080 0 0 0 10699626 6.9 TiB 86911076 35 TiB 0 B 0 B > > total_objects 29718 > total_used 329 GiB > total_avail 7.5 TiB > total_space 7.8 TiB When I combine the usage and the free space shown by `df` we would exceed our cluster size. Our test cluster currently has 7.8TB total space with a replication size of 2 for all pools. With 2.1TB "used" on the cephfs according to `df` + 3.1TB being shows as "free" I get 5.2TB total size. This would mean >10TB of data when accounted for replication. Clearly this can't fit on a cluster with only 7.8TB of capacity. Do you have any ideas why we see so many objects and so much reported usage? Is there any way to fix this without recreating the cephfs? Florian -- Florian Pritz Research Industrial Systems Engineering (RISE) Forschungs-, Entwicklungs- und Großprojektberatung GmbH Concorde Business Park F 2320 Schwechat Austria E-Mail: florian.pritz(a)rise-world.com Web: www.rise-world.com Firmenbuch: FN 280353i Landesgericht Korneuburg UID: ATU62886416

3 15

Re: CephFS with cache-tier kernel-mount client unable to write (Nautilus)
by Ilya Dryomov 23 Jan '20

23 Jan '20

On Thu, Jan 23, 2020 at 2:36 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > On Wed, Jan 22, 2020 at 6:18 PM Hayashida, Mami <mami.hayashida(a)uky.edu> wrote: > > > > Thanks, Ilya. > > > > I just tried modifying the osd cap for client.testuser by getting rid of "tag cephfs data=cephfs_test" part and confirmed this key does work (i.e. lets the CephFS client read/write). It now reads: > > > > [client.testuser] > > key = XXXYYYYZZZ > > caps mds = "allow rw" > > caps mon = "allow r" > > caps osd = "allow rw" // previously "allow rw tag cephfs data=cephfs_test" > > > > I tried removing either "tag cephfs" or "data=cephfs_test" (and leaving the other), but neither worked. > > > > Now, here is my question: will not having the "allow rw tag cephfs data=<file system name>" (under osd caps) result in a security/privacy loophole in a production cluster? (I am still trying to assess whether having a Cache Tier behind CephFS is worth all the headaches...) > > It's probably not worth it. Unless you have a specific tiered > workload in mind and your cache pool is large enough for it, I'd > recommend staying away from cache tiering. > > "allow rw" for osd is only marginally more restrictive than > client.admin's "allow *", allowing the user to read/write every object > in the cluster. Scratch my reply about doing it by hand -- try the > following: > > $ ceph osd pool application enable cephfs-data-cache cephfs > $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test > $ ceph fs authorize cephfs_test ... (as before) > > You will see the same "allow rw tag cephfs data=cephfs_test" cap in > "ceph auth list" output, but it should allow accessing cephfs-data-cache. Dropping ceph-users(a)lists.ceph.com and resending to ceph-users(a)ceph.io. Thanks, Ilya

2 2

Google Summer of Code 2020
by Alastair Dewhurst - UKRI STFC 23 Jan '20

23 Jan '20

Hello I apologies if I have missed an announcement, but I can’t find anything regarding if Ceph will participate in the GSoC this year? The only thing I could find was last years announcement (see below) and webpage. My organisation has some ideas for Ceph projects, however we could submit them under a different banner organisation if necessary. Thanks Alastair On 17 Jan 2019, at 08:12, Mike Perez <miperez(a)redhat.com<mailto:miperez@redhat.com>> wrote: Hello everyone, We're getting ready for the next round of Google Summer of Code and Outreachy. Ali Maredia and I will help organize the Ceph project for each program. We are looking to have project ideas for Google Summer of Code by February 4th as our project application deadline is due by February 6th. https://ceph.com/contribute/gsoc2019/ https://summerofcode.withgoogle.com/how-it-works/ Outreachy applicants in mid-February through the end of March will begin selecting open source projects to work on and make small contributions. On April 12th we hope to have some projects set for Outreachy so we can start selecting applicants. End of May will begin the internships. https://www.outreachy.org/mentor/ You can submit project ideas for both programs on this etherpad. https://pad.ceph.com/p/project-ideas Stay tuned for more updates. -- Mike Perez (thingee) _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

cephfs kernel mount option uid?
by Marc Roos 23 Jan '20

23 Jan '20

Is it possible to mount a cephfs with a specific uid or gid? To make it available to a 'non-root' user?

2 1

Problem : "1 pools have many more objects per pg than average"
by St-Germain, Sylvain (SSC/SPC) 23 Jan '20

23 Jan '20

///////////////////////// Problem /////////////////////////// I've got a Warning on my cluster that I cannot remove : "1 pools have many more objects per pg than average" Does somebody has some insight ? I think it's normal to have this warning because I have just one pool in use, but how can I remove this warning ? Thx ! ///////////////////////// INFORMATION /////////////////////////// *** Here's some information about the cluster # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool default.rgw.buckets.data objects per pg (50567) is more than 14.1249 times cluster average (3580) # sudo ceph-conf -D | grep mon_pg_warn_max_object_skew mon_pg_warn_max_object_skew = 10.000000 # ceph daemon mon.dao-wkr-04 config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "10.000000" # ceph -v ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 873 TiB 823 TiB 50 TiB 51 TiB 5.80 TOTAL 873 TiB 823 TiB 50 TiB 51 TiB 5.80 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 11 3.5 KiB 8 1.5 MiB 0 249 TiB default.rgw.control 12 0 B 8 0 B 0 249 TiB default.rgw.meta 13 52 KiB 186 34 MiB 0 249 TiB default.rgw.log 14 0 B 207 0 B 0 249 TiB default.rgw.buckets.index 15 1.2 GiB 131 1.2 GiB 0 249 TiB cephfs_data 29 915 MiB 202 1.5 GiB 0 467 TiB cephfs_metadata 30 145 KiB 23 2.1 MiB 0 249 TiB default.rgw.buckets.data 31 30 TiB 12.95M 50 TiB 6.32 467 TiB # ceph osd dump | grep default.rgw.buckets.data pool 31 'default.rgw.buckets.data' erasure size 8 min_size 6 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 9502 lfor 0/2191/7577 flags hashpspool stripe_width 20480 target_size_ratio 0.4 application rgw /////////////////////////// SOLUTION TRIED /////////////////////////// 1- I try to increase the value of mon_pg_warn_max_object_skew parameter #sudo ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20' #sudo ceph tell osd.* injectargs '--mon_pg_warn_max_object_skew 20' + And reboot the monitor The parameter didn't change.

1 0

Fwd: OSD crash after change of osd_memory_target
by Martin Mlynář 23 Jan '20

23 Jan '20

Hi Igor, unfortunately same result: # ceph config dump WHO MASK LEVEL OPTION VALUE RO osd basic osd_memory_target 2147483648 # /usr/bin/ceph-osd -d --cluster ceph --id 0 --setuser ceph --setgroup ceph .... 0> 2020-01-23 10:48:04.436 7fc61b5b5c80 -1 *** Caught signal (Aborted) ** in thread 7fc61b5b5c80 thread_name:ceph-osd ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) 1: (()+0x12730) [0x7fc61c05d730] 2: (gsignal()+0x10b) [0x7fc61bb417bb] 3: (abort()+0x121) [0x7fc61bb2c535] 4: (()+0x8c983) [0x7fc61bef4983] 5: (()+0x928c6) [0x7fc61befa8c6] 6: (()+0x92901) [0x7fc61befa901] 7: (()+0x92b34) [0x7fc61befab34] 8: (()+0x5a3f53) [0x55ecdabb2f53] 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x55ecdabb8c91] 10: (BlueStore::_set_cache_sizes()+0x15a) [0x55ecdb033d8a] 11: (BlueStore::_open_bdev(bool)+0x173) [0x55ecdb036b23] 12: (BlueStore::get_devices(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0xef) [0x55ecdb09d7ef] 13: (BlueStore::get_numa_node(int*, std::set<int, std::less<int>, std::allocator<int> >*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0x7b) [0x55ecdb04571b] 14: (main()+0x2870) [0x55ecdab80440] 15: (__libc_start_main()+0xeb) [0x7fc61bb2e09b] 16: (_start()+0x2a) [0x55ecdabb2c6a] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. .... Best Regards, Martin Dne 22. 01. 20 v 16:33 Igor Fedotov napsal(a): > > Hi Martin, > > looks like a bug to me. > > You might want to remove all custom settings from config database and > try to set osd-memory-target only. > > Would it help? > > > Thanks, > > Igor > > On 1/22/2020 3:43 PM, Martin Mlynář wrote: >> >> >> Dne 21. 01. 20 v 21:12 Stefan Kooman napsal(a): >>> Quoting Martin Mlynář (nexus+ceph(a)smoula.net) >>> >>>> Do you think this could help? OSD does not even start, I'm getting a little >>>> lost how flushing caches could help. >>> I might have mis-understood. I though the OSDs crashed when you set the >>> config setting. >>> >>>> According to trace I suspect something around processing config values. >>> I've just set the same config setting on a test cluster and restarted an >>> OSD without problem. So, not sure what is going on there. >>> >>> Gr. Stefan >> >> I've compiled ceph-osd with debug symbols and got better backtrace: >> >> -24> 2020-01-22 13:12:53.614 7f83ed064700 4 set_mon_vals no >> callback set >> -23> 2020-01-22 13:12:53.614 7f83ee867700 10 monclient: discarding >> stray monitor message auth_reply(proto 2 0 (0) Success) v1 >> -22> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_crush_update_on_start = true >> -21> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_max_backfills = 64 >> -20> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_memory_target = 2147483648 >> -19> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_max_active = 40 >> -18> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_max_single_start = 1000 >> -17> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_sleep_hdd = 0.000000 >> -16> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_sleep_hybrid = 0.000000 >> -15> 2020-01-22 13:12:53.627 7f83f0276c40 0 set uid:gid to >> 64045:64045 (ceph:ceph) >> -14> 2020-01-22 13:12:53.627 7f83f0276c40 0 ceph version 14.2.6 >> (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable), process >> ceph-osd, pid 1111622 >> -13> 2020-01-22 13:12:53.649 7f83f0276c40 0 pidfile_write: ignore >> empty --pid-file >> -12> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> init /var/run/ceph/ceph-osd.6.asok >> -11> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> bind_and_listen /var/run/ceph/ceph-osd.6.asok >> -10> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command 0 hook 0x558051872fc0 >> -9> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command version hook 0x558051872fc0 >> -8> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command git_version hook 0x558051872fc0 >> -7> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command help hook 0x558051874220 >> -6> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command get_command_descriptions hook 0x558051874260 >> -5> 2020-01-22 13:12:53.657 7f83ed865700 5 asok(0x5580518fa000) >> entry start >> -4> 2020-01-22 13:12:53.670 7f83f0276c40 5 object store type is >> bluestore >> -3> 2020-01-22 13:12:53.675 7f83f0276c40 1 bdev create path >> /var/lib/ceph/osd/ceph-6/block type kernel >> -2> 2020-01-22 13:12:53.675 7f83f0276c40 1 bdev(0x5580518f3f80 >> /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block >> -1> 2020-01-22 13:12:53.675 7f83f0276c40 1 bdev(0x5580518f3f80 >> /var/lib/ceph/osd/ceph-6/block) open size 3000588304384 >> (0x2baa1000000, 2.7 TiB) block_size 4096 (4 KiB) rotational discard >> not supported >> 0> 2020-01-22 13:12:53.714 7f83f0276c40 -1 *** Caught signal >> (Aborted) ** >> in thread 7f83f0276c40 thread_name:ceph-osd >> >> ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) >> nautilus (stable) >> 1: (()+0x2c19654) [0x558045ec6654] >> 2: (()+0x12730) [0x7f83f0d1f730] >> 3: (gsignal()+0x10b) [0x7f83f08027bb] >> 4: (abort()+0x121) [0x7f83f07ed535] >> 5: (()+0x8c983) [0x7f83f0bb5983] >> 6: (()+0x928c6) [0x7f83f0bbb8c6] >> 7: (()+0x92901) [0x7f83f0bbb901] >> 8: (()+0x92b34) [0x7f83f0bbbb34] >> * 9: (void boost::throw_exception<boost::bad_get>(boost::bad_get >> const&)+0x7b) [0x5580454d5430]* >> * 10: (Option::size_t&& boost::relaxed_get<Option::size_t, >> boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>(boost::variant<boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>&&)+0x5b) [0x5580454d6223]* >> 11: (Option::size_t&& boost::strict_get<Option::size_t, >> boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>(boost::variant<boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>&&)+0x20) [0x5580454d4a39] >> 12: (Option::size_t&& boost::get<Option::size_t, boost::blank, >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >, unsigned long, long, double, bool, >> entity_addr_t, entity_addrvec_t, std::chrono::duration<long, >> std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>(boost::variant<boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>&&)+0x20) [0x5580454d1ed7] >> 13: (Option::size_t const >> md_config_t::get_val<Option::size_t>(ConfigValues const&, >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > const&) const+0x48) [0x5580454ce882] >> 14: (Option::size_t const >> ConfigProxy::get_val<Option::size_t>(std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > const&) const+0x58) >> [0x5580454cb9b8] >> 15: (BlueStore::_set_cache_sizes()+0x159) [0x558045ce2213] >> 16: (BlueStore::_open_bdev(bool)+0x301) [0x558045ce6be3] >> 17: >> (BlueStore::get_devices(std::set<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, >> std::less<std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > >, >> std::allocator<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > > >*)+0xf9) >> [0x558045d0f16d] >> 18: (BlueStore::get_numa_node(int*, std::set<int, std::less<int>, >> std::allocator<int> >*, std::set<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, >> std::less<std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > >, >> std::allocator<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > > >*)+0x79) >> [0x558045d0eb55] >> 19: (main()+0x3aae) [0x5580454c2460] >> 20: (__libc_start_main()+0xeb) [0x7f83f07ef09b] >> 21: (_start()+0x2a) [0x5580454bda2a] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> needed to interpret this. >> >> And managed some gdb debugging (int BlueStore::_set_cache_sizes()): >> >> (gdb) n >> 4116 cache_autotune_interval = >> (gdb) n >> 4117 >> cct->_conf.get_val<double>("bluestore_cache_autotune_interval"); >> (gdb) p cache_autotune_interval >> $3 = 5 >> (gdb) n >> 4118 osd_memory_target = >> cct->_conf.get_val<Option::size_t>("osd_memory_target"); >> (gdb) s >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >::basic_string<std::allocator<char> > >> (this=0x7fffffffc140, __s=0x555558d26c2f "osd_memory_target", __a=...) >> at /usr/include/c++/8/bits/basic_string.h:515 >> 515 : _M_dataplus(_M_local_data(), __a) >> (gdb) n >> 516 { _M_construct(__s, __s ? __s + traits_type::length(__s) >> : __s+npos); } >> (gdb) >> terminate called after throwing an instance of >> 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get> >> >' >> what(): boost::bad_get: failed value get using boost::get >> >> But there I'm stuck. GDBing c++ code is a really dark sorcery for me. >> >> Other get_vals look fine, maybe get_val<*Option::size_t*> is the >> problem? It looks like trouble outside of ceph - what system are you >> testing on? This is debian with official debian build from >> buster-backports. Maybe some of debian's patches? >> >> -- >> Martin Mlynář >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users(a)lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Ceph at DevConf and FOSDEM
by Mike Perez 23 Jan '20

23 Jan '20

Hi Cephers, We will have a booth at DevConf this year! https://www.devconf.info/cz/ If you're interested in helping at the booth at DevConf, please reply to me directly, and I can provide more details. I'm still working on getting a shared booth for FOSDEM with the CentOS table. Also, I will try to get us signed with a BoF session. If you're interested, please also let me know directly. Thanks! https://fosdem.org/2020/ -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 1

Upcoming Ceph Days for 2020
by Mike Perez 23 Jan '20

23 Jan '20

Hi Cephers, We have just posted some upcoming Ceph Days. We are looking for sponsors and content: * Ceph Day Istanbul: March 17 * Ceph Day Oslo: May 13 * Ceph Day Vancouver: May 13 https://ceph.com/cephdays/ Also don't forget about our big event Cephalocon Seoul March 3-5. Registration, schedule. sponsorship and hotel information is available: https://ceph.io/cephalocon/seoul-2020/ If you have any questions about these events, please contact us at events(a)ceph.io. Any help with promoting these events is greatly appreciated. Thanks! -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Migrate Jewel from leveldb to rocksdb
by Robert LeBlanc 22 Jan '20

22 Jan '20

In the last release of Jewel [0] it mentions that omap data can be stored in rocksdb instead of leveldb. We are seeing high latencies from compaction of leveldb on our Jewel cluster (can't upgrade at this time). I installed the latest version, but apparently that is not enough to do the conversion. Is there a way to move OSDs from leveldb to rocksdb without trashing and rebuilding all the osds? [0] https://docs.ceph.com/docs/master/releases/jewel/ Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 3

Auto create rbd snapshots
by Marc Roos 22 Jan '20

22 Jan '20

Is it possible to schedule the creation of snapshots on specific rbd images within ceph?

1 0

Large omap objects in radosgw .usage pool: is there a way to reshard the rgw usage log?
by Ingo Reimann 22 Jan '20

22 Jan '20

Hi All >On 09/10/2019 09:07, Florian Haas wrote: >[...] >the question with about resharding the usage log still stands. (The untrimmed usage log, in my case, would have blasted the old 2M keys threshold, too.) > >Cheers, Florian Is there any new wisdom about resharding the usage log for one user? Since Nautilus we get a HEALTH_WARN after 3 weeks of the month because the usage data of one single user reaches the threshold for large omap warnings - which I already increased to 1M. At start of month, we truncate the usage data so we are save again for a while. Cheers, ingo -- ngo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

1 0

Cephalocon early-bird registration ends today
by Sage Weil 21 Jan '20

21 Jan '20

Hi everyone, Quick reminder that the early-bird registration for Cephalocon Seoul (Mar 3-5) ends tonight! We also have the hotel booking link and code up on the site (finally--sorry for the delay). https://ceph.io/cephalocon/seoul-2020/ Hope to see you there! sage

1 0

PG lock contention? CephFS metadata pool rebalance
by Stefan Kooman 21 Jan '20

21 Jan '20

Hi, Like I said in an earlier mail to this list, we re-balanced ~ 60% of the CephFS metadata pool to NVMe backed devices. Roughly 422 M objects (1.2 Billion replicated). We have 512 PGs allocated to them. While rebalancing we suffered from quite a few SLOW_OPS. Memory, CPU and device IOPS capacity were not a limiting factor as far as we can see (plenty of them available ... nowhere near max capacity). We saw quite a few slow ops with the following events: "time": "2019-12-19 09:41:02.712010", "event": "reached_pg" }, { "time": "2019-12-19 09:41:02.712014", "event": "waiting for rw locks" }, { "time": "2019-12-19 09:41:02.881939", "event": "reached_pg" ... and this repeated 100's of times taking ~ 30 seconds to complete Does this indicate PG lock contention? If so ... would we need to provide more PGs to the metadata pool to avoid this? The metadata pool is only ~ 166 MiB big ... but with loads of OMAPs ... Most advice on PG planning is concerned with the _amount_ of data ... but the metadata pool (and this might also be true for RGW index pools) seem to be a special case. Thanks for your insights, Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

deep-scrub / backfilling: large amount of SLOW_OPS after upgrade to 13.2.8
by Stefan Kooman 21 Jan '20

21 Jan '20

Hi, After the upgrade to 13.2.8 deep-scrub has a big impact on client IO: loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but since the upgrade the impact is so big that we even have OSDs marking each other out (OSD op thread timeout) multiple times during the scrub window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD servers. Has there anything changed in this release that can explain this behaviour? Besides this the impact of rebalance is very severe as well. With only the balancer remapping a couple of PGs at a time there are loads of (MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ... and that triggered a lot of SLOW_OPS. One particular OSD was pegged at 1000% CPU for more than half an hour (not doing that much IO): that's 10 cores going full throttle! After a restart this issue was gone. Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 3

subscribe
by lists 21 Jan '20

21 Jan '20

1 0

Upgrade from Jewel to Luminous resulted 82% misplacement
by 徐蕴 21 Jan '20

21 Jan '20

Hi, We upgraded our cluster from Jewel to Luminous, and it turned out that more than 80% object misplaced. Since our cluster has 130T data, backfilling seems take forever. We didn’t modify any crushmap. Any thoughts about this issue? br, Xu Yun

2 2

lists and gmail
by Sasha Litvak 20 Jan '20

20 Jan '20

It seems that people now split between new and old list servers. Regardless of either one of them, I am missing a number of messages that appear on archive pages but never seem to make to my inbox. And no they are not in my junk folder. I wonder if some of my questions are not getting a response because people don't receive them. Any other reason, like people choosing not to answer, is pretty acceptable. Does anyone else have difficulty communicating with user list using gmail account?

1 0

Re: cephfs kernel mount option uid?
by Kevin Thorpe 20 Jan '20

20 Jan '20

Hi, Pretty certain not. I hit that exact issue. The workaround I was suggested to use was an init container running as root to change the ownership. That works ok but is very hacky. -- *Kevin Thorpe* VP of Enterprise Platform *W* *|* www.predictx.com *P * *|* +44 (0)20 3005 6750 <+44%2020%203005%206750> | +44 (0)808 204 0344 <+44%20808%20204%200344> *A* *|* 7th Floor, 52 Grosvenor Gardens, London SW1W 0AU <https://maps.google.com/?q=7th+Floor,+52+Grosvenor+Gardens,+London+SW1W+0AU…> <http://twitter.com/predictxai> <https://www.linkedin.com/company/1339603/> _________________________________________________________ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. On Mon, 20 Jan 2020 at 16:47, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Is it possible to mount a cephfs with a specific uid or gid? To make it > available to a 'non-root' user? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

backfill / recover logic (OSD included as selection criterion)
by Stefan Kooman 20 Jan '20

20 Jan '20

Hi, Is there any logic / filtering which PGs to backfill at any given time that takes into account the OSD the PG is living on? Our cluster is backfilling a complete pool now (512 PGs) and (currently) of the 7 active+remapped+backfilling there are 4 of them on the same OSD. Which stresses this OSD way more than needed. It would be nice if the selection criteria for which PG to backfill (and / or recover) from would include the OSD as selection criterion in order to spread the load across different OSDs. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 2