- ceph-users - lists.ceph.io

CephFS writes cause system reboot
by Ragan, Tj (Dr.) 24 Jan '20

24 Jan '20

Hello all, I have a problem with my CephFS that I’m stumped on. I recently had to rebuild a node who’s system disk failed. Once I did that, I re-created the osd directory structure in /var/lib/ceph/osd and the osds came back into the cluster, then had to backfill. However, I now have the problem that writing more than a few MB into the filesystem causes the node (I’ve tried it on 6 different nodes now,) to reboot. Note that this did not happen until I brought the osds back into the cluster. Shutting them down (or that entire node,) doesn’t fix the problem. Updating to the latest (Centos7) kernel didn’t fix the problem. This happens with both the kernel and fuse mounts, across several data pools, including a newly created data pool. I’m running 14.2.6. Sometimes I get the following message just before the system reboots (while running an rsync job): Message from syslogd@admin2 at Jan 24 17:21:07 ... kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [rsync:4399] after failing during write to kernel mounted FS, dmesg has (in the abrt crash report): [ 243.911318] ------------[ cut here ]------------ [ 243.911350] kernel BUG at mm/slub.c:3772! [ 243.911368] invalid opcode: 0000 [#1] SMP [ 243.911390] Modules linked in: fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 ceph nfs libceph lockd dns_resolver grace fscache nf_nat_ftp nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt rpcrdma sunrpc rdma_ucm ib_iser rdma_cm ib_umad iw_cm ib_ipoib libiscsi scsi_transport_iscsi [ 243.911780] ib_cm iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass mlx4_ib crc32_pclmul ghash_clmulni_intel ib_uverbs aesni_intel lrw gf128mul ib_core glue_helper ablk_helper cryptd pcspkr joydev lpc_ich i2c_i801 sg mei_me mei ioatdma wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx4_core drm ahci igb libahci crct10dif_pclmul crct10dif_common libata crc32c_intel ptp pps_core devlink drm_panel_orientation_quirks dca i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod [ 243.912143] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Not tainted 3.10.0-1062.4.3.el7.x86_64 #1 [ 243.912177] Hardware name: Supermicro SYS-6018R-WTR/X10DRW-i, BIOS 2.0b 04/13/2017 [ 243.912207] task: ffff96b7e95a20e0 ti: ffff96b7e95b0000 task.ti: ffff96b7e95b0000 [ 243.912236] RIP: 0010:[<ffffffffb322355c>] [<ffffffffb322355c>] kfree+0x13c/0x140 [ 243.912271] RSP: 0018:ffff96c6bf843d60 EFLAGS: 00010246 [ 243.912293] RAX: 002fffff00000000 RBX: ffff96c6aaa60000 RCX: 0000000000000ec0 [ 243.912321] RDX: 002fffff00000000 RSI: 0000000000000001 RDI: ffff96c6aaa60000 [ 243.912349] RBP: ffff96c6bf843d78 R08: ffff96c6aaa60000 R09: ffff96b7e95b3e60 [ 243.912377] R10: 7fffffffffffffff R11: ffffe81100aa9800 R12: ffff96c6aaa60ec0 [ 243.912405] R13: ffffffffb36376c5 R14: 000000000000003d R15: ffff96c7691186a8 [ 243.912433] FS: 0000000000000000(0000) GS:ffff96c6bf840000(0000) knlGS:0000000000000000 [ 243.912465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 243.912488] CR2: 000055ba0b02bfb8 CR3: 000000203972e000 CR4: 00000000003607e0 [ 243.912516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 243.912544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 243.912572] Call Trace: [ 243.912584] <IRQ> [ 243.912598] [<ffffffffb36376c5>] skb_release_data+0xf5/0x140 [ 243.912625] [<ffffffffb3637734>] skb_release_all+0x24/0x30 [ 243.912650] [<ffffffffb3637c1c>] consume_skb+0x2c/0x90 [ 243.912674] [<ffffffffb364cb9d>] __dev_kfree_skb_any+0x3d/0x50 [ 243.912705] [<ffffffffc0ad8bd5>] ipoib_ib_handle_tx_wc+0x75/0x1d0 [ib_ipoib] [ 243.912737] [<ffffffffc0ad8f3a>] ipoib_tx_poll+0x9a/0x100 [ib_ipoib] [ 243.912765] [<ffffffffb364f57f>] net_rx_action+0x26f/0x390 [ 243.912790] [<ffffffffb30a5305>] __do_softirq+0xf5/0x280 [ 243.912815] [<ffffffffb379042c>] call_softirq+0x1c/0x30 [ 243.912840] [<ffffffffb302f715>] do_softirq+0x65/0xa0 [ 243.912862] [<ffffffffb30a5685>] irq_exit+0x105/0x110 [ 243.912884] [<ffffffffb3791886>] do_IRQ+0x56/0xf0 [ 243.913860] [<ffffffffb378336a>] common_interrupt+0x16a/0x16a [ 243.914841] <EOI> [ 243.914853] [<ffffffffb35c10f7>] ? cpuidle_enter_state+0x57/0xd0 [ 243.916792] [<ffffffffb35c124e>] cpuidle_idle_call+0xde/0x230 [ 243.917768] [<ffffffffb3037c6e>] arch_cpu_idle+0xe/0xc0 [ 243.918731] [<ffffffffb3100d3a>] cpu_startup_entry+0x14a/0x1e0 [ 243.919701] [<ffffffffb305a0b7>] start_secondary+0x1f7/0x270 [ 243.920650] [<ffffffffb30000d5>] start_cpu+0x5/0x14 [ 243.921567] Code: 49 8b 03 31 f6 f6 c4 40 74 04 41 8b 73 68 4c 89 df e8 89 2f fa ff eb 84 4c 8b 58 30 48 8b 10 80 e6 80 4c 0f 44 d8 e9 28 ff ff ff <0f> 0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 [ 243.923534] RIP [<ffffffffb322355c>] kfree+0x13c/0x140 [ 243.924435] RSP <ffff96c6bf843d60> Just before the crash, I get the following in my mds log (level 10): 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server straydn is [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server witnesses , have 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker acquire_locks request(client.77200959:238582 cr=0x55d019672580) 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=0 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x1 [...2,head] / auth v34216266 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000001 [...2,head] /home/ auth v98722180 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=0 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=0 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=0 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json) auth_pin by 0x55d01a129400 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x607) auth_pin by 0x55d01a129400 on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x1 [...2,head] / auth v34216266 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x1) auth_pin by 0x55d01a129400 on [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000001 [...2,head] /home/ auth v98722180 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000001) auth_pin by 0x55d01a129400 on [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=0 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000002) auth_pin by 0x55d01a129400 on [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000a52) auth_pin by 0x55d01a129400 on [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x1000268e7cb) auth_pin by 0x55d01a129400 on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x607 10003178634) auth_pin by 0x55d01a129400 on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 authpin=1 0x55d0053208c0] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d01a129400 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d01a129400 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 1 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=0 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json.2465308719) auth_pin by 0x55d01a129400 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=1 0x55d018f10000] now 1 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn lock) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x607 10003178634) auth_pin by 0x55d005320980 on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn lock) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=2 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=2 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=2 authpin=1 0x55d0053208c0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json) auth_pin by 0x55cfeeef3e80 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json.2465308719) auth_pin by 0x55d018f100c0 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 authpin=1 0x55d018f10000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=1 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (ifile sync) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ifile sync) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (ifile lock w=1) [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=1 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (inest lock) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=1 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (inest lock w=1) [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock w=1) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=2 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (ifile excl) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (ifile excl w=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (inest lock) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (inest lock w=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=3 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (iversion lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (iversion lock w=1 last_client=77200959) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (isnap sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (isnap sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (isnap lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d006c3a5e0 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (isnap xlock x=1 by 0x55d01a129400) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (ilink sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ilink sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (ilink lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d006c3a530 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 3 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (ilink xlock x=1 by 0x55d01a129400) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink xlock x=1 by 0x55d01a129400) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (iversion lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (iversion lock w=1 last_client=77200959) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (isnap sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (isnap sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (isnap lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d006c3bae0 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 2 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (isnap xlock x=1 by 0x55d01a129400) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (ilink sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ilink sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (ilink lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d006c3ba30 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 3 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (ilink xlock x=1 by 0x55d01a129400) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink xlock x=1 by 0x55d01a129400) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] 2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore owner 1000:1003 mode 040700) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *] 2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore owner 1000:1003 mode 040700) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *] 2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 owner 1000:1000 mode 0100600) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.snaprealm(0x1 seq 1 0x55cfeff4a280) have_past_parents_open [1,head] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.snaprealm(0x1 seq 1 0x55cfeff4a280) have_past_parents_open [1,head] 2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server _rename_prepare request(client.77200959:238582 cr=0x55d019672580) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 authpin=1 0x55d018f10000] [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] I have no idea if any of these are red herrings (sorry.) Please help, at this point I’m baffled! Thank you. -TJ

1 0

Ceph-volume lvm batch: strategy changed after filtering
by Stolte, Felix 24 Jan '20

24 Jan '20

Hey guys, I’m struggling with the ceph-volume command in nautilus 14.2.6. I have 12 disks on each server, 3 of them ssds (sda,sdb,sdc) and 9 spinning disks (sdd .. sdl). Initial deploy witch ceph-volume batch works fine, one ssd is used for wal and db for 3 spinning disks. But running the ‘ceph-volume lvm batch --report /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl’ after the initial deploy I get 'Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering' Looking at /var/log/ceph/ceph-volume.log I see: [ceph_volume.devices.lvm.batch][INFO ] Ignoring devices already used by ceph: /dev/sdd, /dev/sde, /dev/sdf, /dev/sdg, /dev/sdh, /dev/sdi, /dev/sdj, /dev/sdk, /dev/sdl I do understand that before filtering I have MixedType (ssd + sata) and after filtering (only ssds left) it is single type. So the error message is correct. But why does ceph-volume filter out the spinning disks with data partitions but not the ssds with wal/db? Is this intended? Regards Felix IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

2 1

Map osd to physical disk in a containerized RHCS
by John Molefe 24 Jan '20

24 Jan '20

Hi all, I've been trying out RHCS 3.3 using Bluestore for a few months now, however I came across the challenge below... Can someone please assist with a guide on how to map and replace failed bluestore OSD disks in a containerized Rehat Ceph environment? Thanks in advance. Regards John. Vrywaringsklousule / Disclaimer: http://www.nwu.ac.za/it/gov-man/disclaimer.html

1 0

Re: Problem : "1 pools have many more objects per pg than average"
by St-Germain, Sylvain (SSC/SPC) 23 Jan '20

23 Jan '20

Ok case close, I found the solution : 1- Connect on the active mgr 2- Show config #ceph daemon mgr.`hostname -s` config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "10.000000", 3- Change the value #ceph config set mgr.`hostname -s` mon_pg_warn_max_object_skew 20 4- Confirm the change #ceph daemon mgr.`hostname -s` config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "20.000000", 5- Result #ceph health detail HEALTH_OK Sylvain -----Message d'origine----- De : St-Germain, Sylvain (SSC/SPC) Envoyé : 23 janvier 2020 07:55 À : 'ceph-users(a)ceph.io' <ceph-users(a)ceph.io> Objet : Problem : "1 pools have many more objects per pg than average" ///////////////////////// Problem /////////////////////////// I've got a Warning on my cluster that I cannot remove : "1 pools have many more objects per pg than average" Does somebody has some insight ? I think it's normal to have this warning because I have just one pool in use, but how can I remove this warning ? Thx ! ///////////////////////// INFORMATION /////////////////////////// *** Here's some information about the cluster # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool default.rgw.buckets.data objects per pg (50567) is more than 14.1249 times cluster average (3580) # sudo ceph-conf -D | grep mon_pg_warn_max_object_skew mon_pg_warn_max_object_skew = 10.000000 # ceph daemon mon.dao-wkr-04 config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "10.000000" # ceph -v ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 873 TiB 823 TiB 50 TiB 51 TiB 5.80 TOTAL 873 TiB 823 TiB 50 TiB 51 TiB 5.80 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 11 3.5 KiB 8 1.5 MiB 0 249 TiB default.rgw.control 12 0 B 8 0 B 0 249 TiB default.rgw.meta 13 52 KiB 186 34 MiB 0 249 TiB default.rgw.log 14 0 B 207 0 B 0 249 TiB default.rgw.buckets.index 15 1.2 GiB 131 1.2 GiB 0 249 TiB cephfs_data 29 915 MiB 202 1.5 GiB 0 467 TiB cephfs_metadata 30 145 KiB 23 2.1 MiB 0 249 TiB default.rgw.buckets.data 31 30 TiB 12.95M 50 TiB 6.32 467 TiB # ceph osd dump | grep default.rgw.buckets.data pool 31 'default.rgw.buckets.data' erasure size 8 min_size 6 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 9502 lfor 0/2191/7577 flags hashpspool stripe_width 20480 target_size_ratio 0.4 application rgw /////////////////////////// SOLUTION TRIED /////////////////////////// 1- I try to increase the value of mon_pg_warn_max_object_skew parameter #sudo ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20' #sudo ceph tell osd.* injectargs '--mon_pg_warn_max_object_skew 20' + And reboot the monitor The parameter didn't change.

1 0

ceph-volume lvm filestore OSDs fail to start on reboot. Permission denied on journal partition
by Wesley Dillingham 23 Jan '20

23 Jan '20

After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing the following behavior on OSDs which were created with "ceph-volume lvm create --filestore --osd-id <osd> --data <device> --journal <journal>" Upon restart of the server containing these OSDs they fail to start with the following error in the logs: 2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199) mount(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) Permission denied /var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case and inspecting the ownership on /dev/sdc5 it is root:root, chowning that to ceph:ceph causes the osd to start and come back up and in near instantly. As a note these OSDs we experience this with are OSDs which have previously failed and been replaced using the above ceph-volume, longer running OSDs in the same server created with ceph-disk or ceph-volume simple (that have a corresponding .json in /etc/ceph/osd) start up fine and get ceph:ceph on their journal partition. Bluestore OSDs also do not have any issue. My hope is that I can preemptively fix these OSDs before shutting them down so that reboots happen seamlessly. Thanks for any insight. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

4 4

Re: ceph 14.2.6 problem with default args to rbd (--name)
by Jason Dillaman 23 Jan '20

23 Jan '20

CCing correct mailing list ... On Thu, Jan 23, 2020 at 3:19 PM Jason Dillaman <jdillama(a)redhat.com> wrote: > > On Mon, Jan 20, 2020 at 8:26 AM Rainer Krienke <krienke(a)uni-koblenz.de> wrote: > > > > Hello, > > > > I am fighting with rbd and CEPH_ARGS in order to make typing easier on a > > client. First I created a keyring on one of the ceph nodes: > > > > # ceph auth add client.rainer mon 'profile rbd' osd 'profile rbd' > > added key for client.rainer > > > > Then I added this keyring to /etc/ceph/ceph.keyring on a client machine. > > # cat /etc/ceph/ceph.keyring: > > [client.rainer] > > key = xxyxyxyxyxyxyxyxyxyxyxyxyxyxy== > > > > Now on the client I try: > > > > # rbd ls > > 2020-01-20 14:15:49.124 7fa9191f0700 -1 monclient(hunting): > > handle_auth_bad_method server allowed_methods [2] but i only support [2] > > 2020-01-20 14:15:49.124 7fa9189ef700 -1 monclient(hunting): > > handle_auth_bad_method server allowed_methods [2] but i only support [2] > > rbd: couldn't connect to the cluster! > > rbd: listing images failed: (1) Operation not permitted > > > > Using the --name arg things work good: > > > > # rbd ls --name client.rainer > > <list of rbds> > > ... > > > > Now to make typing easier I tried: > > # export CEPH_ARGS="--name client.rainer --keyring=/etc/ceph/ceph.keyring" > > # rbd ls > > 2020-01-20 14:18:33.367 7f691177c700 -1 monclient(hunting): > > handle_auth_bad_method server allowed_methods [2] but i only support [2] > > rbd: couldn't connect to the cluster! > > rbd: listing images failed: (1) Operation not permitted > > > > According to the documentation this should work, but it seems it > > doesn't. Something I am doing wrong or is this a bug? > > Hmm, it looks like this was unintentionally broken globally throughout > Ceph in commit 7f23142f5ccc5ac8153d32b2c9a8353593831967 back in the > mimic release. Previously, the CEPH_ARGS environment overrides were > combined with the command-line arguments near startup, but now they > are not combined until after numerous early arguments are parsed (e.g. > "name", "id", "cluster", "conf", etc). I'll open a tracker ticket > against this issue. > > > Thanks > > Rainer > > -- > > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 > > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 > > Web: http://userpages.uni-koblenz.de/~krienke > > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > Jason -- Jason

1 0

Re: OSD crash after change of osd_memory_target
by Martin Mlynář 23 Jan '20

23 Jan '20

Hi Igor, Dne 23. 01. 20 v 15:37 Igor Fedotov napsal(a): > > Martin, > > suggest a couple more checks: > > 1) Try different value(s) for memory target. Including one that is > equal to default 4Gb > I've tried many random numbers and even original value acquired by: # ceph daemon osd.0 config get osd_memory_target { "osd_memory_target": "4294967296" } > 2) Make sure all the daemons (MON specifically) are updated and has > the same build as OSDs one. > # ceph versions { "mon": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 4 }, "mgr": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 4 }, "osd": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 24 }, "mds": {}, "overall": { "ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)": 32 } } BTW I've opened offical bug: https://tracker.ceph.com/issues/43766 Regards, Martin > > Thanks, > > Igor > >>>

1 0

Radosgw/Objecter behaviour for homeless session
by Biswajeet Patra 23 Jan '20

23 Jan '20

Hi All, I have a query regarding objecter behaviour for homeless session. In situations when all OSDs containing copies (*let say replication 3*) of an object are down, the objecter assigns a homeless session (OSD=-1) to a client request. This request makes radosgw thread hang indefinitely as the data could never be served because all required OSDs are down. With multiple similar requests, all the radosgw threads gets exhausted and hanged indefinitely waiting for the OSDs to come up. This creates complete service unavailability as no rgw threads are present to process valid requests which could have been directed towards active PGs/OSDs. I think we should have behaviour in objecter or radosgw to terminate request and return early in case of a homeless session. Let me know your thoughts on this. Regards, Biswajeet -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

CephFS ghost usage/inodes
by Florian Pritz 23 Jan '20

23 Jan '20

Hi, When we tried putting some load on our test cephfs setup by restoring a backup in artifactory, we eventually ran out of space (around 95% used in `df` = 3.5TB) which caused artifactory to abort the restore and clean up. However, while a simple `find` no longer shows the files, `df` still claims that we have around 2.1TB of data on the cephfs. `df -i` also shows 2.4M used inodes. When using `du -sh` on a top-level mountpoint, I get 31G used, which is data that is still really here and which is expected to be here. Consequently, we also get the following warning: > MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average > pool cephfs_data objects per pg (38711) is more than 231.802 times cluster average (167) We are running ceph 14.2.5. We have snapshots enabled on cephfs, but there are currently no active snapshots listed by `ceph daemon mds.$hostname dump snaps --server` (see below). I can't say for sure if we created snapshots during the backup restore. > { > "last_snap": 39, > "last_created": 38, > "last_destroyed": 39, > "pending_noop": [], > "snaps": [], > "need_to_purge": {}, > "pending_update": [], > "pending_destroy": [] > } We only have a single CephFS. We use the pool_namespace xattr for our various directory trees on the cephfs. `ceph df` shows: > POOL ID STORED OBJECTS USED %USED MAX AVAIL > cephfs_data 6 2.1 TiB 2.48M 2.1 TiB 24.97 3.1 TiB `ceph daemon mds.$hostname perf dump | grep stray` shows: > "num_strays": 0, > "num_strays_delayed": 0, > "num_strays_enqueuing": 0, > "strays_created": 5097138, > "strays_enqueued": 5097138, > "strays_reintegrated": 0, > "strays_migrated": 0, `rados -p cephfs_data df` shows: > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR > cephfs_data 2.1 TiB 2477540 0 4955080 0 0 0 10699626 6.9 TiB 86911076 35 TiB 0 B 0 B > > total_objects 29718 > total_used 329 GiB > total_avail 7.5 TiB > total_space 7.8 TiB When I combine the usage and the free space shown by `df` we would exceed our cluster size. Our test cluster currently has 7.8TB total space with a replication size of 2 for all pools. With 2.1TB "used" on the cephfs according to `df` + 3.1TB being shows as "free" I get 5.2TB total size. This would mean >10TB of data when accounted for replication. Clearly this can't fit on a cluster with only 7.8TB of capacity. Do you have any ideas why we see so many objects and so much reported usage? Is there any way to fix this without recreating the cephfs? Florian -- Florian Pritz Research Industrial Systems Engineering (RISE) Forschungs-, Entwicklungs- und Großprojektberatung GmbH Concorde Business Park F 2320 Schwechat Austria E-Mail: florian.pritz(a)rise-world.com Web: www.rise-world.com Firmenbuch: FN 280353i Landesgericht Korneuburg UID: ATU62886416

3 15

Re: CephFS with cache-tier kernel-mount client unable to write (Nautilus)
by Ilya Dryomov 23 Jan '20

23 Jan '20

On Thu, Jan 23, 2020 at 2:36 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > On Wed, Jan 22, 2020 at 6:18 PM Hayashida, Mami <mami.hayashida(a)uky.edu> wrote: > > > > Thanks, Ilya. > > > > I just tried modifying the osd cap for client.testuser by getting rid of "tag cephfs data=cephfs_test" part and confirmed this key does work (i.e. lets the CephFS client read/write). It now reads: > > > > [client.testuser] > > key = XXXYYYYZZZ > > caps mds = "allow rw" > > caps mon = "allow r" > > caps osd = "allow rw" // previously "allow rw tag cephfs data=cephfs_test" > > > > I tried removing either "tag cephfs" or "data=cephfs_test" (and leaving the other), but neither worked. > > > > Now, here is my question: will not having the "allow rw tag cephfs data=<file system name>" (under osd caps) result in a security/privacy loophole in a production cluster? (I am still trying to assess whether having a Cache Tier behind CephFS is worth all the headaches...) > > It's probably not worth it. Unless you have a specific tiered > workload in mind and your cache pool is large enough for it, I'd > recommend staying away from cache tiering. > > "allow rw" for osd is only marginally more restrictive than > client.admin's "allow *", allowing the user to read/write every object > in the cluster. Scratch my reply about doing it by hand -- try the > following: > > $ ceph osd pool application enable cephfs-data-cache cephfs > $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test > $ ceph fs authorize cephfs_test ... (as before) > > You will see the same "allow rw tag cephfs data=cephfs_test" cap in > "ceph auth list" output, but it should allow accessing cephfs-data-cache. Dropping ceph-users(a)lists.ceph.com and resending to ceph-users(a)ceph.io. Thanks, Ilya

2 2