- ceph-users - lists.ceph.io

Re: mds crash loop
by Karsten Nielsen 11 Nov '19

11 Nov '19

I started a job that moved some files around in the cephfs cluster that resulted in the mds to go back into the crash loop. Logs are here: http://s3.foo-bar.dk/mds-dumps/mds.log-20191111 Any help would be appriciated. - Karsten -----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 14:20 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs to go. > > And what happened to the missing objects for the dirs. > > > > lost+found files are likely files that were deleted. you can keep the > lost+found dir for a while, then delete the 'lost+found' directory. > > for 'missing object' dirs, mv all of them to a temp directory, such as > /mnt/cephfs/missing_obj_dirs. > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > force recursive repair'. wait a minute, the rm -rf > /mnt/cephfs/missing_obj_dirs > > > Any tool that is able to do that ? > > > > Thanks > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 09:22 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > I have tracked down the root cause. See > https://tracker.ceph.com/issues/42675 > > > > > > Regards > > > Yan, Zheng > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 07:21 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > 13.2.6 > > > > > > > > > > It was running fine for a while but yesterday my mds went > into a > > > crash > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which > is > > > > > running > > > > > > > the > > > > > > > > > same crash loop. > > > > > > > > > > I am running ceph based on > https://hub.docker.com/r/ceph/daemon > > > > > version > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > apperently > > > > > only > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > 13.2.6' upgrade. > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are > in > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > I have had a similar problem with the cluster before where I went through > the > > > cycle of: > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > alternate metadata pool for recovery > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly because > > > cephfs is not that utilized so I thought it was safe to do as after the > upgrade > > > the cluster has not been used much, so data lose would be minimal - > apparently > > > I was wrong. > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > > > _fetched missing object for [dir 0x100013acfcb > > > /nextcloud/custom_apps/carnet/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/carnet) > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > > > _fetched missing object for [dir 0x100013a3156 > /nextcloud/custom_apps/mail/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/mail) > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40700] > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/passwords) > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40e00] > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc41500] > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS > map > > > to > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > - Karsten > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Where rocksdb on my OSD's?
by Andrey Groshev 11 Nov '19

11 Nov '19

Hello, Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working. # ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out. # ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde .... What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.

2 4

Past_interval start interval mismatch (last_clean_epoch reported)
by Huseyin Cotuk 10 Nov '19

10 Nov '19

Hi all, I also hit the bug #24866 in my test environment. According to the logs, the last_clean_epoch in the specified OSD/PG is 17703, but the interval starts with 17895. So the OSD fails to start. There are some other OSDs in the same status. 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=18430/18431 n=1 ec=295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=1 lpr=18431 pi=[17895,18430)/3 crt=18388'4 lcod 0'0 unknown m=1 mbc={}] 18.51 past_intervals [17895,18430) start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the monitor. I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the reported epoch #? Thanks in advance. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

3 2

Re: Wrong %USED and MAX AVAIL stats for pool
by Yordan Yordanov (Innologica) 09 Nov '19

09 Nov '19

The documentation states: https://docs.ceph.com/docs/mimic/rados/operations/monitoring/ The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section DOES NOT reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots. However in our case we are clearly seeing the USAGE field multiplying the total object sizes to the number of replicas. [root@blackmirror ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 34 TiB 46 TiB 46 TiB 58.10 TOTAL 80 TiB 34 TiB 46 TiB 46 TiB 58.10 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.05M 46 TiB 68.32 7.2 TiB bench 5 250 MiB 67 250 MiB 0 22 TiB [root@blackmirror ~]# rbd du -p one NAME PROVISIONED USED ... <TOTAL> 20 TiB 15 TiB This is causing several apps (including ceph dashboard) to display inaccurate percentages, because they calculate the total pool capacity as USED + MAX AVAIL, which in this case yields 53.2TB, which is way off. 7.2TB is about 13% of that, so we receive alarms and this is bugging us for quite some time now.

4 4

Re: mds crash loop
by Karsten Nielsen 08 Nov '19

08 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 14:20 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs to go. > > And what happened to the missing objects for the dirs. > > > > lost+found files are likely files that were deleted. you can keep the > lost+found dir for a while, then delete the 'lost+found' directory. > > for 'missing object' dirs, mv all of them to a temp directory, such as > /mnt/cephfs/missing_obj_dirs. > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > force recursive repair'. wait a minute, the rm -rf > /mnt/cephfs/missing_obj_dirs I got this output from that command. 2019-11-08 14:27:53.110 7f0bda092700 1 mds.k8s-node-01 asok_command: scrub_path (starting...) 2019-11-08 14:27:53.110 7f0bda092700 1 mds.k8s-node-01 asok_command: scrub_path (complete) 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013a3156(/missing_obj_dirs/mail), rewriting it 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013a3156 (/missing_obj_dirs/mail) 2019-11-08 14:27:55.794 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013a3156 [...2,head] /missing_obj_dirs/mail/ auth v77565 dirtyparent f(v0 m2019-11-07 10:10:35.293698 12=3+9) n(v52 rc2019-11-08 14:08:21.585140 b21180195 2561=1820+741) (inest lock dirty) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@5},l=1793815 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc882300]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013a3156:[<0x100026d3cff/mail v77565>,<0x1/missing_obj_dirs v3025317>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 10:10:35.293698 12=3+9)","ondisk_value.rstat":"n(v0 rc2019-11-07 1 0:10:38.061742 b21180195 2561=1820+741)","memory_value.dirrstat":"f(v0 m2019-11-07 10:10:35.293698 12=3+9)","memory_value.rstat":"n(v52 rc2019-11-08 14:08:21.585140 b21180195 2561=1820+741)","error_str":""},"return_code":-61} 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013950d6(/missing_obj_dirs/empty_templates), rewriting it 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013950d6 (/missing_obj_dirs/empty_templates) 2019-11-08 14:27:55.794 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013950d6 [...2,head] /missing_obj_dirs/empty_templates/ auth v77569 ap=2+0 dirtyparent f(v0 m2019-09-17 19:13:12.226251 3=3+0) n(v0 rc2019-11-08 14:09:01.450030 b29712 4=3+1) (inest lock) (ifile sync->lock) (iversion lock) caps={1793815=pAsXs/pAsXsFs/-@4,1804200=pAsLsXs/pAsLsXsFs/AsLsXsFsx@31} | request=1 lock=0 dirfrag=0 caps=1 dirtyparent=1 scrubqueue=0 dirty=1 waiter=1 authpin=1 0x55d4ee870000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013950d6:[<0x100026d3cff/empty_templates v77569>,<0x1/missing_obj_dirs v3025317>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value. dirrstat":"f(v0 m2019-09-17 19:13:12.226251 3=3+0)","memory_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b29712 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2019-11-08 14:27:56.114 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013abb3c(/missing_obj_dirs/passwords), rewriting it 2019-11-08 14:27:56.114 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013abb3c (/missing_obj_dirs/passwords) 2019-11-08 14:27:56.114 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013abb3c [...2,head] /missing_obj_dirs/passwords/ auth v77567 dirtyparent f(v0 m2019-11-07 10:03:19.194734 8=1+7) n(v20 rc2019-11-08 14:08:30.565341 b5715351 413=349+64) (inest lock) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@6},l=1793815 | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc880000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013abb3c:[<0x100026d3cff/passwords v77567>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 10:03:19.194734 8=1+7)","ondisk_value.rstat":"n(v0 rc2019-11-07 10: 03:24.178813 b5715351 413=349+64)","memory_value.dirrstat":"f(v0 m2019-11-07 10:03:19.194734 8=1+7)","memory_value.rstat":"n(v20 rc2019-11-08 14:08:30.565341 b5715351 413=349+64)","error_str":""},"return_code":-61} 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013a2659(/missing_obj_dirs/richdocuments), rewriting it 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013a2659 (/missing_obj_dirs/richdocuments) 2019-11-08 14:27:56.254 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013a2659 [...2,head] /missing_obj_dirs/richdocuments/ auth v77561 dirtyparent f(v0 m2019-11-07 14:54:47.223440 19=9+10) n(v30 rc2019-11-08 14:08:03.876745 b3205323 261=241+20) (inest lock) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@3},l=1793815 | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc87e300]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013a2659:[<0x100026d3cff/richdocuments v77561>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 14:54:47.223440 19=9+10)","ondisk_value.rstat":"n(v0 rc20 19-11-07 14:54:47.223440 b3205323 261=241+20)","memory_value.dirrstat":"f(v0 m2019-11-07 14:54:47.223440 19=9+10)","memory_value.rstat":"n(v30 rc2019-11-08 14:08:03.876745 b3205323 261=241+20)","error_str":""},"return_code":-61} 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013acfcb(/missing_obj_dirs/carnet), rewriting it 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013acfcb (/missing_obj_dirs/carnet) 2019-11-08 14:27:56.254 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013acfcb [...2,head] /missing_obj_dirs/carnet/ auth v77563 ap=2+0 dirtyparent f(v0 m2019-10-21 11:03:29.113564 20=11+9) n(v31 rc2019-11-08 14:08:09.480870 b12005493 1999=1656+343) (inest lock) (ifile sync->lock) (iversion lock) caps={1793815=pAsXs/pAsXsFs/-@4,1794668=pAsXs/pAsXsFs/-@108394} | request=1 lock=0 dirfrag=0 caps=1 dirtyparent=1 scrubqueue=0 dirty=1 waiter=1 authpin=1 0x55d4dc889800]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013acfcb:[<0x100026d3cff/carnet v77563>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirrstat":"f (v0 m2019-10-21 11:03:29.113564 20=11+9)","memory_value.rstat":"n(v31 rc2019-11-08 14:08:09.480870 b12005493 1999=1656+343)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100026d3cff(/missing_obj_dirs), rewriting it 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100026d3cff (/missing_obj_dirs) 2019-11-08 14:27:56.258 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100026d3cff [...2,head] /missing_obj_dirs/ auth v3025512 pv3025514 ap=1+1 dirtyparent f(v0 m2019-11-08 14:09:01.450030 5=0+5) n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4066+1170) (inest lock w=1) (ifile excl) (iversion lock w=1) caps={1793815=pAsLsXsFsx/AsLsXsFsx@7},l=1793815 | request=0 lock=2 dirfrag=1 caps=1 dirtyparent=1 scrubqueue=1 dirty=1 authpin=1 0x55d4ede56700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100026d3cff:[<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-08 14:09:01.450030 5=0+5)","ondisk_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4 066+1170)","memory_value.dirrstat":"f(v0 m2019-11-08 14:09:01.450030 5=0+5)","memory_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4066+1170)","error_str":""},"return_code":-2} 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [INF] : scrub complete I am not sure what to read out of that. Other than data seems to be missing. > > > Any tool that is able to do that ? > > > > Thanks > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 09:22 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > I have tracked down the root cause. See > https://tracker.ceph.com/issues/42675 > > > > > > Regards > > > Yan, Zheng > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 07:21 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > 13.2.6 > > > > > > > > > > It was running fine for a while but yesterday my mds went > into a > > > crash > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which > is > > > > > running > > > > > > > the > > > > > > > > > same crash loop. > > > > > > > > > > I am running ceph based on > https://hub.docker.com/r/ceph/daemon > > > > > version > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > apperently > > > > > only > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > 13.2.6' upgrade. > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are > in > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > I have had a similar problem with the cluster before where I went through > the > > > cycle of: > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > alternate metadata pool for recovery > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly because > > > cephfs is not that utilized so I thought it was safe to do as after the > upgrade > > > the cluster has not been used much, so data lose would be minimal - > apparently > > > I was wrong. > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > > > _fetched missing object for [dir 0x100013acfcb > > > /nextcloud/custom_apps/carnet/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/carnet) > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > > > _fetched missing object for [dir 0x100013a3156 > /nextcloud/custom_apps/mail/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/mail) > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40700] > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/passwords) > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40e00] > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc41500] > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS > map > > > to > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > - Karsten > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Nautilus beast rgw 2 minute delay on startup???
by Marc Roos 08 Nov '19

08 Nov '19

Why is it taking for rgw 2 minutes to be available? Is there some internet lookup being done that is timing out when there is no default gw? [@ ~]# /usr/bin/radosgw -d -f --cluster ceph --name client.rgw2 --keyring=/etc/ceph/ceph.client.rgw2.keyring 2019-11-08 12:32:04.930 7fc9793fb780 0 framework: beast 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_port, val: 7480 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_private_key, val: /etc/pki/tls/private/rgw.xxxxxxx.xxx.key 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_certificate, val: /etc/pki/tls/certs/rgw.xxxxxxx.xxx.crt 2019-11-08 12:32:04.930 7fc9793fb780 0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process radosgw, pid 177335 2019-11-08 12:34:12.541 7fc9793fb780 0 starting handler: beast 2019-11-08 12:34:12.556 7fc9793fb780 1 mgrc service_daemon_register rgw.rgw2 metadata {arch=x86_64,ceph_release=nautilus,ceph_version=ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable),ceph_version_short=14.2.4,cpu=Intel,distro=centos,distro_descri ption=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=beast ssl_port=7480 ssl_private_key=/etc/pki/tls/private/rgw.xxxxxxx.xxx.key ssl_certificate=/etc/pki/tls/certs/rgw.xxxxxxx.xxx.crt,frontend_type#0=b east,hostname=,kernel_description=#1 SMP Fri Sep 13 22:55:44 UTC 2019,kernel_version=3.10.0-1062.1.1.el7.x86_64,mem_swap_kb=10485756,mem_ total_kb=98796656,num_handles=1,os=Linux,pid=177335,zone_id=91613b5d-6b9 b-4848-9546-2479a112cbff,zone_name=default,zonegroup_id=1a766e64-8d85-48 c4-9fae-96391d524a49,zonegroup_name=default}

1 0

Proper way to replace an OSD with a shared SSD for db/wal
by Hayashida, Mami 08 Nov '19

08 Nov '19

We are running the Mimic version of Ceph (13.2.6) and I would like to know a proper way of replacing a defective OSD disk that has its DB and WAL on a separate SSD drive which is shared with 9 other OSDs. More specifically, the failing disk for osd.327 is on /dev/sdai and its wal/db are on /dev/sdc, which is partitioned into 10 LVs, holding wal/db for osd.320-329. When I deployed it, I used pv/vg/lvcreate commands to make VG named ssd1, LV named db320, db321 and so on. Then I used the ceph-deploy command from an admin node (`ceph-deploy osd create --block-db=ssd1/db327 --data=dev/sdai <node>`). My main question is what to do about the separate wal/db data as this page ( https://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-osds/) does not seem to address the issue. 1) Do I need to erase the wal/db data on the ssd1/db327 Logical Volume? If so, how should I do that? 2) Assuming 1) is taken care of (and the "old" OSD is destroyed and the "bad" hard drive has been physically replaced with a new one), does this command look correct? `ceph-volume lvm create --osd-id 327 --bluestore --data /dev/sdai --block.db ssd1/db327` *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

2 1

best schools in sarjapur road
by foundationschool school 08 Nov '19

08 Nov '19

The Foundation schools were established in 2009. The Gunjur branch has become one of the top CBSE schools in varthur, Gunjur, Whitefield and best school in Sarjapur Road areas in a short period of time. The schools follow a methodology that helps children through hands on activities and experiential learning. Children at The Foundation School are seen not as consumers who need external entertainment, but as producers who interact with various learning material with their own hands; construct knowledge and gain immense satisfaction from the process. In keeping with this methodology, The Foundation School follows the Montessori Philosophy at the pre-primary level, and experiential learning from grade 1 onwards. All of these make The Foundation School, one of the foremost leading and progressive schools in Varthur, Gunjur, Schools in Whitefield and also schools near sarjapur Road areas of Bangalore. https://www.foundationschoolindia.com/

1 0

mgr daemons becoming unresponsive
by Oliver Freyermuth 07 Nov '19

07 Nov '19

Dear Cephers, this is a 14.2.4 cluster with device health metrics enabled - since about a day, all mgr daemons go "silent" on me after a few hours, i.e. "ceph -s" shows: cluster: id: 269cf2b2-7e7c-4ceb-bd1b-a33d915ceee9 health: HEALTH_WARN no active mgr 1/3 mons down, quorum mon001,mon002 services: mon: 3 daemons, quorum mon001,mon002 (age 57m), out of quorum: mon003 mgr: no daemons active (since 56m) ... (the third mon has a planned outage and will come back in a few days) Checking the logs of the mgr daemons, I find some "reset" messages at the time when it goes "silent", first for the first mgr: 2019-11-01 21:34:40.286 7f2df6a6b700 0 log_channel(cluster) log [DBG] : pgmap v1798: 1585 pgs: 1585 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail 2019-11-01 21:34:41.458 7f2e0d59b700 0 client.0 ms_handle_reset on v2:10.160.16.1:6800/401248 2019-11-01 21:34:42.287 7f2df6a6b700 0 log_channel(cluster) log [DBG] : pgmap v1799: 1585 pgs: 1585 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail and a bit later, on the standby mgr: 2019-11-01 22:18:14.892 7f7bcc8ae700 0 log_channel(cluster) log [DBG] : pgmap v1798: 1585 pgs: 166 active+clean+snaptrim, 858 active+clean+snaptrim_wait, 561 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail 2019-11-01 22:18:16.022 7f7be9e72700 0 client.0 ms_handle_reset on v2:10.160.16.2:6800/352196 2019-11-01 22:18:16.893 7f7bcc8ae700 0 log_channel(cluster) log [DBG] : pgmap v1799: 1585 pgs: 166 active+clean+snaptrim, 858 active+clean+snaptrim_wait, 561 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail Interestingly, the dashboard still works, but presents outdated information, and for example zero I/O going on. I believe this started to happen mainly after the third mon went into the known downtime, but I am not fully sure if this was the trigger, since the cluster is still growing. It may also have been the addition of 24 more OSDs. I also find other messages in the mgr logs which seem problematic, but I am not sure they are related: ------------------------------ 2019-11-01 21:17:09.849 7f2df4266700 0 mgr[devicehealth] Error reading OMAP: [errno 22] Failed to operate read op for oid Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 396, in put_device_metrics ioctx.operate_read_op(op, devid) File "rados.pyx", line 516, in rados.requires.wrapper.validate_func (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUIL D/ceph-14.2.4/build/src/pybind/rados/pyrex/rados.c:4721) File "rados.pyx", line 3474, in rados.Ioctx.operate_read_op (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/build/src/pybind/rados/pyrex/rados.c:36554) InvalidArgumentError: [errno 22] Failed to operate read op for oid ------------------------------ or: ------------------------------ 2019-11-01 21:33:53.977 7f7bd38bc700 0 mgr[devicehealth] Fail to parse JSON result from daemon osd.51 () 2019-11-01 21:33:53.978 7f7bd38bc700 0 mgr[devicehealth] Fail to parse JSON result from daemon osd.52 () 2019-11-01 21:33:53.979 7f7bd38bc700 0 mgr[devicehealth] Fail to parse JSON result from daemon osd.53 () ------------------------------ The reason why I am cautious about the health metrics is that I observed a crash when trying to query them: ------------------------------ 2019-11-01 20:21:23.661 7fa46314a700 0 log_channel(audit) log [DBG] : from='client.174136 -' entity='client.admin' cmd=[{"prefix": "device get-health-metrics", "devid": "osd.11", "target": ["mgr", ""]}]: dispatch 2019-11-01 20:21:23.661 7fa46394b700 0 mgr[devicehealth] handle_command 2019-11-01 20:21:23.663 7fa46394b700 -1 *** Caught signal (Segmentation fault) ** in thread 7fa46394b700 thread_name:mgr-fin ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (()+0xf5f0) [0x7fa488cee5f0] 2: (PyEval_EvalFrameEx()+0x1a9) [0x7fa48aeb50f9] 3: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] 4: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] 5: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] 6: (PyEval_EvalCodeEx()+0x7ed) [0x7fa48aebe08d] 7: (()+0x709c8) [0x7fa48ae479c8] 8: (PyObject_Call()+0x43) [0x7fa48ae22ab3] 9: (()+0x5aaa5) [0x7fa48ae31aa5] 10: (PyObject_Call()+0x43) [0x7fa48ae22ab3] 11: (()+0x4bb95) [0x7fa48ae22b95] 12: (PyObject_CallMethod()+0xbb) [0x7fa48ae22ecb] 13: (ActivePyModule::handle_command(std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::buffer::v14_2_0::list const&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)+0x20e) [0x55c3c1fefc5e] 14: (()+0x16c23d) [0x55c3c204023d] 15: (FunctionContext::finish(int)+0x2c) [0x55c3c2001eac] 16: (Context::complete(int)+0x9) [0x55c3c1ffe659] 17: (Finisher::finisher_thread_entry()+0x156) [0x7fa48b439cc6] 18: (()+0x7e65) [0x7fa488ce6e65] 19: (clone()+0x6d) [0x7fa48799488d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ------------------------------ I have issued: ceph device monitoring off for now and will keep waiting to see if mgrs go silent again. If there are any better ideas or this issue is known, let me know. Cheers, Oliver

8 25

RGW compression not compressing
by Bryan Stillwell 07 Nov '19

07 Nov '19

Today I tried enabling RGW compression on a Nautilus 14.2.4 test cluster and found it wasn't doing any compression at all. I figure I must have missed something in the docs, but I haven't been able to find out what that is and could use some help. This is the command I used to enable zlib-based compression: # radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zlib I then restarted the radosgw process to activate the change (there's only 1 RGW in this test cluster): # systemctl restart ceph-radosgw(a)radosgw.$(hostname -s) I verified it was enabled correctly with: # radosgw-admin zone get | jq -r '.placement_pools' [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data", "compression_type": "zlib" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } } ] Before starting the test I had nothing in the default.rgw.buckets.data pool: # ceph df | grep default.rgw.buckets.data default.rgw.buckets.data 16 0 B 0 0 B 0 230 TiB I then tried uploading a 1GiB file consisting of all 0s from /dev/zero with both S3 (boto) and Swift (swiftclient) and each time they used 1GiB of data on the cluster: # ceph df -f json | jq -r '.' | grep -A9 default.rgw.buckets.data "name": "default.rgw.buckets.data", "id": 16, "stats": { "stored": 1073741824, "objects": 256, "kb_used": 1048576, "bytes_used": 1073741824, "percent_used": 1.4138463484414387e-06, "max_avail": 253148744646656 } The same thing was reported by bucket stats: # radosgw-admin bucket stats --bucket=bs-test | jq -r '.usage' { "rgw.main": { "size": 1073741824, "size_actual": 1073741824, "size_utilized": 1073741824, "size_kb": 1048576, "size_kb_actual": 1048576, "size_kb_utilized": 1048576, "num_objects": 1 } } What am I missing? Thanks, Bryan

2 4