November 2019 - ceph-users

by naadiamirzablr＠gmail.com

NaadiaMirza &CO was born on the day she realised that creativity was a gift given to her family and business acumen inherited by her father. Thus the company and each project always has a part of him in it . Naadia claims she was lucky to inherit passion and zest which is most important when putting together a big day. The idea is to create spaces that are personal and uniquely yet magical.Today, the team of designers work with local artisans and creative talent to celebrate your special day. <a href="https://www.naadiamirza.com/best-wedding-planner-in-bangalore/">Top wedding decorators in bangalore</a>

4 years, 5 months

1
0
0 0

law schools in bangalore

by abbslaw123＠gmail.com

law courses in bangalore:The ABBS School of Law has started from the academic year 2018 – 19 under the aegis of Samagra Sikshana Samithi Trust. The Trust started the institution with a cherished ambition to impart legal education which is a very important step towards spreading legal awareness and empowering the helpless in the society. The College is affiliated to Karnataka State Law University, Hubballi and recognized by Bar Council of India, New Delhi. The college has a Governing Council duly constituted by Samagra Sikshana Samithi Trust. All the members of the Governing Council are accomplished academicians, legal luminaries, management experts and persons of eminence in public service. Governing Council meets regularly two times a year to review academic and administrative work. https://abbslaw.edu.in/

4 years, 5 months

1
0
0 0

OSD's addrvec, not getting msgr v2 address, PGs stuck unknown or peering

by Wesley Dillingham

Running 14.2.4 (but same issue observed on 14.2.2) we have a problem with, thankfully a testing cluster, where all pgs are failing to peer and are stuck in peering or unknown stale etc states. My working theory is that this is because the OSDs dont seem to be utilizing msgr v2 as "ceph osd find osd.NN" only lists the v1 in the addrvec. This is in contrast to our working 14.2.4 clusters where both v1 and v2 are listed. Our monitors via `ceph mon dump` show each mon running on v1 and v2 on the default ports (3300/6789) and I able to reach each of those ports on all the mons from a few test OSD nodes. OSD logs are filled with heartbeat_check: no reply from <IP> <OSD.XY> ever on either front or back I have attempted to modify the ceph.conf mon_host on the OSDs to use either the standard comma separate ip list and the new bracketed format and then restarting OSD daemons on a number of OSDs but it doesnt seem to impact the addrvec. My desire is to get the OSDs working on V2 and see if they are able to begin peering. How can I force the addrvec to update? Thanks. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com

4 years, 5 months

1
0
0 0

Re: mds crash loop

by Karsten Nielsen

I started a job that moved some files around in the cephfs cluster that resulted in the mds to go back into the crash loop. Logs are here: http://s3.foo-bar.dk/mds-dumps/mds.log-20191111 Any help would be appriciated. - Karsten -----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 14:20 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs to go. > > And what happened to the missing objects for the dirs. > > > > lost+found files are likely files that were deleted. you can keep the > lost+found dir for a while, then delete the 'lost+found' directory. > > for 'missing object' dirs, mv all of them to a temp directory, such as > /mnt/cephfs/missing_obj_dirs. > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > force recursive repair'. wait a minute, the rm -rf > /mnt/cephfs/missing_obj_dirs > > > Any tool that is able to do that ? > > > > Thanks > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 09:22 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > I have tracked down the root cause. See > https://tracker.ceph.com/issues/42675 > > > > > > Regards > > > Yan, Zheng > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 07:21 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > 13.2.6 > > > > > > > > > > It was running fine for a while but yesterday my mds went > into a > > > crash > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which > is > > > > > running > > > > > > > the > > > > > > > > > same crash loop. > > > > > > > > > > I am running ceph based on > https://hub.docker.com/r/ceph/daemon > > > > > version > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > apperently > > > > > only > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > 13.2.6' upgrade. > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are > in > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > I have had a similar problem with the cluster before where I went through > the > > > cycle of: > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > alternate metadata pool for recovery > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly because > > > cephfs is not that utilized so I thought it was safe to do as after the > upgrade > > > the cluster has not been used much, so data lose would be minimal - > apparently > > > I was wrong. > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > > > _fetched missing object for [dir 0x100013acfcb > > > /nextcloud/custom_apps/carnet/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/carnet) > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > > > _fetched missing object for [dir 0x100013a3156 > /nextcloud/custom_apps/mail/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/mail) > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40700] > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/passwords) > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40e00] > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc41500] > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS > map > > > to > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > - Karsten > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

4 years, 5 months

2
1
0 0

Where rocksdb on my OSD's?

by Andrey Groshev

Hello, Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working. # ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out. # ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde .... What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.

4 years, 5 months

2
4
0 0

Past_interval start interval mismatch (last_clean_epoch reported)

by Huseyin Cotuk

Hi all, I also hit the bug #24866 in my test environment. According to the logs, the last_clean_epoch in the specified OSD/PG is 17703, but the interval starts with 17895. So the OSD fails to start. There are some other OSDs in the same status. 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=18430/18431 n=1 ec=295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=1 lpr=18431 pi=[17895,18430)/3 crt=18388'4 lcod 0'0 unknown m=1 mbc={}] 18.51 past_intervals [17895,18430) start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the monitor. I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the reported epoch #? Thanks in advance. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

4 years, 5 months

3
2
0 0

Re: Wrong %USED and MAX AVAIL stats for pool

by Yordan Yordanov (Innologica)

The documentation states: https://docs.ceph.com/docs/mimic/rados/operations/monitoring/ The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section DOES NOT reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots. However in our case we are clearly seeing the USAGE field multiplying the total object sizes to the number of replicas. [root@blackmirror ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 34 TiB 46 TiB 46 TiB 58.10 TOTAL 80 TiB 34 TiB 46 TiB 46 TiB 58.10 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.05M 46 TiB 68.32 7.2 TiB bench 5 250 MiB 67 250 MiB 0 22 TiB [root@blackmirror ~]# rbd du -p one NAME PROVISIONED USED ... <TOTAL> 20 TiB 15 TiB This is causing several apps (including ceph dashboard) to display inaccurate percentages, because they calculate the total pool capacity as USED + MAX AVAIL, which in this case yields 53.2TB, which is way off. 7.2TB is about 13% of that, so we receive alarms and this is bugging us for quite some time now.

4 years, 5 months

4
4
0 0

Re: mds crash loop

by Karsten Nielsen

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 14:20 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs to go. > > And what happened to the missing objects for the dirs. > > > > lost+found files are likely files that were deleted. you can keep the > lost+found dir for a while, then delete the 'lost+found' directory. > > for 'missing object' dirs, mv all of them to a temp directory, such as > /mnt/cephfs/missing_obj_dirs. > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > force recursive repair'. wait a minute, the rm -rf > /mnt/cephfs/missing_obj_dirs I got this output from that command. 2019-11-08 14:27:53.110 7f0bda092700 1 mds.k8s-node-01 asok_command: scrub_path (starting...) 2019-11-08 14:27:53.110 7f0bda092700 1 mds.k8s-node-01 asok_command: scrub_path (complete) 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013a3156(/missing_obj_dirs/mail), rewriting it 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013a3156 (/missing_obj_dirs/mail) 2019-11-08 14:27:55.794 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013a3156 [...2,head] /missing_obj_dirs/mail/ auth v77565 dirtyparent f(v0 m2019-11-07 10:10:35.293698 12=3+9) n(v52 rc2019-11-08 14:08:21.585140 b21180195 2561=1820+741) (inest lock dirty) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@5},l=1793815 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc882300]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013a3156:[<0x100026d3cff/mail v77565>,<0x1/missing_obj_dirs v3025317>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 10:10:35.293698 12=3+9)","ondisk_value.rstat":"n(v0 rc2019-11-07 1 0:10:38.061742 b21180195 2561=1820+741)","memory_value.dirrstat":"f(v0 m2019-11-07 10:10:35.293698 12=3+9)","memory_value.rstat":"n(v52 rc2019-11-08 14:08:21.585140 b21180195 2561=1820+741)","error_str":""},"return_code":-61} 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013950d6(/missing_obj_dirs/empty_templates), rewriting it 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013950d6 (/missing_obj_dirs/empty_templates) 2019-11-08 14:27:55.794 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013950d6 [...2,head] /missing_obj_dirs/empty_templates/ auth v77569 ap=2+0 dirtyparent f(v0 m2019-09-17 19:13:12.226251 3=3+0) n(v0 rc2019-11-08 14:09:01.450030 b29712 4=3+1) (inest lock) (ifile sync->lock) (iversion lock) caps={1793815=pAsXs/pAsXsFs/-@4,1804200=pAsLsXs/pAsLsXsFs/AsLsXsFsx@31} | request=1 lock=0 dirfrag=0 caps=1 dirtyparent=1 scrubqueue=0 dirty=1 waiter=1 authpin=1 0x55d4ee870000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013950d6:[<0x100026d3cff/empty_templates v77569>,<0x1/missing_obj_dirs v3025317>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value. dirrstat":"f(v0 m2019-09-17 19:13:12.226251 3=3+0)","memory_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b29712 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2019-11-08 14:27:56.114 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013abb3c(/missing_obj_dirs/passwords), rewriting it 2019-11-08 14:27:56.114 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013abb3c (/missing_obj_dirs/passwords) 2019-11-08 14:27:56.114 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013abb3c [...2,head] /missing_obj_dirs/passwords/ auth v77567 dirtyparent f(v0 m2019-11-07 10:03:19.194734 8=1+7) n(v20 rc2019-11-08 14:08:30.565341 b5715351 413=349+64) (inest lock) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@6},l=1793815 | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc880000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013abb3c:[<0x100026d3cff/passwords v77567>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 10:03:19.194734 8=1+7)","ondisk_value.rstat":"n(v0 rc2019-11-07 10: 03:24.178813 b5715351 413=349+64)","memory_value.dirrstat":"f(v0 m2019-11-07 10:03:19.194734 8=1+7)","memory_value.rstat":"n(v20 rc2019-11-08 14:08:30.565341 b5715351 413=349+64)","error_str":""},"return_code":-61} 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013a2659(/missing_obj_dirs/richdocuments), rewriting it 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013a2659 (/missing_obj_dirs/richdocuments) 2019-11-08 14:27:56.254 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013a2659 [...2,head] /missing_obj_dirs/richdocuments/ auth v77561 dirtyparent f(v0 m2019-11-07 14:54:47.223440 19=9+10) n(v30 rc2019-11-08 14:08:03.876745 b3205323 261=241+20) (inest lock) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@3},l=1793815 | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc87e300]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013a2659:[<0x100026d3cff/richdocuments v77561>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 14:54:47.223440 19=9+10)","ondisk_value.rstat":"n(v0 rc20 19-11-07 14:54:47.223440 b3205323 261=241+20)","memory_value.dirrstat":"f(v0 m2019-11-07 14:54:47.223440 19=9+10)","memory_value.rstat":"n(v30 rc2019-11-08 14:08:03.876745 b3205323 261=241+20)","error_str":""},"return_code":-61} 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013acfcb(/missing_obj_dirs/carnet), rewriting it 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013acfcb (/missing_obj_dirs/carnet) 2019-11-08 14:27:56.254 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013acfcb [...2,head] /missing_obj_dirs/carnet/ auth v77563 ap=2+0 dirtyparent f(v0 m2019-10-21 11:03:29.113564 20=11+9) n(v31 rc2019-11-08 14:08:09.480870 b12005493 1999=1656+343) (inest lock) (ifile sync->lock) (iversion lock) caps={1793815=pAsXs/pAsXsFs/-@4,1794668=pAsXs/pAsXsFs/-@108394} | request=1 lock=0 dirfrag=0 caps=1 dirtyparent=1 scrubqueue=0 dirty=1 waiter=1 authpin=1 0x55d4dc889800]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013acfcb:[<0x100026d3cff/carnet v77563>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirrstat":"f (v0 m2019-10-21 11:03:29.113564 20=11+9)","memory_value.rstat":"n(v31 rc2019-11-08 14:08:09.480870 b12005493 1999=1656+343)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100026d3cff(/missing_obj_dirs), rewriting it 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100026d3cff (/missing_obj_dirs) 2019-11-08 14:27:56.258 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100026d3cff [...2,head] /missing_obj_dirs/ auth v3025512 pv3025514 ap=1+1 dirtyparent f(v0 m2019-11-08 14:09:01.450030 5=0+5) n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4066+1170) (inest lock w=1) (ifile excl) (iversion lock w=1) caps={1793815=pAsLsXsFsx/AsLsXsFsx@7},l=1793815 | request=0 lock=2 dirfrag=1 caps=1 dirtyparent=1 scrubqueue=1 dirty=1 authpin=1 0x55d4ede56700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100026d3cff:[<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-08 14:09:01.450030 5=0+5)","ondisk_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4 066+1170)","memory_value.dirrstat":"f(v0 m2019-11-08 14:09:01.450030 5=0+5)","memory_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4066+1170)","error_str":""},"return_code":-2} 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [INF] : scrub complete I am not sure what to read out of that. Other than data seems to be missing. > > > Any tool that is able to do that ? > > > > Thanks > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 09:22 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > I have tracked down the root cause. See > https://tracker.ceph.com/issues/42675 > > > > > > Regards > > > Yan, Zheng > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 07:21 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > 13.2.6 > > > > > > > > > > It was running fine for a while but yesterday my mds went > into a > > > crash > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which > is > > > > > running > > > > > > > the > > > > > > > > > same crash loop. > > > > > > > > > > I am running ceph based on > https://hub.docker.com/r/ceph/daemon > > > > > version > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > apperently > > > > > only > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > 13.2.6' upgrade. > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are > in > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > I have had a similar problem with the cluster before where I went through > the > > > cycle of: > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > alternate metadata pool for recovery > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly because > > > cephfs is not that utilized so I thought it was safe to do as after the > upgrade > > > the cluster has not been used much, so data lose would be minimal - > apparently > > > I was wrong. > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > > > _fetched missing object for [dir 0x100013acfcb > > > /nextcloud/custom_apps/carnet/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/carnet) > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > > > _fetched missing object for [dir 0x100013a3156 > /nextcloud/custom_apps/mail/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/mail) > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40700] > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/passwords) > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40e00] > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc41500] > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS > map > > > to > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > - Karsten > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

4 years, 6 months

1
0
0 0

Nautilus beast rgw 2 minute delay on startup???

by Marc Roos

Why is it taking for rgw 2 minutes to be available? Is there some internet lookup being done that is timing out when there is no default gw? [@ ~]# /usr/bin/radosgw -d -f --cluster ceph --name client.rgw2 --keyring=/etc/ceph/ceph.client.rgw2.keyring 2019-11-08 12:32:04.930 7fc9793fb780 0 framework: beast 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_port, val: 7480 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_private_key, val: /etc/pki/tls/private/rgw.xxxxxxx.xxx.key 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_certificate, val: /etc/pki/tls/certs/rgw.xxxxxxx.xxx.crt 2019-11-08 12:32:04.930 7fc9793fb780 0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process radosgw, pid 177335 2019-11-08 12:34:12.541 7fc9793fb780 0 starting handler: beast 2019-11-08 12:34:12.556 7fc9793fb780 1 mgrc service_daemon_register rgw.rgw2 metadata {arch=x86_64,ceph_release=nautilus,ceph_version=ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable),ceph_version_short=14.2.4,cpu=Intel,distro=centos,distro_descri ption=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=beast ssl_port=7480 ssl_private_key=/etc/pki/tls/private/rgw.xxxxxxx.xxx.key ssl_certificate=/etc/pki/tls/certs/rgw.xxxxxxx.xxx.crt,frontend_type#0=b east,hostname=,kernel_description=#1 SMP Fri Sep 13 22:55:44 UTC 2019,kernel_version=3.10.0-1062.1.1.el7.x86_64,mem_swap_kb=10485756,mem_ total_kb=98796656,num_handles=1,os=Linux,pid=177335,zone_id=91613b5d-6b9 b-4848-9546-2479a112cbff,zone_name=default,zonegroup_id=1a766e64-8d85-48 c4-9fae-96391d524a49,zonegroup_name=default}

4 years, 6 months

1
0
0 0

Proper way to replace an OSD with a shared SSD for db/wal

by Hayashida, Mami

We are running the Mimic version of Ceph (13.2.6) and I would like to know a proper way of replacing a defective OSD disk that has its DB and WAL on a separate SSD drive which is shared with 9 other OSDs. More specifically, the failing disk for osd.327 is on /dev/sdai and its wal/db are on /dev/sdc, which is partitioned into 10 LVs, holding wal/db for osd.320-329. When I deployed it, I used pv/vg/lvcreate commands to make VG named ssd1, LV named db320, db321 and so on. Then I used the ceph-deploy command from an admin node (`ceph-deploy osd create --block-db=ssd1/db327 --data=dev/sdai <node>`). My main question is what to do about the separate wal/db data as this page ( https://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-osds/) does not seem to address the issue. 1) Do I need to erase the wal/db data on the ssd1/db327 Logical Volume? If so, how should I do that? 2) Assuming 1) is taken care of (and the "old" OSD is destroyed and the "bad" hard drive has been physically replaced with a new one), does this command look correct? `ceph-volume lvm create --osd-id 327 --bluestore --data /dev/sdai --block.db ssd1/db327` *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

4 years, 6 months

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users November 2019