- ceph-users - lists.ceph.io

scan_links crashing
by Guilherme Geronimo 02 Mar '20

02 Mar '20

Hey guys, I'm trying to solve some Lost+Found errors, but when I try to run the "scan_link" command, it crashes. Any tip? Cheers! ceph cluster version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) cephfs-data-scan version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) root@deployer:/etc/ceph# cephfs-data-scan scan_links 2020-03-02 01:52:42.876 7fda150fa700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2020-03-02 01:52:43.664 7fda23d26f40 -1 datascan.scan_links: Error getting omap from '10000af78b3.00000000': (2) No such file or directory Error ((2) No such file or directory) root@deployer:/etc/ceph# cephfs-data-scan --version ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)

1 0

MAX AVAIL and RAW AVAIL
by konstantin.ilyasov＠mediascope.net 02 Mar '20

02 Mar '20

We have some inexplicable situation. We have ceph cluster on 14.2.4. About 14 nodes with 12 disks (4Tb) in each node. But command ceph df returns for us next report: # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 611 TiB 155 TiB 455 TiB 456 TiB 74.60 TOTAL 611 TiB 155 TiB 455 TiB 456 TiB 74.60 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL poolera01 20 238 TiB 84.59M 362 TiB 76.39 75 TiB poolera01md 21 17 GiB 58.33k 17 GiB 0.01 37 TiB poolera02dt 25 71 TiB 24.56M 92 TiB 45.02 90 TiB poolera02md 26 749 MiB 42.23k 1.2 GiB 0 37 TiB poolera01 - pool Erasure-coded 4+2 poolera02dt - pool Erasure-coded 8+2 poolera01md and poolera02md - both replicated size 3 We are using 2 cephfs. It seems, that all our pools can only utilize about 111TiB raw capacity. But we have 155 TiB! We have enabled balancer with mode upmap # ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Can someone explain why ceph df show less MAX AVAIL than possible from AVAIL 155 TiB?

1 0

Re: Is it ok to add a luminous ceph-disk osd to nautilus still?
by Marc Roos 01 Mar '20

01 Mar '20

I did not have time to convert all drives to lvm yet, so I would like to stick to the use of the partition until I have time to change everything. -----Original Message----- Sent: 01 March 2020 18:17 Subject: Re: [ceph-users] Is it ok to add a luminous ceph-disk osd to nautilus still? So use ceph-volume. ?? The nautilus release notes explain why. > On Mar 1, 2020, at 9:02 AM, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > ceph-disk is not available in Nautilus.elease > > why scrub first? It is a new disk not having any data yet. Scrubbing > is verifying pg's not? > > I just created a vm on the ceph node where I want to add this osd. Did > a passthru of the disk and installed a few rpm's with nodeps to get > the ceph-disk command. > > > > -----Original Message----- > Sent: 01 March 2020 17:47 > Subject: Re: [ceph-users] Is it ok to add a luminous ceph-disk osd to > nautilus still? > > Ensure that it gets scrubbed at least once by Luminous first. But how > and why are you doing this ? Why not use Nautilus binaries ? > >>> On Mar 1, 2020, at 8:36 AM, Marc Roos <M.Roos(a)f1-outsourcing.eu> >> wrote: >> >> >> If I create and osd with luminous 12.0.3 binaries, can I just add it >> to an existing Nautilus cluster? >> >> I sort of did this already, just wondered if there are any drawbacks. >> >> >> [@test2 software]# ceph-disk prepare --bluestore --zap-disk /dev/sdb >> Creating new GPT entries. >> GPT data structures destroyed! You may now partition the disk using >> fdisk or other utilities. >> Creating new GPT entries. >> The operation has completed successfully. >> Setting name! >> partNum is 0 >> REALLY setting name! >> The operation has completed successfully. >> Setting name! >> partNum is 1 >> REALLY setting name! >> The operation has completed successfully. >> The operation has completed successfully. >> meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6400 >> blks >> = sectsz=512 attr=2, projid32bit=1 >> = crc=1 finobt=0, sparse=0 >> data = bsize=4096 blocks=25600, imaxpct=25 >> = sunit=0 swidth=0 blks >> naming =version 2 bsize=4096 ascii-ci=0 ftype=1 >> log =internal log bsize=4096 blocks=864, version=2 >> = sectsz=512 sunit=0 blks, > lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> The operation has completed successfully. >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > >

1 0

Re: Is it ok to add a luminous ceph-disk osd to nautilus still?
by Marc Roos 01 Mar '20

01 Mar '20

ceph-disk is not available in Nautilus. why scrub first? It is a new disk not having any data yet. Scrubbing is verifying pg's not? I just created a vm on the ceph node where I want to add this osd. Did a passthru of the disk and installed a few rpm's with nodeps to get the ceph-disk command. -----Original Message----- Sent: 01 March 2020 17:47 Subject: Re: [ceph-users] Is it ok to add a luminous ceph-disk osd to nautilus still? Ensure that it gets scrubbed at least once by Luminous first. But how and why are you doing this ? Why not use Nautilus binaries ? > On Mar 1, 2020, at 8:36 AM, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > If I create and osd with luminous 12.0.3 binaries, can I just add it > to an existing Nautilus cluster? > > I sort of did this already, just wondered if there are any drawbacks. > > > [@test2 software]# ceph-disk prepare --bluestore --zap-disk /dev/sdb > Creating new GPT entries. > GPT data structures destroyed! You may now partition the disk using > fdisk or other utilities. > Creating new GPT entries. > The operation has completed successfully. > Setting name! > partNum is 0 > REALLY setting name! > The operation has completed successfully. > Setting name! > partNum is 1 > REALLY setting name! > The operation has completed successfully. > The operation has completed successfully. > meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6400 > blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 > data = bsize=4096 blocks=25600, imaxpct=25 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0 ftype=1 > log =internal log bsize=4096 blocks=864, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > The operation has completed successfully. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Is it ok to add a luminous ceph-disk osd to nautilus still?
by Marc Roos 01 Mar '20

01 Mar '20

If I create and osd with luminous 12.0.3 binaries, can I just add it to an existing Nautilus cluster? I sort of did this already, just wondered if there are any drawbacks. [@test2 software]# ceph-disk prepare --bluestore --zap-disk /dev/sdb Creating new GPT entries. GPT data structures destroyed! You may now partition the disk using fdisk or other utilities. Creating new GPT entries. The operation has completed successfully. Setting name! partNum is 0 REALLY setting name! The operation has completed successfully. Setting name! partNum is 1 REALLY setting name! The operation has completed successfully. The operation has completed successfully. meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6400 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=25600, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=864, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 The operation has completed successfully.

1 0

Re: Problems with ragosgw
by Eric Ivancich 29 Feb '20

29 Feb '20

Hi Mohamed, > On Jan 22, 2020, at 10:05 AM, mohamed zayan <mohamed.zayan19(a)gmail.com <mailto:mohamed.zayan19@gmail.com>> wrote: > > Currently I have a cluster of 2 nodes on two raspberrypi 3 devices. > pi1 is admin/mon/mgr/osd > > pi2 is osd > > I am currently trying to run radosgw on pi2. I have failed multiple times > > /var/lib/ceph/radosgw# /usr/bin/radosgw -f --cluster ceph --name > client.rgw.pi2 --setuser ceph --setgroup ceph > Thread::try_create(): pthread_create failed with error > 11/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc <http://thread.cc/>: > In function 'void Thread::create(const char *, size_t)' thread > 71114000 time 2020-01-22 14:58:13.793803 > /tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc <http://thread.cc/>: > 152: FAILED assert(ret == 0) I’m not too familiar with the Raspberry Pi platform or Raspbian. But there are some clues in the output you provided, specifically this text — “ Thread::try_create(): pthread_create failed with error 11”. I assume these low valued error codes are consistent between Fedora and Raspbian, and so the code 11 means: #define EAGAIN 11 /* Try again */ I found the following: https://stackoverflow.com/questions/47078106/pthread-create-fails-with-eaga… <https://stackoverflow.com/questions/47078106/pthread-create-fails-with-eaga…> So that seems to leave 3 possibilities: 1. RGW is asking for more resources that your platform is able to provide. a. There are limits as to the number of threads. b. There may be a memory limit, since each thread needs to maintain its own stack. 2. There is a bug in RGW where completed threads are not thread_join’d by their parents. a. This seems unlikely as this appears to happen during start-up before threads are likely done with their work. 3. Florian Weimer identifies a kernel bug. I suspect it’s #1, so you might want to try reducing the number of threads rgw uses by lowering the value of rgw_thread_pool_size in your configuration. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA

1 0

recover ceph-mon
by xsempresu＠gmail.com 29 Feb '20

29 Feb '20

I have test one-node ceph cluster with 4 osds under assumption to add the second node just before production. Linux 4.19.0-6-amd64 - debian 10 - ceph version 12.2.11 Unfortunately, system drive was broken before it. I recovered the system from full backup. Since no changes was performed to the cluster configuration after that backup, I hoped that it works. For the reasons I can't understand first few seconds after boot ceph status was OK (134 active+clean, 2 active+clean+scrubbing+deep), but a minute later status changed to: # ceph status cluster: id: e02f2885-946b-46c8-91d5-146dd724ecaf health: HEALTH_WARN 1 filesystem is degraded 2 osds down 1 slice (2 osds) down Reduced data availability: 136 pgs inactive, 15 pgs peering services: mon: 1 daemons, quorum rbd0 mgr: rbd0(active) mds: fs-1/1/1 up {0=rbd0=up:replay} osd: 5 osds: 1 up, 3 in data: pools: 2 pools, 136 pgs objects: 118.53k objects, 429GiB usage: 7.15TiB used, 3.77TiB / 10.9TiB avail pgs: 88.971% pgs unknown 11.029% pgs not active 121 unknown 15 peering # ceph osd dump epoch 1983 fsid e02f2885-946b-46c8-91d5-146dd724ecaf created 2019-08-16 15:14:07.783009 modified 2020-02-29 13:55:39.212461 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 27 full_ratio 0.97 backfillfull_ratio 0.94 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client jewel require_osd_release luminous pool 1 'fs_data' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 1595 flags hashpspool stripe_width 0 application cephfs pool 2 'fs_meta' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1595 flags hashpspool stripe_width 0 application cephfs max_osd 5 osd.0 down out weight 0 up_from 1970 up_thru 1973 down_at 1975 last_clean_interval [1949,1963) 192.168.101.111:6806/440 192.168.101.111:6807/440 192.168.101.111:6808/440 192.168.101.111:6809/440 autoout,exists 78eaeb63-47c9-4962-b8ff-46607921f4f6 osd.1 down in weight 1 up_from 1970 up_thru 1970 down_at 1975 last_clean_interval [1952,1963) 192.168.101.111:6801/439 192.168.101.111:6810/439 192.168.101.111:6811/439 192.168.101.111:6812/439 exists c4c4c85d-f537-4199-823b-b7ab01c78f03 osd.2 down in weight 1 up_from 1969 up_thru 1975 down_at 1976 last_clean_interval [1946,1963) 192.168.101.111:6802/441 192.168.101.111:6803/441 192.168.101.111:6804/441 192.168.101.111:6805/441 exists bd66a9c3-bfa4-4352-816e-2e4cd86389f3 osd.3 down out weight 0 up_from 1617 up_thru 1619 down_at 1631 last_clean_interval [1602,1610) 192.168.101.111:6805/933 192.168.101.111:6806/933 192.168.101.111:6807/933 192.168.101.111:6808/933 exists f247115b-c6d5-49b1-9b0e-e799c50be379 osd.4 up in weight 1 up_from 1973 up_thru 1973 down_at 1972 last_clean_interval [1956,1963) 192.168.101.111:6813/442 192.168.101.111:6814/442 192.168.101.111:6815/442 192.168.101.111:6816/442 exists,up c208221e-1228-4247-a742-0c16ce01d38f blacklist 192.168.101.111:6800/2636437603 expires 2020-03-01 13:26:01.809132 "ceph pg query" of any PG didn't response. I can't find any errors in journalctl or in /var/log/ceph/* I wonder why only osd 4 up, what means outoout, why 15 pgs are peering, where to search detail information, is it a way to restore data. Please help me to understand what happend and how to restore data if it possible.

1 0

Question about ceph-balancer and OSD reweights
by shubjero 29 Feb '20

29 Feb '20

Hi all, I'm running a Ceph Mimic cluster 13.2.6 and we use the ceph-balancer in upmap mode. This cluster is fairly old and pre-Mimic we used to set osd reweights to balance the standard deviation of the cluster. Since moving to Mimic about 9 months ago I enabled the ceph-balancer with upmap mode and let it do its thing but I did not think about setting the previously modified reweights back to 1.00000 (not sure if this is fine or would have been a best practice?) Does the ceph-balancer in upmap mode manage the osd reweight dynamically? Just wondering if I need to proactively go back and set all non 1.00000 reweights to 1.00000. Thanks all, I hope that makes sense!

3 4

Stately MDS Transitions
by DHilsbos＠performair.com 29 Feb '20

29 Feb '20

All; We just started really fiddling with CephFS on our production cluster (Nautilus - 14.2.5 / 14.2.6), and I have a question... Is there a command / set of commands that transitions a standby-replay MDS server to the active role, while swapping the active MDS to standby-replay, or even just standby? I'm looking for a way to seamlessly, and without down time, prepare the active MDS to go offline (reboot) as part of planned /periodic maintenance. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 3

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Lincoln Bryant 28 Feb '20

28 Feb '20

Hi all, I just want to report that I was able to recover from the incomplete PG after exporting the PGs and using the dangerous osd_find_best_info_ignore_history_les option on the appropriate OSDs :) Things seem okay for now, fingers crossed. When marking hit set archive objects lost, I was using 'mark_unfound_lost' command with the 'revert' flag rather than 'delete'. I wonder if the objects were reverted to previous versions, which were causing my OSDs to assert? Here's that error again for reference: 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 flags = none tried to add 1365222'2867906(1363674'2866712) flags = delete Anyhow I'd like to +1 the issue Paul reported here: https://tracker.ceph.com/issues/44286 . This seems totally in line with what we are seeing as well. --Lincoln ________________________________ From: Lincoln Bryant <lincolnb(a)uchicago.edu> Sent: Thursday, February 27, 2020 12:05 PM To: Sage Weil <sage(a)newdream.net>; Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 It seems that one of the down PGs was able to recover just fine, but the other OSD went into "incomplete" state after export-and-removing the affected PG from the down OSD. I've still got the exported data from the pg, although re-importing it to the OSD again causes the crashes. What's the best way to proceed here? It would be very very preferable not to lose the data. Thanks, Lincoln ________________________________ From: Lincoln Bryant <lincolnb(a)uchicago.edu> Sent: Thursday, February 27, 2020 9:26 AM To: Sage Weil <sage(a)newdream.net>; Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 Thanks Sage, I can try that. Admittedly I'm not sure how to tell if these two PG can recover without this particular OSD. Note, it seems like there is still an underlying related issue, with hit set archives popping up as unfound objects on my cluster as in Paul's ticket. In total I had about 10-20 such objects in my cluster, most of which were ok with marking them as unfound/lost. I've had this happen once before a few months ago with unfound hit set archives but was able to safely mark the object lost and move on (or so I thought..?). --Lincoln ________________________________ From: Sage Weil <sage(a)newdream.net> Sent: Thursday, February 27, 2020 9:01 AM To: Paul Emmerich <paul.emmerich(a)croit.io> Cc: Lincoln Bryant <lincolnb(a)uchicago.edu>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 If the pg in question can recover without that OSD, I would use use ceph-objectstore-tool to export and remove it, and then move on. I hit a similar issue on my system (due to a bunch in an early octopus build) and it was super tedious to fix up manually (needed patched code and manual modifications of the store). sage On Thu, 27 Feb 2020, Paul Emmerich wrote: > Crash happens in PG::activate, so it's unrelated to IO etc. > > My first approach here would be to read the code and try to understand > why it crashes/what the exact condition is that is violated here. > It looks like something that can probably be fixed by fiddling around > with ceph-objectstore-tool (but you should try to understand what > exactly is happening before running random ceph-objectstore-tool > commands) > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io<http://www.croit.io> > Tel: +49 89 1896585 90 > > On Thu, Feb 27, 2020 at 1:15 PM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > Thanks Paul. > > > > I was able to mark many of the unfound ones as lost, but I'm still stuck with one unfound and OSD assert at this point. > > > > I've tried setting many of the OSD options to pause all cluster I/O, backfilling, rebalancing, tiering agent, etc to try to avoid hitting the assert but alas this one OSD is still crashing. The OSD in question does manage to log quite a bit of things before crashing. > > > > Is there any way for me to delete this or create a dummy object in RADOS that will let this OSD come up, I wonder? > > > > --Lincoln > > > > OBJECT_UNFOUND 1/793053192 objects unfound (0.000%) > > pg 36.1755 has 1 unfound objects > > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs down > > pg 36.1153 is down+remapped, acting [299] > > pg 36.2047 is down+remapped, acting [242] > > > > -2> 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 f > > lags = none tried to add 1365222'2867906(1363674'2866712) flags = delete > > > > > > ________________________________ > > From: Paul Emmerich <paul.emmerich(a)croit.io> > > Sent: Thursday, February 27, 2020 5:27 AM > > To: Lincoln Bryant <lincolnb(a)uchicago.edu> > > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > > Subject: Re: [ceph-users] Cache tier OSDs crashing due to unfound hitset object 14.2.7 > > > > I've also encountered this issue, but luckily without the crashing > > OSDs, so marking as lost resolved it for us. > > > > See https://tracker.ceph.com/issues/44286 > > > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io<http://www.croit.io> > > Tel: +49 89 1896585 90 > > > > On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > > > Hello Ceph experts, > > > > > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > > > > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > > > > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > > > > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > > > > > Thanks, > > > Lincoln Bryant > > > > > > > > > [1] > > > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > > > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > > > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > > > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > > > 2: (()+0x4dddd7) [0x55955fdafdd7] > > > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 15: (()+0x7e25) [0x7ff5669bae25] > > > 16: (clone()+0x6d) [0x7ff565a9a34d] > > > > > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > > > in thread 7ff52edaa700 thread_name:tp_osd_tp > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (()+0xf5e0) [0x7ff5669c25e0] > > > 2: (gsignal()+0x37) [0x7ff5659d71f7] > > > 3: (abort()+0x148) [0x7ff5659d88e8] > > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > > > 5: (()+0x4dddd7) [0x55955fdafdd7] > > > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 18: (()+0x7e25) [0x7ff5669bae25] > > > 19: (clone()+0x6d) [0x7ff565a9a34d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > > > --- logging levels --- > > > 0/ 5 none > > > 0/ 1 lockdep > > > 0/ 1 context > > > 1/ 1 crush > > > 1/ 5 mds > > > 1/ 5 mds_balancer > > > 1/ 5 mds_locker > > > 1/ 5 mds_log > > > 1/ 5 mds_log_expire > > > 1/ 5 mds_migrator > > > 0/ 1 buffer > > > 0/ 1 timer > > > 0/ 1 filer > > > 0/ 1 striper > > > 0/ 1 objecter > > > 0/ 5 rados > > > 0/ 5 rbd > > > 0/ 5 rbd_mirror > > > 0/ 5 rbd_replay > > > 0/ 5 journaler > > > 0/ 5 objectcacher > > > 0/ 5 client > > > 1/ 5 osd > > > 0/ 5 optracker > > > 0/ 5 objclass > > > 1/ 3 filestore > > > 1/ 3 journal > > > 0/ 0 ms > > > 1/ 5 mon > > > 0/10 monc > > > 1/ 5 paxos > > > 0/ 5 tp > > > 1/ 5 auth > > > 1/ 5 crypto > > > 1/ 1 finisher > > > 1/ 1 reserver > > > 1/ 5 heartbeatmap > > > 1/ 5 perfcounter > > > 1/ 5 rgw > > > 1/ 5 rgw_sync > > > 1/10 civetweb > > > 1/ 5 javaclient > > > 1/ 5 asok > > > 1/ 1 throttle > > > 0/ 0 refs > > > 1/ 5 xio > > > 1/ 5 compressor > > > 1/ 5 bluestore > > > 1/ 5 bluefs > > > 1/ 3 bdev > > > 1/ 5 kstore > > > 4/ 5 rocksdb > > > 4/ 5 leveldb > > > 4/ 5 memdb > > > 1/ 5 kinetic > > > 1/ 5 fuse > > > 1/ 5 mgr > > > 1/ 5 mgrc > > > 1/ 5 dpdk > > > 1/ 5 eventtrace > > > 1/ 5 prioritycache > > > -2/-2 (syslog threshold) > > > -1/-1 (stderr threshold) > > > max_recent 10000 > > > max_new 1000 > > > log_file /var/log/ceph/ceph-osd.299.log > > > --- end dump of recent events --- > > > > > > > > > [2] > > > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > > > "key": "", > > > "snapid": -2, > > > "hash": 12827, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1352209'2834660", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > > > "key": "", > > > "snapid": -2, > > > "hash": 12874, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361100'2822063", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > > > "key": "", > > > "snapid": -2, > > > "hash": 4316, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361089'2838543", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0