- ceph-users - lists.ceph.io

continued warnings: Large omap object found
by Seth Galitzer 28 Feb '20

28 Feb '20

I do not have a large ceph cluster, only 4 nodes plus a mon/mgr with 48 OSDs. I have one data pool and one metadata pool with a total of about 140TB of usable storage. I have maybe 30 or so clients. The rest of my systems connect via a host that is a ceph client and then reshares through samba and nfs-ganesha. I'm not using rgw anywhere. I'm running the latest stable release of nautilus (14.2.7) and have had it in production since August 2019. All ceph nodes and the smb/nfs host are running centos7 with latest patches. Other clients are a mix of debian and ubuntu. For the last several weeks, I have been getting the warning "Large omap object found" off and on. I've been resolving it by gradually increasing the value of osd_deep_scrub_large_omap_object_key_threshold and then running a deep scrub on the affected pg. I have now increased this threshold to 1000000 and am wondering if I should keep doing this or if there is another problem that needs to be addressed. The affected pg has been different most times, but they are all on the same osd and with the same mds object. Here's an excerpt from my current set of logs to show what I'm seeing: # zgrep -i "large omap object found" /var/log/ceph/ceph.log* /var/log/ceph/ceph.log:2020-02-27 06:02:01.761641 osd.40 (osd.40) 1578 : cluster [WRN] Large omap object found. Object: 2:654134d2:::mds0_openfiles.0:head PG: 2.4b2c82a6 (2.26) Key count: 1048576 Size (bytes): 46403355 /var/log/ceph/ceph.log:2020-02-27 16:18:00.328869 osd.40 (osd.40) 1585 : cluster [WRN] Large omap object found. Object: 2:654134d2:::mds0_openfiles.0:head PG: 2.4b2c82a6 (2.26) Key count: 1048559 Size (bytes): 46407183 /var/log/ceph/ceph.log-20200227.gz:2020-02-26 19:56:24.972431 osd.40 (osd.40) 1450 : cluster [WRN] Large omap object found. Object: 2:c9647462:::mds0_openfiles.1:head PG: 2.462e2693 (2.13) Key count: 939236 Size (bytes): 40179994 /var/log/ceph/ceph.log-20200227.gz:2020-02-26 21:14:16.497161 osd.40 (osd.40) 1460 : cluster [WRN] Large omap object found. Object: 2:c9647462:::mds0_openfiles.1:head PG: 2.462e2693 (2.13) Key count: 939232 Size (bytes): 40179796 /var/log/ceph/ceph.log-20200227.gz:2020-02-26 21:15:06.399267 osd.40 (osd.40) 1464 : cluster [WRN] Large omap object found. Object: 2:c9647462:::mds0_openfiles.1:head PG: 2.462e2693 (2.13) Key count: 939231 Size (bytes): 40179756 Unfortunately, older logs have already been rotated out, but if memory serves correctly, they had similar messages. As you can see, the key count continues to increase. Last week, I bumped the threshold to 750000 to clear the warning. Before that, I had bumped to 500000. It looks to me like something isn't getting cleaned up like it's supposed to. I haven't been using ceph long enough to figure out what that might be. Do I continue to bump the key threshold and not worry about the warnings, or is there something going on that needs to be corrected? At what point is the threshold too high? If the problem is due to a specific client not closing files, is it possible to identify that client and attempt to reset it? Any advice is welcome. I'm happy to provide additional data if needed. Thanks. Seth -- Seth Galitzer Systems Coordinator Computer Science Department Kansas State University http://www.cs.ksu.edu/~sgsax sgsax(a)ksu.edu 785-532-7790

2 2

Best way to merge crush buckets?
by Adrien Georget 28 Feb '20

28 Feb '20

Hi all, I'm looking for the best way to merge/remap existing host buckets into one. I'm running a Ceph Nautilus cluster used as a Ceph Cinder backend with 2 pools "volume-service" and "volume-recherche" both with dedicated OSDs : |host cccephnd00x-service {|| || id -2 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item osd.0 weight 7.275|| || item osd.6 weight 7.275|| ||}|| ||host cccephnd00x-recherche {|| || id -3 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item osd.11 weight 7.266|| || item osd.17 weight 7.266|| || item osd.22 weight 7.266|| || item osd.27 weight 7.266|| ||} ... ||root service {|| || id -26 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item cccephnd001-service weight 14.550|| || ...|| || item cccephnd006-service weight 14.550|| || ||}|| ||root recherche {|| || id -27 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item cccephnd001-recherche weight 29.064|| || ...|| || item cccephnd006-recherche weight 29.064|| ||}|| ||rule HAService {|| || id 1|| || type replicated|| || min_size 1|| || max_size 10|| || step take service|| || step chooseleaf firstn 0 type host|| || step emit|| ||}|| ||rule Recherche {|| || id 2|| || type replicated|| || min_size 1|| || max_size 10|| || step take recherche|| || step chooseleaf firstn 0 type host|| || step emit|| ||} $ ceph osd pool get volumes-service crush_rule crush_rule: HAService $ ceph osd pool get volumes-recherche crush_rule crush_rule: Recherche | The pool "volume-service" used to work with SSD cache tiering but we decided to stop using it. So I would like to keep these 2 pools but merge the buckets /host cccephnd00X-service/ and /host cccephnd00X-recherche/ into one /cccephnd00X-cinder/ for better performance (more OSDs assigned to pools). In theory, migrate to this kind of crushmap: |host cccephnd00x-cinder {|| || id -22 || || alg straw2|| || hash 0 # rjenkins1|| || item osd.0 weight 7.275|| || item osd.6 weight 7.275||| || item osd.11 weight 7.266|| || item osd.17 weight 7.266|| || item osd.22 weight 7.266|| || item osd.27 weight 7.266|| | ||}|| |... ||root cinder {|| || id -23 || || alg straw2|| || hash 0 # rjenkins1|| || item cccephnd001-|||||cinder| weight 29.064|| || ...|| || item cccephnd006-|||||cinder| weight 29.064|| ||}|| ||rule Cinder {|| || id 24|| || type replicated|| || min_size 1|| || max_size 10|| || step take ||||||cinder|||| || step chooseleaf firstn 0 type host|| || step emit|| ||} |||$ ceph osd pool get volumes-service crush_rule crush_rule: ||||||cinder|||| || $ ceph osd pool get volumes-recherche crush_rule crush_rule: ||||||cinder|||| || |In practice, I'm thinking about creating the new host bucket/root/rule and then change the crush_rule used for the 2 pools to the new one. And then delete the old one. Do you think that's can be done easily? (And without losing existing data?) I guess there will be a huge rebalance activity but I don't have many choice. Or do you have any other suggestions? Cheers, Adrien |||| ||| |||

1 0

SSD considerations for block.db and WAL
by Christian Wahl 28 Feb '20

28 Feb '20

Hi everyone, we currently have 6 OSDs with 8TB HDDs split across 3 hosts. The main usage is KVM-Images. To improve speed we planned on putting the block.db and WAL onto NVMe-SSDs. The plan was to put 2x1TB in each host. One option I thought of was to RAID 1 them for better redundancy, I don't know how high the risk is of corrupting the block.db by one failed SSD block. Or should I just one for WAL+block.db and use the other one as fast storage? Thank you all very much! Christian

3 2

rgw lifecycle process is not fast enough
by quexian da 28 Feb '20

28 Feb '20

ceph version: 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) I set up a ceph cluster and I'm uploading objects through rgw with a speed of 60 objects/s. I added some lifecycle rules to buckets so that my disks will not be used up. However, after I set "debug_rgw" to 5 and run `radosgw-admin lc process`, I found from the logs that the deletion is not parallel (only objects in one bucket is deleted in the same time) and speed is much slower than my upload speed (60 objects/s), the disk usage is still growing bigger and bigger. Could you please tell me what should I do to speed up the lifecycle process? Thanks in advance!

1 0

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Lincoln Bryant 27 Feb '20

27 Feb '20

It seems that one of the down PGs was able to recover just fine, but the other OSD went into "incomplete" state after export-and-removing the affected PG from the down OSD. I've still got the exported data from the pg, although re-importing it to the OSD again causes the crashes. What's the best way to proceed here? It would be very very preferable not to lose the data. Thanks, Lincoln ________________________________ From: Lincoln Bryant <lincolnb(a)uchicago.edu> Sent: Thursday, February 27, 2020 9:26 AM To: Sage Weil <sage(a)newdream.net>; Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 Thanks Sage, I can try that. Admittedly I'm not sure how to tell if these two PG can recover without this particular OSD. Note, it seems like there is still an underlying related issue, with hit set archives popping up as unfound objects on my cluster as in Paul's ticket. In total I had about 10-20 such objects in my cluster, most of which were ok with marking them as unfound/lost. I've had this happen once before a few months ago with unfound hit set archives but was able to safely mark the object lost and move on (or so I thought..?). --Lincoln ________________________________ From: Sage Weil <sage(a)newdream.net> Sent: Thursday, February 27, 2020 9:01 AM To: Paul Emmerich <paul.emmerich(a)croit.io> Cc: Lincoln Bryant <lincolnb(a)uchicago.edu>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 If the pg in question can recover without that OSD, I would use use ceph-objectstore-tool to export and remove it, and then move on. I hit a similar issue on my system (due to a bunch in an early octopus build) and it was super tedious to fix up manually (needed patched code and manual modifications of the store). sage On Thu, 27 Feb 2020, Paul Emmerich wrote: > Crash happens in PG::activate, so it's unrelated to IO etc. > > My first approach here would be to read the code and try to understand > why it crashes/what the exact condition is that is violated here. > It looks like something that can probably be fixed by fiddling around > with ceph-objectstore-tool (but you should try to understand what > exactly is happening before running random ceph-objectstore-tool > commands) > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io<http://www.croit.io> > Tel: +49 89 1896585 90 > > On Thu, Feb 27, 2020 at 1:15 PM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > Thanks Paul. > > > > I was able to mark many of the unfound ones as lost, but I'm still stuck with one unfound and OSD assert at this point. > > > > I've tried setting many of the OSD options to pause all cluster I/O, backfilling, rebalancing, tiering agent, etc to try to avoid hitting the assert but alas this one OSD is still crashing. The OSD in question does manage to log quite a bit of things before crashing. > > > > Is there any way for me to delete this or create a dummy object in RADOS that will let this OSD come up, I wonder? > > > > --Lincoln > > > > OBJECT_UNFOUND 1/793053192 objects unfound (0.000%) > > pg 36.1755 has 1 unfound objects > > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs down > > pg 36.1153 is down+remapped, acting [299] > > pg 36.2047 is down+remapped, acting [242] > > > > -2> 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 f > > lags = none tried to add 1365222'2867906(1363674'2866712) flags = delete > > > > > > ________________________________ > > From: Paul Emmerich <paul.emmerich(a)croit.io> > > Sent: Thursday, February 27, 2020 5:27 AM > > To: Lincoln Bryant <lincolnb(a)uchicago.edu> > > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > > Subject: Re: [ceph-users] Cache tier OSDs crashing due to unfound hitset object 14.2.7 > > > > I've also encountered this issue, but luckily without the crashing > > OSDs, so marking as lost resolved it for us. > > > > See https://tracker.ceph.com/issues/44286 > > > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io<http://www.croit.io> > > Tel: +49 89 1896585 90 > > > > On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > > > Hello Ceph experts, > > > > > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > > > > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > > > > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > > > > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > > > > > Thanks, > > > Lincoln Bryant > > > > > > > > > [1] > > > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > > > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > > > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > > > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > > > 2: (()+0x4dddd7) [0x55955fdafdd7] > > > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 15: (()+0x7e25) [0x7ff5669bae25] > > > 16: (clone()+0x6d) [0x7ff565a9a34d] > > > > > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > > > in thread 7ff52edaa700 thread_name:tp_osd_tp > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (()+0xf5e0) [0x7ff5669c25e0] > > > 2: (gsignal()+0x37) [0x7ff5659d71f7] > > > 3: (abort()+0x148) [0x7ff5659d88e8] > > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > > > 5: (()+0x4dddd7) [0x55955fdafdd7] > > > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 18: (()+0x7e25) [0x7ff5669bae25] > > > 19: (clone()+0x6d) [0x7ff565a9a34d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > > > --- logging levels --- > > > 0/ 5 none > > > 0/ 1 lockdep > > > 0/ 1 context > > > 1/ 1 crush > > > 1/ 5 mds > > > 1/ 5 mds_balancer > > > 1/ 5 mds_locker > > > 1/ 5 mds_log > > > 1/ 5 mds_log_expire > > > 1/ 5 mds_migrator > > > 0/ 1 buffer > > > 0/ 1 timer > > > 0/ 1 filer > > > 0/ 1 striper > > > 0/ 1 objecter > > > 0/ 5 rados > > > 0/ 5 rbd > > > 0/ 5 rbd_mirror > > > 0/ 5 rbd_replay > > > 0/ 5 journaler > > > 0/ 5 objectcacher > > > 0/ 5 client > > > 1/ 5 osd > > > 0/ 5 optracker > > > 0/ 5 objclass > > > 1/ 3 filestore > > > 1/ 3 journal > > > 0/ 0 ms > > > 1/ 5 mon > > > 0/10 monc > > > 1/ 5 paxos > > > 0/ 5 tp > > > 1/ 5 auth > > > 1/ 5 crypto > > > 1/ 1 finisher > > > 1/ 1 reserver > > > 1/ 5 heartbeatmap > > > 1/ 5 perfcounter > > > 1/ 5 rgw > > > 1/ 5 rgw_sync > > > 1/10 civetweb > > > 1/ 5 javaclient > > > 1/ 5 asok > > > 1/ 1 throttle > > > 0/ 0 refs > > > 1/ 5 xio > > > 1/ 5 compressor > > > 1/ 5 bluestore > > > 1/ 5 bluefs > > > 1/ 3 bdev > > > 1/ 5 kstore > > > 4/ 5 rocksdb > > > 4/ 5 leveldb > > > 4/ 5 memdb > > > 1/ 5 kinetic > > > 1/ 5 fuse > > > 1/ 5 mgr > > > 1/ 5 mgrc > > > 1/ 5 dpdk > > > 1/ 5 eventtrace > > > 1/ 5 prioritycache > > > -2/-2 (syslog threshold) > > > -1/-1 (stderr threshold) > > > max_recent 10000 > > > max_new 1000 > > > log_file /var/log/ceph/ceph-osd.299.log > > > --- end dump of recent events --- > > > > > > > > > [2] > > > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > > > "key": "", > > > "snapid": -2, > > > "hash": 12827, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1352209'2834660", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > > > "key": "", > > > "snapid": -2, > > > "hash": 12874, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361100'2822063", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > > > "key": "", > > > "snapid": -2, > > > "hash": 4316, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361089'2838543", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

osdmap::decode crc error -- 13.2.7 -- most osds down
by Dan van der Ster 27 Feb '20

27 Feb '20

Hi, My turn. We suddenly have a big outage which is similar/identical to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036519.html Some of the osds are runnable, but most crash when they start -- crc error in osdmap::decode. I'm able to extract an osd map from a good osd and it decodes well with osdmaptool: # ceph-objectstore-tool --op get-osdmap --data-path /var/lib/ceph/osd/ceph-680/ --file osd.680.map But when I try on one of the bad osds I get: # ceph-objectstore-tool --op get-osdmap --data-path /var/lib/ceph/osd/ceph-666/ --file osd.666.map terminate called after throwing an instance of 'ceph::buffer::malformed_input' what(): buffer::malformed_input: bad crc, actual 822724616 != expected 2334082500 *** Caught signal (Aborted) ** in thread 7f600aa42d00 thread_name:ceph-objectstor ceph version 13.2.7 (71bd687b6e8b9424dd5e5974ed542595d8977416) mimic (stable) 1: (()+0xf5f0) [0x7f5ffefc45f0] 2: (gsignal()+0x37) [0x7f5ffdbae337] 3: (abort()+0x148) [0x7f5ffdbafa28] 4: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f5ffe4be7d5] 5: (()+0x5e746) [0x7f5ffe4bc746] 6: (()+0x5e773) [0x7f5ffe4bc773] 7: (()+0x5e993) [0x7f5ffe4bc993] 8: (OSDMap::decode(ceph::buffer::list::iterator&)+0x160e) [0x7f6000f4168e] 9: (OSDMap::decode(ceph::buffer::list&)+0x31) [0x7f6000f42e31] 10: (get_osdmap(ObjectStore*, unsigned int, OSDMap&, ceph::buffer::list&)+0x1d0) [0x55d30a489190] 11: (main()+0x5340) [0x55d30a3aae70] 12: (__libc_start_main()+0xf5) [0x7f5ffdb9a505] 13: (()+0x3a0f40) [0x55d30a483f40] Aborted (core dumped) I think I want to inject the osdmap, but can't: # ceph-objectstore-tool --op set-osdmap --data-path /var/lib/ceph/osd/ceph-666/ --file osd.680.map osdmap (#-1:b65b78ab:::osdmap.2983572:0#) does not exist. How do I do this? Thanks for any help! dan

2 6

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Paul Emmerich 27 Feb '20

27 Feb '20

Crash happens in PG::activate, so it's unrelated to IO etc. My first approach here would be to read the code and try to understand why it crashes/what the exact condition is that is violated here. It looks like something that can probably be fixed by fiddling around with ceph-objectstore-tool (but you should try to understand what exactly is happening before running random ceph-objectstore-tool commands) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Feb 27, 2020 at 1:15 PM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > Thanks Paul. > > I was able to mark many of the unfound ones as lost, but I'm still stuck with one unfound and OSD assert at this point. > > I've tried setting many of the OSD options to pause all cluster I/O, backfilling, rebalancing, tiering agent, etc to try to avoid hitting the assert but alas this one OSD is still crashing. The OSD in question does manage to log quite a bit of things before crashing. > > Is there any way for me to delete this or create a dummy object in RADOS that will let this OSD come up, I wonder? > > --Lincoln > > OBJECT_UNFOUND 1/793053192 objects unfound (0.000%) > pg 36.1755 has 1 unfound objects > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs down > pg 36.1153 is down+remapped, acting [299] > pg 36.2047 is down+remapped, acting [242] > > -2> 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 f > lags = none tried to add 1365222'2867906(1363674'2866712) flags = delete > > > ________________________________ > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Thursday, February 27, 2020 5:27 AM > To: Lincoln Bryant <lincolnb(a)uchicago.edu> > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] Cache tier OSDs crashing due to unfound hitset object 14.2.7 > > I've also encountered this issue, but luckily without the crashing > OSDs, so marking as lost resolved it for us. > > See https://tracker.ceph.com/issues/44286 > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > Hello Ceph experts, > > > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > > > Thanks, > > Lincoln Bryant > > > > > > [1] > > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > > 2: (()+0x4dddd7) [0x55955fdafdd7] > > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > 15: (()+0x7e25) [0x7ff5669bae25] > > 16: (clone()+0x6d) [0x7ff565a9a34d] > > > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > > in thread 7ff52edaa700 thread_name:tp_osd_tp > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > 1: (()+0xf5e0) [0x7ff5669c25e0] > > 2: (gsignal()+0x37) [0x7ff5659d71f7] > > 3: (abort()+0x148) [0x7ff5659d88e8] > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > > 5: (()+0x4dddd7) [0x55955fdafdd7] > > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > 18: (()+0x7e25) [0x7ff5669bae25] > > 19: (clone()+0x6d) [0x7ff565a9a34d] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > --- logging levels --- > > 0/ 5 none > > 0/ 1 lockdep > > 0/ 1 context > > 1/ 1 crush > > 1/ 5 mds > > 1/ 5 mds_balancer > > 1/ 5 mds_locker > > 1/ 5 mds_log > > 1/ 5 mds_log_expire > > 1/ 5 mds_migrator > > 0/ 1 buffer > > 0/ 1 timer > > 0/ 1 filer > > 0/ 1 striper > > 0/ 1 objecter > > 0/ 5 rados > > 0/ 5 rbd > > 0/ 5 rbd_mirror > > 0/ 5 rbd_replay > > 0/ 5 journaler > > 0/ 5 objectcacher > > 0/ 5 client > > 1/ 5 osd > > 0/ 5 optracker > > 0/ 5 objclass > > 1/ 3 filestore > > 1/ 3 journal > > 0/ 0 ms > > 1/ 5 mon > > 0/10 monc > > 1/ 5 paxos > > 0/ 5 tp > > 1/ 5 auth > > 1/ 5 crypto > > 1/ 1 finisher > > 1/ 1 reserver > > 1/ 5 heartbeatmap > > 1/ 5 perfcounter > > 1/ 5 rgw > > 1/ 5 rgw_sync > > 1/10 civetweb > > 1/ 5 javaclient > > 1/ 5 asok > > 1/ 1 throttle > > 0/ 0 refs > > 1/ 5 xio > > 1/ 5 compressor > > 1/ 5 bluestore > > 1/ 5 bluefs > > 1/ 3 bdev > > 1/ 5 kstore > > 4/ 5 rocksdb > > 4/ 5 leveldb > > 4/ 5 memdb > > 1/ 5 kinetic > > 1/ 5 fuse > > 1/ 5 mgr > > 1/ 5 mgrc > > 1/ 5 dpdk > > 1/ 5 eventtrace > > 1/ 5 prioritycache > > -2/-2 (syslog threshold) > > -1/-1 (stderr threshold) > > max_recent 10000 > > max_new 1000 > > log_file /var/log/ceph/ceph-osd.299.log > > --- end dump of recent events --- > > > > > > [2] > > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > > { > > "num_missing": 1, > > "num_unfound": 1, > > "objects": [ > > { > > "oid": { > > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > > "key": "", > > "snapid": -2, > > "hash": 12827, > > "max": 0, > > "pool": 36, > > "namespace": ".ceph-internal" > > }, > > "need": "1352209'2834660", > > "have": "0'0", > > "flags": "none", > > "locations": [] > > } > > ], > > "more": false > > } > > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > > { > > "num_missing": 1, > > "num_unfound": 1, > > "objects": [ > > { > > "oid": { > > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > > "key": "", > > "snapid": -2, > > "hash": 12874, > > "max": 0, > > "pool": 36, > > "namespace": ".ceph-internal" > > }, > > "need": "1361100'2822063", > > "have": "0'0", > > "flags": "none", > > "locations": [] > > } > > ], > > "more": false > > } > > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > > { > > "num_missing": 1, > > "num_unfound": 1, > > "objects": [ > > { > > "oid": { > > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > > "key": "", > > "snapid": -2, > > "hash": 4316, > > "max": 0, > > "pool": 36, > > "namespace": ".ceph-internal" > > }, > > "need": "1361089'2838543", > > "have": "0'0", > > "flags": "none", > > "locations": [] > > } > > ], > > "more": false > > } > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 3

ceph prometheus module no export content
by 黄明友 27 Feb '20

27 Feb '20

hi,all I had enable the prometheus module on my ceph cluster , the ceph version is ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable). when I eanble this modle, can get exported content from the prometheus module. but , when I restart all the ceph cluster's node , the module can't expor any content , curl http://host/metrics ; get blan; the tcp/9283 port is listen; I had disable and enable this module,but still no content export. 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

4 4

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Paul Emmerich 27 Feb '20

27 Feb '20

I've also encountered this issue, but luckily without the crashing OSDs, so marking as lost resolved it for us. See https://tracker.ceph.com/issues/44286 Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > Hello Ceph experts, > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > Thanks, > Lincoln Bryant > > > [1] > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > 2: (()+0x4dddd7) [0x55955fdafdd7] > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > 15: (()+0x7e25) [0x7ff5669bae25] > 16: (clone()+0x6d) [0x7ff565a9a34d] > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > in thread 7ff52edaa700 thread_name:tp_osd_tp > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > 1: (()+0xf5e0) [0x7ff5669c25e0] > 2: (gsignal()+0x37) [0x7ff5659d71f7] > 3: (abort()+0x148) [0x7ff5659d88e8] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > 5: (()+0x4dddd7) [0x55955fdafdd7] > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > 18: (()+0x7e25) [0x7ff5669bae25] > 19: (clone()+0x6d) [0x7ff565a9a34d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 0 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 rgw_sync > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 kinetic > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > 1/ 5 prioritycache > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.299.log > --- end dump of recent events --- > > > [2] > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > "key": "", > "snapid": -2, > "hash": 12827, > "max": 0, > "pool": 36, > "namespace": ".ceph-internal" > }, > "need": "1352209'2834660", > "have": "0'0", > "flags": "none", > "locations": [] > } > ], > "more": false > } > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > "key": "", > "snapid": -2, > "hash": 12874, > "max": 0, > "pool": 36, > "namespace": ".ceph-internal" > }, > "need": "1361100'2822063", > "have": "0'0", > "flags": "none", > "locations": [] > } > ], > "more": false > } > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > "key": "", > "snapid": -2, > "hash": 4316, > "max": 0, > "pool": 36, > "namespace": ".ceph-internal" > }, > "need": "1361089'2838543", > "have": "0'0", > "flags": "none", > "locations": [] > } > ], > "more": false > } > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1