- ceph-users - lists.ceph.io

Re: OSDs crashing
by Raymond Clotfelter 05 Feb '20

05 Feb '20

I have found that if I set norecovery then I can get almost all OSDs to come up and stay up, but the moment I unset norecovery so that the cluster can heal itself, tons of OSDs go offline again. The OSD host servers have plenty of available RAM, they are not maxing on CPU or I/O near as I can tell. The OSDs just keep crashing with error logs similar what I posted before. Any help would be appreciated. -- ray On Tue, Feb 4, 2020 at 6:22 AM Raymond Clotfelter <ray(a)ksu.edu<mailto:ray@ksu.edu>> wrote: I have 30 or so OSDs on a cluster with 240 that just keep crashing. Below is the last part of one of the log files showing the crash, can anyone please help me read this to figure out what is going on and how to correct it? When I start the OSDs they generally seem to work for 5-30 minutes, and then one by one they will start dropping out with logs similar to this. Thanks. -29> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] exit Started/Stray 1.017145 6 0.000323 -28> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] enter Started/ReplicaActive -27> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] enter Started/ReplicaActive/RepNotRecovering -26> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -25> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -24> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -23> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -22> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -21> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -20> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -19> 2020-02-04 06:00:23.463 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -18> 2020-02-04 06:00:23.463 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -17> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -16> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -15> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.021923 2 0.000098 -14> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepWaitRecoveryReserved -13> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepWaitRecoveryReserved 0.000137 1 0.000080 -12> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepRecovering -11> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -10> 2020-02-04 06:00:23.475 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -9> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/Activating 0.034400 28 0.002950 -8> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/WaitLocalRecoveryReserved -7> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/WaitLocalRecoveryReserved 0.000213 1 0.000221 -6> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/WaitRemoteRecoveryReserved -5> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.002796 5 0.000231 -4> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/Recovering -3> 2020-02-04 06:00:23.491 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -2> 2020-02-04 06:00:23.519 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -1> 2020-02-04 06:00:23.779 7fe300540700 -1 /build/ceph-14.2.7/src/osd/ECBackend.cc: In function 'void ECBackend::do_read_op(ECBackend::ReadOp&)' thread 7fe300540700 time 2020-02-04 06:00:23.774430 /build/ceph-14.2.7/src/osd/ECBackend.cc: 1742: FAILED ceph_assert(!need_attrs) ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x5579b9ecce4c] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5579b9ecd027] 3: (ECBackend::do_read_op(ECBackend::ReadOp&)+0xf76) [0x5579ba295106] 4: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x4d1) [0x5579ba2a6cf1] 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcf6) [0x5579ba2a7f26] 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0xbc) [0x5579ba2a8a8c] 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x97) [0x5579ba17f757] 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x705) [0x5579ba12dee5] 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x5579b9f4ff2f] 10: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x5579ba206e82] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x5579b9f6ea05] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x5579ba5714cc] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5579ba574690] 14: (()+0x7fa3) [0x7fe31ff5cfa3] 15: (clone()+0x3f) [0x7fe31fb0c4cf] 0> 2020-02-04 06:00:23.787 7fe300540700 -1 *** Caught signal (Aborted) ** in thread 7fe300540700 thread_name:tp_osd_tp ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (()+0x12730) [0x7fe31ff67730] 2: (gsignal()+0x10b) [0x7fe31fa4a7bb] 3: (abort()+0x121) [0x7fe31fa35535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x5579b9ecce9d] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5579b9ecd027] 6: (ECBackend::do_read_op(ECBackend::ReadOp&)+0xf76) [0x5579ba295106] 7: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x4d1) [0x5579ba2a6cf1] 8: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcf6) [0x5579ba2a7f26] 9: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0xbc) [0x5579ba2a8a8c] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x97) [0x5579ba17f757] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x705) [0x5579ba12dee5] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x5579b9f4ff2f] 13: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x5579ba206e82] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x5579b9f6ea05] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x5579ba5714cc] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5579ba574690] 17: (()+0x7fa3) [0x7fe31ff5cfa3] 18: (clone()+0x3f) [0x7fe31fb0c4cf] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.168.log --- end dump of recent events --- -- ray _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Re: Cephalocon Seoul is canceled
by Romit Misra 05 Feb '20

05 Feb '20

Hi Sage, Just read the news on Cancellation of Cephlacon 2020, although the site is still status quo/ Double checking that we can proceed with the cancellation of logistics for South Korea Thanks Romit On Tue, Feb 4, 2020 at 11:02 PM <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: More OMAP Issues (Paul Emmerich) > 2. Re: More OMAP Issues (DHilsbos(a)performair.com) > 3. Re: Bluestore cache parameter precedence (Igor Fedotov) > 4. Re: Understanding Bluestore performance characteristics > (vitalif(a)yourcmc.ru) > 5. Cephalocon Seoul is canceled (Sage Weil) > 6. Re: Bluestore cache parameter precedence (Boris Epstein) > 7. Bucket rename with (EDH - Manuel Rios) > > > ---------------------------------------------------------------------- > > Date: Tue, 4 Feb 2020 17:51:40 +0100 > From: Paul Emmerich <paul.emmerich(a)croit.io> > Subject: [ceph-users] Re: More OMAP Issues > To: DHilsbos(a)performair.com > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > < > CAD9yTbEp1BrAagzWzkaAQ-aCq-4ghyEwVJSDdyJzL-So52whBA(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Are you running a multi-site setup? > In this case it's best to set the default shard size to large enough > number *before* enabling multi-site. > > If you didn't do this: well... I think the only way is still to > completely re-sync the second site... > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Tue, Feb 4, 2020 at 5:23 PM <DHilsbos(a)performair.com> wrote: > > > > All; > > > > We're backing to having large OMAP object warnings regarding our RGW > index pool. > > > > This cluster is now in production, so I can simply dump the buckets / > pools and hope everything works out. > > > > I did some additional research on this issue, and it looks like I need > to (re)shard the bucket (index?). I found information that suggests that, > for older versions of Ceph, buckets couldn't be sharded after creation[1]. > Other information suggests the Nautilus (which we are running), can > re-shard dynamically, but not when multi-site replication is configured[2]. > > > > This suggests that a "manual" resharding of a Nautilus cluster should be > possible, but I can't find the commands to do it. Has anyone done this? > Does anyone have the commands to do it? I can schedule down time for the > cluster, and take the RADOSGW instance(s), and dependent user services > offline. > > > > [1]: https://ceph.io/geen-categorie/radosgw-big-index/ > > [2]: https://docs.ceph.com/docs/master/radosgw/dynamicresharding/ > > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director - Information Technology > > Perform Air International Inc. > > DHilsbos(a)PerformAir.com > > www.PerformAir.com > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 4 Feb 2020 17:04:24 +0000 > From: <DHilsbos(a)performair.com> > Subject: [ceph-users] Re: More OMAP Issues > To: <ceph-users(a)ceph.io> > Cc: <paul.emmerich(a)croit.io> > Message-ID: > <0670B960225633449A24709C291A525243605D57(a)COM01.performair.local> > Content-Type: text/plain; charset="utf-8" > > Paul; > > Yes, we are running a multi-site setup. > > Re-sync would be acceptable at this point, as we only have 4 TiB in use > right now. > > Tearing down and reconfiguring the second site would also be acceptable, > except that I've never been able to cleanly remove a zone from a zone > group. The only way I've found to remove a zone completely is to tear down > the entire RADOSGW configuration (delete .rgw.root pool from both clusters). > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > DHilsbos(a)PerformAir.com > www.PerformAir.com > > > > -----Original Message----- > From: Paul Emmerich [mailto:paul.emmerich@croit.io] > Sent: Tuesday, February 04, 2020 9:52 AM > To: Dominic Hilsbos > Cc: ceph-users > Subject: Re: [ceph-users] More OMAP Issues > > Are you running a multi-site setup? > In this case it's best to set the default shard size to large enough > number *before* enabling multi-site. > > If you didn't do this: well... I think the only way is still to > completely re-sync the second site... > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Tue, Feb 4, 2020 at 5:23 PM <DHilsbos(a)performair.com> wrote: > > > > All; > > > > We're backing to having large OMAP object warnings regarding our RGW > index pool. > > > > This cluster is now in production, so I can simply dump the buckets / > pools and hope everything works out. > > > > I did some additional research on this issue, and it looks like I need > to (re)shard the bucket (index?). I found information that suggests that, > for older versions of Ceph, buckets couldn't be sharded after creation[1]. > Other information suggests the Nautilus (which we are running), can > re-shard dynamically, but not when multi-site replication is configured[2]. > > > > This suggests that a "manual" resharding of a Nautilus cluster should be > possible, but I can't find the commands to do it. Has anyone done this? > Does anyone have the commands to do it? I can schedule down time for the > cluster, and take the RADOSGW instance(s), and dependent user services > offline. > > > > [1]: https://ceph.io/geen-categorie/radosgw-big-index/ > > [2]: https://docs.ceph.com/docs/master/radosgw/dynamicresharding/ > > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director - Information Technology > > Perform Air International Inc. > > DHilsbos(a)PerformAir.com > > www.PerformAir.com > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 4 Feb 2020 20:10:20 +0300 > From: Igor Fedotov <ifedotov(a)suse.de> > Subject: [ceph-users] Re: Bluestore cache parameter precedence > To: Boris Epstein <borepstein(a)gmail.com>, ceph-users(a)ceph.io > Message-ID: <0cb36a39-7dba-01b5-5383-dc1116f459a4(a)suse.de> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Boris, > > general settings (unless they are set to zero) override disk-specific > settings . > > I.e. bluestore_cache_size overrides both bluestore_cache_size_hdd and > bluestore_cache_size_ssd. > > Here is the code snippet in case you know C++ > > if (cct->_conf->bluestore_cache_size) { > cache_size = cct->_conf->bluestore_cache_size; > } else { > // choose global cache size based on backend type > if (_use_rotational_settings()) { > cache_size = cct->_conf->bluestore_cache_size_hdd; > } else { > cache_size = cct->_conf->bluestore_cache_size_ssd; > } > } > > Thanks, > > Igor > > On 2/4/2020 2:14 PM, Boris Epstein wrote: > > Hello list, > > > > As stated in this document: > > > > > https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ > > > > there are multiple parameters defining cache limits for BlueStore. You > have > > bluestore_cache_size (presumably controlling the cache size), > > bluestore_cache_size_hdd (presumably doing the same for HDD storage only) > > and bluestore_cache_size_ssd (presumably being the equivalent for SSD). > My > > question is, does bluestore_cache_size override the disk-specific > > parameters, or do I need to set the disk-specific (or, rather, storage > type > > specific ones separately if I want to keep them to a certain value. > > > > Thanks in advance. > > > > Boris. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 04 Feb 2020 20:22:30 +0300 > From: vitalif(a)yourcmc.ru > Subject: [ceph-users] Re: Understanding Bluestore performance > characteristics > To: Bradley Kite <bradley.kite(a)gmail.com> > Cc: ceph-users(a)ceph.io > Message-ID: <c381a59989a4f4f6760d061e745a281a(a)yourcmc.ru> > Content-Type: text/plain; charset=US-ASCII; format=flowed > > SSD (block.db) partition contains object metadata in RocksDB so it > probably loads the metadata before modifying objects (if it's not in > cache yet). Also it sometimes performs compaction which also results in > disk reads and writes. There are other things going on that I'm not > completely aware of. There's the RBD object map... Maybe there are some > locks that come into action when you parallel writes... > > There's a config option to enable RocksDB performance counters. You can > have a look into it. > > However if you're just trying to understand why RBD isn't super fast > then I don't think these reads are the cause... > > > Hi Vitaliy > > > > Yes - I tried this and I can still see a number of reads (~110 iops, > > 440KB/sec) on the SSD, so it is significantly better, but the result > > is still puzzling - I'm trying to understand what is causing the > > reads. The problem is amplified with numjobs >= 2 but it looks like it > > is still there with just 1. > > > > Like some caching parameter is not correct, and the same blocks are > > being read over and over when doing a write? > > > > Could anyone advise on the best way for me to investigate further? > > > > I've tried strace (with -k) and 'perf record' but neither produce any > > useful stack traces to help understand what's going on. > > > > Regards > > -- > > Brad > > ------------------------------ > > Date: Tue, 4 Feb 2020 17:24:37 +0000 (UTC) > From: Sage Weil <sage(a)newdream.net> > Subject: [ceph-users] Cephalocon Seoul is canceled > To: ceph-announce(a)ceph.io, ceph-users(a)ceph.io, dev(a)ceph.io, > ceph-devel(a)vger.kernel.org > Message-ID: <alpine.DEB.2.21.2002041649050.21136(a)piezo.novalocal> > Content-Type: text/plain; charset=US-ASCII > > Hi everyone, > > We are sorry to announce that, due to the recent coronavirus outbreak, we > are canceling Cephalocon for March 3-5 in Seoul. > > More details will follow about how to best handle cancellation of hotel > reservations and so forth. Registrations will of course be > refunded--expect an email with details in the next day or two. > > We are still looking into whether it makes sense to reschedule the event > for later in the year. > > Thank you to everyone who has helped to plan this event, submitted talks, > and agreed to sponsor. It makes us sad to cancel, but the safety of > our community is of the utmost importance, and it was looking increasing > unlikely that we could make this event a success. > > Stay tuned... > > ------------------------------ > > Date: Tue, 4 Feb 2020 12:29:13 -0500 > From: Boris Epstein <borepstein(a)gmail.com> > Subject: [ceph-users] Re: Bluestore cache parameter precedence > To: Igor Fedotov <ifedotov(a)suse.de> > Cc: ceph-users(a)ceph.io > Message-ID: > <CADeF1XHrPzTq1+8S_WG=ZH=SVNAbdLaY= > FR7UaLGHn3O_yWLnw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hi Igor, > > Thanks! > > I think the code needs to be corrected - the choice criteria for which > setting to use when > > cct->_conf->bluestore_cache_size == 0 > > should be as follows: > > 1) See what kind of storage you have. > > 2) Select type-appropriate storage. > > Is this code public-editable? I'll be happy to correct that. > > Regards, > > Boris. > > On Tue, Feb 4, 2020 at 12:10 PM Igor Fedotov <ifedotov(a)suse.de> wrote: > > > Hi Boris, > > > > general settings (unless they are set to zero) override disk-specific > > settings . > > > > I.e. bluestore_cache_size overrides both bluestore_cache_size_hdd and > > bluestore_cache_size_ssd. > > > > Here is the code snippet in case you know C++ > > > > if (cct->_conf->bluestore_cache_size) { > > cache_size = cct->_conf->bluestore_cache_size; > > } else { > > // choose global cache size based on backend type > > if (_use_rotational_settings()) { > > cache_size = cct->_conf->bluestore_cache_size_hdd; > > } else { > > cache_size = cct->_conf->bluestore_cache_size_ssd; > > } > > } > > > > Thanks, > > > > Igor > > > > On 2/4/2020 2:14 PM, Boris Epstein wrote: > > > Hello list, > > > > > > As stated in this document: > > > > > > > > > https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ > > > > > > there are multiple parameters defining cache limits for BlueStore. You > > have > > > bluestore_cache_size (presumably controlling the cache size), > > > bluestore_cache_size_hdd (presumably doing the same for HDD storage > only) > > > and bluestore_cache_size_ssd (presumably being the equivalent for SSD). > > My > > > question is, does bluestore_cache_size override the disk-specific > > > parameters, or do I need to set the disk-specific (or, rather, storage > > type > > > specific ones separately if I want to keep them to a certain value. > > > > > > Thanks in advance. > > > > > > Boris. > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > Date: Tue, 4 Feb 2020 17:29:55 +0000 > From: EDH - Manuel Rios <mriosfer(a)easydatahost.com> > Subject: [ceph-users] Bucket rename with > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <HE1P195MB02521946493264331A2CE2A3B0030(a)HE1P195MB0252.EUR > P195.PROD.OUTLOOK.COM> > Content-Type: text/plain; charset="us-ascii" > > Hi > > Some Customer asked us for a normal easy problem, they want rename a > bucket. > > Checking the Nautilus documentation looks by now its not possible, but I > checked master documentation and a CLI should be accomplish this apparently. > > $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny > > Will be backported to Nautilus? Or its still just for developer/master > users? > > https://docs.ceph.com/docs/master/man/8/radosgw-admin/ > > Regards > Manuel > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 85, Issue 17 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

Performance drops and low oss performance
by quexian da 05 Feb '20

05 Feb '20

Hello, I'm a beginner on ceph. I set up some ceph clusters in google cloud. Cluster1 has three nodes and each node has three disks. Cluster2 has three nodes and each node has two disks. Cluster3 has five nodes and each node has five disks. Disk speed shown by `dd if=/dev/zero of=here bs=1G count=1 oflag=direct` is 117MB/s. The network is 10Gbps. However, I found something strange: 1. The write performance of all clusters drops dramatically after a few minutes. I created a pool named "scbench" with replicated size 1 (I know it is not safe but I want the highest write speed). The write performance (shown by rados bench -p scbench 1000 write) before and after the drop are: cluster1: 297MB/s 94.5MB/s cluster2: 304MB/s 67.4MB/s cluster3: 494MB/s 267.6MB/s It looks like the performance before the drop is nodes_num * 100MB/s, and the performance after the drop is about osds_num * 10MB/s. I have no idea why there is such a drop and why the performances before the drop are linear with nodes_num. 2. The write performance of object storage (shown by swift-bench -c 64 -s 4096000 -n 100000 -g 0 swift.conf) is much lower than that of storage cluster(shown by rados bench -p scbench 1000 write). I have set the replicated size of "default.rgw.buckets.data" and "default.rgw.buckets.index" to 1 The speed of cluster1 oss is 117MB/s (before the drop) and 26MB/s (after the drop), and the speed of cluster3 oss is 118MB/s (the drop does not happen). Is it normal that the oss write performance is worse than rados write performance? If not, how can I solve the problem? Thanks!

1 0

Migrate journal to Nvme from old SSD journal drive?
by Alex L 05 Feb '20

05 Feb '20

Hi, I finally got my Samsung PM983 [1] to use as journal for about 6 drives plus drive cache replacing a consumer SSD - Kingston SV300. But I can't for the life of me figure out how to move an existing journal to this NVME on my Nautilus cluster. # Created a new big partition on the NVME sgdisk --new=1:2048:+33GiB --change-name="1:ceph block.db" --typecode="1:30cd0809-c2b2-499c-8879-2d6b78529876" --mbrtogpt /dev/nvme0n1 partprobe sgdisk -p /dev/nvme0n1 # The below assumes there is already a partition+ fs on the nvme? ceph-bluestore-tool bluefs-bdev-migrate –dev-target /dev/nvme0n1p1 -devs-source /var/lib/ceph/osd/ceph-1/block.db - too many positional options have been specified on the command line ceph-bluestore-tool bluefs-bdev-migrate -–path /var/lib/ceph/osd/ceph-1/block.db –-dev-target /dev/nvme0n1p1 - too many positional options have been specified on the command line # Or should I create a new block device? if yes, will WAL come along ? And how do I remove the SSD journal partition (the old) ceph-bluestore-tool bluefs-bdev-new-db -–path /var/lib/ceph/osd/ceph-1/block.db –-dev-target /dev/nvme0n1p1 The documentation is not very clear on what migration does, nor has the same concept of a DEVICE (/dev/sda is a device for me) it seems. Thanks in advance, Alex ---- [1] - Performance stats: https://docs.google.com/spreadsheets/d/1LXupjEUnNdf011QNr24pkAiDBphzpz5_MwM…

1 0

All pgs peering indefinetely
by Rodrigo Severo - Fábrica 05 Feb '20

05 Feb '20

Hi, I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles. I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time. They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again. Here is some info about my cluster: ------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df) data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering ------------------------------------------- ------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 ------------------------------------------- At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing. What should I do to get my cluster back up? Just wait? Regards, Rodrigo Severo

3 7

Bucket rename with
by EDH - Manuel Rios 04 Feb '20

04 Feb '20

Hi Some Customer asked us for a normal easy problem, they want rename a bucket. Checking the Nautilus documentation looks by now its not possible, but I checked master documentation and a CLI should be accomplish this apparently. $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny Will be backported to Nautilus? Or its still just for developer/master users? https://docs.ceph.com/docs/master/man/8/radosgw-admin/ Regards Manuel

1 0

Cephalocon Seoul is canceled
by Sage Weil 04 Feb '20

04 Feb '20

Hi everyone, We are sorry to announce that, due to the recent coronavirus outbreak, we are canceling Cephalocon for March 3-5 in Seoul. More details will follow about how to best handle cancellation of hotel reservations and so forth. Registrations will of course be refunded--expect an email with details in the next day or two. We are still looking into whether it makes sense to reschedule the event for later in the year. Thank you to everyone who has helped to plan this event, submitted talks, and agreed to sponsor. It makes us sad to cancel, but the safety of our community is of the utmost importance, and it was looking increasing unlikely that we could make this event a success. Stay tuned...

1 0

More OMAP Issues
by DHilsbos＠performair.com 04 Feb '20

04 Feb '20

All; We're backing to having large OMAP object warnings regarding our RGW index pool. This cluster is now in production, so I can simply dump the buckets / pools and hope everything works out. I did some additional research on this issue, and it looks like I need to (re)shard the bucket (index?). I found information that suggests that, for older versions of Ceph, buckets couldn't be sharded after creation[1]. Other information suggests the Nautilus (which we are running), can re-shard dynamically, but not when multi-site replication is configured[2]. This suggests that a "manual" resharding of a Nautilus cluster should be possible, but I can't find the commands to do it. Has anyone done this? Does anyone have the commands to do it? I can schedule down time for the cluster, and take the RADOSGW instance(s), and dependent user services offline. [1]: https://ceph.io/geen-categorie/radosgw-big-index/ [2]: https://docs.ceph.com/docs/master/radosgw/dynamicresharding/ Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

Doubt about AVAIL space on df
by German Anders 04 Feb '20

04 Feb '20

Hello Everyone, I would like to understand if this output is right: *# ceph df* GLOBAL: SIZE AVAIL RAW USED %RAW USED 85.1TiB 43.7TiB 41.4TiB 48.68 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS volumes 13 13.8TiB 64.21 7.68TiB 3620495 I only have (1) pool called 'volumes' which is using 13.8TiB (we have a replica of 3) so it's actually using 41,4TiB and that would be the RAW USED, at this point is fine, but, then it said on the GLOBAL section that the AVAIL space is 43.7TiB and the %RAW USED is only 48.68%. So if I use the 7.68TiB of MAX AVAIL and the pool goes up to 100% of usage, that would not lead to the total space of the cluster, right? I mean were are those 43.7TiB of AVAIL space? I'm using Luminous 12.2.12 release. Sorry if it's a silly question or if it has been answered before. Thanks in advance, Best regards,

3 2

Re: Doubt about AVAIL space on df
by German Anders 04 Feb '20

04 Feb '20

Manuel, find the output of ceph osd df tree command: # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME -7 84.00099 - 85.1TiB 41.6TiB 43.6TiB 48.82 1.00 - root root -5 12.00000 - 13.1TiB 5.81TiB 7.29TiB 44.38 0.91 - rack rack1 -1 12.00000 - 13.1TiB 5.81TiB 7.29TiB 44.38 0.91 - node cpn01 0 nvme 1.00000 1.00000 1.09TiB 496GiB 621GiB 44.40 0.91 143 osd.0 1 nvme 1.00000 1.00000 1.09TiB 489GiB 629GiB 43.72 0.90 141 osd.1 2 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.03 0.98 155 osd.2 3 nvme 1.00000 1.00000 1.09TiB 474GiB 644GiB 42.40 0.87 137 osd.3 4 nvme 1.00000 1.00000 1.09TiB 532GiB 586GiB 47.57 0.97 153 osd.4 5 nvme 1.00000 1.00000 1.09TiB 476GiB 642GiB 42.60 0.87 137 osd.5 6 nvme 1.00000 1.00000 1.09TiB 467GiB 650GiB 41.82 0.86 135 osd.6 7 nvme 1.00000 1.00000 1.09TiB 544GiB 574GiB 48.65 1.00 157 osd.7 8 nvme 1.00000 1.00000 1.09TiB 482GiB 636GiB 43.12 0.88 139 osd.8 9 nvme 1.00000 1.00000 1.09TiB 458GiB 660GiB 40.96 0.84 133 osd.9 10 nvme 1.00000 1.00000 1.09TiB 514GiB 604GiB 45.97 0.94 148 osd.10 11 nvme 1.00000 1.00000 1.09TiB 484GiB 633GiB 43.34 0.89 140 osd.11 -6 12.00099 - 13.1TiB 6.02TiB 7.08TiB 45.98 0.94 - rack rack2 -2 12.00099 - 13.1TiB 6.02TiB 7.08TiB 45.98 0.94 - node cpn02 12 nvme 1.00000 1.00000 1.09TiB 499GiB 619GiB 44.61 0.91 144 osd.12 13 nvme 1.00000 1.00000 1.09TiB 561GiB 557GiB 50.19 1.03 162 osd.13 14 nvme 1.00000 1.00000 1.09TiB 577GiB 541GiB 51.60 1.06 167 osd.14 15 nvme 1.00000 1.00000 1.09TiB 546GiB 572GiB 48.84 1.00 158 osd.15 16 nvme 1.00000 1.00000 1.09TiB 537GiB 580GiB 48.07 0.98 155 osd.16 17 nvme 1.00000 1.00000 1.09TiB 508GiB 610GiB 45.41 0.93 147 osd.17 18 nvme 1.00000 1.00000 1.09TiB 490GiB 628GiB 43.86 0.90 142 osd.18 19 nvme 1.00000 1.00000 1.09TiB 534GiB 584GiB 47.76 0.98 155 osd.19 20 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.80 0.86 134 osd.20 21 nvme 1.00000 1.00000 1.09TiB 447GiB 671GiB 40.01 0.82 129 osd.21 22 nvme 1.00099 1.00000 1.09TiB 561GiB 556GiB 50.21 1.03 162 osd.22 23 nvme 1.00000 1.00000 1.09TiB 441GiB 677GiB 39.45 0.81 127 osd.23 -15 12.00000 - 13.1TiB 5.92TiB 7.18TiB 45.20 0.93 - rack rack3 -3 12.00000 - 13.1TiB 5.92TiB 7.18TiB 45.20 0.93 - node cpn03 24 nvme 1.00000 1.00000 1.09TiB 500GiB 617GiB 44.77 0.92 145 osd.24 25 nvme 1.00000 1.00000 1.09TiB 462GiB 655GiB 41.37 0.85 133 osd.25 26 nvme 1.00000 1.00000 1.09TiB 446GiB 672GiB 39.88 0.82 129 osd.26 27 nvme 1.00000 1.00000 1.09TiB 565GiB 553GiB 50.54 1.04 162 osd.27 28 nvme 1.00000 1.00000 1.09TiB 513GiB 605GiB 45.89 0.94 148 osd.28 29 nvme 1.00000 1.00000 1.09TiB 554GiB 564GiB 49.55 1.01 160 osd.29 30 nvme 1.00000 1.00000 1.09TiB 527GiB 591GiB 47.12 0.97 152 osd.30 31 nvme 1.00000 1.00000 1.09TiB 484GiB 634GiB 43.31 0.89 140 osd.31 32 nvme 1.00000 1.00000 1.09TiB 505GiB 612GiB 45.21 0.93 146 osd.32 33 nvme 1.00000 1.00000 1.09TiB 551GiB 567GiB 49.28 1.01 159 osd.33 34 nvme 1.00000 1.00000 1.09TiB 498GiB 620GiB 44.52 0.91 143 osd.34 35 nvme 1.00000 1.00000 1.09TiB 457GiB 660GiB 40.93 0.84 132 osd.35 -16 12.00000 - 13.1TiB 6.00TiB 7.10TiB 45.77 0.94 - rack rack4 -4 12.00000 - 13.1TiB 6.00TiB 7.10TiB 45.77 0.94 - node cpn04 36 nvme 1.00000 1.00000 1.09TiB 540GiB 578GiB 48.29 0.99 156 osd.36 37 nvme 1.00000 1.00000 1.09TiB 517GiB 601GiB 46.25 0.95 149 osd.37 38 nvme 1.00000 1.00000 1.09TiB 519GiB 599GiB 46.42 0.95 149 osd.38 39 nvme 1.00000 1.00000 1.09TiB 457GiB 661GiB 40.85 0.84 132 osd.39 40 nvme 1.00000 1.00000 1.09TiB 527GiB 590GiB 47.17 0.97 152 osd.40 41 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.01 0.98 155 osd.41 42 nvme 1.00000 1.00000 1.09TiB 522GiB 596GiB 46.68 0.96 151 osd.42 43 nvme 1.00000 1.00000 1.09TiB 459GiB 658GiB 41.09 0.84 132 osd.43 44 nvme 1.00000 1.00000 1.09TiB 550GiB 568GiB 49.17 1.01 158 osd.44 45 nvme 1.00000 1.00000 1.09TiB 570GiB 548GiB 51.00 1.04 164 osd.45 46 nvme 1.00000 1.00000 1.09TiB 451GiB 667GiB 40.32 0.83 130 osd.46 47 nvme 1.00000 1.00000 1.09TiB 492GiB 626GiB 44.03 0.90 142 osd.47 -20 12.00000 - 10.9TiB 5.77TiB 5.15TiB 52.84 1.08 - rack rack5 -19 12.00000 - 10.9TiB 5.77TiB 5.15TiB 52.84 1.08 - node cpn05 48 nvme 1.00000 1.00000 931GiB 551GiB 380GiB 59.19 1.21 159 osd.48 49 nvme 1.00000 1.00000 931GiB 469GiB 462GiB 50.39 1.03 136 osd.49 50 nvme 1.00000 1.00000 931GiB 548GiB 384GiB 58.83 1.20 158 osd.50 51 nvme 1.00000 1.00000 931GiB 380GiB 551GiB 40.83 0.84 109 osd.51 52 nvme 1.00000 1.00000 931GiB 487GiB 445GiB 52.24 1.07 141 osd.52 53 nvme 1.00000 1.00000 931GiB 503GiB 429GiB 53.98 1.11 146 osd.53 54 nvme 1.00000 1.00000 931GiB 479GiB 452GiB 51.47 1.05 139 osd.54 55 nvme 1.00000 1.00000 931GiB 522GiB 410GiB 55.99 1.15 150 osd.55 56 nvme 1.00000 1.00000 931GiB 571GiB 360GiB 61.31 1.26 165 osd.56 57 nvme 1.00000 1.00000 931GiB 405GiB 527GiB 43.46 0.89 117 osd.57 58 nvme 1.00000 1.00000 931GiB 456GiB 475GiB 48.97 1.00 132 osd.58 59 nvme 1.00000 1.00000 931GiB 535GiB 396GiB 57.45 1.18 154 osd.59 -23 12.00000 - 10.9TiB 5.98TiB 4.93TiB 54.79 1.12 - rack rack6 -24 12.00000 - 10.9TiB 5.98TiB 4.93TiB 54.79 1.12 - node cpn06 60 nvme 1.00000 1.00000 931GiB 499GiB 432GiB 53.61 1.10 144 osd.60 61 nvme 1.00000 1.00000 931GiB 447GiB 485GiB 47.94 0.98 129 osd.61 62 nvme 1.00000 1.00000 931GiB 505GiB 426GiB 54.24 1.11 146 osd.62 63 nvme 1.00000 1.00000 931GiB 563GiB 368GiB 60.47 1.24 162 osd.63 64 nvme 1.00000 1.00000 931GiB 605GiB 326GiB 65.01 1.33 175 osd.64 65 nvme 1.00000 1.00000 931GiB 476GiB 455GiB 51.15 1.05 138 osd.65 66 nvme 1.00000 1.00000 931GiB 461GiB 471GiB 49.44 1.01 133 osd.66 67 nvme 1.00000 1.00000 931GiB 483GiB 448GiB 51.86 1.06 140 osd.67 68 nvme 1.00000 1.00000 931GiB 521GiB 411GiB 55.92 1.15 151 osd.68 69 nvme 1.00000 1.00000 931GiB 481GiB 450GiB 51.69 1.06 139 osd.69 70 nvme 1.00000 1.00000 931GiB 506GiB 426GiB 54.29 1.11 146 osd.70 71 nvme 1.00000 1.00000 931GiB 576GiB 355GiB 61.87 1.27 166 osd.71 -27 12.00000 - 10.9TiB 6.06TiB 4.85TiB 55.56 1.14 - rack rack7 -28 12.00000 - 10.9TiB 6.06TiB 4.85TiB 55.56 1.14 - node cpn07 72 nvme 1.00000 1.00000 931GiB 554GiB 378GiB 59.43 1.22 160 osd.72 73 nvme 1.00000 1.00000 931GiB 443GiB 489GiB 47.52 0.97 128 osd.73 74 nvme 1.00000 1.00000 931GiB 450GiB 481GiB 48.33 0.99 130 osd.74 75 nvme 1.00000 1.00000 931GiB 529GiB 403GiB 56.77 1.16 153 osd.75 76 nvme 1.00000 1.00000 931GiB 489GiB 443GiB 52.48 1.08 141 osd.76 77 nvme 1.00000 1.00000 931GiB 571GiB 360GiB 61.32 1.26 165 osd.77 78 nvme 1.00000 1.00000 931GiB 474GiB 458GiB 50.87 1.04 137 osd.78 79 nvme 1.00000 1.00000 931GiB 536GiB 395GiB 57.58 1.18 155 osd.79 80 nvme 1.00000 1.00000 931GiB 492GiB 440GiB 52.79 1.08 142 osd.80 81 nvme 1.00000 1.00000 931GiB 511GiB 421GiB 54.84 1.12 148 osd.81 82 nvme 1.00000 1.00000 931GiB 563GiB 368GiB 60.48 1.24 162 osd.82 83 nvme 1.00000 1.00000 931GiB 599GiB 332GiB 64.32 1.32 173 osd.83 TOTAL 85.1TiB 41.6TiB 43.6TiB 48.82 MIN/MAX VAR: 0.81/1.33 STDDEV: 6.30 Is there any documentation or scripts regarding manually distribution for rebalancing disk with minimal I/O impact on clients io? On Tue, Feb 4, 2020 at 10:25 AM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > With “ceph osd df tree” will be clear but right now I can see that some > %USE osd between 44% and 65%. > > > > Ceph osd df tree give also the balance at host level. > > > > Do you have balancer enabled ?No “perfect” distribution cause that you > cant use the full space. > > > > In our case we gain space manually rebalancing disk, that cause some > objects moves to other osd but you can so fast space available. > > > > Regards > > > > > > *De:* German Anders <yodasbunker(a)gmail.com> > *Enviado el:* martes, 4 de febrero de 2020 14:20 > *Para:* EDH - Manuel Rios <mriosfer(a)easydatahost.com> > *CC:* ceph-users(a)ceph.com > *Asunto:* Re: [ceph-users] Doubt about AVAIL space on df > > > > Hi Manuel, > > > > Sure thing: > > > > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 0 nvme 1.00000 1.00000 1.09TiB 496GiB 622GiB 44.35 0.91 143 > 1 nvme 1.00000 1.00000 1.09TiB 488GiB 630GiB 43.63 0.89 141 > 2 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.05 0.99 155 > 3 nvme 1.00000 1.00000 1.09TiB 473GiB 644GiB 42.36 0.87 137 > 4 nvme 1.00000 1.00000 1.09TiB 531GiB 587GiB 47.52 0.97 153 > 5 nvme 1.00000 1.00000 1.09TiB 476GiB 642GiB 42.55 0.87 137 > 6 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.77 0.86 135 > 7 nvme 1.00000 1.00000 1.09TiB 543GiB 574GiB 48.61 1.00 157 > 8 nvme 1.00000 1.00000 1.09TiB 481GiB 636GiB 43.08 0.88 139 > 9 nvme 1.00000 1.00000 1.09TiB 457GiB 660GiB 40.92 0.84 133 > 10 nvme 1.00000 1.00000 1.09TiB 513GiB 604GiB 45.92 0.94 148 > 11 nvme 1.00000 1.00000 1.09TiB 484GiB 634GiB 43.29 0.89 140 > 12 nvme 1.00000 1.00000 1.09TiB 498GiB 620GiB 44.57 0.91 144 > 13 nvme 1.00000 1.00000 1.09TiB 560GiB 557GiB 50.13 1.03 162 > 14 nvme 1.00000 1.00000 1.09TiB 576GiB 542GiB 51.55 1.06 167 > 15 nvme 1.00000 1.00000 1.09TiB 545GiB 572GiB 48.78 1.00 158 > 16 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.02 0.98 155 > 17 nvme 1.00000 1.00000 1.09TiB 507GiB 611GiB 45.36 0.93 147 > 18 nvme 1.00000 1.00000 1.09TiB 490GiB 628GiB 43.86 0.90 142 > 19 nvme 1.00000 1.00000 1.09TiB 533GiB 584GiB 47.72 0.98 155 > 20 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.75 0.86 134 > 21 nvme 1.00000 1.00000 1.09TiB 447GiB 671GiB 39.97 0.82 129 > 22 nvme 1.00099 1.00000 1.09TiB 561GiB 557GiB 50.16 1.03 162 > 23 nvme 1.00000 1.00000 1.09TiB 441GiB 677GiB 39.46 0.81 127 > 24 nvme 1.00000 1.00000 1.09TiB 500GiB 618GiB 44.72 0.92 145 > 25 nvme 1.00000 1.00000 1.09TiB 462GiB 656GiB 41.30 0.85 133 > 26 nvme 1.00000 1.00000 1.09TiB 445GiB 672GiB 39.85 0.82 129 > 27 nvme 1.00000 1.00000 1.09TiB 564GiB 554GiB 50.45 1.03 162 > 28 nvme 1.00000 1.00000 1.09TiB 512GiB 605GiB 45.84 0.94 148 > 29 nvme 1.00000 1.00000 1.09TiB 553GiB 565GiB 49.49 1.01 160 > 30 nvme 1.00000 1.00000 1.09TiB 526GiB 592GiB 47.07 0.97 152 > 31 nvme 1.00000 1.00000 1.09TiB 484GiB 633GiB 43.34 0.89 140 > 32 nvme 1.00000 1.00000 1.09TiB 504GiB 613GiB 45.13 0.93 146 > 33 nvme 1.00000 1.00000 1.09TiB 550GiB 567GiB 49.23 1.01 159 > 34 nvme 1.00000 1.00000 1.09TiB 497GiB 620GiB 44.51 0.91 143 > 35 nvme 1.00000 1.00000 1.09TiB 457GiB 661GiB 40.88 0.84 132 > 36 nvme 1.00000 1.00000 1.09TiB 539GiB 578GiB 48.25 0.99 156 > 37 nvme 1.00000 1.00000 1.09TiB 516GiB 601GiB 46.19 0.95 149 > 38 nvme 1.00000 1.00000 1.09TiB 518GiB 600GiB 46.35 0.95 149 > 39 nvme 1.00000 1.00000 1.09TiB 456GiB 662GiB 40.81 0.84 132 > 40 nvme 1.00000 1.00000 1.09TiB 527GiB 591GiB 47.13 0.97 152 > 41 nvme 1.00000 1.00000 1.09TiB 536GiB 581GiB 47.98 0.98 155 > 42 nvme 1.00000 1.00000 1.09TiB 521GiB 597GiB 46.62 0.96 151 > 43 nvme 1.00000 1.00000 1.09TiB 459GiB 659GiB 41.05 0.84 132 > 44 nvme 1.00000 1.00000 1.09TiB 549GiB 569GiB 49.12 1.01 158 > 45 nvme 1.00000 1.00000 1.09TiB 569GiB 548GiB 50.95 1.04 164 > 46 nvme 1.00000 1.00000 1.09TiB 450GiB 668GiB 40.28 0.83 130 > 47 nvme 1.00000 1.00000 1.09TiB 491GiB 626GiB 43.97 0.90 142 > 48 nvme 1.00000 1.00000 931GiB 551GiB 381GiB 59.13 1.21 159 > 49 nvme 1.00000 1.00000 931GiB 469GiB 463GiB 50.34 1.03 136 > 50 nvme 1.00000 1.00000 931GiB 548GiB 384GiB 58.78 1.21 158 > 51 nvme 1.00000 1.00000 931GiB 380GiB 552GiB 40.79 0.84 109 > 52 nvme 1.00000 1.00000 931GiB 486GiB 445GiB 52.20 1.07 141 > 53 nvme 1.00000 1.00000 931GiB 502GiB 429GiB 53.93 1.11 146 > 54 nvme 1.00000 1.00000 931GiB 479GiB 452GiB 51.42 1.05 139 > 55 nvme 1.00000 1.00000 931GiB 521GiB 410GiB 55.93 1.15 150 > 56 nvme 1.00000 1.00000 931GiB 570GiB 361GiB 61.25 1.26 165 > 57 nvme 1.00000 1.00000 931GiB 404GiB 527GiB 43.43 0.89 117 > 58 nvme 1.00000 1.00000 931GiB 455GiB 476GiB 48.89 1.00 132 > 59 nvme 1.00000 1.00000 931GiB 535GiB 397GiB 57.39 1.18 154 > 60 nvme 1.00000 1.00000 931GiB 499GiB 433GiB 53.56 1.10 144 > 61 nvme 1.00000 1.00000 931GiB 446GiB 485GiB 47.92 0.98 129 > 62 nvme 1.00000 1.00000 931GiB 505GiB 427GiB 54.18 1.11 146 > 63 nvme 1.00000 1.00000 931GiB 563GiB 369GiB 60.39 1.24 162 > 64 nvme 1.00000 1.00000 931GiB 605GiB 326GiB 64.99 1.33 175 > 65 nvme 1.00000 1.00000 931GiB 476GiB 455GiB 51.10 1.05 138 > 66 nvme 1.00000 1.00000 931GiB 460GiB 471GiB 49.38 1.01 133 > 67 nvme 1.00000 1.00000 931GiB 483GiB 449GiB 51.82 1.06 140 > 68 nvme 1.00000 1.00000 931GiB 520GiB 411GiB 55.86 1.15 151 > 69 nvme 1.00000 1.00000 931GiB 481GiB 450GiB 51.64 1.06 139 > 70 nvme 1.00000 1.00000 931GiB 505GiB 426GiB 54.24 1.11 146 > 71 nvme 1.00000 1.00000 931GiB 576GiB 356GiB 61.81 1.27 166 > 72 nvme 1.00000 1.00000 931GiB 552GiB 379GiB 59.30 1.22 160 > 73 nvme 1.00000 1.00000 931GiB 442GiB 489GiB 47.47 0.97 128 > 74 nvme 1.00000 1.00000 931GiB 450GiB 482GiB 48.28 0.99 130 > 75 nvme 1.00000 1.00000 931GiB 529GiB 403GiB 56.77 1.16 153 > 76 nvme 1.00000 1.00000 931GiB 488GiB 443GiB 52.44 1.08 141 > 77 nvme 1.00000 1.00000 931GiB 570GiB 361GiB 61.25 1.26 165 > 78 nvme 1.00000 1.00000 931GiB 473GiB 458GiB 50.79 1.04 137 > 79 nvme 1.00000 1.00000 931GiB 536GiB 396GiB 57.54 1.18 155 > 80 nvme 1.00000 1.00000 931GiB 491GiB 440GiB 52.74 1.08 142 > 81 nvme 1.00000 1.00000 931GiB 510GiB 421GiB 54.78 1.12 148 > 82 nvme 1.00000 1.00000 931GiB 563GiB 369GiB 60.42 1.24 162 > 83 nvme 1.00000 1.00000 931GiB 599GiB 333GiB 64.28 1.32 173 > TOTAL 85.1TiB 41.5TiB 43.6TiB 48.77 > MIN/MAX VAR: 0.81/1.33 STDDEV: 6.30 > > > > > > Thanks in advance, > > > > Best regards, > > > > > > > > On Tue, Feb 4, 2020 at 10:15 AM EDH - Manuel Rios < > mriosfer(a)easydatahost.com> wrote: > > Hi German, > > Can you post , ceph osd df tree ? > > Looks like your usage distribution is not perfect and that's why you got > less space than real. > Regards > > > -----Mensaje original----- > De: German Anders <yodasbunker(a)gmail.com> > Enviado el: martes, 4 de febrero de 2020 14:00 > Para: ceph-users(a)ceph.com > Asunto: [ceph-users] Doubt about AVAIL space on df > > Hello Everyone, > > I would like to understand if this output is right: > > *# ceph df* > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 85.1TiB 43.7TiB 41.4TiB 48.68 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > volumes 13 13.8TiB 64.21 7.68TiB 3620495 > > I only have (1) pool called 'volumes' which is using 13.8TiB (we have a > replica of 3) so it's actually using 41,4TiB and that would be the RAW > USED, at this point is fine, but, then it said on the GLOBAL section that > the AVAIL space is 43.7TiB and the %RAW USED is only 48.68%. > > So if I use the 7.68TiB of MAX AVAIL and the pool goes up to 100% of > usage, that would not lead to the total space of the cluster, right? I mean > were are those 43.7TiB of AVAIL space? > > I'm using Luminous 12.2.12 release. > > Sorry if it's a silly question or if it has been answered before. > > Thanks in advance, > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0