- ceph-users - lists.ceph.io

osd crashing and rocksdb corruption
by Francois Legrand 27 Apr '20

27 Apr '20

Hi all, *** Short version *** Is there a way to repair a rocksdb from errors "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ? *** Long version *** We operate a nautilus ceph cluster (with 100 disks of 8TB in 6 servers + 4 mons/mgr + 3 mds). We recently (Monday 20) upgraded from 14.2.7 to 14.2.8. This triggered a rebalancing of some data. Two days later (Wednesday 22) we had a very short power outage. Only one of the osd servers went down (and unfortunately died). This triggered a reconstruction of the losts osds. Operations went fine until Saturday 25 where some osds in the 5 remaining servers started to crash apparently with no reasons. We tryed to restart them, but they crashed again. We ended with 18 osd down (+ 16 in the dead server so 34 osd downs out of 100). Looking at the logs we found for all the crashed osd : -237> 2020-04-25 16:32:51.835 7f1f45527a80 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2729370997 in db/181355.sst offset 18446744073709551615 size 18446744073709551615 and 2020-04-25 16:05:47.251 7fcbd1e46a80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db erroring opening db: We also noticed that the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" was present few days before the crash. We also have some osd with this error but still up. We tryed to repair with : ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-3 repair But no success (it ends with _open_db erroring opening db). Thus does somebody have an idea to fix this or at least know if it's possible to repair and correct the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ! Thanks for your help (we are desperate because we will loose datas and are fighting to save something) !!! F.

1 0

Re: HBase/HDFS on Ceph/CephFS
by Xiaoxi Chen 27 Apr '20

27 Apr '20

RBD is never a workable solution unless you want to pay the cost of double-replication in both HDFS and Ceph. I think the right approach is thinking about other implementation of the FileSystem interface, like s3a and localfs. s3a is straight forward, ceph rgw provide s3 interface and s3a is stable and well tested in Hadoop ecosystem, just run it. There are a few other in-house solution offered by some vendor that integrating librgw into the s3a driver so it saves one extra hop and the management/LB cost of maintaining an RGW cluster. local filesystem is a bit tricky, we just tried a POC that mounting CephFS into every hadoop , configure Hadoop using LocalFS with Replica = 1. Which end up with each data only write once into cephfs and cephfs take care of the data durability. There was a libcephfs-jni but it is significantly out of date and seems be abandoned, which is a pity. For both solutions for sure you lost the locality , but trading for better scalability and compute/storage separation. -Xiaoxi Marc Roos <M.Roos(a)f1-outsourcing.eu> 于2020年4月24日周五下午4:00写道： > > I think the idea behind pool size of 1, is that hadoop already writes > copies to 2 other pools(?). > > However that leaves the possibility that pg's of these 3 pools can maybe > share an osd, and if that osd fails, you loose data in these pools. I > have no idea what the chances are that the same data of different pools > can end up on the same osd. > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] HBase/HDFS on Ceph/CephFS > > Hi > > We have an 3 year old Hadoop cluster - up for refresh - so it is time to > evaluate options. The "only" usecase is running an HBase installation > which is important for us and migrating out of HBase would be a hazzle. > > Our Ceph usage has expanded and in general - we really like what we see. > > Thus - Can this be "sanely" consolidated somehow? I have seen this: > https://docs.ceph.com/docs/jewel/cephfs/hadoop/ > But it seem really-really bogus to me. > > It recommends that you set: > pool 3 'hadoop1' rep size 1 min_size 1 > > Which would - if I understand correct - be disastrous. The Hadoop end > would replicated in 3 across - but within Ceph the replication would be > 1. > The 1 replication in ceph means pulling the OSD node would "gaurantee" > the pg's to go inactive - which could be ok - but there is nothing > gauranteeing that the other Hadoop replicas are not served out of the > same OSD-node/pg? In which case - rebooting an OSD node would bring the > hadoop cluster unavailable. > > Is anyone serving HBase out of Ceph - how does the stadck and > configuration look? If I went for 3 x replication in both Ceph and HDFS > then it would definately work, but 9x copies of the dataset is a bit > more than what looks feasible at the moment. > > Thanks for your reflections/input. > > Jesper > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

active+remapped+backfilling keeps going .. and going
by Kyriazis, George 27 Apr '20

27 Apr '20

Hello, I have a Proxmox ceph cluster with 5 nodes and 3 OSDs each (total 15 OSDs), on a 10G network. The cluster started small, and I’ve progressively added OSDs over time. Problem is…. The cluster never rebalances completely. There is always progress on backfilling, but PGs that used to be in active+clean state jump back into the active+remapped+backfilling (or active+remapped+backfill_wait) state, to be moved to different OSDs. Initially I had a 1G network (recently upgraded to 10G), and I was holding on the backfill settings (osd_max_backfills and osd_recovery_sleep_hdd). I just recently (last few weeks) upgraded to 10G, with osd_max_backfills = 50 and osd_recovery_sleep_hdd = 0 (only HDDs, no SSDs). Cluster has been backfilling for months now with no end in sight. Is this normal behavior? Is there any setting that I can look at that till give me an idea as to why PGs are jumping back into remapped from clean? Below is output of “ceph osd tree” and “ceph osd df”: # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72472 root default -9 40.01666 host vis-hsw-01 3 hdd 10.91309 osd.3 up 1.00000 1.00000 6 hdd 14.55179 osd.6 up 1.00000 1.00000 10 hdd 14.55179 osd.10 up 1.00000 1.00000 -13 40.01666 host vis-hsw-02 0 hdd 10.91309 osd.0 up 1.00000 1.00000 7 hdd 14.55179 osd.7 up 1.00000 1.00000 11 hdd 14.55179 osd.11 up 1.00000 1.00000 -11 40.01666 host vis-hsw-03 4 hdd 10.91309 osd.4 up 1.00000 1.00000 8 hdd 14.55179 osd.8 up 1.00000 1.00000 12 hdd 14.55179 osd.12 up 1.00000 1.00000 -3 40.01666 host vis-hsw-04 5 hdd 10.91309 osd.5 up 1.00000 1.00000 9 hdd 14.55179 osd.9 up 1.00000 1.00000 13 hdd 14.55179 osd.13 up 1.00000 1.00000 -15 43.65807 host vis-hsw-05 1 hdd 14.55269 osd.1 up 1.00000 1.00000 2 hdd 14.55269 osd.2 up 1.00000 1.00000 14 hdd 14.55269 osd.14 up 1.00000 1.00000 -5 0 host vis-ivb-07 -7 0 host vis-ivb-10 # # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 3 hdd 10.91309 1.00000 11 TiB 8.2 TiB 8.2 TiB 552 MiB 25 GiB 2.7 TiB 75.08 1.19 131 up 6 hdd 14.55179 1.00000 15 TiB 9.1 TiB 9.1 TiB 1.2 GiB 30 GiB 5.5 TiB 62.47 0.99 148 up 10 hdd 14.55179 1.00000 15 TiB 8.1 TiB 8.1 TiB 1.5 GiB 20 GiB 6.4 TiB 55.98 0.89 142 up 0 hdd 10.91309 1.00000 11 TiB 7.5 TiB 7.4 TiB 504 MiB 24 GiB 3.5 TiB 68.34 1.09 120 up 7 hdd 14.55179 1.00000 15 TiB 8.7 TiB 8.7 TiB 1.0 GiB 31 GiB 5.8 TiB 60.07 0.95 144 up 11 hdd 14.55179 1.00000 15 TiB 9.4 TiB 9.3 TiB 819 MiB 20 GiB 5.2 TiB 64.31 1.02 147 up 4 hdd 10.91309 1.00000 11 TiB 7.0 TiB 7.0 TiB 284 MiB 25 GiB 3.9 TiB 64.35 1.02 112 up 8 hdd 14.55179 1.00000 15 TiB 9.3 TiB 9.2 TiB 1.8 GiB 29 GiB 5.3 TiB 63.65 1.01 157 up 12 hdd 14.55179 1.00000 15 TiB 8.6 TiB 8.6 TiB 623 MiB 19 GiB 5.9 TiB 59.14 0.94 136 up 5 hdd 10.91309 1.00000 11 TiB 8.6 TiB 8.6 TiB 542 MiB 29 GiB 2.3 TiB 79.01 1.26 134 up 9 hdd 14.55179 1.00000 15 TiB 8.2 TiB 8.2 TiB 707 MiB 27 GiB 6.3 TiB 56.56 0.90 138 up 13 hdd 14.55179 1.00000 15 TiB 8.7 TiB 8.7 TiB 741 MiB 18 GiB 5.8 TiB 59.85 0.95 134 up 1 hdd 14.55269 1.00000 15 TiB 9.8 TiB 9.8 TiB 1.3 GiB 20 GiB 4.8 TiB 67.18 1.07 158 up 2 hdd 14.55269 1.00000 15 TiB 8.7 TiB 8.7 TiB 936 MiB 18 GiB 5.8 TiB 60.04 0.95 148 up 14 hdd 14.55269 1.00000 15 TiB 8.3 TiB 8.3 TiB 673 MiB 18 GiB 6.3 TiB 56.97 0.90 131 up TOTAL 204 TiB 128 TiB 128 TiB 13 GiB 350 GiB 75 TiB 62.95 MIN/MAX VAR: 0.89/1.26 STDDEV: 6.44 # Thank you! George

5 9

Re: rbd perf image iostat - rbdmap write latency
by Jason Dillaman 27 Apr '20

27 Apr '20

On Mon, Apr 27, 2020 at 7:38 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > I guess this is not good for ssd (samsung sm863)? Or do I need to devide > 14.8 by 40? > The 14.8 ms number is the average latency coming from the OSDs, so no need to divide the number by anything. What is the size of your writes? At 40 writes/sec against an SSD-backed cluster, I can only hope they are large IOs. > > rbd perf image iostat > > NAME WR RD WR_BYTES RD_BYTES WR_LAT > RD_LAT > rbd.ssd/vps-test 40/s 0/s 5.0 MiB/s 0 B/s 14.84 ms > 0.00 ns > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Jason

1 0

rbd perf image iostat - rbdmap write latency
by Marc Roos 27 Apr '20

27 Apr '20

I guess this is not good for ssd (samsung sm863)? Or do I need to devide 14.8 by 40? rbd perf image iostat NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT rbd.ssd/vps-test 40/s 0/s 5.0 MiB/s 0 B/s 14.84 ms 0.00 ns

1 0

Not able to start object gateway
by Sailaja Yedugundla 27 Apr '20

27 Apr '20

I am trying to setup a single node cluster using cephadm. I was able to start the cluster with 1 monitor and 3 osds. When I try to create users for rados gateway using the command radosgw-admin user create, the command hangs. Here is my cluster status. cluster: id: 5a03d7e2-85e4-11ea-bba9-021f94750a41 health: HEALTH_WARN Reduced data availability: 44 pgs inactive Degraded data redundancy: 6/15 objects degraded (40.000%), 2 pgs degraded, 80 pgs undersized services: mon: 1 daemons, quorum ip-172-31-9-253.ec2.internal (age 9h) mgr: ip-172-31-9-253.ec2.internal.umudgc(active, since 9h) osd: 3 osds: 3 up (since 74m), 3 in (since 74m); 41 remapped pgs data: pools: 4 pools, 97 pgs objects: 5 objects, 1.2 KiB usage: 3.0 GiB used, 27 GiB / 30 GiB avail pgs: 45.361% pgs not active 6/15 objects degraded (40.000%) 2/15 objects misplaced (13.333%) 44 undersized+peered 24 active+undersized+remapped 17 active+clean 10 active+undersized 2 active+undersized+degraded progress: Rebalancing after osd.1 marked in (74m) [=========...................] (remaining: 2h) Rebalancing after osd.2 marked in (74m) [............................] Can someone help me resolving this issue.

2 2

Device /dev/rbd0 excluded by a filter.
by Marc Roos 27 Apr '20

27 Apr '20

Why is this info not available??? https://access.redhat.com/solutions/4009431

2 5

best hair transplant clinic in bangalore
by maachairtransplant123＠gmail.com 27 Apr '20

27 Apr '20

Best hair transplant in bangalore:Get advanced hair loss treatment by best Hair specialists in Bangalore! We have award winning doctors for Hair transplantation. Contact: https://www.maachairtransplantbangalore.com/

1 0

Sporadic mgr segmentation fault
by XuYun 26 Apr '20

26 Apr '20

Dear ceph users, We are experiencing sporadic mgr crash in all three ceph clusters (version 14.2.6 and version 14.2.8), the crash log is: 2020-04-17 23:10:08.986 7fed7fe07700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: In function 'const char* ceph::buffer::v14_2_0::ptr::c_str() const' thread 7fed7fe07700 time 2020-04-17 23:10:08.984887 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: 578: FAILED ceph_assert(_raw) ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fed8605c325] 2: (()+0x2534ed) [0x7fed8605c4ed] 3: (()+0x5a21ed) [0x7fed863ab1ed] 4: (PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)+0xbd) [0x7fed863840ed] 5: (AsyncConnection::_try_send(bool)+0xb6) [0x7fed8632fc76] 6: (ProtocolV2::write_message(Message*, bool)+0x832) [0x7fed8635bf52] 7: (ProtocolV2::write_event()+0x175) [0x7fed863718c5] 8: (AsyncConnection::handle_write()+0x40) [0x7fed86332600] 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1397) [0x7fed8637f997] 10: (()+0x57c977) [0x7fed86385977] 11: (()+0x80bdaf) [0x7fed86614daf] 12: (()+0x7e65) [0x7fed8394ce65] 13: (clone()+0x6d) [0x7fed825fa88d] 2020-04-17 23:10:08.990 7fed7ee05700 -1 *** Caught signal (Segmentation fault) ** in thread 7fed7ee05700 thread_name:msgr-worker-2 ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) 1: (()+0xf5f0) [0x7fed839545f0] 2: (ceph::buffer::v14_2_0::ptr::release()+0x8) [0x7fed863aafd8] 3: (ceph::crypto::onwire::AES128GCM_OnWireTxHandler::~AES128GCM_OnWireTxHandler()+0x59) [0x7fed86388669] 4: (ProtocolV2::reset_recv_state()+0x11f) [0x7fed8635f5af] 5: (ProtocolV2::stop()+0x77) [0x7fed8635f857] 6: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr<AsyncConnection>)+0x5ef) [0x7fed86374f8f] 7: (ProtocolV2::handle_client_ident(ceph::buffer::v14_2_0::list&)+0xd9c) [0x7fed8637673c] 8: (ProtocolV2::handle_frame_payload()+0x1fb) [0x7fed86376c1b] 9: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x7fed86376e70] 10: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)+0x44d) [0x7fed863773cd] 11: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x7fed86360534] 12: (AsyncConnection::process()+0x186) [0x7fed86330656] 13: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7fed8637f015] 14: (()+0x57c977) [0x7fed86385977] 15: (()+0x80bdaf) [0x7fed86614daf] 16: (()+0x7e65) [0x7fed8394ce65] 17: (clone()+0x6d) [0x7fed825fa88d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Any thoughts about this issue? Xu Yun

2 4

Is "." a legal characted in device class or not?
by Frank Schilder 25 Apr '20

25 Apr '20

Dear all, I created https://tracker.ceph.com/issues/45253 on an inconsistency in allowed characters in device class names. Short story: I created a device class with name "rbd.meta" and all worked fine until I did [root@gnosis ~]# ceph osd crush class ls-osd rbd.meta Invalid command: invalid chars . in rbd.meta osd crush class ls-osd <class> : list all osds belonging to the specific <class> Error EINVAL: invalid command Question is now, is "." legal or should I change? Note: I really like the dot notation here and everything seems to work as expected, except the "class ls-osd" command. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0