- ceph-users - lists.ceph.io

Large RocksDB (db_slow_bytes) on OSD which is marked as out
by Wido den Hollander 31 Aug '20

31 Aug '20

Hello, On a Nautilus 14.2.8 cluster I am seeing large RocksDB database with many slow DB bytes in use. To investigate this further I marked one OSD as out and waited for the all the backfilling to complete. Once the backfilling was completed I exported BlueFS and investigated the RocksDB using 'ceph-kvstore-tool'. This resulted in 22GB of data. Listing all the keys in the RocksDB shows me there are 747.000 keys in the DB. A small portion are osdmaps, but the biggest amount are keys prefixed with 'b'. I dumped the stats of the RocksDB and this shows me: L1: 1/0: 439.32 KB L2: 1/0: 2.65 MB L3: 5/0: 14.36 MB L4: 127/0: 7.22 GB L5: 217/0: 13.73 GB Sum: 351/0: 20.98 GB So there is almost 21GB of data in this RocksDB database. Why? Where is this coming from? Throughout this cluster OSDs are suffering from many slow bytes used and I can't figure out why. Has anybody seen this or has a clue on what is going on? I have an external copy of this RocksDB database to do investigations on. Thank you, Wido

2 3

How to query status of scheduled commands.
by Frank Schilder 31 Aug '20

31 Aug '20

Hi all, can anyone help me with this? In mimic, for any of these commands: ceph osd [deep-]scrub ID ceph pg [deep-]scrub ID ceph pg repair ID an operation is scheduled asynchronously. How can I check the following states: 1) Operation is pending (scheduled, not started). 2) Operation is running. 3) Operation has completed. 4) Exit code and error messages if applicable. Many thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Re: osd regularly wrongly marked down
by Francois Legrand 31 Aug '20

31 Aug '20

We tried to rise the osd_memory_target from 4 to 8G but the problem still occurs (osd wrongly marked down few times a day). Does somebody have any clue ? F. > On Fri, Aug 28, 2020 at 10:34 AM Francois Legrand > <fleg(a)lpnhe.in2p3.fr <mailto:fleg@lpnhe.in2p3.fr>> wrote: > > Hi all, > > We have a ceph cluster in production with 6 osds servers (with > 16x8TB > disks), 3 mons/mgrs and 3 mdss. Both public and cluster > networks are in > 10GB and works well. > > After a major crash in april, we turned the option > bluefs_buffered_io to > false to workaround the large write bug when > bluefs_buffered_io was > true (we were in version 14.2.8 and the default value at this > time was > true). > Since that time, we regularly have some osds wrongly marked > down by the > cluster after heartbeat timeout (heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15). > > Generally the osd restart and the cluster is back healthy, but > several > time, after many of these kick-off the osd reach the > osd_op_thread_suicide_timeout and goes down definitely. > > We increased the osd_op_thread_timeout and > osd_op_thread_suicide_timeout... The problems still occurs > (but less > frequently). > > Few days ago, we upgraded to 14.2.11 and revert the timeout to > their > default value, hoping that it will solve the problem (we > thought that it > should be related to this bug > https://tracker.ceph.com/issues/45943) > but it didn't. We still have some osds wrongly marked down. > > Can somebody help us to fix this problem ? > Thanks. > > Here is an extract of an osd log at failure time: > > --------------------------------- > 2020-08-28 02:19:05.019 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 44.7d scrub starts > 2020-08-28 02:19:25.755 7f040e43d700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:19:25.755 7f040dc3c700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > this last line is repeated more than 1000 times > ... > 2020-08-28 02:20:17.484 7f040d43b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:20:17.551 7f03f1384700 0 > bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow > operation > observed for _collection_list, latency = 67.3532s, lat = 67s cid > =44.7d_head start GHMAX end GHMAX max 25 > ... > 2020-08-28 02:20:22.600 7f040dc3c700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:21:20.774 7f03f1384700 0 > bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow > operation > observed for _collection_list, latency = 63.223s, lat = 63s cid > =44.7d_head start > #44:beffc78d:::rbd_data.1e48e8ab988992.00000000000011bd:0# end > #MAX# max > 2147483647 > 2020-08-28 02:21:20.774 7f03f1384700 1 heartbeat_map > reset_timeout > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:21:20.805 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 44.7d scrub ok > 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) > log [WRN] : > Monitor daemon marked osd.16 down, but it is still running > 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) > log [DBG] : > map e609411 wrongly marked me down at e609410 > 2020-08-28 02:21:21.099 7f03fd997700 1 osd.16 609411 > start_waiting_for_healthy > 2020-08-28 02:21:21.119 7f03fd997700 1 osd.16 609411 start_boot > 2020-08-28 02:21:21.124 7f03f0b83700 1 osd.16 pg_epoch: 609410 > pg[36.3d0( v 609409'481293 (449368'478292,609409'481293] > local-lis/les=609403/609404 n=154651 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609410/609410/608752) > [25,72] r=-1 > lpr=609410 pi=[609403,609410)/1 luod=0'0 lua=609392'481198 > crt=609409'481293 lcod 609409'481292 active mbc={}] > start_peering_interval up [25,72,16] -> [25,72], acting > [25,72,16] -> > [25,72], acting_primary 25 -> 25, up_primary 25 -> 25, role 2 > -> -1, > features acting 4611087854031667199 upacting 4611087854031667199 > ... > 2020-08-28 02:21:21.166 7f03f0b83700 1 osd.16 pg_epoch: 609411 > pg[36.56( v 609409'480511 (449368'477424,609409'480511] > local-lis/les=609403/609404 n=153854 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609410/609410/609410) > [103,102] > r=-1 lpr=609410 pi=[609403,609410)/1 crt=609409'480511 lcod > 609409'480510 unknown NOTIFY mbc={}] state<Start>: > transitioning to Stray > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > public network em1 numa node 0 > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > cluster network em2 numa node 0 > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > objectstore and network numa nodes do not match > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > not setting numa affinity > 2020-08-28 02:21:21.566 7f040a435700 1 osd.16 609413 tick > checking mon > for new map > 2020-08-28 02:21:22.515 7f03fd997700 1 osd.16 609414 state: > booting -> > active > 2020-08-28 02:21:22.515 7f03f0382700 1 osd.16 pg_epoch: 609414 > pg[36.20( v 609409'483167 (449368'480117,609409'483167] > local-lis/les=609403/609404 n=155171 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609414/609414/609361) > [97,16,72] > r=1 lpr=609414 pi=[609403,609414)/1 crt=609409'483167 lcod > 609409'483166 > unknown NOTIFY mbc={}] start_peering_interval up [97,72] -> > [97,16,72], > acting [97,72] -> [97,16,72], acting_primary 97 -> 97, > up_primary 97 -> > 97, role -1 -> 1, features acting 4611087854031667199 upacting > 4611087854031667199 > ... > 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 > pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] > local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) > [16,34,21] > r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod > 609409'479795 > mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up > [34,21] -> > [16,34,21], acting [34,21] -> [16,34,21], acting_primary 34 -> > 16, > up_primary 34 -> 16, role -1 -> 0, features acting > 4611087854031667199 > upacting 4611087854031667199 > 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 > pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] > local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) > [16,34,21] > r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod > 609409'479795 > mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary > 2020-08-28 02:21:24.738 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 36.2f3 scrub starts > 2020-08-28 02:22:18.857 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 36.2f3 scrub ok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> >

1 0

How to repair rbd image corruption
by Jared 31 Aug '20

31 Aug '20

Hi, I have a rook cluster running with ceph 12.2.7 for almost one year. Recently some pvc couldn’t be attached with error as below, Warning FailedMount 7m19s kubelet, 192.168.34.119 MountVolume.SetUp failed for volume "pvc-8f4ca7ac-42ab-11ea-99d7-005056b84936" : mount command failed, status: Failure, reason: failed to mount volume /dev/rbd1 [ext4] to /var/lib/kubelet/plugins/rook.io/rook-ceph/mounts/pvc-8f4ca7ac-42ab-11ea-99d7-005056b84936, error 'fsck' found errors on device /dev/rbd1 but could not correct them: fsck from util-linux 2.23.2 /dev/rbd1: recovering journal /dev/rbd1 contains a file system with errors, check forced. /dev/rbd1: Inode 393244, end of extent exceeds allowed value (logical block 512, physical block 12091904, len 4388) After mapping storage to block device and run "fsck -y" on it, it indicated that filesystem was clean. Then retry to mount the storage, it still reported the same error as above. Response from "ceph status” is as below, cluster: id: 54a729b6-7b59-4e5b-bc09-7dc99109cbad health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set Degraded data redundancy: 50711/152133 objects degraded (33.333%), 100 pgs degraded, 100 pgs undersized mons rook-ceph-mon41,rook-ceph-mon44 are low on available space services: mon: 3 daemons, quorum rook-ceph-mon44,rook-ceph-mon47,rook-ceph-mon41 mgr: a(active) osd: 3 osds: 3 up, 3 in flags noscrub,nodeep-scrub data: pools: 1 pools, 100 pgs objects: 50711 objects, 190 GB usage: 383 GB used, 1111 GB / 1495 GB avail pgs: 50711/152133 objects degraded (33.333%) 100 active+undersized+degraded io: client: 78795 B/s wr, 0 op/s rd, 11 op/s wr From output, why is scrub disabled? Could I trigger it manually? And how could I check which pgs or objects is corrupted? Any advice for fixing the issue? Thanks for your help. Jared

1 0

Re: Erasure coding RBD pool for OpenStack
by Lazuardi Nasution 31 Aug '20

31 Aug '20

Hi Konstantin, I hope you or anybody still follows this old thread. Can this EC data pool be configured per pool, not per client? If we follow https://docs.ceph.com/docs/master/rbd/rbd-openstack/ we may see that cinder client will access vms and volumes pools, both with read and write permission. How can we handle this? If we config with different clients for nova (vms) and cinder (volumes), I think there will be a problem if there is cross pool access, especially on write. Let's say that client nova will create volume on instance creation for booting from that volume. Any thoughts? Best regards, > Date: Wed, 11 Jul 2018 11:16:27 +0700 > From: Konstantin Shalygin <k0ste(a)k0ste.ru> > To: ceph-users(a)lists.ceph.com > Subject: Re: [ceph-users] Erasure coding RBD pool for OpenStack > Glance, Nova and Cinder > Message-ID: <069ac368-22b0-3d18-937b-70ce39287cb1(a)k0ste.ru> > Content-Type: text/plain; charset=utf-8; format=flowed > > > So if you want, two more questions to you : > > > > - How do you handle your ceph.conf configuration (default data pool by > > user) / distribution ? Manually, config management, openstack-ansible... > > ? > > - Did you made comparisons, benchmarks between replicated pools and EC > > pools, on the same hardware / drives ? I read that small writes are not > > very performant with EC. > > > ceph.conf with default data pool is only need for Cinder at image > creation time, after this luminous+ rbd client will be found feature > "data-pool" and will perform data-io to this pool. > > > > # rbd info > > erasure_rbd_meta/volume-09ed44bf-7d16-453a-b712-a636a0d3d812 <----- > > meta pool ! > > rbd image 'volume-09ed44bf-7d16-453a-b712-a636a0d3d812': > > ??????? size 1500 GB in 384000 objects > > ??????? order 22 (4096 kB objects) > > ??????? data_pool: erasure_rbd_data??????? <----- our data pool > > ??????? block_name_prefix: rbd_data.6.a2720a1ec432bf > > ??????? format: 2 > > ??????? features: layering, exclusive-lock, object-map, fast-diff, > > deep-flatten, data-pool????????? <----- "data-pool" feature > > ??????? flags: > > ??????? create_timestamp: Sat Jan 27 20:24:04 2018 > > > > k > > > > ------------------------------ >

2 8

Re: Speeding up reconnection
by William Edwards 31 Aug '20

31 Aug '20

I replaced the VMs taking care of routing between clients and MDSes by physical machines. Problems below are solved. It seems to have been related to issues with the virtual NIC. It seemed to work well with E1000 instead of VirtIO... Met vriendelijke groeten, William Edwards ----- Original Message ----- From: William Edwards (wedwards(a)cyberfusion.nl) Date: 08/11/20 11:38 To: ceph-users(a)ceph.io Subject: Speeding up reconnection Hello, When connection is lost between kernel client, a few things happen: 1. Caps become stale: Aug 11 11:08:14 admin-cap kernel: [308405.227718] ceph: mds0 caps stale 2. MDS evicts client for being unresponsive: MDS log: 2020-08-11 11:12:08.923 7fd1f45ae700 0 log_channel(cluster) log [WRN] : evicting unresponsive client admin-cap.cf.ha.cyberfusion.cloud:DB0001-cap (144786749), after 300.978 seconds Client log: Aug 11 11:12:11 admin-cap kernel: [308643.051006] ceph: mds0 hung 3. Socket is closed: Aug 11 11:22:57 admin-cap kernel: [309289.192705] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state OPEN) I am not sure whether the kernel client or MDS closes the connection. I think the kernel client does so, because nothing is logged at the MDS side at 11:22:57 4. Connection is reset by MDS: MDS log: 2020-08-11 11:22:58.831 7fd1f9e49700 0 --1- [v2:[fdb7:b01e:7b8e:0:10:10:10:1]:6800/3619156441,v1:[fdb7:b01e:7b8e:0:10:10:10:1]:6849/3619156441] >> v1:[fc00:b6d:cfc:951::7]:0/133007863 conn(0x55bfaf1c2880 0x55c16cb47000 :6849 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 1), sending RESETSESSION Client log: Aug 11 11:22:58 admin-cap kernel: [309290.058222] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 connection reset 5. Kernel client reconnects: Aug 11 11:22:58 admin-cap kernel: [309290.058972] ceph: mds0 closed our session Aug 11 11:22:58 admin-cap kernel: [309290.058973] ceph: mds0 reconnect start Aug 11 11:22:58 admin-cap kernel: [309290.069979] ceph: mds0 reconnect denied Aug 11 11:22:58 admin-cap kernel: [309290.069996] ceph: dropping file locks for 000000006a23d9dd 1099625041446 Aug 11 11:22:58 admin-cap kernel: [309290.071135] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state NEGOTIATING) Question: As you can see, there's 10 minutes between losing the connection and the reconnection attempt (11:12:08 - 11:22:58). I could not find any settings related to the period after which reconnection is attempted. I would like to change this value from 10 minutes to something like 1 minute. I also tried searching the Ceph docs for the string '600' (10 minutes), but did not find anything useful. Hope someone can help. Environment details: Client kernel: 4.19.0-10-amd64 Ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable) Met vriendelijke groeten, William Edwards

1 0

issues with object-map in benji
by Pavel Vondřička 31 Aug '20

31 Aug '20

Hello, Lately, we upgraded Ceph to version 15.2.4 and shortly after that we had a blackout, which caused a restart of all servers at once (BTW, Ceph did not come up well itself). Since then we were receiving lots of complaints about problems with "object-maps" with every benji backup (tool for differential backups of RBD images). So, I turned on "exclusive-lock" and "fast-diff" and rebuilt object-maps for all RBD images. Since then I am receiving the following message for every RBD image backup: 2020-08-24T06:25:10.451+0200 7f0b857fa700 -1 librbd::object_map::InvalidateRequest: 0x7f0b8000a690 invalidating object map in-memory 2020-08-24T06:25:10.451+0200 7f0b857fa700 -1 librbd::object_map::InvalidateRequest: 0x7f0b8000a690 invalidating object map on-disk 2020-08-24T06:25:10.451+0200 7f0b857fa700 -1 librbd::object_map::InvalidateRequest: 0x7f0b8000a690 should_complete: r=0 I am not sure whether it indicates some real problem limiting the functionality of the backups, or whether it is just natural part of the process. Is it a problem of Ceph and the unexpected blackout causing some corruption, or a problem of Benji and some new feature/issue of Ceph version 15? I wonder whether the object-maps were used before, since there were never any reports about any problems of this kind, and even the features mentioned above were (probably?) never turned on before (so there were no object maps at all?). I am quite confused. Thanks for any explanation, Pavel

1 0

Best Assignment Help Service In Australia
by jeniferbrooks027＠gmail.com 31 Aug '20

31 Aug '20

MyAssignmentHelpNow is famous for providing quality assignment help services at reasonable prices. We have been serving quality writing service on various subjects to reach out the maximum number of students in Australia by ensuring that our prices remain cheap and affordable. Our experts are quite skilled and knowledgeable to accomplish challenging tasks and we always strive to help you with best and cheap assignment writing service in Australia. https://myassignmenthelpnow.com/assignment-help-sydney/

2 1

Undo ceph osd destroy?
by Michael Fladischer 31 Aug '20

31 Aug '20

Hi, I accidentally destroyed the wrong OSD in my cluster. It is now marked as "destroyed" but the HDD is still there and data was not touch AFAICT. I was able to avtivate it again using ceph-volume lvm activate and I can make the OSD as "in" but it's status is not changing from "destroyed". Is there a way to unmark it so I can reintegrate it in the cluster? Regards, Michael

2 3

Why online thesis help is the best option?
by kayleebrown402＠gmail.com 30 Aug '20

30 Aug '20

When you choose the best thesis help to write your thesis, you can either hire a person or a professional writing agency like https://www.edumagnate.com/thesis-help.html. But whom to choose for your case! Well, both of them are equally good and advantageous. You can choose any one of them as per your requirement and as per your pocket permits. In the case of a writing agency, lots of writers are available. Some of those writers are high end, and some are affordable. You will get service as per the money paid by you. But in the case of the freelancer, you need to work with him only. But lots of freelancers are out there, and you can choose someone that fits your bill. Maximum thesis helpers work on several projects simultaneously. So, when you approach the person with your project, he will take that up and may not provide you the project before the deadline ends. There’s always a risk factor going around these people. You can always check online thesis help reviews on different platforms. Check what others are saying about the writer before hiring him for your project. But if you choose the writing agency, they will replace the writer or refund you if the project is not complete before the deadline.

1 0