Hi all,
can anyone help me with this? In mimic, for any of these commands:
ceph osd [deep-]scrub ID
ceph pg [deep-]scrub ID
ceph pg repair ID
an operation is scheduled asynchronously. How can I check the following states:
1) Operation is pending (scheduled, not started).
2) Operation is running.
3) Operation has completed.
4) Exit code and error messages if applicable.
Many thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
We tried to rise the osd_memory_target from 4 to 8G but the problem
still occurs (osd wrongly marked down few times a day).
Does somebody have any clue ?
F.
> On Fri, Aug 28, 2020 at 10:34 AM Francois Legrand
> <fleg(a)lpnhe.in2p3.fr <mailto:fleg@lpnhe.in2p3.fr>> wrote:
>
> Hi all,
>
> We have a ceph cluster in production with 6 osds servers (with
> 16x8TB
> disks), 3 mons/mgrs and 3 mdss. Both public and cluster
> networks are in
> 10GB and works well.
>
> After a major crash in april, we turned the option
> bluefs_buffered_io to
> false to workaround the large write bug when
> bluefs_buffered_io was
> true (we were in version 14.2.8 and the default value at this
> time was
> true).
> Since that time, we regularly have some osds wrongly marked
> down by the
> cluster after heartbeat timeout (heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15).
>
> Generally the osd restart and the cluster is back healthy, but
> several
> time, after many of these kick-off the osd reach the
> osd_op_thread_suicide_timeout and goes down definitely.
>
> We increased the osd_op_thread_timeout and
> osd_op_thread_suicide_timeout... The problems still occurs
> (but less
> frequently).
>
> Few days ago, we upgraded to 14.2.11 and revert the timeout to
> their
> default value, hoping that it will solve the problem (we
> thought that it
> should be related to this bug
> https://tracker.ceph.com/issues/45943),
> but it didn't. We still have some osds wrongly marked down.
>
> Can somebody help us to fix this problem ?
> Thanks.
>
> Here is an extract of an osd log at failure time:
>
> ---------------------------------
> 2020-08-28 02:19:05.019 7f03f1384700 0 log_channel(cluster)
> log [DBG] :
> 44.7d scrub starts
> 2020-08-28 02:19:25.755 7f040e43d700 1 heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15
> 2020-08-28 02:19:25.755 7f040dc3c700 1 heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15
> this last line is repeated more than 1000 times
> ...
> 2020-08-28 02:20:17.484 7f040d43b700 1 heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15
> 2020-08-28 02:20:17.551 7f03f1384700 0
> bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow
> operation
> observed for _collection_list, latency = 67.3532s, lat = 67s cid
> =44.7d_head start GHMAX end GHMAX max 25
> ...
> 2020-08-28 02:20:22.600 7f040dc3c700 1 heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15
> 2020-08-28 02:21:20.774 7f03f1384700 0
> bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow
> operation
> observed for _collection_list, latency = 63.223s, lat = 63s cid
> =44.7d_head start
> #44:beffc78d:::rbd_data.1e48e8ab988992.00000000000011bd:0# end
> #MAX# max
> 2147483647
> 2020-08-28 02:21:20.774 7f03f1384700 1 heartbeat_map
> reset_timeout
> 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15
> 2020-08-28 02:21:20.805 7f03f1384700 0 log_channel(cluster)
> log [DBG] :
> 44.7d scrub ok
> 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster)
> log [WRN] :
> Monitor daemon marked osd.16 down, but it is still running
> 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster)
> log [DBG] :
> map e609411 wrongly marked me down at e609410
> 2020-08-28 02:21:21.099 7f03fd997700 1 osd.16 609411
> start_waiting_for_healthy
> 2020-08-28 02:21:21.119 7f03fd997700 1 osd.16 609411 start_boot
> 2020-08-28 02:21:21.124 7f03f0b83700 1 osd.16 pg_epoch: 609410
> pg[36.3d0( v 609409'481293 (449368'478292,609409'481293]
> local-lis/les=609403/609404 n=154651 ec=435353/435353 lis/c
> 609403/609403 les/c/f 609404/609404/0 609410/609410/608752)
> [25,72] r=-1
> lpr=609410 pi=[609403,609410)/1 luod=0'0 lua=609392'481198
> crt=609409'481293 lcod 609409'481292 active mbc={}]
> start_peering_interval up [25,72,16] -> [25,72], acting
> [25,72,16] ->
> [25,72], acting_primary 25 -> 25, up_primary 25 -> 25, role 2
> -> -1,
> features acting 4611087854031667199 upacting 4611087854031667199
> ...
> 2020-08-28 02:21:21.166 7f03f0b83700 1 osd.16 pg_epoch: 609411
> pg[36.56( v 609409'480511 (449368'477424,609409'480511]
> local-lis/les=609403/609404 n=153854 ec=435353/435353 lis/c
> 609403/609403 les/c/f 609404/609404/0 609410/609410/609410)
> [103,102]
> r=-1 lpr=609410 pi=[609403,609410)/1 crt=609409'480511 lcod
> 609409'480510 unknown NOTIFY mbc={}] state<Start>:
> transitioning to Stray
> 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413
> set_numa_affinity
> public network em1 numa node 0
> 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413
> set_numa_affinity
> cluster network em2 numa node 0
> 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413
> set_numa_affinity
> objectstore and network numa nodes do not match
> 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413
> set_numa_affinity
> not setting numa affinity
> 2020-08-28 02:21:21.566 7f040a435700 1 osd.16 609413 tick
> checking mon
> for new map
> 2020-08-28 02:21:22.515 7f03fd997700 1 osd.16 609414 state:
> booting ->
> active
> 2020-08-28 02:21:22.515 7f03f0382700 1 osd.16 pg_epoch: 609414
> pg[36.20( v 609409'483167 (449368'480117,609409'483167]
> local-lis/les=609403/609404 n=155171 ec=435353/435353 lis/c
> 609403/609403 les/c/f 609404/609404/0 609414/609414/609361)
> [97,16,72]
> r=1 lpr=609414 pi=[609403,609414)/1 crt=609409'483167 lcod
> 609409'483166
> unknown NOTIFY mbc={}] start_peering_interval up [97,72] ->
> [97,16,72],
> acting [97,72] -> [97,16,72], acting_primary 97 -> 97,
> up_primary 97 ->
> 97, role -1 -> 1, features acting 4611087854031667199 upacting
> 4611087854031667199
> ...
> 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414
> pg[36.2f3( v 609409'479796 (449368'476712,609409'479796]
> local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c
> 609403/609403 les/c/f 609404/609404/0 609414/609414/609414)
> [16,34,21]
> r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod
> 609409'479795
> mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up
> [34,21] ->
> [16,34,21], acting [34,21] -> [16,34,21], acting_primary 34 ->
> 16,
> up_primary 34 -> 16, role -1 -> 0, features acting
> 4611087854031667199
> upacting 4611087854031667199
> 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414
> pg[36.2f3( v 609409'479796 (449368'476712,609409'479796]
> local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c
> 609403/609403 les/c/f 609404/609404/0 609414/609414/609414)
> [16,34,21]
> r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod
> 609409'479795
> mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary
> 2020-08-28 02:21:24.738 7f03f1384700 0 log_channel(cluster)
> log [DBG] :
> 36.2f3 scrub starts
> 2020-08-28 02:22:18.857 7f03f1384700 0 log_channel(cluster)
> log [DBG] :
> 36.2f3 scrub ok
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> <mailto:ceph-users-leave@ceph.io>
>
Hi,
I have a rook cluster running with ceph 12.2.7 for almost one year.
Recently some pvc couldn’t be attached with error as below,
Warning FailedMount 7m19s kubelet, 192.168.34.119 MountVolume.SetUp failed for volume "pvc-8f4ca7ac-42ab-11ea-99d7-005056b84936" : mount command failed, status: Failure, reason: failed to mount volume /dev/rbd1 [ext4] to /var/lib/kubelet/plugins/rook.io/rook-ceph/mounts/pvc-8f4ca7ac-42ab-11ea-99d7-005056b84936, error 'fsck' found errors on device /dev/rbd1 but could not correct them: fsck from util-linux 2.23.2
/dev/rbd1: recovering journal
/dev/rbd1 contains a file system with errors, check forced.
/dev/rbd1: Inode 393244, end of extent exceeds allowed value
(logical block 512, physical block 12091904, len 4388)
After mapping storage to block device and run "fsck -y" on it, it indicated that filesystem was clean. Then retry to mount the storage, it still reported the same error as above.
Response from "ceph status” is as below,
cluster:
id: 54a729b6-7b59-4e5b-bc09-7dc99109cbad
health: HEALTH_WARN
noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 50711/152133 objects degraded (33.333%), 100 pgs degraded, 100 pgs undersized
mons rook-ceph-mon41,rook-ceph-mon44 are low on available space
services:
mon: 3 daemons, quorum rook-ceph-mon44,rook-ceph-mon47,rook-ceph-mon41
mgr: a(active)
osd: 3 osds: 3 up, 3 in
flags noscrub,nodeep-scrub
data:
pools: 1 pools, 100 pgs
objects: 50711 objects, 190 GB
usage: 383 GB used, 1111 GB / 1495 GB avail
pgs: 50711/152133 objects degraded (33.333%)
100 active+undersized+degraded
io:
client: 78795 B/s wr, 0 op/s rd, 11 op/s wr
From output, why is scrub disabled? Could I trigger it manually?
And how could I check which pgs or objects is corrupted?
Any advice for fixing the issue?
Thanks for your help.
Jared
Hi Konstantin,
I hope you or anybody still follows this old thread.
Can this EC data pool be configured per pool, not per client? If we follow
https://docs.ceph.com/docs/master/rbd/rbd-openstack/ we may see that cinder
client will access vms and volumes pools, both with read and write
permission. How can we handle this?
If we config with different clients for nova (vms) and cinder (volumes), I
think there will be a problem if there is cross pool access, especially on
write. Let's say that client nova will create volume on instance creation
for booting from that volume. Any thoughts?
Best regards,
> Date: Wed, 11 Jul 2018 11:16:27 +0700
> From: Konstantin Shalygin <k0ste(a)k0ste.ru>
> To: ceph-users(a)lists.ceph.com
> Subject: Re: [ceph-users] Erasure coding RBD pool for OpenStack
> Glance, Nova and Cinder
> Message-ID: <069ac368-22b0-3d18-937b-70ce39287cb1(a)k0ste.ru>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> > So if you want, two more questions to you :
> >
> > - How do you handle your ceph.conf configuration (default data pool by
> > user) / distribution ? Manually, config management, openstack-ansible...
> > ?
> > - Did you made comparisons, benchmarks between replicated pools and EC
> > pools, on the same hardware / drives ? I read that small writes are not
> > very performant with EC.
>
>
> ceph.conf with default data pool is only need for Cinder at image
> creation time, after this luminous+ rbd client will be found feature
> "data-pool" and will perform data-io to this pool.
>
>
> > # rbd info
> > erasure_rbd_meta/volume-09ed44bf-7d16-453a-b712-a636a0d3d812 <-----
> > meta pool !
> > rbd image 'volume-09ed44bf-7d16-453a-b712-a636a0d3d812':
> > ??????? size 1500 GB in 384000 objects
> > ??????? order 22 (4096 kB objects)
> > ??????? data_pool: erasure_rbd_data??????? <----- our data pool
> > ??????? block_name_prefix: rbd_data.6.a2720a1ec432bf
> > ??????? format: 2
> > ??????? features: layering, exclusive-lock, object-map, fast-diff,
> > deep-flatten, data-pool????????? <----- "data-pool" feature
> > ??????? flags:
> > ??????? create_timestamp: Sat Jan 27 20:24:04 2018
>
>
>
> k
>
>
>
> ------------------------------
>
I replaced the VMs taking care of routing between clients and MDSes by physical machines. Problems below are solved. It seems to have been related to issues with the virtual NIC. It seemed to work well with E1000 instead of VirtIO...
Met vriendelijke groeten,
William Edwards
----- Original Message -----
From: William Edwards (wedwards(a)cyberfusion.nl)
Date: 08/11/20 11:38
To: ceph-users(a)ceph.io
Subject: Speeding up reconnection
Hello,
When connection is lost between kernel client, a few things happen:
1.
Caps become stale:
Aug 11 11:08:14 admin-cap kernel: [308405.227718] ceph: mds0 caps stale
2.
MDS evicts client for being unresponsive:
MDS log: 2020-08-11 11:12:08.923 7fd1f45ae700 0 log_channel(cluster) log [WRN] : evicting unresponsive client admin-cap.cf.ha.cyberfusion.cloud:DB0001-cap (144786749), after 300.978 seconds
Client log: Aug 11 11:12:11 admin-cap kernel: [308643.051006] ceph: mds0 hung
3.
Socket is closed:
Aug 11 11:22:57 admin-cap kernel: [309289.192705] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state OPEN)
I am not sure whether the kernel client or MDS closes the connection. I think the kernel client does so, because nothing is logged at the MDS side at 11:22:57
4.
Connection is reset by MDS:
MDS log: 2020-08-11 11:22:58.831 7fd1f9e49700 0 --1- [v2:[fdb7:b01e:7b8e:0:10:10:10:1]:6800/3619156441,v1:[fdb7:b01e:7b8e:0:10:10:10:1]:6849/3619156441] >> v1:[fc00:b6d:cfc:951::7]:0/133007863 conn(0x55bfaf1c2880 0x55c16cb47000 :6849 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 1), sending RESETSESSION
Client log: Aug 11 11:22:58 admin-cap kernel: [309290.058222] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 connection reset
5.
Kernel client reconnects:
Aug 11 11:22:58 admin-cap kernel: [309290.058972] ceph: mds0 closed our session
Aug 11 11:22:58 admin-cap kernel: [309290.058973] ceph: mds0 reconnect start
Aug 11 11:22:58 admin-cap kernel: [309290.069979] ceph: mds0 reconnect denied
Aug 11 11:22:58 admin-cap kernel: [309290.069996] ceph: dropping file locks for 000000006a23d9dd 1099625041446
Aug 11 11:22:58 admin-cap kernel: [309290.071135] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state NEGOTIATING)
Question:
As you can see, there's 10 minutes between losing the connection and the reconnection attempt (11:12:08 - 11:22:58). I could not find any settings related to the period after which reconnection is attempted. I would like to change this value from 10 minutes to something like 1 minute. I also tried searching the Ceph docs for the string '600' (10 minutes), but did not find anything useful.
Hope someone can help.
Environment details:
Client kernel: 4.19.0-10-amd64
Ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable)
Met vriendelijke groeten,
William Edwards
Hello,
Lately, we upgraded Ceph to version 15.2.4 and shortly after that we had
a blackout, which caused a restart of all servers at once (BTW, Ceph did
not come up well itself). Since then we were receiving lots of
complaints about problems with "object-maps" with every benji backup
(tool for differential backups of RBD images). So, I turned on
"exclusive-lock" and "fast-diff" and rebuilt object-maps for all RBD
images. Since then I am receiving the following message for every RBD
image backup:
2020-08-24T06:25:10.451+0200 7f0b857fa700 -1
librbd::object_map::InvalidateRequest: 0x7f0b8000a690 invalidating
object map in-memory
2020-08-24T06:25:10.451+0200 7f0b857fa700 -1
librbd::object_map::InvalidateRequest: 0x7f0b8000a690 invalidating
object map on-disk
2020-08-24T06:25:10.451+0200 7f0b857fa700 -1
librbd::object_map::InvalidateRequest: 0x7f0b8000a690 should_complete: r=0
I am not sure whether it indicates some real problem limiting the
functionality of the backups, or whether it is just natural part of the
process. Is it a problem of Ceph and the unexpected blackout causing
some corruption, or a problem of Benji and some new feature/issue of
Ceph version 15?
I wonder whether the object-maps were used before, since there were
never any reports about any problems of this kind, and even the features
mentioned above were (probably?) never turned on before (so there were
no object maps at all?). I am quite confused.
Thanks for any explanation,
Pavel
MyAssignmentHelpNow is famous for providing quality assignment help services at reasonable prices. We have been serving quality writing service on various subjects to reach out the maximum number of students in Australia by ensuring that our prices remain cheap and affordable. Our experts are quite skilled and knowledgeable to accomplish challenging tasks and we always strive to help you with best and cheap assignment writing service in Australia.
https://myassignmenthelpnow.com/assignment-help-sydney/
Hi,
I accidentally destroyed the wrong OSD in my cluster. It is now marked
as "destroyed" but the HDD is still there and data was not touch AFAICT.
I was able to avtivate it again using ceph-volume lvm activate and I can
make the OSD as "in" but it's status is not changing from "destroyed".
Is there a way to unmark it so I can reintegrate it in the cluster?
Regards,
Michael
When you choose the best thesis help to write your thesis, you can either hire a person or a professional writing agency like https://www.edumagnate.com/thesis-help.html. But whom to choose for your case! Well, both of them are equally good and advantageous. You can choose any one of them as per your requirement and as per your pocket permits. In the case of a writing agency, lots of writers are available. Some of those writers are high end, and some are affordable. You will get service as per the money paid by you. But in the case of the freelancer, you need to work with him only. But lots of freelancers are out there, and you can choose someone that fits your bill.
Maximum thesis helpers work on several projects simultaneously. So, when you approach the person with your project, he will take that up and may not provide you the project before the deadline ends. There’s always a risk factor going around these people. You can always check online thesis help reviews on different platforms. Check what others are saying about the writer before hiring him for your project. But if you choose the writing agency, they will replace the writer or refund you if the project is not complete before the deadline.
Cash app comes with the Visa debit card to make your transaction reliable as well as safe. Here we describe you to resolve your query get money off Cash app without card. Actually, if you lost your card or you didn’t get a Cash card yet, then you have to make a request for getting a Cash card.https://www.cashapp-customerservice.com/blog/get-money-cash-app-withou…
