New subject: Failed to get omap key when mirroring of image is enabled

23 Aug 2019

On Fri, Aug 23, 2019 at 7:38 AM Ajitha Robert &lt;ajitharobert01(a)gmail.com&gt; wrote:
...

 Sir,

 I have a running DR setup with ceph.. but i did the same for another two sites.. Its
actually direct L2 connectivity link between sites.. I m getting repeated error

 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequestfinish: resending after timeout

That is just indicative that you are having issues talking to your
local cluster. Assuming you only have a single rbd-mirror daemon
running, it seems like it cannot even sent a message through the OSD
to itself within 5 seconds. Perhaps your cluster is too slow to
respond?

...

 its coming continously. so not getting replicated to other site. Whether direct L2
connectivity is a concern?? whether rbd-mirror expects a L3 layer link for two sites?

 On Wed, Jul 24, 2019 at 12:42 AM Ajitha Robert &lt;ajitharobert01(a)gmail.com&gt; wrote:

 Thanks for your reply.

 Regarding rbd mirroring, Can you please check the logs for rbd image creation. Second[2]
one started syncing and no progress further.

 1)Log for manual rbd image creation

 http://paste.openstack.org/show/754766/

 2)Log for 16gb volume created from cinder, status in cinder volume is available

 http://paste.openstack.org/show/754767/

 3)Log for 100gb volume created from cinder, status in cinder volume is error

 http://paste.openstack.org/show/754769/

 On Tue, Jul 23, 2019 at 1:13 AM Jason Dillaman &lt;jdillama(a)redhat.com&gt; wrote:

 On Mon, Jul 22, 2019 at 3:26 PM Ajitha Robert &lt;ajitharobert01(a)gmail.com&gt; wrote:

 Thanks for your reply

 1) In scenario 1, I didnt attempt to delete the cinder volume. Please find the cinder
volume log.
 http://paste.openstack.org/show/754731/ 
 It might be better to ping Cinder folks about that one. It doesn't
 really make sense to me from a quick glance.

 2) In scenario 2. I will try with debug. But i m having a test setup with one OSD in
primary and one OSD in secondary. distance between two ceph clusters is 300 km

 3)I have disabled ceph authentication totally for all including rbd-mirror daemon. Also i
have deployed the ceph cluster using ceph-ansible. Will these both  create any issue to
the entire setup 
 Not to my knowledge.

  4)The image which was in syncing mode, showed
read only status in secondary. 
 Mirrored images are either primary or non-primary. It is the expected
 (documented) behaviour that non-primary images are read-only.

  5)In a presentation i found as journaling feature
is causing poor performance in IO operations and we can skip the journaling process for
mirroring... Is it possible.. By enabling mirroring to entire cinder pool as pool mode
instead of mirror mode of rbd mirroring.. And we can skip the replication_enabled is true
spec in cinder type.. 
 Journaling is required for RBD mirroring.

 On Mon, Jul 22, 2019 at 11:13 PM Jason Dillaman &lt;jdillama(a)redhat.com&gt; wrote:
>
> On Mon, Jul 22, 2019 at 10:49 AM Ajitha Robert &lt;ajitharobert01(a)gmail.com&gt;
wrote:
> >
> > No error log in rbd-mirroring except some connection timeout came once,
> > Scenario 1:
> >   when I create a bootable volume of 100 GB with a glance image.Image get
downloaded and from cinder, volume log throws with "volume is busy deleting volume
that has snapshot" . Image was enabled with exclusive lock, journaling, layering,
object-map, fast-diff and deep-flatten
> > Cinder volume is in error state but the rbd image is created in primary but not
in secondary.
>
> Any chance you know where in Cinder that error is being thrown? A
> quick grep of the code doesn't reveal that error message. If the image
> is being synced to the secondary site when you attempt to delete it,
> it's possible you could hit this issue. Providing debug log messages
> from librbd on the Cinder controller might also be helpful for this.
>
> > Scenario 2:
> > but when i create a 50gb volume with another glance image. Volume  get created.
and in the backend i could see the rbd images both in primary and secondary
> >
> > From rbd mirror image status i found secondary cluster starts copying , and
syncing was struck at around 14 %... It will be in 14 % .. no progress at all. should I
set any parameters for this like timeout??
> >
> > I manually checked rbd --cluster primary object-map check <object-name>.. 
No results came for the objects and the command was in hanging.. Thats why got worried on
the failed to map object key log. I couldnt even rebuild the object map.
>
> It sounds like one or more of your primary OSDs are not reachable from
> the secondary site. If you run w/ "debug rbd-mirror = 20" and "debug
> rbd = 20", you should be able to see the last object it attempted to
> copy. From that, you could use "ceph osd map" to figure out the
> primary OSD for that object.
>
> > the image which was in syncing mode, showed read only status in secondary.
> >
> >
> >
> > On Mon, 22 Jul 2019, 17:36 Jason Dillaman, &lt;jdillama(a)redhat.com&gt; wrote:
> >>
> >> On Sun, Jul 21, 2019 at 8:25 PM Ajitha Robert
&lt;ajitharobert01(a)gmail.com&gt; wrote:
> >> >
> >> >  I have a rbd mirroring setup with primary and secondary clusters as
peers and I have a pool enabled image mode.., In this i created a rbd image , enabled with
journaling.
> >> >
> >> > But whenever i enable mirroring on the image,  I m getting error in
osd.log. I couldnt trace it out. please guide me to solve this error.
> >> >
> >> > I think initially it worked fine. but after ceph process restart. these
error coming
> >> >
> >> >
> >> > Secondary.osd.0.log
> >> >
> >> > 2019-07-22 05:36:17.371771 7ffbaa0e9700  0 <cls>
/build/ceph-12.2.12/src/cls/journal/cls_journal.cc:61: failed to get omap key:
client_a5c76849-ba16-480a-a96b-ebfdb7f6ac65
> >> > 2019-07-22 05:36:17.388552 7ffbaa0e9700  0 <cls>
/build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than
minimum: 0 < 1
> >> > 2019-07-22 05:36:17.413102 7ffbaa0e9700  0 <cls>
/build/ceph-12.2.12/src/cls/journal/cls_journal.cc:61: failed to get omap key: order
> >> > 2019-07-22 05:36:23.341490 7ffbab8ec700  0 <cls>
/build/ceph-12.2.12/src/cls/rbd/cls_rbd.cc:4125: error retrieving image id for global id
'9e36b9f8-238e-4a54-a055-19b19447855e': (2) No such file or directory
> >> >
> >> >
> >> > primary-osd.0.log
> >> >
> >> > 2019-07-22 05:16:49.287769 7fae12db1700  0 log_channel(cluster) log
[DBG] : 1.b deep-scrub ok
> >> > 2019-07-22 05:16:54.078698 7fae125b0700  0 log_channel(cluster) log
[DBG] : 1.1b scrub starts
> >> > 2019-07-22 05:16:54.293839 7fae125b0700  0 log_channel(cluster) log
[DBG] : 1.1b scrub ok
> >> > 2019-07-22 05:17:04.055277 7fae12db1700  0 <cls>
/build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than
minimum: 0 < 1
> >> >
> >> > 2019-07-22 05:33:21.540986 7fae135b2700  0 <cls>
/build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than
minimum: 0 < 1
> >> > 2019-07-22 05:35:27.447820 7fae12db1700  0 <cls>
/build/ceph-12.2.12/src/cls/rbd/cls_rbd.cc:4125: error retrieving image id for global id
'8a61f694-f650-4ba1-b768-c5e7629ad2e0': (2) No such file or directory
> >>
> >> Those don't look like errors, but the log level should probably be
> >> reduced for those OSD cls methods. If you look at your rbd-mirror
> >> daemon log, do you see any errors? That would be the important place
> >> to look.
> >>
> >> >
> >> > --
> >> > Regards,
> >> > Ajitha R
> >> > _______________________________________________
> >> > ceph-users mailing list
> >> > ceph-users(a)lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >>
> >> --
> >> Jason
>
>
>
> --
> Jason

 --
 Regards,
 Ajitha R 

 --
 Jason 

 --
 Regards,
 Ajitha R 

 --
 Regards,
 Ajitha R 

-- 
Jason

Re: Failed to get omap key when mirroring of image is enabled