- ceph-users - lists.ceph.io

Re: Strange Ceph architect with SAN storages
by Brett Chancellor 23 Aug '19

23 Aug '19

It's certainly possible. It makes things a little more complex though. Some questions you may want to consider during the design.. - Is the customer aware this won't preserve any data on the luns they are hoping to reuse. - Is the plan to eventually replace the SAN with JBOD, in the same systems? If so you may want to make your luns look like the eventual drive size and count. - Is the plan to use a few systems with SAN and add standalone systems later? Then you need to calculate expected speeds and divide between failure domains. - Is the plan to use a couple of hosts with SAN to save money, and have the rest be traditional Ceph storage? If so consider putting the SAN hosts all in one failure domain. - Depending on the SAN you may consider aligning your failure domains to different arrays, switches, or even array directors. - Remember to take the hosts network speed into consideration when calculating how many luns to put on each host. Hope that helps. -Brett On Thu, Aug 22, 2019, 4:14 AM Mohsen Mottaghi <mohsenmottaghi(a)outlook.com> wrote: > Hi > > > Yesterday one of our customers asked us a strange request. He asked us to > use SAN as the Ceph storage space to add the SAN storages it currently has > to the cluster and reduce other disk purchase costs. > > > Anybody know can we do this or not?! And if this is possible how we should > start to architect this Strange Ceph?! Is it good or not?! > > > > Thanks for your help. > > Mohsen Mottaghi > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

4 4

Ceph Tech Talk Cancelled for August
by Mike Perez 22 Aug '19

22 Aug '19

Hi all, We're cancelling our Ceph tech talk today. In the mean time check out our archive and consider reaching out to me about giving your own talk in the upcoming months. https://ceph.com/ceph-tech-talks/ -- Mike Perez (thingee)

1 0

Re: About image migration
by Jason Dillaman 22 Aug '19

22 Aug '19

On Wed, Aug 21, 2019 at 9:33 PM Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> wrote: > > I tested and combed the current migration process. If I read and write the new image during the migration process and then use migration_abort, the newly written data will be lost. Do we have a solution to this problem? That's a good point that we didn't consider. I've opened a tracker ticket against the issue [1]. > > -----邮件原件----- > 发件人: Jason Dillaman [mailto:jdillama@redhat.com] > 发送时间: 2019年8月22日 8:38 > 收件人: Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> > 抄送: ceph-users <ceph-users(a)ceph.com> > 主题: Re: About image migration > > On Wed, Aug 21, 2019 at 8:35 PM Zaharo Bai (白战豪)-云数据中心集团 > > <baizhanhao01(a)inspur.com> wrote: > > > > So, what is the current usage scenario for our online migration? I understand that the current online migration of the community has not been truly online, and the upper layer (iscsi target, openstack, etc.) must be required to do some operations such as switching and data maintenance. > > Is there a way to achieve full online migration in the rbd layer, the upper application is not aware, just need to call librbd's CLI or API, or is it necessary to do so, because doing so will inevitably change the architecture of ceph. > > If using RBD under a higher-level layer, the upper layers would need to know about the migration to update their internal data structures to point to the correct (new) image. The only case where this really isn't necessary is when live-migrating an image "in-place" (i.e. you keep it in the same pool w/ the same name). > > > -----邮件原件----- > > 发件人: Jason Dillaman [mailto:jdillama@redhat.com] > > 发送时间: 2019年8月21日 20:44 > > 收件人: Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> > > 抄送: ceph-users <ceph-users(a)ceph.com> > > 主题: Re: About image migration > > > > On Tue, Aug 20, 2019 at 10:04 PM Zaharo Bai (白战豪)-云数据中心集团 > > <baizhanhao01(a)inspur.com> wrote: > > > > > > Hi jason: > > > > > > I have a question I would like to ask you, Is the current image migration and openstack adapted? according to my understanding, openstack’s previous live-migration logic is implemented in cinder, just call librbd rbd_read/write API to do Data migration. > > > > I believe the existing Cinder volume block live-migration is just a wrapper around QEMU's block live-migration functionality, so indeed it would just be a series of RBD read/write API calls between two volumes. The built-in RBD live-migration is similar but it copies snapshots and preserves image sparseness during the migration process. > > Because Cinder is using the QEMU block live-migration functionality, it's not tied into RBD's live-migration. > > > > > > > > > > > > > > Best wishes > > > > > > Zaharo > > > > > > > > -- > > Jason > > > > -- > Jason [1] https://tracker.ceph.com/issues/41394 -- Jason

1 0

Strange Ceph architect with SAN storages
by Mohsen Mottaghi 22 Aug '19

22 Aug '19

Hi Yesterday one of our customers asked us a strange request. He asked us to use SAN as the Ceph storage space to add the SAN storages it currently has to the cluster and reduce other disk purchase costs. Anybody know can we do this or not?! And if this is possible how we should start to architect this Strange Ceph?! Is it good or not?! Thanks for your help. Mohsen Mottaghi

1 0

deep-scrub stat mismatch after PG merge
by Daniel Schreiber 22 Aug '19

22 Aug '19

Hi everyone, few days ago I reduced the number og PGs on a small pool. The cluster runs 14.2.2, was upgraded from jewel to 14.2.1 and the to 14.2.2. I did a ceph-bluestore-tool repair on all OSDs to update statistics. Today I got a scrub error reporting: 4.3 scrub : stat mismatch, got 68/68 objects, 0/0 clones, 68/68 dirty, 2/2 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 545259538/511705106 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. This message appears on the primary OSD only. rados list-inconsistent-obj 4.3 {"epoch":228717,"inconsistents":[]} What can I do to find the reason for the scrub error? Is this a hardware failure or a bug? Is it safe to do a pg repair? Thanks for your help. Daniel -- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444 Fax: +49 371 531 835444

1 0

Re: About image migration
by Jason Dillaman 21 Aug '19

21 Aug '19

On Wed, Aug 21, 2019 at 8:35 PM Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> wrote: > > So, what is the current usage scenario for our online migration? I understand that the current online migration of the community has not been truly online, and the upper layer (iscsi target, openstack, etc.) must be required to do some operations such as switching and data maintenance. > Is there a way to achieve full online migration in the rbd layer, the upper application is not aware, just need to call librbd's CLI or API, or is it necessary to do so, because doing so will inevitably change the architecture of ceph. If using RBD under a higher-level layer, the upper layers would need to know about the migration to update their internal data structures to point to the correct (new) image. The only case where this really isn't necessary is when live-migrating an image "in-place" (i.e. you keep it in the same pool w/ the same name). > -----邮件原件----- > 发件人: Jason Dillaman [mailto:jdillama@redhat.com] > 发送时间: 2019年8月21日 20:44 > 收件人: Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> > 抄送: ceph-users <ceph-users(a)ceph.com> > 主题: Re: About image migration > > On Tue, Aug 20, 2019 at 10:04 PM Zaharo Bai (白战豪)-云数据中心集团 > <baizhanhao01(a)inspur.com> wrote: > > > > Hi jason: > > > > I have a question I would like to ask you, Is the current image migration and openstack adapted? according to my understanding, openstack’s previous live-migration logic is implemented in cinder, just call librbd rbd_read/write API to do Data migration. > > I believe the existing Cinder volume block live-migration is just a wrapper around QEMU's block live-migration functionality, so indeed it would just be a series of RBD read/write API calls between two volumes. The built-in RBD live-migration is similar but it copies snapshots and preserves image sparseness during the migration process. > Because Cinder is using the QEMU block live-migration functionality, it's not tied into RBD's live-migration. > > > > > > > > > Best wishes > > > > Zaharo > > > > -- > Jason -- Jason

1 0

Re: About image migration
by Jason Dillaman 21 Aug '19

21 Aug '19

On Tue, Aug 20, 2019 at 10:04 PM Zaharo Bai (白战豪)-云数据中心集团 <baizhanhao01(a)inspur.com> wrote: > > Hi jason: > > I have a question I would like to ask you, Is the current image migration and openstack adapted? according to my understanding, openstack’s previous live-migration logic is implemented in cinder, just call librbd rbd_read/write API to do Data migration. I believe the existing Cinder volume block live-migration is just a wrapper around QEMU's block live-migration functionality, so indeed it would just be a series of RBD read/write API calls between two volumes. The built-in RBD live-migration is similar but it copies snapshots and preserves image sparseness during the migration process. Because Cinder is using the QEMU block live-migration functionality, it's not tied into RBD's live-migration. > > > > Best wishes > > Zaharo -- Jason

1 0

Multisite RGW data corruption (not 14.2.1 curl issue)
by vladimir＠bashkirtsev.com 19 Aug '19

19 Aug '19

Hello, I have setup two separate Ceph clusters with RGW instance each and trying to achieve multisite data synchronization. Primary runs 13.2.5, slave runs 14.2.2 (I have upgraded slave side from 14.2.1 due to known data corruption during transfer due to curl errors). I have emptied slave zone and allowed sync to run from beginning to the end. Then I have recalculated MD5 hashes over original data and over data in slave zones and found that in some cases they do not match. Data corruption is evident. Comparison of data byte for byte shows that some parts of data just moved around (for example file has correct bytes from 0 to 513k and then exactly the same bytes from position of 513k up to 1026k - feels like as some sort of buffer issue). File size is correct. I have read the RGW sources and could not find anything what can cause such sort of behavior however I could not find one single piece of code which I would consider critical: during FetchRemote RGW obtains object's etag from remote RGW but apparently there nothing what recalculates MD5 from actual data and comparing it to etag received to ensure that data was correctly transferred. Even broken data then stored to local cluster and into bucket index. Then there just nothing further to prevent wrong data to reach end user downloading from slave zone. Such check should happen regardless of sync backend in use: objects with broken data should not be stored in bucket index at all. No one needs broken data and it is better just to fail object sync to allow it to try again later instead of storing faulty data and then allowing end users to download it. Is it me who cannot see such sort of check or it is actually missing?! And if it is not there at all I think it should be quite high on TODO list. Regards, Vladimir

2 2

Re: Failing heartbeats when no backfill is running
by Lorenz Kiefner 19 Aug '19

19 Aug '19

Oh no, it's not that bad. It's $ ping -s 65000 dest.inati.on on a VPN connection that has a MTU of 1300 via IPv6. So I suspect that I only get an answer, when all 51 fragments get fully returned. It's clear that big packets with lots of fragments are more affected by packet loss than 64 byte pings. I just (at 9 o'clock in the morning) repeated this ping test and got hardly any drops (less than 1%), even with the size of 64k. So it's really dependent on the time of the day. Seems like some ISPs are dropping some packets, especially in the evening... A few minutes ago I restarted all down-marked OSDs, but they are getting marked down again... Seems like Ceph is tolerable against packet loss (it surely affects performance, but this irrelevant for me). Could erasure coded pools pose some problems? Thank you all for every hint! Lorenz Am 15.08.19 um 08:51 schrieb Janne Johansson: > Den ons 14 aug. 2019 kl 17:46 skrev Lorenz Kiefner > <root+cephusers(a)deinadmin.de <mailto:root%2Bcephusers@deinadmin.de>>: > > Is ceph sensitive to packet loss? On some VPN links I have up to 20% > packet loss on 64k packets but less than 3% on 5k packets in the > evenings. > > > 20% seems crazy high, there must be something really wrong there. > > At 20%, you would get tons of packet timeouts to wait for on all those > lost frames, > then resends of (at least!) those 20% extra, which in turn would lead > to 20% of those > resends getting lost, all while the main streams of data try to move > forward when some > older packet do get over. This is a really bad situation to design for, > > I think you should look for a link solution that doesn't drop that > many packets, instead of changing > the software you try to run over that link, all others will notice > this too and act badly in some way or other. > > Heck, 20% is like taking a math schoolbook and remove all instances of > "3" and "8" and see if kids can learn to count from it. 8-/ > > -- > May the most significant bit of your life be positive.

2 3

Re: rbd image journal performance
by Jason Dillaman 18 Aug '19

18 Aug '19

On Mon, Aug 12, 2019 at 10:03 PM yangjun(a)cmss.chinamobile.com <yangjun(a)cmss.chinamobile.com> wrote: > > Hi Jason, > > I was recently testing the RBD mirror feature(ceph12.2.8), my test environment is a single-node cluster, which including 10 3T hdd OSDs + 800G pcie ssd + bluestore, and the wal and db partition of the OSD is 30G. > The test result of a 100G image is as follows： > > disable journal enable journal decline percentage > iops： 1000 877 12.3% > bw： 402MB/s 129MB/s 67% > > > Why does the bandwidth decline so much after starting journal of the RBD image? I'm very appreciate if you could give me some suggestions for optimization. Thank you very much. The use of the journal requires first writing to the journal and, once committed, writing to the image (i.e. doubling the latency). Therefore, the expected worst-case performance should be around 2x slower [1]. There was a recent bug fix [2] in the master branch that will be backported to older releases which greatly increases small IO journal performance -- since it was nearly 10x slower due to the bug instead of the expected 2x [3]. > ________________________________ > yangjun(a)cmss.chinamobile.com [1] https://www.slideshare.net/JasonDillaman/disaster-recovery-and-ceph-block-s… [2] https://tracker.ceph.com/issues/40072 [3] https://youtu.be/ZifNGprBUTA?t=1687 -- Jason

1 0