- ceph-users - lists.ceph.io

krbd crc
by Seena Fallah 11 Feb '21

11 Feb '21

Hi, I have a few questions about krbd on kernel 4.15 1. Does it support msgr v2? (If not which kernel supports msgr v2?) 2. If krbd is using msgr v1, does it checksum (CRC) the messages that it sends to see for example if the write is correct or not? and if it does checksums, If there were a problem in write how does it react to that? For example, does it raise I/O Error or retry or...? Thanks.

3 4

how far can we go using vstart.sh script for fake dev cluster-HELP
by Bobby 11 Feb '21

11 Feb '21

Hi, Ceph source code contains a script called vstart.sh which allows developers to quickly test their code using a simple deployment on your development system. Here: https://docs.ceph.com/en/latest//dev/quick_guide/ I am really curious that how far we can go with vstart.sh script. While my development cluster is running, I use tools like rados bench, rbd and rbd-nbd to benchmark simple workload and test my code. Do we have options to change the network settings in the fake cluster built from vstart script and later benchmark it? For example , trying 1gbit ethernet and 10gbit ethernet. Thanks

1 0

Re: Dashboard : Block image listing and infos
by Gilles Mocellin 11 Feb '21

11 Feb '21

Hi ! I respond to the list, as it may help others. I also reorder the response. > On Mon, Jan 18, 2021 at 2:41 PM Gilles Mocellin < > > gilles.mocellin(a)nuagelibre.org> wrote: > > Hello Cephers, > > > > On a new cluster, I only have 2 RBD block images, and the Dashboard > > doesn't manage to list them correctly. > > > > I have this message : > > Warning > > Displaying previously cached data for pool veeam-repos. > > > > Sometime it disappears, but as soon as I reload or return to the listing > > page, it's there. > > > > What I've seen, is a high CPU load due to ceph-mgr on the active > > manager. > > And also stack-traces like this : [...] > > dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to > > retrieve data > > > > I also see that, when I try to edit an image : > > > > 2021-01-18T11:13:26.383+0100 7f00199ca700 0 [dashboard ERROR > > frontend.error] > > (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-> > repos%252Fveeam-repo2-vol1 > > <https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-repos% > > 252Fveeam-repo2-vol1>): Cannot read property 'features_name' of > > undefined > > > > TypeError: Cannot read property 'features_name' of undefined [...] > > > > But that's perhaps just becaus I open an Edit window on the image and it > > does not have the datas. > > The Edit window is empty, and I can't edit things, especially, I wan't > > to resize the image. > > [...] > > -- > > Gilles Le jeudi 21 janvier 2021, 21:56:58 CET Ernesto Puerta a écrit : > Hey Gilles, > > If I'm not wrong, that exception (ViewCacheNoDataException) happens when > the dashboard is unable to gather all required data from Ceph within a > defined timeout (5 secs I think, since the UI refreshes the data every ~5 > seconds). > > It'd be great if you could provide the steps to reproduce it and some > insights into your environment (number of RBD pools, number of RBD images, > snapshots, etc.). > > Kind Regards, > > Ernesto OK, As it is now, it always hapens, on the image listing, I have the Warning and the list is not always up to date, if I create an image, I must wait very long to see it. Also, I can not edit the 2 big images I have. Perhaps the size is important, they are 2 images of 40 TB. If I create a 1 GB test image, I can edit and resize it. But impossible withe the big image, the windows opens but all the fields are empty. Also, if it can matter, the images use a data pool (EC 3+2). I have 2 pools, a replicated one for metadatas veeam-repos (replic x3), and a data pool veeam-repos.data (EC 3+2). My cluster has 6 nodes with AMD 16 cores CPU, 128 GB RAM, 10 8 TB HDD. So 60 OSD. Soon doubling everything to 12 nodes. Usage, as the pool and image names can tell, is to mount RBD image as a XFS filesystem for a Veeam Backup Repository (krbd, because nbd-rbd tailed regularly, especially during fstrim).

2 2

Data sync init vs bucket sync init
by Szabo, Istvan (Agoda) 11 Feb '21

11 Feb '21

Hi, What’s the difference between data sync init and bucket sync init? Data initialise the complete cluster? Bucket only bucket? I see when initialise finished, have shards behind but doesn’t do anything with it? What is the proper steps to bring things back to sync? Init Run Restart ?? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Data sync init vs bucket sync init
by Szabo, Istvan (Agoda) 11 Feb '21

11 Feb '21

Hi, What’s the difference between data sync init and bucket sync init? Data initialise the complete cluster? Bucket only bucket? I see when initialise finished, have shards behind but doesn’t do anything with it? What is the proper steps to bring things back to sync? Init Run Restart ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

How to debug container upgrade issue?
by Darrin Hodges 10 Feb '21

10 Feb '21

HI all, Upgrading ceph containers is failing, how do I debug it? 'cephadm pull' seems to work on the node, but the upgrade fails. $ ceph orch upgrade start --ceph-version 15.2.8 "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

3 2

Device is not available after zap
by Tony Liu 10 Feb '21

10 Feb '21

Hi, With v15.2.8, after zap a device on OSD node, it's still not available. The reason is "locked, LVM detected". If I reboot the whole OSD node, then the device will be available. There must be something no being cleaned up. Any clues? Thanks! Tony

4 7

Some slow writes over time
by João Victor Mafra 10 Feb '21

10 Feb '21

I am running a simple workload (just 1 client and 1 file) of random writings on Ceph FS and I noticed that approximately 3% of the operations (well spread over time) show latencies higher than the other 97% (100 ms x 10 ms). Is there any reason for this to happen? - I'm using fio with O_DIRECT to avoid cache buffer, so it is expected that the operations will only be completed after writing to the disk. - My WAL is also disabled, so there is no reason for ceph to be doing deferred writing. - I performed the same workload on the gluster fs and the latencies were uniform over time.

1 0

using secondary OSDs for reading
by mj 10 Feb '21

10 Feb '21

Hi, Quoting the page https://docs.ceph.com/en/latest/architecture/ > location query over a chatty session. The CRUSH algorithm allows a > client to compute where objects should be stored, and enables the > client to contact the primary OSD to store or retrieve the objects. So clients contact the *primary* OSD to store/retrieve objects. Why do clients not contact secondary (or even tertiary) OSDs for reading data? Would that not (potentially) result in greatly improved performance..? I'm sure there are good reasons for the current behaviour, but it seems logical, since you have multiple copies of the same data, to try and use the nearest copy? Curious :-) MJ

3 5

File I/O with mixed read/write and high streaming performance
by perkins＠stephen-perkins.com 10 Feb '21

10 Feb '21

Hi all, I'm trying to understand if the CephFS is a good approach for the following scenario. From some of the OLD benchmarks, GlusterFS significantly beat the CephFS when many file I/Os were required. But... this was an OLD benchmark. I'd like your thoughts on the matter. What I need to perform are the following two steps: Re-organize files (Step one) I need to take a large directory structure (assumed to reside on CephFS) and "re-arrange" it via a copy or link mechanism. I will want to make a full copy of the directory structure but do a simple disk span chunking so that all the files in the original copy end up in a set of folders where each folder is no larger than a fixed size. This is like what we did back in the days where we needed to write data in CDROM sized chunks. There is a set of tools that will do this in the genisoimage package (dssplit and dirsplit). Folder Axe was the MS Windows equivalent Presumably, this would put a large random read and random write load on the cluster. Since the size can be large (hundreds of G (maybe up to 1TB) with 10s to 100s of thousands of small files), I would need for this to be well optimized. One mechanism that might be available is to use hard or soft links so that no actual copying is done (Don't know if CephFS/POSIX supports this). The linking approach would probably put a large strain on the MDS servers but not so much on the storage. Write to media (Step two) I need to stream the chunked folders to a set of media devices (think tape drive) that can ingest at high speed (about 200 megabytes per second... yes bytes). I'd like to make sure that we can feed the ingest at the max rate (if possible). Whether we can write the folder chunks one at a time or in parallel (to multiple tape drives) remains to be seen. Presumably, this would put a large random read load on the cluster. Once the media has been successfully written, the chunked copy can be deleted. Notes: Currently, planning for all access to be done via Linux servers. I'm eagerly watching the windows native CephFS beta. The server performing the chunking job will be the only reader/writer of the data. The server performing the streaming job will also be the only reader/writer of the data. If we can support parallel, then there may be 2-3 chunking servers and 2-3 streaming servers operating concurrently. There are only a few system in play... NOT hundreds of concurrent clients accessing the data. One might assume that we could keep the raw data on cheaper disk and then "reconstruct" the copy on flash. In this scenario, we can stream from flash. I'd definitely appreciate your feedback on whether CephFS would be a good fit. Thanks in advance for your thoughts! - Steve

1 0