February 2021 - ceph-users

How to debug container upgrade issue?
by Darrin Hodges 10 Feb '21

10 Feb '21

HI all, Upgrading ceph containers is failing, how do I debug it? 'cephadm pull' seems to work on the node, but the upgrade fails. $ ceph orch upgrade start --ceph-version 15.2.8 "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

3 2

Device is not available after zap
by Tony Liu 10 Feb '21

10 Feb '21

Hi, With v15.2.8, after zap a device on OSD node, it's still not available. The reason is "locked, LVM detected". If I reboot the whole OSD node, then the device will be available. There must be something no being cleaned up. Any clues? Thanks! Tony

4 7

Some slow writes over time
by João Victor Mafra 10 Feb '21

10 Feb '21

I am running a simple workload (just 1 client and 1 file) of random writings on Ceph FS and I noticed that approximately 3% of the operations (well spread over time) show latencies higher than the other 97% (100 ms x 10 ms). Is there any reason for this to happen? - I'm using fio with O_DIRECT to avoid cache buffer, so it is expected that the operations will only be completed after writing to the disk. - My WAL is also disabled, so there is no reason for ceph to be doing deferred writing. - I performed the same workload on the gluster fs and the latencies were uniform over time.

1 0

using secondary OSDs for reading
by mj 10 Feb '21

10 Feb '21

Hi, Quoting the page https://docs.ceph.com/en/latest/architecture/ > location query over a chatty session. The CRUSH algorithm allows a > client to compute where objects should be stored, and enables the > client to contact the primary OSD to store or retrieve the objects. So clients contact the *primary* OSD to store/retrieve objects. Why do clients not contact secondary (or even tertiary) OSDs for reading data? Would that not (potentially) result in greatly improved performance..? I'm sure there are good reasons for the current behaviour, but it seems logical, since you have multiple copies of the same data, to try and use the nearest copy? Curious :-) MJ

3 5

File I/O with mixed read/write and high streaming performance
by perkins＠stephen-perkins.com 10 Feb '21

10 Feb '21

Hi all, I'm trying to understand if the CephFS is a good approach for the following scenario. From some of the OLD benchmarks, GlusterFS significantly beat the CephFS when many file I/Os were required. But... this was an OLD benchmark. I'd like your thoughts on the matter. What I need to perform are the following two steps: Re-organize files (Step one) I need to take a large directory structure (assumed to reside on CephFS) and "re-arrange" it via a copy or link mechanism. I will want to make a full copy of the directory structure but do a simple disk span chunking so that all the files in the original copy end up in a set of folders where each folder is no larger than a fixed size. This is like what we did back in the days where we needed to write data in CDROM sized chunks. There is a set of tools that will do this in the genisoimage package (dssplit and dirsplit). Folder Axe was the MS Windows equivalent Presumably, this would put a large random read and random write load on the cluster. Since the size can be large (hundreds of G (maybe up to 1TB) with 10s to 100s of thousands of small files), I would need for this to be well optimized. One mechanism that might be available is to use hard or soft links so that no actual copying is done (Don't know if CephFS/POSIX supports this). The linking approach would probably put a large strain on the MDS servers but not so much on the storage. Write to media (Step two) I need to stream the chunked folders to a set of media devices (think tape drive) that can ingest at high speed (about 200 megabytes per second... yes bytes). I'd like to make sure that we can feed the ingest at the max rate (if possible). Whether we can write the folder chunks one at a time or in parallel (to multiple tape drives) remains to be seen. Presumably, this would put a large random read load on the cluster. Once the media has been successfully written, the chunked copy can be deleted. Notes: Currently, planning for all access to be done via Linux servers. I'm eagerly watching the windows native CephFS beta. The server performing the chunking job will be the only reader/writer of the data. The server performing the streaming job will also be the only reader/writer of the data. If we can support parallel, then there may be 2-3 chunking servers and 2-3 streaming servers operating concurrently. There are only a few system in play... NOT hundreds of concurrent clients accessing the data. One might assume that we could keep the raw data on cheaper disk and then "reconstruct" the copy on flash. In this scenario, we can stream from flash. I'd definitely appreciate your feedback on whether CephFS would be a good fit. Thanks in advance for your thoughts! - Steve

1 0

firewall config for ceph fs client
by Frank Schilder 10 Feb '21

10 Feb '21

Hi all, we plan to add a kernel client mount to a server in our DMZ. I can't find information on how to allow a ceph client to access a ceph cluster through a firewall. Does somebody have a link or sample configs for both, iptables on the host itself and for a transparent firewall between host and cluster? Note: performance is not an issue here. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 10 Feb '21

10 Feb '21

Hi Tom, This is great! will look into the PR. Regarding the tests, the unit tests for amqp are actually here [1]. However, they are testing against a mock amqp library [2] and not a real broker, so, I don't think it is critical to cover SSL there. The disabled tests you pointed out, are the integration tests, currently they are disabled because we don't have the infrastructure in teuthology (auto test automation framework) to run them. To run them locally, you would need to set "skip_push_tests" to "False". Note that the amqp tests there are running the rabbitmq server locally. If you want to test against an existing broker running on a different machine, you should comment out the "init_rabbitmq()" call, and set the endpoint currently in the topic. As part of the tests we are also creating the consumer for the notification (so we can validate the results), there [3] also we assume that the broker runs locally, so if it is not the case. it will have to be changed as well. Information on how to run them locally is here [4]. Let me know if you need help with that. Yuval [1] https://github.com/ceph/ceph/blob/master/src/test/rgw/test_rgw_amqp.cc [2] https://github.com/ceph/ceph/blob/master/src/test/rgw/amqp_mock.cc [3] https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/tests_ps.py… [4] https://github.com/ceph/ceph/blob/master/src/test/rgw/test_multi.md On Wed, Feb 10, 2021 at 11:03 AM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > I opened a PR <https://github.com/ceph/ceph/pull/39392> on Github to add > SSL support for AMQP connections. I haven’t been able to test it though as > the AMQP unit tests all look disabled > <https://github.com/ceph/ceph/blob/fe69a2abf44ad38e9c3523d24cb173dbb2ccd026/…>. > Could you provide some guidance please? > > Many thanks, > > Tom > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 29 Jan 2021, at 09:14, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Fri, Jan 29, 2021 at 9:18 AM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> What do I need to do if I want to switch to using a different exchange on >> the RabbitMQ endpoint? Or change the amqp-ack-level option that was used? >> Would you expect the same problem again? Will the existing connections to >> the RabbitMQ server be cleanly terminated? >> >> I think that changing the ack level would take effect on the next publish > (as this is not a feature of the connection, but the calling code), but to > change the exchange (or any other parameter of the connection itself), or > even creating a new topic to the same endpoint with a different exchange, > you would need a restart :-( > (tracking this here: https://tracker.ceph.com/issues/46127) > > >> I tried the topic example >> <https://www.rabbitmq.com/tutorials/tutorial-five-python.html> from the >> RabbitMQ tutorial and I actually got the same behaviour as with Ceph: >> messages sent before the consumer queue is attached are lost. From what I >> understand this is a *feature* of this type of exchange. See also this >> <https://stackoverflow.com/questions/6148381/rabbitmq-persistent-message-wit…> stackoverflow >> post. >> >> > this is interesting. and looks like a better model than what we > currently have. > we should declare our own fanout "gateway" exchange, connected to an "eat > all" queue. the users may then connect the consumers directly to it, or via > a topic exchange they declare. that would actually fix 2 issues: > - durability of messages before clients are connected > - exchange name configuration issues > > >> Best, >> >> Tom >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 28 Jan 2021, at 18:16, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> >> On Thu, Jan 28, 2021 at 7:34 PM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Hi Yuval, >>> >>> >>> Together with Tom Byrne I ran some more tests today while keeping an eye >>> on the logs as well. >>> >>> We immediately noticed that the nodes were logging errors when uploading >>> files like: >>> >>> 2021-01-28 16:10:45.825 7f56ff5cf700 1 ====== starting new request req=0x7f56ff5c87f0 ===== >>> 2021-01-28 16:10:45.828 7f5721e14700 1 AMQP connect: exchange mismatch >>> 2021-01-28 16:10:45.828 7f5721e14700 1 ERROR: failed to create push endpoint: amqp://<username>:<password>@<my.rabbitmq.server>:5672 due to: pubsub endpoint configuration error: AMQP: failed to create connection to: amqp://<username>:<password>@<my.rabbitmq.server>:5672 >>> 2021-01-28 16:10:45.828 7f571ee0e700 1 ====== req done req=0x7f571ee077f0 op status=0 http_status=200 latency=0.0569997s ====== >>> >>> >>> Which resulted in no connections being established to the RabbitMQ >>> server. >>> >>> Tom restarted then the Ceph services on one gateway node, which led to >>> events being sent to RabbitMQ without blocking, but only if this particular >>> node was picked up by the boto3 upload request in the round-robin DNS. >>> >>> Restarting the Ceph service on all nodes fixed the problem and I got a >>> nice steady stream of events to my consumer Python script! >>> >>> >> we should fix it. no restart should be needed if one of the connection >> parameters was wrong >> >> >> >>> I did notice that any events that were sent while my consumer script was >>> not running are lost, as they are not picked up after I restart the script. >>> Any thoughts on this? >>> >>> >> this is strange. in our code [1] we don't require immediate transfer of >> messages. >> how is the exchange declared? >> can you check if this is happening when you send messages from a python >> producer as well? >> >> [1] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_amqp.cc#L575 >> >> >> >>> Many thanks!! >>> >>> Best, >>> >>> Tom >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 27 Jan 2021, at 16:21, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> >>> On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < >>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>> >>>> Looks like there’s already a ticket open for AMQP SSL support: >>>> https://tracker.ceph.com/issues/42902 (you opened it ;-)) >>>> >>>> I will give a try myself if I have some time, but don’t hold your >>>> breath with lockdown and home schooling. Also I am not much of a C++ coder. >>>> >>>> I need to go over the logs with Tom Byrne to see why it is not working >>>> properly. And perhaps I will be able to come up with a fix then. >>>> >>>> However this is what I have run into so far today: >>>> >>>> 1. After configuring a bucket with a topic using the non-SSL port, I >>>> tried a couple of uploads to this bucket. They all hanged, which seemed >>>> like something was very wrong, so I Ctrl-C’ed every time. After some time I >>>> figured out from the RabbitMQ admin UI that Ceph was indeed connecting to >>>> it, and the connections remained so I killed them from the UI. >>>> >>> >>> sending the notification to the rabbitmq server is synchronous with the >>> upload to the bucket. so, if the server is slow or not acking the >>> notification, the upload request would hang. not that the upload itself is >>> done first, but the reply to the client does not happen until rabbitmq >>> server acks. >>> >>> would be great if you can share the radosgw logs. >>> maybe the issue is related to the user/password method we use? we use: >>> AMQP_SASL_METHOD_PLAIN >>> >>> one possible workaround would be to set "amqp-ack-level" to "none". in >>> this case the radosgw does not wait for an ack >>> >>> in "pacific" you could use "persistent topics" where the notifications >>> are sent asynchronously to the endpoint. >>> >>> 2. I then wrote a python script with Pika to consume the events, hoping >>>> that would stop the blocking. I had some minor success with this. Usually >>>> the first three or four uploaded files would generate events that I could >>>> consume with my script. >>>> >>> >>> the radosgw is waiting for an ack from the broker, not the end consumer, >>> so this should not have mattered... >>> did you actually see any notifications delivered to the consumer? >>> >>> >>>> However, the rest would block for ever. I repeated this a couple of >>>> times but always the same result. I noticed that after I stopped uploading, >>>> removed the bucket and the topic, the connection from Ceph in the RabbitMQ >>>> UI remained. I killed it but it came back seconds later from another port >>>> on the Ceph cluster. I ended up playing whack-a-mole with this until no >>>> more connections would be established from Ceph to RabbitMQ. I probably >>>> killed a 100 or so of them. >>>> >>> >>> once you remove the bucket there cannot be new notification sent. if you >>> create the bucket again you may see notifications again (this is fixed in >>> "pacific"). >>> either way, even if the connection to the rabbitmq server would still be >>> open, but no new notifications should be sent there. just having the >>> connection should not be an issue but would be nice to fix that as well: >>> https://tracker.ceph.com/issues/49033 >>> >>> 3. After this I couldn’t get any events sent anymore. There is no more >>>> blocking when uploading, files get written but nothing else happens. No >>>> connections are made anymore from Ceph to RabbitMQ. >>>> >>>> Hope this helps… >>>> >>> >>> yes, this is very helpful! >>> >>> >>>> Best, >>>> >>>> Tom >>>> >>>> >>>> >>>> >>>> Dr Tom Schoonjans >>>> >>>> Research Software Engineer - HPC and Cloud >>>> >>>> Rosalind Franklin Institute >>>> Harwell Science & Innovation Campus >>>> Didcot >>>> Oxfordshire >>>> OX11 0FA >>>> United Kingdom >>>> >>>> https://www.rfi.ac.uk >>>> >>>> The Rosalind Franklin Institute is a registered charity in England and >>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>> the Engineering and Physical Sciences Research Council. >>>> >>>> On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>> >>>> >>>> >>>> On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < >>>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>>> >>>>> Hi Yuval, >>>>> >>>>> >>>>> Switching to non-SSL connections to RabbitMQ allowed us to get things >>>>> working, although currently it’s not very reliable. >>>>> >>>> >>>> can you please add more about that? what reliability issues did you see? >>>> >>>> >>>>> I will open a new ticket over this if we can’t fix things ourselves. >>>>> >>>>> >>>> this would be great. we have ssl support for kafka and http endpoint, >>>> so, if you decide to give it a try you can look at them as examples. >>>> and let me know if you have questions or need help. >>>> >>>> >>>> >>>>> I will open an issue on the tracker as soon as my account request has >>>>> been approved :-) >>>>> >>>>> Best, >>>>> >>>>> Tom >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Dr Tom Schoonjans >>>>> >>>>> Research Software Engineer - HPC and Cloud >>>>> >>>>> Rosalind Franklin Institute >>>>> Harwell Science & Innovation Campus >>>>> Didcot >>>>> Oxfordshire >>>>> OX11 0FA >>>>> United Kingdom >>>>> >>>>> https://www.rfi.ac.uk >>>>> >>>>> The Rosalind Franklin Institute is a registered charity in England and >>>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>>> the Engineering and Physical Sciences Research Council. >>>>> >>>>> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>>> >>>>> >>>>> >>>>> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >>>>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>>>> >>>>>> Hi Yuval, >>>>>> >>>>>> >>>>>> I worked on this earlier today with Tom Byrne and I think I may be >>>>>> able to provide some more information. >>>>>> >>>>>> I set up the RabbitMQ server myself, and created the exchange with >>>>>> type ’topic’ before configuring the bucket. >>>>>> >>>>>> Not sure if this matters, but the RabbitMQ endpoint is reached over >>>>>> SSL, using certificates generated with Letsencrypt. >>>>>> >>>>>> >>>>> it actually does. we don't support amqp over ssl. >>>>> feel free to open a tracker for that - as we should probably support >>>>> that! >>>>> but note that it would probably be backported only to later versions >>>>> than nautilus. >>>>> >>>>> >>>>> >>>>>> Many thanks, >>>>>> >>>>>> Tom >>>>>> >>>>>> >>>>>> >>>>>> Dr Tom Schoonjans >>>>>> >>>>>> Research Software Engineer - HPC and Cloud >>>>>> >>>>>> Rosalind Franklin Institute >>>>>> Harwell Science & Innovation Campus >>>>>> Didcot >>>>>> Oxfordshire >>>>>> OX11 0FA >>>>>> United Kingdom >>>>>> >>>>>> https://www.rfi.ac.uk >>>>>> >>>>>> The Rosalind Franklin Institute is a registered charity >>>>>> in England and Wales, No. 1179810 Company Limited by Guarantee Registered >>>>>> in England and Wales, No.11266143. Funded by UK Research and >>>>>> Innovation through the Engineering and Physical Sciences Research Council. >>>>>> >>>>>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>>>> >>>>>> Hi Tom, >>>>>> Did you create the exchange in rabbitmq? The RGW does not create it >>>>>> and assume it is already created? >>>>>> Could you increase the log level in RGW and see if there are more log >>>>>> messages that have "AMQP" in them? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yuval >>>>>> >>>>>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>>>>> tom.byrne(a)stfc.ac.uk> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We've been trying to get RGW Bucket notifications working with a >>>>>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>>>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>>>>> a message to the endpoint, the message never appears in the queue, and we >>>>>>> get this error from in the RGW logs: >>>>>>> >>>>>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>>>>> Endpoint >>>>>>> URI: amqp://user:pass@host:5671 >>>>>>> Topic: ceph-topic-test >>>>>>> Exchange: ceph-test >>>>>>> Ack Level: broker failed, with error: -4098 >>>>>>> >>>>>>> We've confirmed the URI is correct, and that the gateway host can >>>>>>> send messages to the RabbitMQ via a standalone script (using the same >>>>>>> information as in the URI). Does anyone have any hints about how to dig >>>>>>> into this? >>>>>>> >>>>>>> Cheers, >>>>>>> Tom >>>>>>> >>>>>>> This email and any attachments are intended solely for the use of >>>>>>> the named recipients. If you are not the intended recipient you must not >>>>>>> use, disclose, copy or distribute this email or any of its attachments and >>>>>>> should notify the sender immediately and delete this email from your >>>>>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>>>>> precaution to minimise risk of this email or any attachments containing >>>>>>> viruses or malware but the recipient should carry out its own virus and >>>>>>> malware checks before opening the attachments. UKRI does not accept any >>>>>>> liability for any losses or damages which the recipient may sustain due to >>>>>>> presence of any viruses. Opinions, conclusions or other information in this >>>>>>> message and attachments that are not related directly to UKRI business are >>>>>>> solely those of the author and do not represent the views of UKRI. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> >>>>>> >> >

1 0

How is DB handled when remove/replace and add OSD?
by Tony Liu 10 Feb '21

10 Feb '21

Hi, I'd like to know how DB device is expected to be handled by "orch osd rm". What I see is that, DB device on SSD is untouched when OSD on HDD is removed or replaced. "orch device zap" removes PV, VG and LV of the device. It doesn't touch the DB LV on SSD. To remove an OSD permanently, do I need to manually clean up the DB LV on SSD? To replace and OSD, is the old DB LV going to be reused for the new OSD, or a new DB LV will be created? I am asking this because, to replace an OSD, when the OSD is removed, I manually removed DB LV on SSD. Now, I try to add new OSD, but --try-run doesn't show DB device. ``` # cat osd-spec.yaml service_type: osd service_id: osd-spec placement: hosts: - ceph-osd-1 spec: #objectstore: bluestore #block_db_size: 32212254720 #block_db_size: 64424509440 data_devices: rotational: 1 db_devices: #rotational: 0 size: ":500GB" #unmanaged: true # ceph orch apply osd -i osd-spec.yaml --dry-run +---------+----------+------------+----------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+------------+----------+----+-----+ |osd |osd-spec |ceph-osd-1 |/dev/sdd |- |- | +---------+----------+------------+----------+----+-----+ ``` Any clues? Thanks! Tony

1 0

Announcing go-ceph v0.8.0
by John Mulligan 09 Feb '21

09 Feb '21

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.8.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Speed of S3 Ceph gateways
by michal.strnad＠cesnet.cz 09 Feb '21

09 Feb '21

Hi all, We are testing our S3 Ceph endpoints and we are not satisfied with its speed. Our results are something between around 120 - 150 MB/s depending on small/bigger files. This is good for 1Gbps connection, but not for 10GE or more. We've tried the most recent versions of the AWS CLI, s3cmd, s4cmd, s3fs ... programs. Of course we are using multipart upload/download which is precondition for parallel upload/download. Also we tried multi-thread (25 or more threads) transfer in s4cmd but still we don't get proper results. For proof of concept that high speed can be achieved we have written small script in bash which uses multi-part & parallel transfer and can saturate at least 10GE without problem. I would like to ask you, if you know proper program and its parameters, so we can saturate n x 10GE if needed? We are using the latest nautilus. S3 gateways have much more computer power and bandwidth to internet then it is used right now. Thank you Regards Michal Strnad

4 3

Can see objects with "rados ls" but cannot delete them with "rados rm"
by James, GleSYS 09 Feb '21

09 Feb '21

Hi, We have in issue in our cluster (octopus 15.2.7) where we’re unable to remove orphaned objects from a pool, despite the fact these objects can be listed with “rados ls”. Here is an example of an orphaned object which we can list (not sure why multiple objects are returned with the same name…related to the issue perhaps?) rados ls -p default.rgw.buckets.data | grep -i 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 And the error message when we try to stat / rm the object: rados stat -p default.rgw.buckets.data 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 error stat-ing default.rgw.buckets.data/5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6: (2) No such file or directory rados -p default.rgw.buckets.data rm 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 error removing default.rgw.buckets.data>5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6: (2) No such file or directory The bucket with id "5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83” was deleted from radosgw a few months ago, but we still have approximately 450,000 objects with this bucket id that are orphaned: cat orphan-list-202101191211.out | grep -i 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83 | wc -l 448683 I can also see from our metrics that prior to deletion there was about 10TB of compressed data stored in this bucket, and this has not been reclaimed in the pool usage after the bucket was deleted. Anyone have any suggestions on how we can remove these objects and reclaim the space? We’re not using snapshots or cache tiers in our environment. Thanks, James.

3 3

one PG is stuck on single OSD and does not sync back
by Boris Behrens 09 Feb '21

09 Feb '21

Good day together, I've got an issue after rebooting an osd node. It looks like there is one PG that does not sync back to the other UP osds. I've tried to restart the ceph processes for all three OSDs and when I stopped the one on OSD.14 the PG went down. Any ideas what I can do? # ceph pg ls degraded PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 10.1a 2078 766 0 0 0 9071695544 28506523 3031 active+undersized+degraded+remapped+backfilling 2m 301810'52824786 301818:1346392477 [18,14,54]p18 [14]p14 2021-01-02 06:28:01.124161 2021-01-02 06:28:01.124161 PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg down pg 10.1a is down, acting [18,54] Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 6

ceph monitors using 6-96GB RAM and crashing [nautilus]
by Nico Schottelius 08 Feb '21

08 Feb '21

Hello, we have recently moved our ceph monitors from small, 4GB RAM servers to big servers, because we saw memory pressure on the machines. However even on our big machines (64GB ~ 1TB RAM) we are seeing ceph-mon processes being killed at around 90-94GB of RAM. Now, my understanding is that a ceph-mon should be a rather "small" process, so allocating 90GB RAM seems to be a bit.. atypical? This cluster runs ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable), about 90 osds, about 620TB data. Any input is appreciated. Best regards, Nico PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/1 36244 - 29 0 7312K 15156K 95.1G 912K 95.3G 94.9G 0K 0K 0K 0K ceph ceph 9% ceph-mon -- Renewable hosting with full IPv6 support. Checkout www.datacenterlight.ch for details.

2 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

ceph-volume bluestore _read_fsid unparsable uuid
by Frank Schilder 06 Feb '21

06 Feb '21

Hi Dave and everyone else affected, I'm responding to a thread you opened on an issue with lvm OSD creation: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YYH3VANVV22… https://tracker.ceph.com/issues/43868 Most important question: is there a workaround? My observations: I'm running into the exact same issue on mimic 13.2.10. The strange thing is, that some OSDs get created and others fail. I can't see a pattern here. I have 1 host where every create worked out and another where half failed. The important lines in the log are probably: stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bluestore(/var/lib/ceph/osd/ceph-342/) _read_fsid unparsable uuid stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bdev(0x561db199c700 /var/lib/ceph/osd/ceph-342//block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bluestore(/var/lib/ceph/osd/ceph-342/) mkfs failed, (11) Resource temporarily unavailable stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (11) Resource temporarily unavailable stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-342/: (11) Resource temporarily unavailable[0m I really need to get a decent number of disks up very soon. Any help is appreciated. I can provide more output if that helps. Best regards and good weekend! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 2

Worst thing that can happen if I have size= 2
by Mario Giammarco 06 Feb '21

06 Feb '21

Hello, Imagine this situation: - 3 servers with ceph - a pool with size 2 min 1 I know perfectly the size 3 and min 2 is better. I would like to know what is the worst thing that can happen: - a disk breaks and another disk breaks before ceph has reconstructed second replica, ok I lose data - if network goes down and so monitors lose quorum do ceph still write on disks? What else? Thanks, Mario

16 28

mon db high iops
by Seena Fallah 05 Feb '21

05 Feb '21

Hi all, My monitor nodes are getting up and down because of paxos lease timeout and there is a high iops (2k iops) and 500MB/s throughput on /var/lib/ceph/mon/ceph.../store.db/. My cluster is in a recovery state and there is a bunch of degraded pgs on my cluster. It seems it's doing a 200k block size io on rocksdb. Is that okay?! Also is there any solution to fix these downtimes for monitors? Thanks for your help!

2 4

NVMe and 2x Replica
by Adam Boyhan 05 Feb '21

05 Feb '21

I know there is already a few threads about 2x replication but I wanted to start one dedicated to discussion on NVMe. There are some older threads, but nothing recent that addresses how the vendors are now pushing the idea of 2x. We are in the process of considering Ceph to replace our Nimble setup. We will have two completely separate clusters at two different sites that we are using rbd-mirror snapshot replication. The plan would be to run 2x replication on each cluster. 3x is still an option, but for obvious reasons 2x is enticing. Both clusters will be spot on to the super micro example in the white paper below. It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise. Trying to wrap my head around were the disconnect is between the big players and the community. I could be missing something, but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe. Appreciate the input! [ https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf | https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referen… ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referen… | https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-referen… ] [ https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/red… | https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/red… ]

12 22

ceph-volume deactivate errs out with "findmnt: invalid option -- M"
by Frank Schilder 05 Feb '21

05 Feb '21

Hi all, I'm experimenting with ceph-volume on Centos7, ceph mimic 13.2.10. When I execute "ceph-volume deactivate ..." on a previousy activated OSD, I get this error: # ceph-volume lvm deactivate 12 0bbf481c-6a3d-4724-9a27-3a845eb05911 stderr: /usr/bin/findmnt: invalid option -- 'M' stderr: Usage: Help summary of findmnt at the bottom. I could hack a script together that forwards this call to findmnt and translates the option. Can anyone tell me if this is a bug or if I'm running with the wrong version? In the mean time, is ceph-volume deactivate just unmounting /var/lib/ceph/osd/ceph-ID directories, or is there more to it (no dmcrypt in use)? Thanks! # findmnt --help Usage: findmnt [options] findmnt [options] <device> | <mountpoint> findmnt [options] <device> <mountpoint> findmnt [options] [--source <device>] [--target <mountpoint>] Options: -s, --fstab search in static table of filesystems -m, --mtab search in table of mounted filesystems -k, --kernel search in kernel table of mounted filesystems (default) -p, --poll[=<list>] monitor changes in table of mounted filesystems -w, --timeout <num> upper limit in milliseconds that --poll will block -A, --all disable all built-in filters, print all filesystems -a, --ascii use ASCII chars for tree formatting -c, --canonicalize canonicalize printed paths -D, --df imitate the output of df(1) -d, --direction <word> direction of search, 'forward' or 'backward' -e, --evaluate convert tags (LABEL,UUID,PARTUUID,PARTLABEL) to device names -F, --tab-file <path> alternative file for --fstab, --mtab or --kernel options -f, --first-only print the first found filesystem only -i, --invert invert the sense of matching -l, --list use list format output -N, --task <tid> use alternative namespace (/proc/<tid>/mountinfo file) -n, --noheadings don't print column headings -u, --notruncate don't truncate text in columns -O, --options <list> limit the set of filesystems by mount options -o, --output <list> the output columns to be shown -P, --pairs use key="value" output format -r, --raw use raw output format -t, --types <list> limit the set of filesystems by FS types -v, --nofsroot don't print [/dir] for bind or btrfs mounts -R, --submounts print all submounts for the matching filesystems -S, --source <string> the device to mount (by name, maj:min, LABEL=, UUID=, PARTUUID=, PARTLABEL=) -T, --target <string> the mountpoint to use -h, --help display this help and exit -V, --version output version information and exit Available columns: SOURCE source device TARGET mountpoint FSTYPE filesystem type OPTIONS all mount options VFS-OPTIONS VFS specific mount options FS-OPTIONS FS specific mount options LABEL filesystem label UUID filesystem UUID PARTLABEL partition label PARTUUID partition UUID MAJ:MIN major:minor device number ACTION action detected by --poll OLD-TARGET old mountpoint saved by --poll OLD-OPTIONS old mount options saved by --poll SIZE filesystem size AVAIL filesystem size available USED filesystem size used USE% filesystem use percentage FSROOT filesystem root TID task ID ID mount ID OPT-FIELDS optional mount fields PROPAGATION VFS propagation flags FREQ dump(8) frequency in days [fstab only] PASSNO pass number on parallel fsck(8) [fstab only] For more details see findmnt(8). ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

can't query most pgs after restart
by Jeremy Austin 05 Feb '21

05 Feb '21

I was in the middle of a rebalance on a small test cluster with about 1% of pgs degraded, and shut the cluster entirely down for maintenance. On startup, many pgs are entirely unknown, and most stale. In fact most pgs can't be queried! No mon failures. Would osd logs tell me why pgs aren't even moving to an inactive state? I'm not concerned about data loss due to the shutdown (all activity to the cluster had been stopped), so should I be setting some or all OSDs " osd_find_best_info_ignore_history_les = true"? Thank you, -- Jeremy Austin jhaustin(a)gmail.com

2 2

Re: Worst thing that can happen if I have size= 2
by Dan van der Ster 05 Feb '21

05 Feb '21

On Thu, Feb 4, 2021 at 10:30 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > >IMO with a cluster this size, you should not ever mark out any OSDs -- > >rather, you should leave the PGs degraded, replace the disk (keep the > >same OSD ID), then recover those objects to the new disk. > >Or, keep it <40% used (which sounds like a waste). > > Dear Dan, > > I particularly like your idea of " leave the PGs degraded, and replace the disk with the same OSD ID ". This is a wonderful thing i really want to do. > > Could you please share some more details on how to achieve this, or some scripts already being tested? Hi Samuel, To do this you'd first set a config so that down osds aren't automatically marked out, either by setting `mon_osd_down_out_interval` to a really large value, or perhaps mon_osd_down_out_subtree_limit = osd would work too. Then when the osd fails, PGs will become degraded. You just zap, then replace, then recreate the OSD. Here are the ceph-volume commands for that: * ceph-volume lvm zap --osd-id 1234 --destroy * (then you might need `ceph osd destroy 1234`) * (then replace the drive) * ceph-volume lvm create <according to your hw> --osd-id 1234 Cheers, Dan > > thanks a lot, > > Samuel > > > > ________________________________ > huxiaoyu(a)horebdata.cn > > > From: Dan van der Ster > Date: 2021-02-04 11:57 > To: Mario Giammarco > CC: Ceph Users > Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 > On Thu, Feb 4, 2021 at 11:30 AM Mario Giammarco <mgiammarco(a)gmail.com> wrote: > > > > > > > > Il giorno mer 3 feb 2021 alle ore 21:22 Dan van der Ster <dan(a)vanderster.com> ha scritto: > >> > >> > >> Lastly, if you can't afford 3x replicas, then use 2+2 erasure coding if possible. > >> > > > > I will investigate I heard that erasure coding is slow. > > > > Anyway I will write here the reason of this thread: > > In my customers I have usually proxmox+ceph with: > > > > - three servers > > - three monitors > > - 6 osd (two per server) > > - size=3 and min_size=2 > > > > I followed the recommendations to stay safe. > > But one day one disk of one server has broken, osd where at 55%. > > What happened then? > > Ceph started filling the remaining OSD to maintain size=3 > > OSD reached 90% ceph stopped all. > > Customer VMs froze and customer lost time and some data that was not written on disk. > > > > So I got angry.... size=3 and customer still loses time and data? > > You should size the osd fullness config in such a way that failure you > expect would still leave sufficient capacity. > In our case, we plan so that we could lose and re-replicate an entire > rack and still have enough space left. -- (IOW, with 5-6 racks, we > start to add capacity when the clusters reach ~70-75% full) > > In your case, the issue is more extreme: > Because you have 3 hosts, 2 osds each, and 3 replicas: when one OSD > fails and is marked out, you are telling ceph that *all* of the > objects will need to be written to the last remaining disk on that > host with the failure. > So unless your cluster was under 40-50% used, that osd is going to > become overfull. (But BTW, ceph will get backfillfull on the loaded > OSD before stopping IO -- this should not have blocked your user > unless they *also* filled the disk with new data at the same time). > > IMO with a cluster this size, you should not ever mark out any OSDs -- > rather, you should leave the PGs degraded, replace the disk (keep the > same OSD ID), then recover those objects to the new disk. > Or, keep it <40% used (which sounds like a waste). > > -- dan > > > > > > > > > > > > > > > > >> > >> Cheers, Dan > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> On Wed, Feb 3, 2021, 8:49 PM Mario Giammarco <mgiammarco(a)gmail.com> wrote: > >>> > >>> Thanks Simon and thanks to other people that have replied. > >>> Sorry but I try to explain myself better. > >>> It is evident to me that if I have two copies of data, one brokes and while > >>> ceph creates again a new copy of the data also the disk with the second > >>> copy brokes you lose the data. > >>> It is obvious and a bit paranoid because many servers on many customers run > >>> on raid1 and so you are saying: yeah you have two copies of the data but > >>> you can broke both. Consider that in ceph recovery is automatic, with raid1 > >>> some one must manually go to the customer and change disks. So ceph is > >>> already an improvement in this case even with size=2. With size 3 and min 2 > >>> it is a bigger improvement I know. > >>> > >>> What I ask is this: what happens with min_size=1 and split brain, network > >>> down or similar things: do ceph block writes because it has no quorum on > >>> monitors? Are there some failure scenarios that I have not considered? > >>> Thanks again! > >>> Mario > >>> > >>> > >>> > >>> Il giorno mer 3 feb 2021 alle ore 17:42 Simon Ironside < > >>> sironside(a)caffetine.org> ha scritto: > >>> > >>> > On 03/02/2021 09:24, Mario Giammarco wrote: > >>> > > Hello, > >>> > > Imagine this situation: > >>> > > - 3 servers with ceph > >>> > > - a pool with size 2 min 1 > >>> > > > >>> > > I know perfectly the size 3 and min 2 is better. > >>> > > I would like to know what is the worst thing that can happen: > >>> > > >>> > Hi Mario, > >>> > > >>> > This thread is worth a read, it's an oldie but a goodie: > >>> > > >>> > > >>> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-December/014846.ht… > >>> > > >>> > Especially this post, which helped me understand the importance of > >>> > min_size=2 > >>> > > >>> > > >>> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-December/014892.ht… > >>> > > >>> > Cheers, > >>> > Simon > >>> > _______________________________________________ > >>> > ceph-users mailing list -- ceph-users(a)ceph.io > >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

replace OSD failed
by Tony Liu 05 Feb '21

05 Feb '21

Hi, With 15.2.8, run "ceph orch rm osd 12 --replace --force", PGs on osd.12 are remapped, osd.12 is removed from "ceph osd tree", the daemon is removed from "ceph orch ps", the device is "available" in "ceph orch device ls". Everything seems good at this point. Then dry-run service spec. ``` # cat osd-spec.yaml service_type: osd service_id: osd-spec placement: hosts: - ceph-osd-1 data_devices: rotational: 1 db_devices: rotational: 0 # ceph orch apply osd -i osd-spec.yaml --dry-run +---------+----------+------------+----------+----------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+------------+----------+----------+-----+ |osd |osd-spec |ceph-osd-3 |/dev/sdd |/dev/sdb |- | +---------+----------+------------+----------+----------+-----+ ``` It looks as expected. Then "ceph orch apply osd -i osd-spec.yaml". Here is the log of cephadm. ``` /bin/docker:stderr --> relative data size: 1.0 /bin/docker:stderr --> passed block_db devices: 1 physical, 0 LVM /bin/docker:stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/docker:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json /bin/docker:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b05c3c90-b7d5-4f13-8a58-f72761c1971b 12 /bin/docker:stderr Running command: /usr/sbin/vgcreate --force --yes ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 /dev/sdd /bin/docker:stderr stdout: Physical volume "/dev/sdd" successfully created. /bin/docker:stderr stdout: Volume group "ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64" successfully created /bin/docker:stderr Running command: /usr/sbin/lvcreate --yes -l 572318 -n osd-block-b05c3c90-b7d5-4f13-8a58-f72761c1971b ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 /bin/docker:stderr stderr: Volume group "ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64" has insufficient free space (572317 extents): 572318 required. /bin/docker:stderr --> Was unable to complete a new OSD, will rollback changes ``` Q1, why VG name (ceph-<id>) is different from others (ceph-block-<id>)? Q2, where is that 572318 from? Since all HDDs are the same model, VG "Total PE" of all HDDs is 572317. Has anyone seen similar issues? Anything I am missing? Thanks! Tony

1 2

Multisite reshard stale instances
by Szabo, Istvan (Agoda) 05 Feb '21

05 Feb '21

Hi, I found 6-700 stale instances with the reshard stale instances list command. Is there a way to clean it up (or actually should I clean it up)? The stale instance rm doesn't work in multisite. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

log_meta log_data was turned off in multisite and deleted
by Szabo, Istvan (Agoda) 05 Feb '21

05 Feb '21

Hi, Is there a way to reinitialize the stored data and make it sync from the logs? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Re: Using RBD to pack billions of small files
by Loïc Dachary 04 Feb '21

04 Feb '21

Hi Frederico, On 04/02/2021 05:51, Federico Lucifredi wrote: > Hi Loïc, > I am intrigued, but am missing something: why not using RGW, and store the source code files as objects? RGW has native compression and can take care of that behind the scenes. Excellent question! > > Is the desire to use RBD only due to minimum allocation sizes? I *assume* that since RGW does have specific strategies to take advantage of the fact that objects are immutable and will never be removed: * It will be slower to add artifacts in RGW than in an RBD image + index * The metadata in RGW will be larger than an RBD image + index However I have not verified this and if you have an opinion I'd love to hear it :-) Cheers > > Best -F > > > -- "'Problem' is a bleak word for challenge" - Richard Fish > _________________________________________ > Federico Lucifredi > Product Management Director, Ceph Storage Platform > Red Hat > A273 4F57 58C0 7FE8 838D 4F87 AEEB EC18 4A73 88AC > redhat.com <http://redhat.com> TRIED. TESTED. TRUSTED. > > > On Sat, Jan 30, 2021 at 10:01 AM Loïc Dachary <loic(a)dachary.org <mailto:loic@dachary.org>> wrote: > > Bonjour, > > In the context Software Heritage (a noble mission to preserve all source code)[0], artifacts have an average size of ~3KB and there are billions of them. They never change and are never deleted. To save space it would make sense to write them, one after the other, in an every growing RBD volume (more than 100TB). An index, located somewhere else, would record the offset and size of the artifacts in the volume. > > I wonder if someone already implemented this idea with success? And if not... does anyone see a reason why it would be a bad idea? > > Cheers > > [0] https://docs.softwareheritage.org/ <https://docs.softwareheritage.org/> > > -- > Loïc Dachary, Artisan Logiciel Libre > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> > -- Loïc Dachary, Artisan Logiciel Libre

3 3

NFS version 4.0
by Jens Hyllegaard (Soft Design A/S) 04 Feb '21

04 Feb '21

Hi. We are trying to set up an NFS server using ceph which needs to be accessed by an IBM System i. As far as I can tell the IBM System i only supports nfs v. 4. Looking at the nfs-ganesha deployments it seems that these only support 4.1 or 4.2. I have tried editing the configuration file to support 4.0 and it seems to work. Is there a reason than it currently only support 4.1 and 4.2? I can of course edit the configuration file, but I would have to do that after any deployment or upgrade of the nfs servers. Regards Jens Hyllegaard

2 1

Re: Worst thing that can happen if I have size= 2
by Mario Giammarco 04 Feb '21

04 Feb '21

Hi Federico, here I am not mixing raid1 with ceph. I am doing a comparison: is it safer to have a server with raid1 disks or two servers with ceph and size=2 min_size=1 ? We are talking about real world examples where a customer is buying a new server and want to choose. Il giorno gio 4 feb 2021 alle ore 05:52 Federico Lucifredi < federico(a)redhat.com> ha scritto: > Ciao Mario, > > >> It is obvious and a bit paranoid because many servers on many customers >> run >> on raid1 and so you are saying: yeah you have two copies of the data but >> you can broke both. Consider that in ceph recovery is automatic, with >> raid1 >> some one must manually go to the customer and change disks. So ceph is >> already an improvement in this case even with size=2. With size 3 and min >> 2 >> it is a bigger improvement I know. >> > > Generally speaking, users running Ceph at any scale do not use RAID to > mirror their drives. They rely on data resiliency as delivered by Ceph > (three replicas on HDD, two replicas on solid state media). > > It is expensive to run RAID underneath Ceph, and in some cases even > counter-productive. We do use RAID controllers whenever we can because they > are battery-backed and insure writes hit the local disk even on a power > failure, but that is (ideally) the only case where you hear the words RAID > and Ceph together. > > -- "'Problem' is a bleak word for challenge" - Richard Fish > _________________________________________ > Federico Lucifredi > Product Management Director, Ceph Storage Platform > Red Hat > A273 4F57 58C0 7FE8 838D 4F87 AEEB EC18 4A73 88AC > redhat.com TRIED. TESTED. TRUSTED. >

1 0

replace OSD without PG remapping
by Tony Liu 04 Feb '21

04 Feb '21

Hi, There are multiple different procedures to replace an OSD. What I want is to replace an OSD without PG remapping. #1 I tried "orch osd rm --replace", which sets OSD reweight 0 and status "destroyed". "orch osd rm status" shows "draining". All PGs on this OSD are remapped. Checked "pg dump", can't find this OSD any more. 1) Given [1], setting weight 0 seems better than setting reweight 0. Is that right? If yes, should we change the behavior of "orch osd rm --replace"? 2) "ceph status" doesn't show anything about OSD draining. Is there any way to see the progress of draining? Is there actually copy happening? The PG on this OSD is remapped and copied to another OSD, right? 3) When OSD is replaced, there will be remapping and backfilling. 4) There is remapping in #2 and remapping again in #3. I want to avoid it. #2 Is there any procedure that doesn't mark OSD out (set reweight 0), neither set weight 0, which should keep PG map unchanged, but just warn about less redundancy (one out of 3 OSDs of PG is down), and when OSD is replaced, no remapping, just data backfilling? [1] https://ceph.com/geen-categorie/difference-between-ceph-osd-reweight-and-ce… Thanks! Tony

2 3

Using RBD to pack billions of small files
by Loïc Dachary 03 Feb '21

03 Feb '21

Bonjour, In the context Software Heritage (a noble mission to preserve all source code)[0], artifacts have an average size of ~3KB and there are billions of them. They never change and are never deleted. To save space it would make sense to write them, one after the other, in an every growing RBD volume (more than 100TB). An index, located somewhere else, would record the offset and size of the artifacts in the volume. I wonder if someone already implemented this idea with success? And if not... does anyone see a reason why it would be a bad idea? Cheers [0] https://docs.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre

8 16

15.2.9 ETA?
by David Orman 03 Feb '21

03 Feb '21

Hi, We're looking forward to a few of the major bugfixes for breaking mgr issues with larger clusters that are merged into the Octopus branch, as well as the updated cheroot pushed to EPEL that should make it into the next container build. It's been quite some time since the last (15.2.8) release, and we were curious if there was an ETA on the 15.2.9 release being cut, or at least a place to look and determine status. We checked the docs and didn't see a way to gauge estimated release dates, so thought we'd ask here. Thanks!

1 0

XFS block size on RBD / EC vs space amplification
by Gilles Mocellin 03 Feb '21

03 Feb '21

Hello, As we know, with 64k for bluestore_min_alloc_size_hdd (I'm only using HDDs), in certain conditions, especially with erasure coding, there's a leak of space while writing objects smaller than 64k x k (EC:k+m). Every object is divided in k elements, written on different OSD. My main use case is big (40TB) RBD images mounted as XFS filesystems on Linux servers, exposed to our backup software. So, it's mainly big files. My though, but I'd like some other point of view, is that I could deal with the amplification by using bigger block sizes on my XFS filesystems. Instead of reducing bluestore_min_alloc_size_hdd on all OSDs. What do you think ?

2 2

Re: XFS block size on RBD / EC vs space amplification
by Gilles Mocellin 03 Feb '21

03 Feb '21

Hello, thank you for your response. Erasure Coding gets better and we really cannot afford the storage overhead of x3 replication. Anyway, as I understand the problem, it is also present with replication, just less amplified (blocks are not divided between OSDs, just replicated fully). Le 2021-02-02 16:50, Steven Pine a écrit : > You are unlikely to avoid the space amplification bug by using larger > block sizes. I honestly do not recommend using an EC pool, it is > generally less performant and EC pools are not as well supported by > the ceph development community. > > On Tue, Feb 2, 2021 at 5:11 AM Gilles Mocellin > <gilles.mocellin(a)nuagelibre.org> wrote: > >> Hello, >> >> As we know, with 64k for bluestore_min_alloc_size_hdd (I'm only >> using >> HDDs), >> in certain conditions, especially with erasure coding, >> there's a leak of space while writing objects smaller than 64k x k >> (EC:k+m). >> >> Every object is divided in k elements, written on different OSD. >> >> My main use case is big (40TB) RBD images mounted as XFS filesystems >> on >> Linux servers, >> exposed to our backup software. >> So, it's mainly big files. >> >> My though, but I'd like some other point of view, is that I could >> deal >> with the amplification by using bigger block sizes on my XFS >> filesystems. >> Instead of reducing bluestore_min_alloc_size_hdd on all OSDs. >> >> What do you think ? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > > Steven Pine > > E steven.pine(a)webair.com | P 516.938.4100 x > > Webair | 501 Franklin Avenue Suite 200, Garden City NY, 11530 > > webair.com [1] > > [2] [3] [4] > > NOTICE: This electronic mail message and all attachments transmitted > with it are intended solely for the use of the addressee and may > contain legally privileged proprietary and confidential information. > If the reader of this message is not the intended recipient, or if you > are an employee or agent responsible for delivering this message to > the intended recipient, you are hereby notified that any > dissemination, distribution, copying, or other use of this message or > its attachments is strictly prohibited. If you have received this > message in error, please notify the sender immediately by replying to > this message and delete it from your computer. > > > > Links: > ------ > [1] http://webair.com > [2] https://www.facebook.com/WebairInc/ > [3] https://twitter.com/WebairInc > [4] https://www.linkedin.com/company/webair

1 0

Module 'cephadm' has failed: 'NoneType' object has no attribute 'split'
by Tony Liu 03 Feb '21

03 Feb '21

Hi, After upgrading from 15.2.5 to 15.2.8, I see this health error. Has anyone seen this? "ceph log last cephadm" doesn't show anything about it. How can I trace it? Thanks! Tony

1 1

no device listed after adding host
by Tony Liu 02 Feb '21

02 Feb '21

Hi, I added a host by "ceph orch host add ceph-osd-5 10.6.10.84 ceph-osd". I can see the host by "ceph orch host ls", but no devices listed by "ceph orch device ls ceph-osd-5". I tried "ceph orch device zap ceph-osd-5 /dev/sdc --force", which works fine. Wondering why no devices listed? What I am missing here? Thanks! Tony

2 5

is unknown pg going to be active after osds are fixed?
by Tony Liu 02 Feb '21

02 Feb '21

Hi, With 3 replicas, a pg hs 3 osds. If all those 3 osds are down, the pg becomes unknow. Is that right? If those 3 osds are replaced and in and on, is that pg going to be eventually back to active? Or anything else has to be done to fix it? Thanks! Tony

3 3

radosgw bucket index issue
by Fox, Kevin M 02 Feb '21

02 Feb '21

We have a fairly old cluster that has over time been upgraded to nautilus. We were digging through some things and found 3 bucket indexes without a corresponding bucket. They should have been deleted but somehow were left behind. When we try and delete the bucket index, it will not allow it as the bucket is not found. The bucket index list command works fine though without the bucket. Is there a way to delete the indexes? Maybe somehow relink the bucket so it can be deleted again? Thanks, Kevin

1 1

Re: `cephadm` not deploying OSDs from a storage spec
by Juan Miguel Olmo Martinez 02 Feb '21

02 Feb '21

Hi Davor, Use "ceph orch ls osd --format yaml" to have more info about the problems deploying the osd service, probably that will give you clues about what is happening. Share the input if you cannot solve the problem:-) The same command can be used for other services like the node-exporter, although in that case I think that the problem was a bug fixed a few days ago. https://github.com/ceph/ceph/pull/38946 The fix was backported to pacific last week. BR -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

Re: no device listed after adding host
by Juan Miguel Olmo Martinez 02 Feb '21

02 Feb '21

Hi Eugen Block <https://lists.ceph.io/hyperkitty/users/d8d92a6469954bcd82ced59fcbf701d7/> useful tips to create OSDs: 1. Check devices availability in your cluster hosts: # ceph orch device ls 2. Devices not available: This usually means that you have created lvs in these devices, (I mean the devices are not cleaned.) A ""cepr orch zap <device>" will fix that. 3. The OSD does not start. Check what is the status with: ceph orch osd ls --format yaml -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

pg repair or pg deep-scrub does not start
by Marcel Kuiper 02 Feb '21

02 Feb '21

Hi I've got an old cluster running ceph 10.2.11 with filestore backend. Last week a PG was reported inconsistent with a scrub error # ceph health detail HEALTH_ERR 1 pgs inconsistent; 1 scrub errors pg 38.20 is active+clean+inconsistent, acting [1778,1640,1379] 1 scrub errors I first tried 'ceph pg repair' but nothing seemed to happen, then # rados list-inconsistent-obj 38.20 --format=json-pretty showed that the problem was on osd 1379. The logs showed that that osd had read errors so I decided to mark that osd out for replacement. Later on removed it from the crush map en deleted the osd. My thoughts were that the missing replica gets backfilled on another osd and everything would be ok again. It got another osd assigned but the health error stayed # ceph health detail HEALTH_ERR 1 pgs inconsistent; 1 scrub errors pg 38.20 is active+clean+inconsistent, acting [1778,1640,1384] 1 scrub errors Now I get an error on: # rados list-inconsistent-obj 38.20 --format=json-pretty No scrub information available for pg 38.20 error 2: (2) No such file or directory And if I try # ceph pg deep-scrub 38.20 instructing pg 38.20 on osd.1778 to deep-scrub The deepscrub does not get scheduled. Same goes for # ceph daemon osd.1778 trigger_scrub 38.20 on the storage node Nothing appears in the logs concerning the scrubbing of PG 38.20. I see in the log that other PG's get (deep) scrubbed according to the automatic scheduling There is no recovery going on but just to be sure I'd set ceph daemon osd.1778 config set osd_scrub_during_recovery true Also the load limit is set way higher then the actual system load I checked the other osds en there are no scrubs going on on these when I schedule the deep-scrub I found some report of people that had the same problem. However no solution was found (for example https://tracker.ceph.com/issues/15781) Even in mimic and luminous there were sort of the same cases - Does anyone know what logging I should incraese in order to get more information as to why my deep-scrub does not get scheduled - Is there a way in jewel to see the list of scheduled scrubs and their dates for an osd - Does someone have advice on how to proceed in clearing this PG error Thanks for any help Marcel

1 0

osd recommended scheduler
by Andrei Mikhailovsky 02 Feb '21

02 Feb '21

Hello everyone, Could some one please let me know what is the recommended modern kernel disk scheduler that should be used for SSD and HDD osds? The information in the manuals is pretty dated and refer to the schedulers which have been deprecated from the recent kernels. Thanks Andrei

4 5

CephFS per client monitoring
by Erwin Bogaard 02 Feb '21

02 Feb '21

Hi, we're using mainly CephFS to give access to storage. At all times we can see that all clients combines use "X MiB/s" and "y op/s" for read and write by using the cli or ceph dashboard. With a tool like iftop, I can get a bit of insight to which clients most data 'flows', but it isn't really precise. is there any way to get a MiB/s and op/s number per CephFS client? Thanks, Erwin

2 1

radosgw process crashes multiple times an hour
by Andrei Mikhailovsky 02 Feb '21

02 Feb '21

Hello, I am experiencing very frequent crashes of the radosgw service. It happens multiple times every hour. As an example, over the last 12 hours we've had 35 crashes. Has anyone experienced similar behaviour of the radosgw octopus release service? More info below: Radosgw service is running on two Ubuntu servers. I have tried upgrading OS on one of the servers to Ubuntu 20.04 with latest updates. The second server is still running Ubuntu 18.04. Both services crash occasionally, but the service which is running on Ubuntu 20.04 crashes far more often it seems. The ceph cluster itself is pretty old and was initially setup around 2013. The cluster was updated pretty regularly with every major release. Currently, I've got Octopus 15.2.8 running on all osd, mon, mgr and radosgw servers. Crash Backtrace: ceph crash info 2021-01-28T11:36:48.912771Z_08f80efd-c0ad-4551-88ce-905ca9cd3aa8 |less { "backtrace": [ "(()+0x46210) [0x7f815a49a210]", "(gsignal()+0xcb) [0x7f815a49a18b]", "(abort()+0x12b) [0x7f815a479859]", "(()+0x9e951) [0x7f8150ee9951]", "(()+0xaa47c) [0x7f8150ef547c]", "(()+0xaa4e7) [0x7f8150ef54e7]", "(()+0xaa799) [0x7f8150ef5799]", "(()+0x344ba) [0x7f815a1404ba]", "(()+0x71e04) [0x7f815a17de04]", "(librados::v14_2_0::IoCtx::nobjects_begin(librados::v14_2_0::ObjectCursor const&, ceph::buffer::v15_2_0::list const&)+0x5d) [0x7f815a18c7bd]", "(RGWSI_RADOS::Pool::List::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWAccessListFilter*)+0x115) [0x7f815b0d9935]", "(RGWSI_SysObj_Core::pool_list_objects_init(rgw_pool const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWSI_SysObj::Pool::ListCtx*)+0x255) [0x7f815abd7035]", "(RGWSI_MetaBackend_SObj::list_init(RGWSI_MetaBackend::Context*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x206) [0x7f815b0ccfe6]", "(RGWMetadataHandler_GenericMetaBE::list_keys_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void**)+0x41) [0x7f815ad23201]", "(RGWMetadataManager::list_keys_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void**)+0x71) [0x7f815ad254d1]", "(AsyncMetadataList::_send_request()+0x9b) [0x7f815b13c70b]", "(RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x25) [0x7f815ae60f25]", "(RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x11) [0x7f815ae69401]", "(ThreadPool::worker(ThreadPool::WorkThread*)+0x5bb) [0x7f81517b072b]", "(ThreadPool::WorkThread::entry()+0x15) [0x7f81517b17f5]", "(()+0x9609) [0x7f815130d609]", "(clone()+0x43) [0x7f815a576293]" ], "ceph_version": "15.2.8", "crash_id": "2021-01-28T11:36:48.912771Z_08f80efd-c0ad-4551-88ce-905ca9cd3aa8", "entity_name": "client.radosgw1.gateway", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "20.04.1 LTS (Focal Fossa)", "os_version_id": "20.04", "process_name": "radosgw", "stack_sig": "347474f09a756104ac2bb99d80e0c1fba3e9dc6f26e4ef68fe55946c103b274a", "timestamp": "2021-01-28T11:36:48.912771Z", "utsname_hostname": "arh-ibstorage1-ib", "utsname_machine": "x86_64", "utsname_release": "5.4.0-64-generic", "utsname_sysname": "Linux", "utsname_version": "#72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021" } radosgw.log file (file names were redacted): -25> 2021-01-28T11:36:48.794+0000 7f8043fff700 1 civetweb: 0x7f814c0cf010: 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-u115134.JPG HTTP/1.1" 400 460 - - -24> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 ====== starting new request req=0x7f80437f5780 ===== -23> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s initializing for trans_id = tx000000000000000001431-006012a1d0-31197b5c-default -22> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s getting op 1 -21> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj verifying requester -20> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj normalizing buckets and tenants -19> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj init permissions -18> 2021-01-28T11:36:48.814+0000 7f80437fe700 0 req 5169 0s NOTICE: invalid dest placement: default-placement/REDUCED_REDUNDANCY -17> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 op->ERRORHANDLER: err_no=-22 new_err_no=-22 -16> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj op status=0 -15> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj http status=400 -14> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 ====== req done req=0x7f80437f5780 op status=0 http_status=400 latency=0s ====== -13> 2021-01-28T11:36:48.822+0000 7f80437fe700 1 civetweb: 0x7f814c0cf9e8: 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-d20201223-u115132.JPG HTTP/1.1" 400 460 - - -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request req=0x7f8043ff6780 ===== -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for trans_id = tx000000000000000001432-006012a1d0-31197b5c-default -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1 -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying requester -8> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj normalizing buckets and tenants -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request req=0x7f8043ff6780 ===== -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for trans_id = tx000000000000000001432-006012a1d0-31197b5c-default -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1 -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying requester -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request req=0x7f8043ff6780 ===== -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for trans_id = tx000000000000000001432-006012a1d0-31197b5c-default -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1 -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying requester -8> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj normalizing buckets and tenants -7> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj init permissions -6> 2021-01-28T11:36:48.878+0000 7f8043fff700 0 req 5170 0s NOTICE: invalid dest placement: default-placement/REDUCED_REDUNDANCY -5> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 op->ERRORHANDLER: err_no=-22 new_err_no=-22 -4> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj op status=0 -3> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj http status=400 -2> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== req done req=0x7f8043ff6780 op status=0 http_status=400 latency=0s ====== -1> 2021-01-28T11:36:48.886+0000 7f8043fff700 1 civetweb: 0x7f814c0cf010: 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-223-u115136.JPG HTTP/1.1" 400 460 - - 0> 2021-01-28T11:36:48.910+0000 7f8128ff9700 -1 *** Caught signal (Aborted) ** 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 30417 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 framework: civetweb 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 framework conf key: port, val: 443s Could someone help me troubleshoot and fix the issue? Thanks Andrei

3 4

Issue with cephadm upgrading containers.
by Darrin Hodges 02 Feb '21

02 Feb '21

Hi all, I'm attempting to upgrade our octopus 15.2.4 containers to 15.2.8. If I run 'ceph orch upgrade start --ceph-version 15.2.8' it eventually errors with: '"message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", The documentation suggests that this is caused by specifying the incorrect version or the registry is not reachable. If I run 'cephadm pull' it doesn't complain: Using recent ceph image ceph/ceph:v15.2.8 Pulling container image ceph/ceph:v15.2.8... { "ceph_version": "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)", "image_id": "5553b0cb212ca2aa220d33ba39d9c602c8412ce6c5febc57ef9cdc9c5844b185", "repo_digest": "ceph/ceph@sha256:37939a3739e4e037dcf1b1f5828058d721d8c6de958212609f9e7d920b9c62bf" } Not sure what the issue is with upgrading. many thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

1 1

`cephadm` not deploying OSDs from a storage spec
by Davor Cubranic 02 Feb '21

02 Feb '21

Hello, I am trying to set up a test cluster with the cephadm tool on Ubuntu 20.04 nodes. Following the directions at https://docs.ceph.com/en/octopus/cephadm/install/, I have set up the monitor and manager on a management node, and added two hosts that I want to use for storage. All storage devices present on those nodes are included in the output of `ceph orch device ls`, and all are marked “available”. However, when I try to deploy OSDs with `ceph orch apply osd -i spec.yml`, following the example for HDD+SSD storage spec at https://docs.ceph.com/en/latest/cephadm/drivegroups/#the-simple-case, I see the new service in the output of `ceph orch ls`, but it is not running anywhere (“0/2”), and no OSDs get created. I am not sure how to debug this, and any pointers would be much appreciated. Thank you, Davor Output: ``` # ceph orch host ls INFO:cephadm:Inferring fsid 150b5f1a-64bf-11eb-a7e9-d96bd5ac4db3 INFO:cephadm:Inferring config /var/lib/ceph/150b5f1a-64bf-11eb-a7e9-d96bd5ac4db3/mon.sps-head/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 HOST ADDR LABELS STATUS sps-head sps-head mon sps-st1 sps-st1 mon sps-st2 sps-st2 # ceph orch device ls INFO:cephadm:Inferring fsid 150b5f1a-64bf-11eb-a7e9-d96bd5ac4db3 INFO:cephadm:Inferring config /var/lib/ceph/150b5f1a-64bf-11eb-a7e9-d96bd5ac4db3/mon.sps-head/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 Hostname Path Type Serial Size Health Ident Fault Available sps-head /dev/nvme0n1 ssd S5JXNS0N504446R 1024G Unknown N/A N/A Yes sps-st1 /dev/nvme0n1 ssd S5JXNS0N504948D 1024G Unknown N/A N/A Yes sps-st1 /dev/nvme1n1 ssd S5JXNS0N504958T 1024G Unknown N/A N/A Yes sps-st1 /dev/sdb hdd 5000cca28ed36018 14.0T Unknown N/A N/A Yes sps-st1 /dev/sdc hdd 5000cca28ed353e5 14.0T Unknown N/A N/A Yes […] # cat /mnt/osd_spec.yml service_type: osd service_id: default_drive_group placement: host_pattern: 'sps-st[1-6]' data_devices: rotational: 1 db_devices: rotational: 0 [**After running `ceph orch apply osd -i spec.yml`:**] # ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID alertmanager 1/1 9m ago 6h count:1 docker.io/prom/alertmanager:v0.20.0 0881eb8f169f crash 3/3 9m ago 6h * docker.io/ceph/ceph:v15 5553b0cb212c grafana 1/1 9m ago 6h count:1 docker.io/ceph/ceph-grafana:6.6.2 a0dce381714a mgr 2/2 9m ago 6h count:2 docker.io/ceph/ceph:v15 5553b0cb212c mon 1/2 9m ago 3h label:mon docker.io/ceph/ceph:v15 5553b0cb212c node-exporter 0/3 - - * <unknown> <unknown> osd.default_drive_group 0/2 - - sps-st[1-6] <unknown> <unknown> prometheus 1/1 9m ago 6h count:1 docker.io/prom/prometheus:v2.18.1 de242295e225 [** I am not sure why neither “osd.default_drive_group” nor “node-exporter” are running anywhere. How do I check that? **] # ceph osd tree INFO:cephadm:Inferring fsid 150b5f1a-64bf-11eb-a7e9-d96bd5ac4db3 INFO:cephadm:Inferring config /var/lib/ceph/150b5f1a-64bf-11eb-a7e9-d96bd5ac4db3/mon.sps-head/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0 root default # ceph orch --version ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable) ```

1 0

Bucket synchronization works only after disable/enable, once finished, some operation maxes out SSDs/nvmes and sync degraded.
by Szabo, Istvan (Agoda) 02 Feb '21

02 Feb '21

Hello, We have a 3 geo locational freshly installed multisite setup with an upgraded octopus from 15.2.5 to 15.2.7. We have 6 osd nodes, 3 mon/mgr/rgw in each dc, full SSD, 3 ssd is using 1 nvme for journaling. Each zone backed with 3 RGW, one on each mon/mgr node. The goal is to replicate 2 (currently) big buckets in the zonegroup but it only works if I disable and reenable the bucket sync. Big buckets means, one bucket is presharded for 9000 shards (9 billions objects), the 2nd bucket that I'm detailing in this ticket 24000 (24 billions objects) shards. Once picked up the objects (not all, only the ones that was on the source site at that given time when it was enabled) it will slows down a lot from 100.000 objects / 15 minutes in and 10GB/15 minutes to 50 objects/4 hours. Once it synchronized after enabled/disabled, it maxing out the osd nodes with NVME/SSD drives with some operation which I don't know what is it. Let me show you the symptoms below. Let me summarize as much as I can. We have 1 realm, in this realm we have 1 zonegroup (please help me to check if the sync policies are ok) and in this zonegroup we have 1 cluster in US, 1 in Hong Kong (master) and 1 in Singapore. Here is the realm, zonegroup and zones definition: https://pastebin.com/raw/pu66tqcf Let me show you one enable/disable operation when I've disabled on the HKG master site the pix-bucket and enabled it. In this screenshot: https://i.ibb.co/WNC0gNQ/6nodes6day.png the highlighted area is when the data sync is running after disable enable. You can see almost no operation. You can see also when sync is not running, the green and yellow is the NVME drive rocksdb+wal drives. The screenshot represents the 6 Singapore nodes SSD/NVME disk utilizations. The first node you can see in the last hours no green and yellow, it's because I've reinstalled in that nodes all the osds to not use NVME. In the following 1st screenshot you can see the HKG object usage where the user is uploading the objects. 2nd screenshot the SGP one where you can see the highlighted area is the disable/enable operation. HKG where user upload: https://i.ibb.co/vj2VFYP/pixhkg6d.png SGP where sync happened: https://i.ibb.co/w41rmQT/pixsgp6d.png Let me show you some troubleshooting logs regarding bucket sync status, cluster sync status, reshard list (which might be because of previous testing), sync error list https://pastebin.com/raw/TdwiZFC1 The issue might be very similar to this issue: https://tracker.ceph.com/issues/21591 Where I should move forward or how can I help you to provide more logs to help me please? Thank you in advance ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

rbd resize progress report?
by Jorge Garcia 01 Feb '21

01 Feb '21

I'm trying to resize a block device using "rbd --resize". The block device is pretty huge (100+ TB). The resize has been running for over a week, and I have no idea if it's actually doing anything, or if it's just hanging or in some infinite loop. Is there any way of getting a progress report from the resize to get an idea if this is ever going to finish? Thanks!

1 0

Unable to enable RBD-Mirror Snapshot on image when VM is using RBD
by Adam Boyhan 01 Feb '21

01 Feb '21

This is a odd one. I don't hit it all the time so I don't think its expected behavior. Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM. root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument

2 4

Balancing with upmap
by Francois Legrand 01 Feb '21

01 Feb '21

Hi all, I have a cluster with 116 disks (24 new disks of 16TB added in december and the rest of 8TB) running nautilus 14.2.16. I moved (8 month ago) from crush_compat to upmap balancing. But the cluster seems not well balanced, with a number of pgs on the 8TB disks varying from 26 to 52 ! And an occupation from 35 to 69%. The recent 16 TB disks are more homogeneous with 48 to 61 pgs and space between 30 and 43%. Last week, I realized that some osd were maybe not using upmap because I did a ceph osd crush weight-set ls and got (compat) as result. Thus I ran a ceph osd crush weight-set rm-compat which triggered some rebalancing. Now there is no more recovery for 2 days, but the cluster is still unbalanced. As far as I understand, upmap is supposed to reach an equal number of pgs on all the disks (I guess weighted by their capacity). Thus I would expect more or less 30 pgs on the 8TB disks and 60 on the 16TB and around 50% usage on all. Which is not the case (by far). The problem is that it impact the free available space in the pools (264Ti while there is more than 578Ti free in the cluster) because free space seems to be based on space available before the first osd will be full ! Is it normal ? Did I missed something ? What could I do ? F.

3 11

Multisite recovering shards
by Szabo, Istvan (Agoda) 01 Feb '21

01 Feb '21

Hi, I’ve never seen in our multisite sync status healthy output, almost all the sync shards are recovering. What can I do with recovering shards? We have 1 realm, 1 zonegroup and inside the zonegroup we have 3 zones in 3 different geo location. We are using octopus 15.2.7 for bucket sync with symmetrical replication. The user is at the moment migrating their data and the sites are always behind which is replicated from the place where it was uploaded. I’ve restarted all rgw and disable / enable bucket sync, it started to work, but I think when it comes to close sync it will stop again due to the recovering shards. Any idea? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4