- ceph-users - lists.ceph.io

Ceph Ansible fails on check if monitor initial keyring already exists
by Jared Jacob 09 Jun '21

09 Jun '21

I am running the Ceph ansible script to install ceph version Stable-6.0 (Pacific). When running the sample yml file that was supplied by the github repo it runs fine up until the "ceph-mon : check if monitor initial keyring already exists" step. There it will hang for 30-40 minutes before failing. From my understanding ceph ansible should be creating this keyring and using it for communication between monitors, so does anyone know why the playbook would have a hard time with this step? Thanks in advance!

1 0

DocuBetter Meeting -- 09 June 2021 1730 UTC
by John Zachary Dover 08 Jun '21

08 Jun '21

A DocuBetter Meeting will be held on 09 June 2021 at 1730 UTC. This is the monthly DocuBetter Meeting that is more convenient for European and North American Ceph contributors than the other meeting, which is convenient for people in Australia and Asia (and which is very rarely attended). Topics: - cephadm docs rewrite (ongoing) - ceph.io copy rewrite and information architecture restructure (ongoing) - rgw manual install procedure (prospective) - new ceph-docs mailing list Bring your docs complaints and requests to this meeting. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

Announcing go-ceph v0.10.0
by John Mulligan 08 Jun '21

08 Jun '21

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.10.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. In addition to our regular release this week, we're also participating in this June's "Ceph Month" event with the "go-ceph get together" Birds-of-a-Feather session on Thursday June 10th at 10:10 Eastern time. It should be visible in the Ceph Community calendar [1]. If you can't make the BoF, questions, comments, bugs etc are best directed at our github issues tracker or github discussions forum. [1] - https://ceph.io/contribute/#community-calendar -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

ceph buckets
by Rok Jaklič 08 Jun '21

08 Jun '21

Hi, I try to create buckets through rgw in following order: - *bucket1* with *user1* with *access_key1* and *secret_key1* - *bucket1* with *user2* with *access_key2* and *secret_key2* when I try to create a second bucket1 with user2 I get *Error response code BucketAlreadyExists.* Why? Should not buckets relate only to users? Is this by design and is there any particular reason that it follows this concept? Regards, Rok

3 4

Bucket creation on RGW Multisite env.
by mhnx 08 Jun '21

08 Jun '21

Hello. I have a multisite RGW environment. When I create a new bucket, the bucket is directly created on master and secondary. If I don't want to sync a bucket, I need to stop sync after creation. Is there any global option as "Do not sync directly, only start if I want to" ?

3 6

Index pool hasn't been cleaned up and caused large omap, safe to delete the index file?
by Szabo, Istvan (Agoda) 08 Jun '21

08 Jun '21

Hi, In my multisite setup 1 big bucket has been deleted and seems like hasn't been cleaned up on one of the secondary site. Is it safe to delete the 11 shard objects from the index pool which holding the omaps of that bucket files? Also a quick question, is it a problem if we use like this? Create a bucket which means create in all dc Don't create any bucket sync the user upload different files in different dcs. When a bucket deletion happens would this usage behavior cause issue that different files are in the buckets? If yes, how to prevent this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

Re: OT: How to Build a poor man's storage with ceph
by Sebastian Knust 08 Jun '21

08 Jun '21

Hi Michael, On 08.06.21 11:38, Ml Ml wrote: > Now i was asked if i could also build a cheap 200-500TB Cluster > Storage, which should also scale. Just for Data Storage such as > NextCloud/OwnCloud. With similar requirements (server primarily for Samba and NextCloud, some RBD use, very limited budget) I am using HDD for data and SSD for system and CephFS metadata. Note that I am running NextCloud on CephFS storage. If you want to go with RGW/S3 as a storage backend instead, the following might not apply to your use case. My nodes (bought end of 2020) are: - 2U chassis with 12 3.5" SATA slots - Intel Xeon Silver 4208 - 128 GB RAM - 2 x 480 GB Samsung PM883 SSD -> 50 GB in MD-RAID1 for system -> 430 GB OSD (one per SSD) - initially 6 x 14 TB Enterprise HDD - 4 x 10 GBase-T (active/passive bonded, dedicated backend network) Each node with this configuration cost about 4k EUR net at the end of 2020. Due to increasing prices for storage, it will be a bit more expensive now. I am running five nodes now and have added a few more disks (ranging 8-14 TB), nearly filling up the nodes. My experience so far: - I had to throttle scrubbing (see below for details) - For purely NextCloud and Samba performance is sufficient for a few hundred concurrent users with a handful of power users - Migration of the mail server to this cluster was a disaster due to limited IOPS, had to add some more SSDs and place the mail server in an SSD-only pool. - MDS needs a lot of memory for larger CephFS installs, I will move it to a dedicated server probably next year. 128 GB per node works but I would not recommend any less. - Rebalancing takes an eternity (2-3 weeks), so make sure that your PG nums are okay from the start - I have all but given up on snapshots with CephFS due to severe performance degradation with kernel client during backup My scrubbing config looks like this: osd_backfill_scan_max 16 osd_backfill_scan_min 4 osd_deep_scrub_interval 2592000.000000 osd_deep_scrub_randomize_ratio 0.030000 osd_recovery_max_active_hdd 1 osd_recovery_max_active_ssd 5 osd_recovery_sleep_hdd 0.050000 osd_scrub_begin_hour 18 osd_scrub_end_hour 7 osd_scrub_chunk_max 1 osd_scrub_chunk_min 1 osd_scrub_max_interval 2419200.000000 osd_scrub_min_interval 172800.000000 osd_scrub_sleep 0.100000 My data is in a replicated pool with n=3 without compression. You might also consider EC and then want to aim for more nodes. Cheers Sebastian

1 0

OT: How to Build a poor man's storage with ceph
by Ml Ml 08 Jun '21

08 Jun '21

Hello List, i used to build 3 Node Cluster with spinning Rust and later with (Enterprise) SSDs. All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks and i was done. The Requirements were just 10/15TB Disk usage (30-45TB Raw). Now i was asked if i could also build a cheap 200-500TB Cluster Storage, which should also scale. Just for Data Storage such as NextCloud/OwnCloud. Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x 45k EUR = 135k EUR. Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD) How do the "big boys" do this? Just throw money at it? Would a mix of OSD SSD Metadata + Spinning Rust do the job? My experience so far is that each time i had a crash/problem it was always such a pain to wait for the spinning rust. Do you have any experience/hints on this? Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which would speed up the performance, but have a bigger impact on a dying disk. My requirements are more or less low IO Traffic but loads of disk space usage. Any hints/ideas/links are welcome. Cheers, Michael

4 3

How to enable lazyio under kcephfs?
by opengers 08 Jun '21

08 Jun '21

ceph: 14.2.x kernel: 4.15 In cephfs, due to the need for cache consistency, When a client is executing buffer IO, another client will hang when reading and writing the same file It seems that lazyio can solve this problem, lazyio allows multiple clients to execute buffer IO at the same time(relax consistency), But I am not sure how to enable lazyio under the kernel mount, the test found that the "client_force_lazyio" parameter does not work My final requirement is to use lazyio to implement multiple clients to read and write the same file(buffer IO mode) Can someone explain how to enable lazyio under kcephfs, thanks

2 1

Only 2/5 mon services running
by Jeremy Hansen 08 Jun '21

08 Jun '21

In an attempt to troubleshoot why only 2/5 mon services were running, I believe I’ve broke something: [ceph: root@cn01 /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager 1/1 81s ago 9d count:1 crash 6/6 7m ago 9d * grafana 1/1 80s ago 9d count:1 mds.testfs 2/2 81s ago 9d cn01.ceph.la1.clx.corp;cn02.ceph.la1.clx.corp;cn03.ceph.la1.clx.corp;cn04.ceph.la1.clx.corp;cn05.ceph.la1.clx.corp;cn06.ceph.la1.clx.corp;count:2 mgr 2/2 81s ago 9d count:2 mon 2/5 81s ago 9d count:5 node-exporter 6/6 7m ago 9d * osd.all-available-devices 20/26 7m ago 9d * osd.unmanaged 7/7 7m ago - <unmanaged> prometheus 2/2 80s ago 9d count:2 I tried to stop and start the mon service, but now the cluster is pretty much unresponsive, I’m assuming because I stopped mon: [ceph: root@cn01 /]# ceph orch stop mon Scheduled to stop mon.cn01 on host 'cn01.ceph.la1.clx.corp' Scheduled to stop mon.cn02 on host 'cn02.ceph.la1.clx.corp' Scheduled to stop mon.cn03 on host 'cn03.ceph.la1.clx.corp' Scheduled to stop mon.cn04 on host 'cn04.ceph.la1.clx.corp' Scheduled to stop mon.cn05 on host 'cn05.ceph.la1.clx.corp' [ceph: root@cn01 /]# ceph orch start mon ^CCluster connection aborted Now even after a reboot of the cluster, it’s unresponsive. How do I get mon started again? I’m going through Ceph and breaking things left and right, so I apologize for all the questions. I learn best from breaking things and figuring out how to resolve the issues. Thank you -jeremy

1 1