- ceph-users - lists.ceph.io

warn if acting set violates failure domain
by Dan van der Ster 30 May '20

30 May '20

Hi, (In nautilus) I noticed that the acting set for a pg in acting+remapped+backfilling (or backfill_wait) can violate the failure domain rule. We have a 3x pool replicated across racks. Following some host outage I noticed several PGs like: 75.200 7570 0 7570 0 31583745536 0 0 3058 active+remapped+backfill_wait 55m 3208334'13480873 3208334:23611310 [596,502,717]p596 [596,502,424]p596 2020-05-28 05:44:42.604307 2020-05-28 05:44:42.604307 Checking those up and acting sets, I have: OK: PG 75.200 has no failure domain problem in up set [596, 502, 717] with racks ['BA09', 'BA10', 'BA12'] WARN: PG 75.200 has a failure domain problem in acting set [596, 502, 424] with racks ['BA09', 'BA10', 'BA09'] If BA09 were to go down, the pg would drop below min_size. (The full pg query output is at https://pastebin.com/rrEQMnyC) Is this https://tracker.ceph.com/issues/3360 ? # ceph osd dump | grep 75.200 pg_temp 75.200 [596,502,424] Shouldn't we mark a PG degraded or similar to raise HEALTH_WARN if the failure domain is violated? Cheers, Dan

1 0

v15.2.3 Octopus released
by Josh Durgin 30 May '20

30 May '20

We’re happy to announce the availability of the third Octopus stable release series. This release mainly is a workaround for a potential OSD corruption in v15.2.2. We advise users to upgrade to v15.2.3 directly. For users running v15.2.2 please execute the following:: ceph config set osd bluefs_preextend_wal_files false Changelog ~~~~~~~~~ * bluestore: common/options.cc: disable bluefs_preextend_wal_files Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: d289bbdec69ed7c1f516e0a093594580a76b78d0

1 0

OSD backups and recovery
by Ludek Navratil 30 May '20

30 May '20

HI all, what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local filesystem and backup the content using Amanda. I believe that there must a better way to do this but I couldn't find it in docs. I know that one option is to setup an archive zone, but it requires an additional ceph cluster that needs to be maintained and looked after. I would rather avoid that. How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me to recover the data correctly? Many thanks,Ludek

4 5

rocksdb tuning
by Frank R 30 May '20

30 May '20

Hi all, I am attempting to prevent bluestore rocksdb Level 3/4 spillover with a 150GB logical volume for the db/wal. I am thinking of setting max_bytes_for_level_base to about 1.3G (1342177280). This should let Level 3 fill up the 150GB logical volume. I don't expect to ever actually need L4. Another option would be to set max_bytes_for_level_base to 128m. This db/wal device is a fast SSD. OSD is HDD. Any opinions on whether it would be better to increase max_bytes_for_level_base to over 1G or decrease it to 128m? thanks, Frank

1 0

General question CephFS or RBD
by Willi Schiegel 30 May '20

30 May '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

2 1

Ceph and iSCSI
by Bobby 30 May '20

30 May '20

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

2 1

Is 2 osds per disk, encryption possible with cephadm on 15.2.2?
by Marco Pizzolo 29 May '20

29 May '20

Hello Everyone, I'm still having issues getting the OSDs to properly create on a brand new Ceph 15.2.2 cluster. I don't see to be able to have OSDs created based on service definition of 2 osds per disk and encryption. It seems to hang and/or I see "No Deployments..." Has anyone had luck with this at this point? If no, how are you doing it? Thanks, Marco

1 0

CEPH failure domain - power considerations
by Phil Regnauld 29 May '20

29 May '20

Hi, in our production cluster, we have the following setup - 10 nodes - 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe - dual 10G in LACP, linked to two different switches (Cisco vPC) - OSDs, MONs and MGRs are colocated - A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2 - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2) - switches have dual PSUs and are connected to both rails - CEPH nodes - single power supply - Odd nodes (1,3,5...) are connected to PDU1 - Even nodes (2,4,6...) are connected to PDU2 ... I can provide a drawing if it helps :) Now, the default crush map ensures that multiple copies of the same object won't find their way on the same host, which is fine. But I'm thinking that in case of power failure [1] of either ATS or PDU, we'd be losing half the nodes in the cluster at the same time. How would I go about tuning our map so it took into account that, for a 3 copy replicated pool, we don't have those stored on hosts, say, 5,7,9 ? And, what about when using EC pools ? We currently have 5+2 SSD pools - how would we avoid losing availability in case of a power loss where 50% of the server are offline ? I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/ but don't believe I'm at the stage where I dare make changes without incurring a huge data migration (probably can't be avoided). Any input appreciated. Cheers, Phil [1] both power feeds lost at the same time is really hard to protect against :)

8 10

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 29 May '20

29 May '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 2

Ceph and Windows - experiences or suggestions
by Lars Täuber 29 May '20

29 May '20

Hi there! I got the task to connect a Windows client to our existing ceph cluster. I'm looking for experiences or suggestions from the community. There came two possibilities to my mind: 1. iSCSI Target on RBD exported to Windows 2. NFS-Ganesha on CephFS exported to Windows Is there a third way exporting a ceph cluster to a windows machine? I have some experiences with CephFS. We have a small cluster successfully running for linux clients. I don't have experiences with RBD or iSCSI. The Windows machine will use the space for backups. The kind of data is unknown. I expect the data to be a MS-SQL dump and user data from a sharepoint system. The Windows admin does not care whether NFS or iSCSI is used. I'd be happy if some of you could share experiences. Thanks Lars

10 10