May 2020 - ceph-users - lists.ceph.io

by Josh Durgin

We’re happy to announce the availability of the third Octopus stable release series. This release mainly is a workaround for a potential OSD corruption in v15.2.2. We advise users to upgrade to v15.2.3 directly. For users running v15.2.2 please execute the following:: ceph config set osd bluefs_preextend_wal_files false Changelog ~~~~~~~~~ * bluestore: common/options.cc: disable bluefs_preextend_wal_files Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: d289bbdec69ed7c1f516e0a093594580a76b78d0

3 years, 11 months

1
0
0 0

OSD backups and recovery

by Ludek Navratil

HI all, what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local filesystem and backup the content using Amanda. I believe that there must a better way to do this but I couldn't find it in docs. I know that one option is to setup an archive zone, but it requires an additional ceph cluster that needs to be maintained and looked after. I would rather avoid that. How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me to recover the data correctly? Many thanks,Ludek

3 years, 11 months

4
5
0 0

rocksdb tuning

by Frank R

Hi all, I am attempting to prevent bluestore rocksdb Level 3/4 spillover with a 150GB logical volume for the db/wal. I am thinking of setting max_bytes_for_level_base to about 1.3G (1342177280). This should let Level 3 fill up the 150GB logical volume. I don't expect to ever actually need L4. Another option would be to set max_bytes_for_level_base to 128m. This db/wal device is a fast SSD. OSD is HDD. Any opinions on whether it would be better to increase max_bytes_for_level_base to over 1G or decrease it to 128m? thanks, Frank

3 years, 11 months

1
0
0 0

General question CephFS or RBD

by Willi Schiegel

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

3 years, 11 months

2
1
0 0

Ceph and iSCSI

by Bobby

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

3 years, 11 months

2
1
0 0

Is 2 osds per disk, encryption possible with cephadm on 15.2.2?

by Marco Pizzolo

Hello Everyone, I'm still having issues getting the OSDs to properly create on a brand new Ceph 15.2.2 cluster. I don't see to be able to have OSDs created based on service definition of 2 osds per disk and encryption. It seems to hang and/or I see "No Deployments..." Has anyone had luck with this at this point? If no, how are you doing it? Thanks, Marco

3 years, 11 months

1
0
0 0

CEPH failure domain - power considerations

by Phil Regnauld

Hi, in our production cluster, we have the following setup - 10 nodes - 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe - dual 10G in LACP, linked to two different switches (Cisco vPC) - OSDs, MONs and MGRs are colocated - A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2 - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2) - switches have dual PSUs and are connected to both rails - CEPH nodes - single power supply - Odd nodes (1,3,5...) are connected to PDU1 - Even nodes (2,4,6...) are connected to PDU2 ... I can provide a drawing if it helps :) Now, the default crush map ensures that multiple copies of the same object won't find their way on the same host, which is fine. But I'm thinking that in case of power failure [1] of either ATS or PDU, we'd be losing half the nodes in the cluster at the same time. How would I go about tuning our map so it took into account that, for a 3 copy replicated pool, we don't have those stored on hosts, say, 5,7,9 ? And, what about when using EC pools ? We currently have 5+2 SSD pools - how would we avoid losing availability in case of a power loss where 50% of the server are offline ? I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/ but don't believe I'm at the stage where I dare make changes without incurring a huge data migration (probably can't be avoided). Any input appreciated. Cheers, Phil [1] both power feeds lost at the same time is really hard to protect against :)

3 years, 11 months

8
10
0 0

Very bad performance on a ceph rbd pool via iSCSI to VMware esx

by Salsa

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa Sent with [ProtonMail](https://protonmail.com) Secure Email.

3 years, 11 months

2
2
0 0

Ceph and Windows - experiences or suggestions

by Lars Täuber

Hi there! I got the task to connect a Windows client to our existing ceph cluster. I'm looking for experiences or suggestions from the community. There came two possibilities to my mind: 1. iSCSI Target on RBD exported to Windows 2. NFS-Ganesha on CephFS exported to Windows Is there a third way exporting a ceph cluster to a windows machine? I have some experiences with CephFS. We have a small cluster successfully running for linux clients. I don't have experiences with RBD or iSCSI. The Windows machine will use the space for backups. The kind of data is unknown. I expect the data to be a MS-SQL dump and user data from a sharepoint system. The Windows admin does not care whether NFS or iSCSI is used. I'd be happy if some of you could share experiences. Thanks Lars

3 years, 11 months

10
10
0 0

Re: RBD Mirroring down+unknown

by Jason Dillaman

On Fri, May 29, 2020 at 12:09 PM Miguel Castillo <Miguel.Castillo(a)centro.net> wrote: > Happy New Year Ceph Community! > > I'm in the process of figuring out RBD mirroring with Ceph and having a > really tough time with it. I'm trying to set up just one way mirroring > right now on some test systems (baremetal servers, all Debian 9). The first > cluster is 3 nodes, and the 2nd cluster is 2 nodes (not worried about a > properly performing setup, just the functionality of RBD mirroring right > now). The purpose is to have a passive failover ceph cluster in a separate > DC. Mirroring seems like the best solution, but if we can't get it working, > we'll end up resorting to a scheduled rsync which is less than ideal. I've > followed several guides, read through a lot of documentation, and nothing > has worked for me thus far. If anyone can offer some troubleshooting help > or insight into what I might have missed in this setup, I'd greatly > appreciate it! I also don't fully understand the relationship between > images and pools and how you're supposed to configure statically sized > images for a pool that has a variable amount of dat > a, but that's a question for afterwards, I think :) > > Once RBD mirroring is set up, the mirror test image status shows as > down+unknown: > > On ceph1-dc2: > rbd --cluster dc1ceph mirror pool status fs_data --verbose > health: WARNING > images: 1 total > 1 unknown > > mirror_test: > global_id: c335017c-9b8f-49ee-9bc1-888789537c47 > state: down+unknown > description: status not found > last_update: > > Here are the commands I run using ceph-deploy on both clusters to get > everything up and running (run from a deploy directory on the first node of > each cluster). The clusters are created at the same time, and rbd setup > commands are only run after the clusters are up and healthy, and the > fs_data pool is created. > > ----------------------------------------------------------- > > Cluster 1 (dc1ceph): > > ceph-deploy new ceph1-dc1 ceph2-dc1 ceph3-dc1 > sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf > ceph-deploy install ceph1-dc1 ceph2-dc1 ceph3-dc1 --release luminous > ceph-deploy mon create-initial > ceph-deploy admin ceph1-dc1 ceph2-dc1 ceph3-dc1 > ceph-deploy mgr create ceph1-dc1 ceph2-dc1 ceph3-dc1 > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph1-dc1 ; done > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph2-dc1 ; done > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph3-dc1 ; done > ceph-deploy mds create ceph1-dc1 ceph2-dc1 ceph3-dc1 > ceph-deploy rgw create ceph1-dc1 ceph2-dc1 ceph3-dc1 > for f in 1 2 ; do scp ceph.client.admin.keyring > ceph$f-dc2:/etc/ceph/dc1ceph.client.admin.keyring ; done > for f in 1 2 ; do scp ceph.conf ceph$f-dc2:/etc/ceph/dc1ceph.conf ; done > for f in 1 2 ; do ssh ceph$f-dc2 "chown ceph.ceph /etc/ceph/dc1ceph*" ; > done > ceph osd pool create fs_data 512 512 replicated > rbd --cluster ceph mirror pool enable fs_data image > rbd --cluster dc2ceph mirror pool enable fs_data image > rbd --cluster ceph mirror pool peer add fs_data client.admin@dc2ceph > (generated id: b5e347b3-0515-4142-bc49-921a07636865) > rbd create fs_data/mirror_test --size=1G > rbd feature enable fs_data/mirror_test journaling > rbd mirror image enable fs_data/mirror_test > chown ceph.ceph ceph.client.admin.keyring > > Cluster 2 (dc2ceph): > > ceph-deploy new ceph1-dc2 ceph2-dc2 > sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf > ceph-deploy install ceph1-dc2 ceph2-dc2 --release luminous > ceph-deploy mon create-initial > ceph-deploy admin ceph1-dc2 ceph2-dc2 > ceph-deploy mgr create ceph1-dc2 ceph2-dc2 > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph1-dc2 ; done > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph2-dc2 ; done > ceph-deploy mds create ceph1-dc2 ceph2-dc2 > ceph-deploy rgw create ceph1-dc2 ceph2-dc2 > apt install rbd-mirror > for f in 1 2 3 ; do scp ceph.conf ceph$f-dc1:/etc/ceph/dc2ceph.conf ; done > for f in 1 2 3 ; do scp ceph.client.admin.keyring > ceph$f-dc1:/etc/ceph/dc2ceph.client.admin.keyring ; done > for f in 1 2 3 ; do ssh ceph$f-dc1 "chown ceph.ceph /etc/ceph/dc2ceph*" ; > done > ceph osd pool create fs_data 512 512 replicated > rbd --cluster ceph mirror pool peer add fs_data client.admin@dc1ceph > (generated id: e486c401-e24d-49bc-9800-759760822282) > systemctl enable ceph-rbd-mirror@admin > systemctl start ceph-rbd-mirror@admin > rbd --cluster dc1ceph mirror pool status fs_data --verbose > > > Cluster 1: > > ls /etc/ceph: > ceph.client.admin.keyring > ceph.conf > dc2ceph.client.admin.keyring > dc2ceph.conf > rbdmap > tmpG36OYs > > cat /etc/ceph/ceph.conf: > [global] > fsid = 8fede407-50e1-4487-8356-3dc98b30c500 > mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 > mon_host = *.*.*.1,*.*.*.27,*.*.*.41 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > cat /etc/ceph/dc2ceph.conf > [global] > fsid = 813ff410-02dc-47bd-b678-38add38495bb > mon_initial_members = ceph1-dc2, ceph2-dc2 > mon_host = *.*.*.56,*.*.*.0 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > > Cluster 2: > > ls /etc/ceph: > ceph.client.admin.keyring > ceph.conf > dc1ceph.client.admin.keyring > dc1ceph.conf > rbdmap > tmp_yxkPs > > cat /etc/ceph/ceph.conf > [global] > fsid = 813ff410-02dc-47bd-b678-38add38495bb > mon_initial_members = ceph1-dc2, ceph2-dc2 > mon_host = *.*.*.56,*.*.*.70 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > cat /etc/ceph/dc1ceph.conf > [global] > fsid = 8fede407-50e1-4487-8356-3dc98b30c500 > mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 > mon_host = *.*.*.1,*.*.*.27,*.*.*.41 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > > RBD Mirror daemon status: > > ceph-rbd-mirror(a)admin.service - Ceph rbd mirror daemon > Loaded: loaded (/lib/systemd/system/ceph-rbd-mirror@.service; enabled; > vendor preset: enabled) > Active: inactive (dead) since Mon 2020-01-06 16:21:44 EST; 3s ago > Process: 910178 ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} > --id admin --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS) > Main PID: 910178 (code=exited, status=0/SUCCESS) > > Jan 06 16:21:44 ceph1-dc2 systemd[1]: Started Ceph rbd mirror daemon. > Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462916 > 7f76ecf88780 -1 auth: unable to find a keyring on > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory > Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462949 > 7f76ecf88780 -1 monclient: ERROR: missing keyring, cannot use cephx for > authentication > Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: failed to initialize: (2) No > such file or directory2020-01-06 16:21:44.463874 7f76ecf88780 -1 > rbd::mirror::Mirror: 0x558d3ce6ce20 init: error connecting to local cluster > It seems like it's saying that "rbd-mirror" cannot access your keyring for DC2. Does the "ceph" user have read permission to the keyring? Can you run "sudo -u ceph ceph health" successfully? > > ------------------------------------------- > > I also tried running the ExecStart command manually, substituting in > different values for the parameters, and just never got it to work. If more > info is needed, please don't hesitate to ask. Thanks in advance! > > -Miguel > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Jason

3 years, 11 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users May 2020