- ceph-users - lists.ceph.io

Can I fix the devive name,when using image map?
by sz_cuitao＠163.com 06 Apr '20

06 Apr '20

Hi: I use image map to mount a ceph device to localhost,but I found that,the device name I can not control. This is a problem,when it's name changed,the FS on it or the database using this device may get wrongs. Can I control the device name? For exapme: [root@gate2 ~]# rbd showmapped id pool namespace image snap device 0 testpool test_img - /dev/rbd0 Can I map the device like this,just directly give the deveice name?like this: rbd map testpool/test_img /dev/rbd0 sz_cuitao(a)163.com

2 2

Removing OSDs in Mimic
by ADRIAN NICOLAE 06 Apr '20

06 Apr '20

Hi all, I have a Ceph cluster with ~ 70 OSDs of different sizes running on Mimic . I'm using ceph-deploy for managing the cluster size. I have to remove some smaller drives and replace them with bigger drives. From your experience, are the removing an OSD guidelines from Mimic docs accurate ? I know that there were some changes from the older versions and I want to avoid any confusions. I'm talking about the following procedure : - take the OSD out of the cluster with "ceph osd out osd_no" - stopping the osd daemon if it's still running - purging the OSD from the cluster map running from the ceph-deploy host (or one of the mons ?) : ceph osd purge {id} --yes-i-really-mean-it I don't have any specific entries for these OSDs in ceph.conf so I guess I shouldn't change any conf file. Thank you.

1 0

Radosgw WAF
by m.kefayati＠afranet.com 06 Apr '20

06 Apr '20

Hi we deploy ceph object storage and want secure RGW. Is there any solution or any user experience about it? Is it common to use WAF ? tnx

2 1

v14.2.8 Nautilus released
by Abhishek Lekshmanan 06 Apr '20

06 Apr '20

This is the eighth update to the Ceph Nautilus release series. This release fixes issues across a range of subsystems. We recommend that all users upgrade to this release. Please note the following important changes in this release; as always the full changelog is posted at: https://ceph.io/releases/v14-2-8-nautilus-released Notable Changes --------------- * The default value of `bluestore_min_alloc_size_ssd` has been changed to 4K to improve performance across all workloads. * The following OSD memory config options related to bluestore cache autotuning can now be configured during runtime: - osd_memory_base (default: 768 MB) - osd_memory_cache_min (default: 128 MB) - osd_memory_expected_fragmentation (default: 0.15) - osd_memory_target (default: 4 GB) The above options can be set with:: ceph config set osd <option> <value> * The MGR now accepts `profile rbd` and `profile rbd-read-only` user caps. These caps can be used to provide users access to MGR-based RBD functionality such as `rbd perf image iostat` an `rbd perf image iotop`. * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap balancing has been removed. Instead use the mgr balancer config `upmap_max_deviation` which now is an integer number of PGs of deviation from the target PGs per OSD. This can be set with a command like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default `upmap_max_deviation` is 1. There are situations where crush rules would not allow a pool to ever have completely balanced PGs. For example, if crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of the racks. In those cases, the configuration value can be increased. * RGW: a mismatch between the bucket notification documentation and the actual message format was fixed. This means that any endpoints receiving bucket notification, will now receive the same notifications inside a JSON array named 'Records'. Note that this does not affect pulling bucket notification from a subscription in a 'pubsub' zone, as these are already wrapped inside that array. * CephFS: multiple active MDS forward scrub is now rejected. Scrub currently only is permitted on a file system with a single rank. Reduce the ranks to one via `ceph fs set <fs_name> max_mds 1`. * Ceph now refuses to create a file system with a default EC data pool. For further explanation, see: https://docs.ceph.com/docs/nautilus/cephfs/createfs/#creating-pools * Ceph will now issue a health warning if a RADOS pool has a `pg_num` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.8.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 2d095e947a02261ce61424021bb43bd3022d35cb -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer HRB 21284 (AG Nürnberg)

6 7

Cann't create ceph cluster
by sz_cuitao＠163.com 06 Apr '20

06 Apr '20

hi! why can not create custer using ceph-deploy? Please give me some advice,thanks! [root@cadmin ~]# cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [root@cadmin ~]# [root@cadmin ~]# python --version Python 2.7.5 [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# su - cephuser Last login: Sat Apr 4 23:44:23 CST 2020 on pts/0 [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ cd cluster/ [cephuser@cadmin cluster]$ [cephuser@cadmin cluster]$ ceph-deploy new cadmin [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephuser/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.25): /bin/ceph-deploy new cadmin [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph_deploy][ERROR ] Traceback (most recent call last): [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/util/decorators.py", line 69, in newfunc [ceph_deploy][ERROR ] return f(*a, **kw) [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/cli.py", line 162, in _main [ceph_deploy][ERROR ] return args.func(args) [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/new.py", line 141, in new [ceph_deploy][ERROR ] ssh_copy_keys(host, args.username) [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/new.py", line 35, in ssh_copy_keys [ceph_deploy][ERROR ] if ssh.can_connect_passwordless(hostname): [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/util/ssh.py", line 15, in can_connect_passwordless [ceph_deploy][ERROR ] if not remoto.connection.needs_ssh(hostname): [ceph_deploy][ERROR ] AttributeError: 'module' object has no attribute 'needs_ssh' [ceph_deploy][ERROR ] sz_cuitao(a)163.com

3 3

Re: Questions on Ceph cluster without OS disks
by Martin Verges 05 Apr '20

05 Apr '20

Hello Samuel, we from croit.io don't use NFS to boot up Servers. We copy the OS directly into the RAM (approximately 0.5-1GB). Think of it like a container, you start it and throw it away when you no longer need it. This way we can save the slots of OS harddisks to add more storage per node and reduce overall costs as 1GB ram is cheaper then an OS disk and consumes less power. If our management node is down, nothing will happen to the cluster. No impact, no downtime. However, you do need the mgmt node to boot up the cluster. So after a very rare total power outage, your first system would be the mgmt node and then the cluster itself. But again, if you configure your systems correct, no manual work is required to recover from that. For everything else, it is possible (but definitely not needed) to deploy our mgmt node in active/passive HA. We have multiple hundred installations worldwide in production environments. Our strong PXE knowledge comes from more than 20 years of datacenter hosting experience and it never ever failed us in the last >10 years. The main benefits out of that: - Immutable OS freshly booted: Every host has exactly the same version, same library, kernel, Ceph versions,... - OS is heavily tested by us: Every croit deployment has exactly the same image. We can find errors much faster and hit much fewer errors. - Easy Update: Updating OS, Ceph or anything else is just a node reboot. No cluster downtime, No service Impact, full automatic handling by our mgmt Software. - No need to install OS: No maintenance costs, no labor required, no other OS management required. - Centralized Logs/Stats: As it is booted in memory, all logs and statistics are collected on a central place for easy access. - Easy to scale: It doesn't matter if you boot 3 oder 300 nodes, all boot the exact same image in a few seconds. .. lots more Please do not hesitate to contact us directly. We always try to offer an excellent service and are strongly customer oriented. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 21. März 2020 um 13:53 Uhr schrieb huxiaoyu(a)horebdata.cn < huxiaoyu(a)horebdata.cn>: > Hello， Martin， > > I notice that Croit advocate the use of ceph cluster without OS disks, but > with PXE boot. > > Do you use a NFS server to serve the root file system for each node? such > as hosting configuration files, user and password, log files, etc. My > question is, will the NFS server be a single point of failure? If the NFS > server goes down, the network experience any outage, ceph nodes may not be > able to write to the local file systems, possibly leading to service outage. > > How do you deal with the above potential issues in production? I am a bit > worried... > > best regards, > > samuel > > > > > ------------------------------ > huxiaoyu(a)horebdata.cn > > >

6 14

Is there a better way to make a samba/nfs gateway?
by Seth Galitzer 04 Apr '20

04 Apr '20

I have a hybrid environment and need to share with both Linux and Windows clients. For my previous iterations of file storage, I exported nfs and samba shares directly from my monolithic file server. All Linux clients used nfs and all Windows clients used samba. Now that I've switched to ceph, things are a bit more complicated. I built a gateway to export nfs and samba as needed, and connect that as a client to my ceph cluster. After having file locking problems with kernel nfs, I made the switch to nfs-ganesha, which has helped immensely. For Linux clients that have high I/O needs, like desktops and some web servers, I connect to ceph directly for those shares. For all other Linux needs, I use nfs from the gateway. For all Windows clients (desktops and a small number of servers), I use samba exported from the gateway. Since my ceph cluster went live in August, I have had some kind of strange (to me) error at least once a week, almost always related to the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since we're on Spring Break at my university and not very busy, I decided to unmount/remount the ceph share, requiring stopping nfs and samba services. Stopping nfs-ganesha took a while, but it finally completed with no complaints from the ceph cluster. Stopping samba took longer and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds. It finally finished, and I was able to unmount/remount the ceph share and that finally cleared all the errors. This is leading me to believe that samba on the gateway and all the clients attaching to that is putting a strain on the connection back to ceph. Which finally brings me to my question: is there a better way to export samba to my clients using the ceph back end? Or is this as good as it gets and I just have to put up with the seemingly frequent errors? I can live with the errors and have been able to handle them so far, but I know people who have much bigger clusters and many more clients than me (by an order of magnitude) and don't see nearly as many errors as I do. Which is why I'm trying to figure out what is special about my setup. All my ceph nodes are running latest nautilus on Centos 7 (I just updated last week to 14.2.8), as is the gateway host. I'm mounting ceph directly on the gateway (by way of the kernel using cephfs, not rados/rbd) to a single mount point and exporting from there. My searches so far have not turned up anything extraordinarily useful, so I'm asking for some guidance here. Any advice is welcome. Thanks. Seth -- Seth Galitzer Systems Coordinator Computer Science Department Kansas State University http://www.cs.ksu.edu/~sgsax sgsax(a)ksu.edu 785-532-7790

8 11

Resize Bluestore i.e. shrink?
by Udo Waechter 04 Apr '20

04 Apr '20

Hi all, I'm currently building a little ceph-cluster, with embedded devices. My OSD Nodes are constrained in RAM (1GB, but 5 SATA Ports, please don't kill me ;) ). Anyways. Each of those nodes have 2x256 GB SSD and 2x 1TB HDDs. I'm using nautilus and deployd the whole thing with ceph-deploy. Now, I'd like to add more mem (a.k.a SWAP space, as raid1 on either ssds/hdds) to those OSDs. My question is: Can I shrink a bluestore lvm? I've seen 'ceph-bluestore-tool bluefs-bdev-expand' which suggests to expand the underlying lvm. It seem to me that I would have to remove the specific OSD and then add it manually after resizing the underlying lvm volume. And yes, I know that ceph need RAM ;) Cheers, udo.

2 1

Upgrading from Mimic to Nautilus
by Paul Choi 03 Apr '20

03 Apr '20

Hi, How is the experience of upgrading from Mimic to Nautilus? Anything to watch out for? And is it a smooth transition for the ceph-fuse clients on Mimic when interacting with a Nautilus MDS? I'm reading the docs on https://docs.ceph.com/docs/nautilus/install/upgrading-ceph/ and it looks pretty straightforward. I'm currently on Mimic 13.2.8, and have been putting off upgrading to Nautilus, but I think it's time. Thanks in advance, -Paul Choi

3 2

Purpose of crush_ln() function
by Bobby 03 Apr '20

03 Apr '20

Hi, I am trying to understand Straw2 Bucket of Ceph CRUSH algorithm. Can someone please tell what is the purpose of* crush_ln?* And what does "*compute 2^44*log2(input+1)*" means in the comment section above above *crush_ln* function? Thanks in advance

1 0