- ceph-users - lists.ceph.io

Re: SSD recommendations for RBD and VM's
by mhnx 06 Jun '21

06 Jun '21

Hello Samuel. Thanks for the answer. Yes the Intel S4510 series is a good choice but it's expensive. I have 21 server and data distribution is quite well. At power loss I don't think I'll lose data. All the VM's using same image and the rest is cookie. In this case I'm not sure I should spend extra money on PLP. Actually I like Samsung 870 EVO. It's cheap and I think 300TBW will be enough for 5-10years. Do you know any better ssd with the same price range as 870 EVO? Samsung 870 evo (500GB) = 5 Years or 300 TBW - $64.99 Samsung 860 pro (512GB) = 5 Years or 600 TBW - $99 > > I would recommend Intel S4510 series, which has power loss protection (PLP). > > If you do not care about PLP, lower-cost Samsung 870EVO and Crucial MX500 should also be OK (with separate DB/WAL on enterprise SSD with PLP) > > Samuel > > ________________________________ > huxiaoyu(a)horebdata.cn > > > From: by morphin > Date: 2021-05-30 02:48 > To: Anthony D'Atri > CC: Ceph Users > Subject: [ceph-users] Re: SSD recommendations for RBD and VM's > Hello Anthony. > > I use Qemu and I don't need size. > I've 1000 vm and usually they're clones from the same rbd image. The > image is 30GB. > Right now I've 7TB Stored data. rep x3 = 20TB data. It's mostly read > intensive. Usage is stable and does not grow. > So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's. > I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB > will be enough. Actually 120PCS 256GB will be better but the price > goes up. > I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've > 40PCS 100GB. I'm gonna make them OSD as well. > 7 years and DC S3700 still rocks. Not even one of them is dead. > The SSD must be Low price & High TBW life span. Rest is not important. > > > Anthony D'Atri <anthony.datri(a)gmail.com>, 30 May 2021 Paz, 02:26 > tarihinde şunu yazdı: > > > > The choice depends on scale, your choice of chassis / form factor, budget, workload and needs. > > > > The sizes you list seem awfully small. Tell us more about your use-case. OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ? > > —aad > > > > > > > On May 29, 2021, at 2:10 PM, by morphin <morphinwithyou(a)gmail.com> wrote: > > > > > > Hello. > > > > > > I have virtualization env and I'm looking new SSD for HDD replacement. > > > What are the best Performance / Price SSDs in the market right now? > > > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. > > > > > > Is there a SSD recommendation list for ceph? > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

5 9

RBD + ZFS + NFS = bad performance. How to speed up?
by mhnx 05 Jun '21

05 Jun '21

Hello. I'm using rbd disks for zfs and on the zfs I use NFS. I see 10-50MB/s top. When I use the rbd disk in the VM environment it can reach 1GB/s RW so there is nothing wrong with the rbd and the cluster. Also I'm sure about ZFS+NFS performance when zfs on the real device. But when they work together I can't get even %10 performance. I think something is wrong, it shouldn't be that bad. Ceph version: Nautilus 14.2.16 RBD Data = EC Pool RBD Metadata = SSD replicated zpool create $testpool rbd0 exportfs /testpool Is there anyone who has tried it before?

1 1

Zabbix sender issue
by Bob Loi 05 Jun '21

05 Jun '21

Hi, I managed to build a Ceph cluster with the help of cephadm tool. It works like a charm. I have a problem that i’m still not able to fix: I know that zabbix-sender executable is not integrated into the cephadm image of ceph-mgr pulled and started by podman because of this choice. https://github.com/ceph/ceph-container/issues/1651 I’m a total newbie about containers tecnologies, but i managed to install zabbix sender manually by executing this with podman podman ps -a -> looking for the container ID podman exec -ti [id_container] /bin/bash -> to Then install repo via rpm and dnf install zabbix-sender. Two considerations. 1) https://github.com/ceph/ceph-container/issues/1651 -> This answer still leaves me confused. It’s pretty no-sense giving a Zabbix module if then you have to install the executable by yourself and the container it’s overwritten everytime you reboot the physical host giving minimal info about avoiding this behavior. I know this is open source and you have to know well the environment before doing everything, i know devs want less troubles as possible, but this is about user experience. If you have to choose between the final user or you having an annoying problem, IMHO, i prefer to have a happy user. End of consideration. I think including zabbix-sender executable into the container wouldn’t kill nobody. 2) Since i’m a total noob about containers, podman, docker etc, do you have any info about "fix" this behavior and avoid overwrite the mgr containers everytime i reboot the host? Please forgive me, i’m totally a newbie about containers. Thank you in advance for the support. Best, Roberto Kirecom.net

1 0

Rolling upgrade model to new OS
by Drew Weaver 04 Jun '21

04 Jun '21

Hello, I need to upgrade the OS that our Ceph cluster is running on to support new versions of Ceph. Has anyone devised a model for how you handle this? Do you just: Install some new nodes with the new OS Install the old version of Ceph on the new nodes Add those nodes/osds to the cluster Remove the old nodes Upgrade Ceph on the new nodes Are there any specific OS that Ceph has said that will have longer future version support? Would like to only touch the OS every 3-4 years if possible. Thanks, -Drew

4 3

CRUSH rule for EC 6+2 on 6-node cluster
by Bryan Stillwell 04 Jun '21

04 Jun '21

I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host. If I use the default rule with an osd failure domain like this: step take default step choose indep 0 type osd step emit I get clustering of 3-4 chunks on some of the hosts: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do > echo $pg > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do > ceph osd find $osd | jq -r '.host' > done | sort | uniq -c | sort -n -k1 8.0 1 harrahs 3 paris 4 aladdin 8.1 1 aladdin 1 excalibur 2 mandalaybay 4 paris 8.2 1 harrahs 2 aladdin 2 mirage 3 paris ... However, if I change the rule to use: step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit I get the data spread across 4 hosts with 2 chunks per host: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do > echo $pg > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do > ceph osd find $osd | jq -r '.host' > done | sort | uniq -c | sort -n -k1 > done 8.0 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.1 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.2 2 harrahs 2 mandalaybay 2 mirage 2 paris ... Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2. Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab). Thanks, Bryan

5 12

SAS vs SATA for OSD
by Dave Hall 04 Jun '21

04 Jun '21

Hello, We're planning another batch of OSD nodes for our cluster. Our prior nodes have been 8 x 12TB SAS drives plus 500GB NVMe per HDD. Due to market circumstances and the shortage of drives those 12TB SAS drives are in short supply. Our integrator has offered an option of 8 x 14TB SATA drives (still Enterprise). For Ceph, will the switch to SATA carry a performance difference that I should be concerned about? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 7

OSD Won't Start - LVM IOCTL Error - Read-only
by Dave Hall 04 Jun '21

04 Jun '21

Hello, I had an OSD drop out a couple days ago. This is 14.2.16, Bluestore, HDD + NVMe, non-container. The HDD sort of went away. I powered down the node, reseated the drive, and it came back. However, the OSD won't start. Systemctl --failed shows that the lvm2 pvscan failed, preventing the OSD unit from starting. Running the pvscan activate command manually with with verbose gave 'device-mapper: reload ioctl on (253:7) failed: Read-only file system'. I have been looking at this for a while, but I can't figure out what is read-only that is causing the problem. The full output of the pvscan is: # pvscan --cache --activate ay --verbose '8:48' pvscan devices on command line. activation/auto_activation_volume_list configuration setting not defined: All logical volumes will be auto-activated. Activating logical volume ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79/osd-block-425faf92-449e-4b57-98f2-a90a7f60e2a4. activation/volume_list configuration setting not defined: Checking only host tags for ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79/osd-block-425faf92-449e-4b57-98f2-a90a7f60e2a4. Creating ceph--block--b1fea172--71a4--463e--a3e3--8cdcc1bc7b79-osd--block--425faf92--449e--4b57--98f2--a90a7f60e2a4 Loading table for ceph--block--b1fea172--71a4--463e--a3e3--8cdcc1bc7b79-osd--block--425faf92--449e--4b57--98f2--a90a7f60e2a4 (253:7). device-mapper: reload ioctl on (253:7) failed: Read-only file system Removing ceph--block--b1fea172--71a4--463e--a3e3--8cdcc1bc7b79-osd--block--425faf92--449e--4b57--98f2--a90a7f60e2a4 (253:7) Activated 0 logical volumes in volume group ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79. 0 logical volume(s) in volume group "ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79" now active ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79: autoactivation failed. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

radosgw-admin bucket delete linear memory growth?
by Janne Johansson 03 Jun '21

03 Jun '21

I am seeing a huge usage of ram, while my bucket delete is churning over left-over multiparts, and while I realize there are *many* being done a 1000 at a time, like this: 2021-06-03 07:29:06.408 7f9b7f633240 0 abort_bucket_multiparts WARNING : aborted 254000 incomplete multipart uploads ..my first run ended radosgw-admin with out-of-mem, so it seems some part of this keeps data or forgets to free up old parts of the lists after they have been cancelled? I also don't know if restarting means it iterates over the same ones again, or if this log line means 254k are processed and if I need to restart again, it will at least have made a few hours of progress. At this point, RES is 2.1g so roughly 10000 bytes per "entry" if it is linear somehow. ceph 13.2.10 in this case. -- May the most significant bit of your life be positive.

1 0

ceph configuration using ubuntu 18.04
by Michel Niyoyita 03 Jun '21

03 Jun '21

Dear Ceph user, I want to configure ceph in our production environment using ubuntu . anyone who used it can help for the tutorial used . Best regards.

1 0

ceph-client homebrew for MacOS
by kefu chai 03 Jun '21

03 Jun '21

hi folks, i am glad to announce that with the help of Sven and Chris, ceph-client homebrew for MacOS was updated with a recent master commit. this homebrew provides some essential client side tools and libraries which allow us to talk to a ceph cluster from MacOS machine, like: ceph ceph-conf ceph-fuse rados rbd in addition to the executables and libraries, the brew also packages the header files. so, if you are up to do some development with librados, librbd and libcephfs, you can also have the necessary bits from the brew. see https://github.com/mulbc/homebrew-ceph-client it comes with a pre-bottled brew for BigSur on amd64. the last release[0] of the brew was packaged 3 years ago with mimic-13.2.2 . please note, this is not official - do not expect frequent updates or support from Ceph. cheers, -- [0] https://ceph-users.ceph.narkive.com/bnyGalkH/ceph-client-libraries-for-osx -- Regards Kefu Chai

1 0