- ceph-users - lists.ceph.io

Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)
by VELARTIS Philipp Dürhammer 06 Sep '20

06 Sep '20

I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals?

6 16

damaged cephfs
by Magnus HAGDORN 06 Sep '20

06 Sep '20

Hi there, we reconfigured our ceph cluster yesterday to remove the cluster network and things didn't quite go to plan. I am trying to figure out what went wrong and also what to do next. We are running nautilus 14.2.10 on Scientific Linux 7.8. So, we are using a mixture of RBDs and cephfs. For the transition we switched off all machines that are using the RBDs and switched off the cephfs using ceph fs set one down true Once no more MDS were running we reconfigured ceph to remove the cluster network and set various flags ceph osd set noout ceph osd set nodown ceph osd set pause ceph osd set nobackfill ceph osd set norebalance ceph osd set norecover We then restarted the OSDs one host at a time. During this process ceph was mostly happy, except for two PGs. After all OSDs had been restarted we switched off the cluster network switches to make sure it was totally gone. ceph was still happy. The PG error also disappeared. We then unset all those errors and re-enabled cephfs. We then switched on the servers using the RBDs with no issues. So far so good. We then started using the cephfs (we keep VM images on the cephfs). The MDS were showing an error. I restarted the MDS but they didn't come back. We then followed the instructions here: https://docs.ceph.com/docs/nautilus/cephfs/disaster-recovery-experts/#disas… up to truncating the journal. The MDS started again. However, as soon as we started writing the cephfs the MDS crashed. A scrub of the cephfs revealed backtrace damage. We have now followed the remaining steps of the disaster recovery procedure and are waiting for the cephfs-data-scan scan_extents to complete. It would be really helpful if you could give an indication of how long this process will take (we have ~40TB in our cephfs) and how many workers to use. The other missing bit of documentation is the cephfs scrubbing. Is that something we should run routinely? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 3

ceph fs reset situation
by Alexander B. Ustinov 06 Sep '20

06 Sep '20

Hello, list. Have anybody been in the situation when after "ceph fs reset" filesystem becomes blank (mounts OK, ls shows no files/directories), but data and metadata pools still hold something (698G and 400M respectively by "ceph fs status"). Would be grateful for documentation vectors and/or suggestions. Maybe i remember wrong, but few times in the past same "ceph fs reset" produced minor corruption to recent filesystem changes.

1 0

Re: PG number per OSD
by Anthony D'Atri 06 Sep '20

06 Sep '20

I think there are multiple variables there. My advice is for HDDs to aim for an average of 150-200 as I wrote before. The limitation is the speed of the device, throw a thousand PGs on there and you won’t get any more out of it, you’ll just have more peering and more RAM used. NVMe is a different story. > > Is there any rules for computing RAM requeirements in terms of the number of PGs? > > Just curious abount what is the fundamental limitations on the number of PGs per OSD for bigger capacity HDD > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn > > From: Anthony D'Atri > Date: 2020-09-05 20:00 > To: huxiaoyu(a)horebdata.cn > CC: ceph-users > Subject: Re: [ceph-users] PG number per OSD > One factor is RAM usage, that was IIRC the motivation for the lowering of the recommendation of the ratio from 200 to 100. Memory needs also increase during recovery and backfill. > > When calculating, be sure to consider repllicas. > > ratio = (pgp_num x replication) / num_osds > > As HDDs grow the interface though isn’t becoming faster (with SATA at least), and there are only so many IOPS and MB/s that you’re going to get out of one no matter how you slice it. Everything always depends on your use-case and workload, but I suspect that often the bottleneck is the drive, not PG or OSD serialization. > > For example, do you prize IOPS more, latency, or MB/s? If you don’t care about latency, then you can drive your HDDs harder and get more MB/s throughput out of them, though your average latency might climb to 100ms. Which eg. RBD VM clients probably wouldn’t be too happy about, but which an object service *might* tolerate. > > Basically in the absence of more info, I would personally suggest aiming at the 150-200 average range, with pgp_num a power of 2. If you aim a bit high, the ratio will come down a bit when you add nodes/OSDs to your cluster to gain capacity. Be sure to balance usage and watch your mon_max_pg_per_osd setting — allowing some headroom for natural variation and for when components fail. > > YMMV. > > — aad > >> On Sep 5, 2020, at 10:34 AM, huxiaoyu(a)horebdata.cn wrote: >> >> Dear Ceph folks, >> >> As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? >> >> thanks a lot, >> >> Samuel >> >> >> >> >> huxiaoyu(a)horebdata.cn >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: PG number per OSD
by Dan van der Ster 06 Sep '20

06 Sep '20

Good question! Did you already observe some performance impact of very large PGs? Which PG locks are you speaking of? Is there perhaps some way to improve this with the op queue shards? (I'm cc'ing Mark in case this is something that the performance team has already looked into). With a 20TB osd, we'll have up to 200GB PGs following the current suggestions -- but even then, backfilling those huge PGs would still be done in under an hour, which seems pretty reasonable IMHO. -- dan On Sat, Sep 5, 2020 at 7:35 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Ceph folks, > > As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? > > thanks a lot, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: PG number per OSD
by Anthony D'Atri 06 Sep '20

06 Sep '20

One factor is RAM usage, that was IIRC the motivation for the lowering of the recommendation of the ratio from 200 to 100. Memory needs also increase during recovery and backfill. When calculating, be sure to consider repllicas. ratio = (pgp_num x replication) / num_osds As HDDs grow the interface though isn’t becoming faster (with SATA at least), and there are only so many IOPS and MB/s that you’re going to get out of one no matter how you slice it. Everything always depends on your use-case and workload, but I suspect that often the bottleneck is the drive, not PG or OSD serialization. For example, do you prize IOPS more, latency, or MB/s? If you don’t care about latency, then you can drive your HDDs harder and get more MB/s throughput out of them, though your average latency might climb to 100ms. Which eg. RBD VM clients probably wouldn’t be too happy about, but which an object service *might* tolerate. Basically in the absence of more info, I would personally suggest aiming at the 150-200 average range, with pgp_num a power of 2. If you aim a bit high, the ratio will come down a bit when you add nodes/OSDs to your cluster to gain capacity. Be sure to balance usage and watch your mon_max_pg_per_osd setting — allowing some headroom for natural variation and for when components fail. YMMV. — aad > On Sep 5, 2020, at 10:34 AM, huxiaoyu(a)horebdata.cn wrote: > > Dear Ceph folks, > > As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? > > thanks a lot, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

librados: rados_cache_pin returning Invalid argument. need help
by Daniel Mezentsev 06 Sep '20

06 Sep '20

Hi ceph-users, Im bit stuck with librados, partcularly rados_cache_pin finction. For some reason it returns "Invalid argument", error code 22. Can't find what im doing wrong. in rados_cache_pin im using the same ioctx and obj name like i use with rados_write - which works just fine. Any help is more then welcome. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

1 0

cephadm orch thinks hosts are offline
by xahare＠gmail.com 06 Sep '20

06 Sep '20

We have a 5 node cluster, all monitors, installed with cephadm. recently, the hosts needed to be rebooted for upgrades, but as we rebooted them, hosts fail their cephadm check. as you can see ceph1 is in quorum and is the host the command is run from. following is the output of ceph -s and ceph orch host ls. ceph orch pause and resume only removed the "offline" status of cephmon-temp, which really is offline. how do we fix ceph orchs confusion? the third is a temp node we had that ceph orch remove host couldnt get rid of. ceph1:~# ceph -s health: HEALTH_WARN 3 hosts fail cephadm check services: mon: 5 daemons, quorum ceph5,ceph4,ceph3,ceph2,ceph1 (age 2d) mgr: ceph3.dmpmih(active, since 3w), standbys: ceph5.pwseyi osd: 30 osds: 30 up (since 2d), 30 in (since 3w) data: pools: 2 pools, 129 pgs objects: 2.29M objects, 8.7 TiB usage: 22 TiB used, 87 TiB / 109 TiB avail pgs: 129 active+clean io: client: 149 KiB/s wr, 0 op/s rd, 14 op/s wr ceph1:~# ceph orch host ls HOST ADDR LABELS STATUS ceph1 ceph1 mon Offline ceph2 ceph2 mon Offline ceph3 ceph3 mon ceph4 ceph4 mon ceph5 ceph5 mon cephmon-temp cephmon-temp Offline

1 0

Re: bug of the year (with compressed omap and lz 1.7(?))
by Dan van der Ster 06 Sep '20

06 Sep '20

Hi Marc, As far as I know, the osdmap corruption occurs with this osd config: bluestore_compression_mode=aggressive bluestore_compression_algorithm=lz4 (My understanding is that if you don't have the above settings, but use pool-specific compression settings instead, then the osdmaps are not compressed, so they would not be corruptable). I prefer to stay on the safe side, though: since we know that lz4 < 1.8.2 can corrupt data if comes from fragmented memory, and we don't know all of the cases where that might be the case, I suggest to use another algorithm or turn off compression until you have upgraded to v14.2.10 or newer. And btw, there was another bug earlier in nautilus where those pool-specific compression settings weren't applied correctly anyway, so I'm not sure they even work yet in 14.2.9. -- dan On Sat, Sep 5, 2020 at 6:12 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I am still running 14.2.9 with lz4-1.7.5-3. Will I run into this bug > enabling compression on a pool with: > > ceph osd pool set POOL_NAME compression_algorithm COMPRESSION_ALGORITHM > ceph osd pool set POOL_NAME compression_mode COMPRESSION_MODE > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

PG number per OSD
by huxiaoyu＠horebdata.cn 05 Sep '20

05 Sep '20

Dear Ceph folks, As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? thanks a lot, Samuel huxiaoyu(a)horebdata.cn

1 0