July 2019 - Dev - lists.ceph.io

by Ulrich Weigand

Hello, we've been trying to get Ceph to work on IBM Z, a big-endian system, and have been running into various serious issues relating to endian conversion code. The main issue we've been seeing is that while the old-style decode/encode machinery in include/encoding.h automatically byte-swaps all integers on big-endian systems (to ensure the serialized format is always little endian), the *new-style* machinery in include/denc.h does not. This seemed confusing at first since there is quite a bit of code there that appears intended to perform exactly that function, e.g. template<typename T> struct ExtType<T, std::enable_if_t<std::is_same_v<T, int32_t> || std::is_same_v<T, uint32_t>>> { using type = __le32; }; However, it turns out that at this point __le32 is actually just an alias for __u32, so this whole machinery doesn't really do anything at all. Looking at the old code in encoding.h, I notice that it works similarly, but uses ceph_le32 instead of __le32. The former is a C++ class that actually does perform byte-swap on access. Even more confusing, there is this code in include/types.h: // temporarily remap __le* to ceph_le* for benefit of shared kernel/userland headers #define __le16 ceph_le16 #define __le32 ceph_le32 #define __le64 ceph_le64 #include "ceph_fs.h" #include "ceph_frag.h" #include "rbd_types.h" #undef __le16 #undef __le32 #undef __le64 which --sometimes-- redefines __le32 as ceph_le32, but those redefines are not active at the point denc.h is included. So it would appear that the usage of __le32 in denc.h is incorrect, and this code should be using ceph_le32 instead. Is this right? But even so, grepping for __le32 throughout the code base shows quite a bit of additional places where it is used, most of which also appear to make the assumption that byte-swaps automatically happen. In addition, there appear to be some places where e.g. ceph_fs.h is included directly, without going via types.h -- and in those places, we suddenly no longer get the byte-swaps ... Now I was wondering whether the best way forward might be to just have __le32 always be defined as ceph_le32 when compiling user space code. But then I noticed that it used to be that way, but that was deliberated changed by this commit back in 2010: commit 737b5043576153817a6b4195b292672585df10d3 Author: Sage Weil <sage(a)newdream.net> Date: Fri May 7 13:45:00 2010 -0700 endian: simplify __le* type hackery Instead of preventing linux/types.h from being included, instead name our types ceph_le*, and remap using #define _only_ when including the shared kernel/userspace headers. So I'm a bit at a loss to understand how all this is supposed to be working. Any suggestions would be welcome -- we'd be willing to implement whatever's needed, but would like some guidance as to how the solution should look like ... Bye, Ulrich

4 years, 7 months

2
6
1 0

[crimson] bluestore in an alien world

by kefu chai

hi Chunmei, i am reviewing your change of https://github.com/ceph/ceph/compare/master...liu-chunmei:ceph_seastar_alie…. it looks good in general. i think the simplest way to co-locate different versions of alien-common, ceph-common and crimson-common is to introduce different namespaces. because we need to have alien-common and crimson-common in the same binary, and to have all of these three versions in the same repository. but this divergence concerns me, as it introduces yet another condition in the shared infrastructure in our code base. and in the long run, this #ifdef won't go away if we want to go this way, so i need to at least give it a try. what is "it"? to port rocksdb to seastar. as seastar offers "seastar::thread" which makes it relative simpler to wrap the blocking calls with ucontext. and rocksdb offers a abstraction machinery allowing one to port it to a new platform. and seastar is a "platform" to some degree, i'd say. will update you guys with my progress and findings. -- Regards Kefu Chai

4 years, 8 months

3
6
0 0

refcounting chunks vs snapshots

by Sage Weil

Hi Myoungwon, I was thinking about how a refcounted cas pool would interact with snapshots and it occurred to me that dropping refs when an object is deleted may break snapshotted versions of that object. If object A has a ref to chunk X, is snapshotted, then A is deleted, we'll (currently) drop the ref to X and remove it. That means that A can't be read. One way to get around that would be to mirror snaps from the source pool to the chunk pool--this is how cache tiering works. The problem I see there is that I'd hoped to allow multiple pools to share/consume the same chunk pool, but each pool has its own snapid namespace. Another would be to bake the refs more deepling into the source rados pool so that the refs are only dropped after all clones also drop the ref. That is harder to track, though, since I think you'd need to examine all of the clones to know whether the ref is truly gone. Unless we embed even more metadata in the SnapSet--something analogous to clone_overlap to identifying the chunks. That seems like it will bloat that structure, though. Other ideas? sage

4 years, 8 months

3
4
0 0

libcephfs and supplimentary groups

by David Disseldorp

Hi, Without calling ceph_mount_perms_set(), libcephfs consumers such as Samba can rely upon UserPerm::uid() and UserPerm::gid() to fallback to geteuid() and setegid() respectively for things such as ACL enforcement. However, there is no such fallback for supplementary groups, so ACL checks for a user which is only permitted path access via a supplementary group will result in a permission denied error. Samba ticket: https://bugzilla.samba.org/show_bug.cgi?id=14053 I've written a patch to address this (it currently omits the get_gids() codepath): https://github.com/ddiss/ceph/commit/035a1785ec73d803fead42c7240df01b755a81… Does this approach make sense, or should Samba go down the ceph_mount_perms_set() route to avoid this bug? The latter would likely be problematic, as user/group details for a mount will remain static. Cheers, David

4 years, 8 months

3
4
0 0

Adventures with large RGW buckets

by Paul Emmerich

Hi, we are seeing a trend towards rather large RGW S3 buckets lately. we've worked on several clusters with 100 - 500 million objects in a single bucket, and we've been asked about the possibilities of buckets with several billion objects more than once. From our experience: buckets with tens of million objects work just fine with no big problems usually. Buckets with hundreds of million objects require some attention. Buckets with billions of objects? "How about indexless buckets?" - "No, we need to list them". A few stories and some questions: 1. The recommended number of objects per shard is 100k. Why? How was this default configuration derived? It doesn't really match my experiences. We know a few clusters running with larger shards because resharding isn't possible for various reasons at the moment. They sometimes work better than buckets with lots of shards. So we've been considering to at least double that 100k target shard size for large buckets, that would make the following point far less annoying. 2. Many shards + ordered object listing = lots of IO Unfortunately telling people to not use ordered listings when they don't really need them doesn't really work as their software usually just doesn't support that :( A listing request for X objects will retrieve up to X objects from each shard for ordering them. That will lead to quite a lot of traffic between the OSDs and the radosgw instances, even for relatively innocent simple queries as X defaults to 1000 usually. Simple example: just getting the first page of a bucket listing with 4096 shards fetches around 1 GB of data from the OSD to return ~300kb or so to the S3 client. I've got two clusters here that are only used for some relatively low-bandwidth backup use case here. However, there are a few buckets with hundreds of millions of objects that are sometimes being listed by the backup system. The result is that this cluster has an average read IO of 1-2 GB/s, all going to the index pool. Not a big deal since that's coming from SSDs and goes over 80 Gbit/s LACP bonds. But it does pose the question about scalability as the user- visible load created by the S3 clients is quite low. 3. Deleting large buckets Someone accidentaly put 450 million small objects into a bucket and only noticed when the cluster ran full. The bucket isn't needed, so just delete it and case closed? Deleting is unfortunately far slower than adding objects, also radosgw-admin leaks memory during deletion: https://tracker.ceph.com/issues/40700 Increasing --max-concurrent-ios helps with deletion speed (option does effect deletion concurrency, documentation says it's only for other specific commands). Since the deletion is going faster than new data is being added to that cluster the "solution" was to run the deletion command in a memory-limited cgroup and restart it automatically after it gets killed due to leaking. How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level? 4. Common prefixes could filtered in the rgw class on the OSD instead of in radosgw Consider a bucket with 100 folders with 1000 objects in each and only one shard /p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000 Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100. How much data will be transfered between the OSDs and radosgw for this request? How many omap entries does the OSD scan? radosgw will ask the (single) index object to list the first 1000 objects. It'll return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000 radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on. So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either. This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something? IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

4 years, 8 months

9
11
0 0

Re: [ceph-users] details about cloning objects using librados

by Brad Hubbard

On Thu, Jun 27, 2019 at 8:58 PM nokia ceph <nokiacephusers(a)gmail.com> wrote: > > Hi Team, > > We have a requirement to create multiple copies of an object and currently we are handling it in client side to write as separate objects and this causes huge network traffic between client and cluster. > Is there possibility of cloning an object to multiple copies using librados api? > Please share the document details if it is feasible. It may be possible to use an object class to accomplish what you want to achieve but the more we understand what you are trying to do, the better the advice we can offer (at the moment your description sounds like replication which is already part of RADOS as you know). More on object classes from Cephalocon Barcelona in May this year: https://www.youtube.com/watch?v=EVrP9MXiiuU > > Thanks, > Muthu > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad

4 years, 8 months

4
14
0 0

cephfs metadata recovery

by Wyllys Ingersoll

I had a bad experience upgrading from Luminous to Mimic, in no small part due to a few critical errors on my part. Some data was lost (none of it critical), but the majority was unaffected. At this point I'd just like to get access to the data that is still there, but I cannot get the MDS servers to start. They initially were failing with the following errors: 2019-07-29 11:38:55.481 7f6167a35700 -1 /build/ceph-13.2.6/src/mds/MDCache.cc: In function 'void MDCache::add_inode(CInode*)' thread 7f6167a35700 time 2019-07-29 11:38:55.478864 /build/ceph-13.2.6/src/mds/MDCache.cc: 290: FAILED assert(!p) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f616f6f397e] 2: (()+0x2fab07) [0x7f616f6f3b07] 3: /usr/bin/ceph-mds() [0x5a9d0e] 4: (Server::prepare_new_inode(boost::intrusive_ptr<MDRequestImpl>&, CDir*, inodeno_t, unsigned int, file_layout_t*)+0xfcf) [0x55e37f] 5: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0xd5d) [0x560a3d] 6: (Server::handle_client_request(MClientRequest*)+0x49b) [0x563beb] 7: (Server::dispatch(Message*)+0x2fb) [0x5678cb] 8: (MDSRank::handle_deferrable_message(Message*)+0x434) [0x4da3c4] 9: (MDSRank::_dispatch(Message*, bool)+0x89b) [0x4f17db] 10: (MDSRankDispatcher::ms_dispatch(Message*)+0xa3) [0x4f1e43] 11: (MDSDaemon::ms_dispatch(Message*)+0xd3) [0x4d2073] 12: (DispatchQueue::entry()+0xb92) [0x7f616f7b48c2] 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f616f85172d] 14: (()+0x76ba) [0x7f616ef6f6ba] 15: (clone()+0x6d) [0x7f616e79841d] 2019-07-29 11:38:55.485 7f6167a35700 -1 *** Caught signal (Aborted) ** in thread 7f6167a35700 thread_name:ms_dispatch ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0x11390) [0x7f616ef79390] 2: (gsignal()+0x38) [0x7f616e6c6428] 3: (abort()+0x16a) [0x7f616e6c802a] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7f616f6f3a86] 5: (()+0x2fab07) [0x7f616f6f3b07] 6: /usr/bin/ceph-mds() [0x5a9d0e] 7: (Server::prepare_new_inode(boost::intrusive_ptr<MDRequestImpl>&, CDir*, inodeno_t, unsigned int, file_layout_t*)+0xfcf) [0x55e37f] 8: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0xd5d) [0x560a3d] 9: (Server::handle_client_request(MClientRequest*)+0x49b) [0x563beb] 10: (Server::dispatch(Message*)+0x2fb) [0x5678cb] 11: (MDSRank::handle_deferrable_message(Message*)+0x434) [0x4da3c4] 12: (MDSRank::_dispatch(Message*, bool)+0x89b) [0x4f17db] 13: (MDSRankDispatcher::ms_dispatch(Message*)+0xa3) [0x4f1e43] 14: (MDSDaemon::ms_dispatch(Message*)+0xd3) [0x4d2073] 15: (DispatchQueue::entry()+0xb92) [0x7f616f7b48c2] 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f616f85172d] 17: (()+0x76ba) [0x7f616ef6f6ba] 18: (clone()+0x6d) [0x7f616e79841d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. I then followed all of the MDS recovery instructions ( http://docs.ceph.com/docs/master/cephfs/disaster-recovery-experts/#disaster…) which took several days to complete. Now, when I attempt to start the MDS, I get a different crash and the mds still fails to start up: -10000> 2019-07-30 09:29:31.943 7f6ba2c21700 -1 /build/ceph-13.2.6/src/mds/MDCache.cc: In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f6ba2c21700 time 2019-07-30 09:29:31.943654 /build/ceph-13.2.6/src/mds/MDCache.cc: 1680: FAILED assert(follows >= realm->get_newest_seq()) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6bada9497e] 2: (()+0x2fab07) [0x7f6bada94b07] 3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xd3f) [0x5f821f] 4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xc0) [0x5f8450] 5: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0x4b1) [0x5f9141] 6: (Locker::scatter_writebehind(ScatterLock*)+0x465) [0x64a615] 7: (Locker::simple_sync(SimpleLock*, bool*)+0x176) [0x64e506] 8: (Locker::scatter_nudge(ScatterLock*, MDSInternalContextBase*, bool)+0x3dd) [0x652f6d] 9: (Locker::scatter_tick()+0x1e4) [0x6535a4] 10: (Locker::tick()+0x9) [0x6538b9] 11: (MDSRankDispatcher::tick()+0x1e9) [0x4f00d9] 12: (FunctionContext::finish(int)+0x2c) [0x4d52dc] 13: (Context::complete(int)+0x9) [0x4d31d9] 14: (SafeTimer::timer_thread()+0x18b) [0x7f6bada9120b] 15: (SafeTimerThread::entry()+0xd) [0x7f6bada9286d] 16: (()+0x76ba) [0x7f6bad3106ba] 17: (clone()+0x6d) [0x7f6bacb3941d] -10000> 2019-07-30 09:29:31.951 7f6ba2c21700 -1 *** Caught signal (Aborted) ** in thread 7f6ba2c21700 thread_name:safe_timer ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0x11390) [0x7f6bad31a390] 2: (gsignal()+0x38) [0x7f6baca67428] 3: (abort()+0x16a) [0x7f6baca6902a] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7f6bada94a86] 5: (()+0x2fab07) [0x7f6bada94b07] 6: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xd3f) [0x5f821f] 7: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xc0) [0x5f8450] 8: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0x4b1) [0x5f9141] 9: (Locker::scatter_writebehind(ScatterLock*)+0x465) [0x64a615] 10: (Locker::simple_sync(SimpleLock*, bool*)+0x176) [0x64e506] 11: (Locker::scatter_nudge(ScatterLock*, MDSInternalContextBase*, bool)+0x3dd) [0x652f6d] 12: (Locker::scatter_tick()+0x1e4) [0x6535a4] 13: (Locker::tick()+0x9) [0x6538b9] 14: (MDSRankDispatcher::tick()+0x1e9) [0x4f00d9] 15: (FunctionContext::finish(int)+0x2c) [0x4d52dc] 16: (Context::complete(int)+0x9) [0x4d31d9] 17: (SafeTimer::timer_thread()+0x18b) [0x7f6bada9120b] 18: (SafeTimerThread::entry()+0xd) [0x7f6bada9286d] 19: (()+0x76ba) [0x7f6bad3106ba] 20: (clone()+0x6d) [0x7f6bacb3941d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Is there any hope of getting my MDS servers back up and accessing the cephfs data at this point? None of the guides I've followed thus far have been successful. thanks, Wyllys Ingersoll

4 years, 8 months

2
5
0 0

Re: build status with boost-1.70

by Willem Jan Withagen

On 25-4-2019 22:44, Casey Bodley wrote: > > On 4/25/19 10:50 AM, Willem Jan Withagen wrote: >> On 25-4-2019 11:56, Willem Jan Withagen wrote: >>> On 24-4-2019 20:30, Patrick McLean wrote: >>>> On Wed, 24 Apr 2019 09:35:57 -0400 >>>> Casey Bodley <cbodley(a)redhat.com> wrote: >>>> >>>>> On 4/20/19 6:25 AM, Willem Jan Withagen wrote: >>>>>> On 19-4-2019 03:46, Patrick McLean wrote: >>>>>>> On Wed, 17 Apr 2019 09:50:00 -0400 >>>>>>> Casey Bodley <cbodley(a)redhat.com> wrote: >>>>>>>> That's good to know, thanks for testing! This one is documented >>>>>>>> as a breaking change in >>>>>>>> https://www.boost.org/doc/libs/1_70_0/doc/html/boost_asio/history.html: >>>>>>>> >>>>>>>> >>>>> I have a branch building against 1.70 in >>>>> https://github.com/ceph/ceph/pull/27730. I ended up adding '#if >>>>> BOOST_VERSION < 107000' in a couple places, so we could merge the rgw >>>>> bits without updating the minimum required version to 1.70. I don't >>>>> see any immediate benefit to switching now; I just can't guarantee >>>>> that rgw won't break things in the meantime if we're only testing >>>>> builds against 1.67. What do you all think? >>>> This branch builds for me as well, I haven't done any functional >>>> testing. >>>> >>>> Perhaps there could be an option to use either until a full >>>> switch to 1.70 (or later) is required (I am not sure what kind of cmake >>>> magic it would take to allow either, but I could take a stab at it if >>>> it's desired). >>> I've asked one of the FreeBSD ports maintainers on the plans for boost. >>> >>> I'd like to make 2 comments: >>> 1) The cmake foo is usually the more annoying work, since cmake and >>> boots do not really run in sync. >>> 2) it would be sort of silly not to already prepare for boost going >>> to 1.7 and on, if somebody has done the work. >>> Question will be how to do the jenkins/QA/teutology testing for >>> this, otherwise the fault will only show when testing gets to 1.7 >> >> I checked with the guys doing boost porting, and 1.70 is on the virge >> of arriving in the stable ports tree. Meaning that I'll be needing the >> same fixes.... to keep head building. >> >> --WjW >> > Thanks Willem. > > I updated https://github.com/ceph/ceph/pull/27794 to remove the cmake > change that required boost 1.70, so we can revisit that upgrade later. > If rgw breaks anything else in the meantime, you guys know where to find > me. I finally got around to do some more work on Ceph, and getting Boost 1.70 working on FreeBSD was the first thing that was needed as FreeBSD 13.0-Current only had Boost 1.70. It was as "simple" as replacing our in-tree FindBoost.cmake with a more recent one that understands 1.7.... So atm. HEAD is compiling and testing oke again on FreeBSD: http://cephdev.digiware.nl:8180/jenkins/job/ceph-master/3585/console I guess that it would also allow other platforms to use 1.7 as well. --WjW

4 years, 8 months

1
0
0 0

Migrate old EC pool to new EC pool (mimic)

by Valentin Bajrami

Hello There, Recently, I've been reading about how to migrate an existing EC pool which has the following profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 To a new EC pool with the following profile: crush-device-class= crush-failure-domain=rack crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 So, I've been reading the following blog but not sure if this will work with an EC pool. The header which says "The simple way" shows the steps how it's done but there is a small comment beneath it which says and I quote: || "But it does not work in all cases. For example with EC pools : “error copying pool testpool => newpool: (95) Operation not supported”." My setup is as follows: DC1 = 4 servers each with 1 OSD running DC2 = 4 servers each with 1 OSD running Since my current EC profile crush-failure-domain is set to 'host' (which seems to be the default) I wanted to change that to 'rack' so I can ensures that no two /chunks/ are stored in the same rack. The idea here is to place 4 OSD's on DC1 in rack1 and 4 OSD's on DC2 in rack2. I am aware I'd need to modify the crush ruleset to achieve, however, from this starting point, what's best practice to achieve this? Please see a diagram (Thanks to lordcirth_ on IRC oftc #ceph) attached to this email.// Please let me know if you need additional information from my side. -- Met vriendelijke groeten / Kind regards, Valentin Bajrami Target Holding

4 years, 8 months

3
6
0 0

What are the deleterious effects when using lower shard number on ceph osd?

by 刘朝洋

The document of ceph osd config( http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#caveats) said that using lower shard number may have deleterious effects. I want to know what the deleterious effects are. I test the performance of ceph cluster using different osd_op_num_shards and different osd_op_num_threads_per_shard configuration. I found that Ceph cluster will get bad performance if using lower shard number. And I also found that using many thread number with lower shard number can get the same improvement of performance with multi-shard. So I think I can use lower shard number with many thread number per shard to replace many shard number. But I don't know if there are other bad effects besides bad-performance. I want to use dmclock feature in my ceph cluster but it can get good effect only when using lower shard number. The environment of my ceph cluster: - 2 nodes - 3 osds per node - nvme ssd as osd - CPU：Intel(R) Xeon(R) Gold 6130 - Mem：187GB The main configuration of my ceph cluster: auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 filestore xattr use omap = true mon_allow_pool_delete=true objecter_inflight_ops = 10240 objecter_inflight_op_bytes = 104857600 osd_pool_default_size=2 osd_pool_default_min_size=1 osd_op_queue_cut_off=high osd_op_num_shards_ssd=1 osd_op_num_threads_per_shard_ssd=1 osd_op_queue=wpq Thanks for assistance! Liu Chaoyang

4 years, 8 months

1
0
0 0

2024

2023

2022

2021

2020

2019

Dev July 2019