- ceph-users - lists.ceph.io

Re: one read/write, many read only
by iwesley＠mail.de 02 Oct '19

02 Oct '19

There is a lock for object exists. If the file was not writing close, other ones can read only. Regards > Am Oct 2, 2019 - 12:09 AM schrieb khaled.atteya(a)gmail.com: > > > Hi, > > Is it possible to do this scenario : > If one open a file first , he will get read/write permissions and other will get read-only permission if they open the file after the first one. > > Thanks >

1 0

Re: RAM recommendation with large OSDs?
by Paul Emmerich 01 Oct '19

01 Oct '19

The problem with lots of OSDs per node is that this usually means you have too few nodes. It's perfectly fine to run 60 OSDs per node if you got a total of 1000 OSDs or so. But I've seen too many setups with 3-5 nodes where each node runs 60 OSDs which makes no sense (and usually isn't even cheaper than more nodes, especially once you consider the lost opportunity for running erasure coding). The usual backup cluster we are seeing is in the single-digit petabyte range with about 12 to 24 disks per server running ~8+3 erasure coding. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Oct 2, 2019 at 12:53 AM Darrell Enns <darrelle(a)knowledge.ca> wrote: > > Thanks Paul. I was speaking more about total OSDs and RAM, rather than a single node. However, I am considering building a cluster with a large OSD/node count. This would be for archival use, with reduced performance and availability requirements. What issues would you anticipate with a large OSD/node count? Is the concern just the large rebalance if a node fails and takes out a large portion of the OSDs at once? > > -----Original Message----- > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Tuesday, October 01, 2019 3:00 PM > To: Darrell Enns <darrelle(a)knowledge.ca> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] RAM recommendation with large OSDs? > > On Tue, Oct 1, 2019 at 6:12 PM Darrell Enns <darrelle(a)knowledge.ca> wrote: > > > > The standard advice is “1GB RAM per 1TB of OSD”. Does this actually still hold with large OSDs on bluestore? > > No > > > Can it be reasonably reduced with tuning? > > Yes > > > > From the docs, it looks like bluestore should target the “osd_memory_target” value by default. This is a fixed value (4GB by default), which does not depend on OSD size. So shouldn’t the advice really by “4GB per OSD”, rather than “1GB per TB”? Would it also be reasonable to reduce osd_memory_target for further RAM savings? > > Yes > > > For example, suppose we have 90 12TB OSD drives: > > Please don't put 90 drives in one node, that's not a good idea in 99.9% of the use cases. > > > > > “1GB per TB” rule: 1080GB RAM > > “4GB per OSD” rule: 360GB RAM > > “2GB per OSD” (osd_memory_target reduced to 2GB): 180GB RAM > > > > > > > > Those are some massively different RAM values. Perhaps the old advice was for filestore? Or there is something to consider beyond the bluestore memory target? What about when using very dense nodes (for example, 60 12TB OSDs on a single node)? > > Keep in mind that it's only a target value, it will use more during recovery if you set a low value. > We usually set a target of 3 GB per OSD and recommend 4 GB of RAM per OSD. > > RAM saving trick: use fewer PGs than recommended. > > > Paul > > > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io

1 0

Re: RAM recommendation with large OSDs?
by Paul Emmerich 01 Oct '19

01 Oct '19

On Tue, Oct 1, 2019 at 6:12 PM Darrell Enns <darrelle(a)knowledge.ca> wrote: > > The standard advice is “1GB RAM per 1TB of OSD”. Does this actually still hold with large OSDs on bluestore? No > Can it be reasonably reduced with tuning? Yes > From the docs, it looks like bluestore should target the “osd_memory_target” value by default. This is a fixed value (4GB by default), which does not depend on OSD size. So shouldn’t the advice really by “4GB per OSD”, rather than “1GB per TB”? Would it also be reasonable to reduce osd_memory_target for further RAM savings? Yes > For example, suppose we have 90 12TB OSD drives: Please don't put 90 drives in one node, that's not a good idea in 99.9% of the use cases. > > “1GB per TB” rule: 1080GB RAM > “4GB per OSD” rule: 360GB RAM > “2GB per OSD” (osd_memory_target reduced to 2GB): 180GB RAM > > > > Those are some massively different RAM values. Perhaps the old advice was for filestore? Or there is something to consider beyond the bluestore memory target? What about when using very dense nodes (for example, 60 12TB OSDs on a single node)? Keep in mind that it's only a target value, it will use more during recovery if you set a low value. We usually set a target of 3 GB per OSD and recommend 4 GB of RAM per OSD. RAM saving trick: use fewer PGs than recommended. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

one read/write, many read only
by khaled atteya 01 Oct '19

01 Oct '19

Hi, Is it possible to do this scenario : If one open a file first , he will get read/write permissions and other will get read-only permission if they open the file after the first one. Thanks

2 1

Doubt about ceph-iscsi and Vmware
by Gesiel Galvão Bernardes 01 Oct '19

01 Oct '19

Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points: - If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct? - In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata? - How much memory ceph-iscsi use? Which is a good number of RAM? Regards Gesiel * https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/

4 5

Nautilus pg autoscale, data lost?
by Raymond Berg Hansen 01 Oct '19

01 Oct '19

Hi. I am new to ceph but have set it up on my homelab and started using it. It seemed very good intil I desided to try pg autoscale. After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the number of PGs and the pools are now unaccessible. I have tried to turn it off again, but no luck! Please help. ceph status: https://pastebin.com/88qNivJi (do not know why it lists 4 pools, I have 3. Maybe one of the pools I created after and deleted are in limbo?) ceph osd pool ls detail: https://pastebin.com/HZLz6yHL ceph health detail: https://pastebin.com/Kqd2YMtm

5 9

ceph-osd@n crash dumps
by Del Monaco, Andrea 01 Oct '19

01 Oct '19

Hi list, After the nodes ran OOM and after reboot, we are not able to restart the ceph-osd@x services anymore. (Details about the setup at the end). I am trying to do this manually, so we can see the error but all i see is several crash dumps - this is just one of the OSDs which is not starting. Any idea how to get past this?? [root@ceph001 ~]# /usr/bin/ceph-osd --debug_osd 10 -f --cluster ceph --id 83 --setuser ceph --setgroup ceph > /tmp/dump 2>&1 starting osd.83 at - osd_data /var/lib/ceph/osd/ceph-83 /var/lib/ceph/osd/ceph-83/journal /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] 2019-10-01 14:19:49.500 2aaaaaaf5540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] 2019-10-01 14:19:49.509 2aaaaaaf5540 -1 *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -693> 2019-10-01 14:19:49.500 2aaaaaaf5540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/ x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] -693> 2019-10-01 14:19:49.509 2aaaaaaf5540 -1 *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -693> 2019-10-01 14:19:49.500 2aaaaaaf5540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] -693> 2019-10-01 14:19:49.509 2aaaaaaf5540 -1 *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Environment: [root@ceph001 ~]# uname -r 3.10.0-957.27.2.el7.x86_64 [root@ceph001 ~]# cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [root@ceph001 ~]# rpm -qa | grep -i ceph cm-config-ceph-release-mimic-8.2-73_cm8.2.noarch ceph-13.2.6-0.el7.x86_64 ceph-selinux-13.2.6-0.el7.x86_64 ceph-base-13.2.6-0.el7.x86_64 ceph-osd-13.2.6-0.el7.x86_64 cm-config-ceph-radosgw-systemd-8.2-6_cm8.2.noarch libcephfs2-13.2.6-0.el7.x86_64 ceph-common-13.2.6-0.el7.x86_64 ceph-mgr-13.2.6-0.el7.x86_64 cm-config-ceph-systemd-8.2-12_cm8.2.noarch ceph-mon-13.2.6-0.el7.x86_64 python-cephfs-13.2.6-0.el7.x86_64 ceph-mds-13.2.6-0.el7.x86_64 ceph osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 785.95801 root default -5 261.98599 host ceph001 1 hdd 7.27699 osd.1 up 1.00000 1.00000 3 hdd 7.27699 osd.3 down 1.00000 1.00000 6 hdd 7.27699 osd.6 down 1.00000 1.00000 9 hdd 7.27699 osd.9 down 0 1.00000 12 hdd 7.27699 osd.12 down 1.00000 1.00000 15 hdd 7.27699 osd.15 up 1.00000 1.00000 18 hdd 7.27699 osd.18 down 1.00000 1.00000 21 hdd 7.27699 osd.21 down 1.00000 1.00000 24 hdd 7.27699 osd.24 up 1.00000 1.00000 27 hdd 7.27699 osd.27 down 1.00000 1.00000 30 hdd 7.27699 osd.30 down 1.00000 1.00000 35 hdd 7.27699 osd.35 down 1.00000 1.00000 37 hdd 7.27699 osd.37 down 1.00000 1.00000 40 hdd 7.27699 osd.40 down 1.00000 1.00000 44 hdd 7.27699 osd.44 down 1.00000 1.00000 47 hdd 7.27699 osd.47 up 1.00000 1.00000 50 hdd 7.27699 osd.50 up 1.00000 1.00000 53 hdd 7.27699 osd.53 down 1.00000 1.00000 56 hdd 7.27699 osd.56 down 1.00000 1.00000 59 hdd 7.27699 osd.59 up 1.00000 1.00000 62 hdd 7.27699 osd.62 down 0 1.00000 65 hdd 7.27699 osd.65 down 1.00000 1.00000 68 hdd 7.27699 osd.68 down 1.00000 1.00000 71 hdd 7.27699 osd.71 down 1.00000 1.00000 74 hdd 7.27699 osd.74 down 1.00000 1.00000 77 hdd 7.27699 osd.77 up 1.00000 1.00000 80 hdd 7.27699 osd.80 down 1.00000 1.00000 83 hdd 7.27699 osd.83 up 1.00000 1.00000 86 hdd 7.27699 osd.86 down 1.00000 1.00000 88 hdd 7.27699 osd.88 down 1.00000 1.00000 91 hdd 7.27699 osd.91 down 1.00000 1.00000 94 hdd 7.27699 osd.94 down 1.00000 1.00000 97 hdd 7.27699 osd.97 down 1.00000 1.00000 100 hdd 7.27699 osd.100 down 0 1.00000 103 hdd 7.27699 osd.103 down 1.00000 1.00000 106 hdd 7.27699 osd.106 up 1.00000 1.00000 -3 261.98599 host ceph002 0 hdd 7.27699 osd.0 down 0 1.00000 4 hdd 7.27699 osd.4 up 1.00000 1.00000 7 hdd 7.27699 osd.7 up 1.00000 1.00000 11 hdd 7.27699 osd.11 down 1.00000 1.00000 13 hdd 7.27699 osd.13 up 1.00000 1.00000 16 hdd 7.27699 osd.16 down 1.00000 1.00000 19 hdd 7.27699 osd.19 down 0 1.00000 23 hdd 7.27699 osd.23 up 1.00000 1.00000 26 hdd 7.27699 osd.26 down 0 1.00000 29 hdd 7.27699 osd.29 down 0 1.00000 32 hdd 7.27699 osd.32 down 0 1.00000 33 hdd 7.27699 osd.33 down 0 1.00000 36 hdd 7.27699 osd.36 down 0 1.00000 39 hdd 7.27699 osd.39 down 1.00000 1.00000 43 hdd 7.27699 osd.43 up 1.00000 1.00000 46 hdd 7.27699 osd.46 up 1.00000 1.00000 49 hdd 7.27699 osd.49 down 1.00000 1.00000 52 hdd 7.27699 osd.52 down 1.00000 1.00000 55 hdd 7.27699 osd.55 down 0 1.00000 58 hdd 7.27699 osd.58 up 1.00000 1.00000 61 hdd 7.27699 osd.61 down 1.00000 1.00000 64 hdd 7.27699 osd.64 down 1.00000 1.00000 67 hdd 7.27699 osd.67 up 1.00000 1.00000 70 hdd 7.27699 osd.70 down 1.00000 1.00000 73 hdd 7.27699 osd.73 down 1.00000 1.00000 76 hdd 7.27699 osd.76 up 1.00000 1.00000 78 hdd 7.27699 osd.78 down 1.00000 1.00000 81 hdd 7.27699 osd.81 down 1.00000 1.00000 84 hdd 7.27699 osd.84 down 0 1.00000 87 hdd 7.27699 osd.87 down 1.00000 1.00000 90 hdd 7.27699 osd.90 down 0 1.00000 93 hdd 7.27699 osd.93 down 1.00000 1.00000 96 hdd 7.27699 osd.96 down 0 1.00000 99 hdd 7.27699 osd.99 down 0 1.00000 102 hdd 7.27699 osd.102 down 0 1.00000 105 hdd 7.27699 osd.105 up 1.00000 1.00000 -7 261.98599 host ceph003 2 hdd 7.27699 osd.2 up 1.00000 1.00000 5 hdd 7.27699 osd.5 down 1.00000 1.00000 8 hdd 7.27699 osd.8 up 1.00000 1.00000 10 hdd 7.27699 osd.10 down 0 1.00000 14 hdd 7.27699 osd.14 down 0 1.00000 17 hdd 7.27699 osd.17 up 1.00000 1.00000 20 hdd 7.27699 osd.20 down 0 1.00000 22 hdd 7.27699 osd.22 down 0 1.00000 25 hdd 7.27699 osd.25 up 1.00000 1.00000 28 hdd 7.27699 osd.28 up 1.00000 1.00000 31 hdd 7.27699 osd.31 down 0 1.00000 34 hdd 7.27699 osd.34 down 0 1.00000 38 hdd 7.27699 osd.38 down 0 1.00000 41 hdd 7.27699 osd.41 down 1.00000 1.00000 42 hdd 7.27699 osd.42 down 0 1.00000 45 hdd 7.27699 osd.45 up 1.00000 1.00000 48 hdd 7.27699 osd.48 up 1.00000 1.00000 51 hdd 7.27699 osd.51 down 1.00000 1.00000 54 hdd 7.27699 osd.54 up 1.00000 1.00000 57 hdd 7.27699 osd.57 down 1.00000 1.00000 60 hdd 7.27699 osd.60 down 1.00000 1.00000 63 hdd 7.27699 osd.63 up 1.00000 1.00000 66 hdd 7.27699 osd.66 down 1.00000 1.00000 69 hdd 7.27699 osd.69 up 1.00000 1.00000 72 hdd 7.27699 osd.72 up 1.00000 1.00000 75 hdd 7.27699 osd.75 down 1.00000 1.00000 79 hdd 7.27699 osd.79 up 1.00000 1.00000 82 hdd 7.27699 osd.82 down 1.00000 1.00000 85 hdd 7.27699 osd.85 down 1.00000 1.00000 89 hdd 7.27699 osd.89 down 0 1.00000 92 hdd 7.27699 osd.92 down 1.00000 1.00000 95 hdd 7.27699 osd.95 down 0 1.00000 98 hdd 7.27699 osd.98 down 0 1.00000 101 hdd 7.27699 osd.101 down 1.00000 1.00000 104 hdd 7.27699 osd.104 down 0 1.00000 107 hdd 7.27699 osd.107 up 1.00000 1.00000 Ceph status; [root@ceph001 ~]# ceph status cluster: id: 54052e72-6835-410e-88a9-af4ac17a8113 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 48 osds down Reduced data availability: 2053 pgs inactive, 2043 pgs down, 7 pgs peering, 3 pgs incomplete, 126 pgs stale Degraded data redundancy: 18473/27200783 objects degraded (0.068%), 106 pgs degraded, 103 pgs undersized too many PGs per OSD (258 > max 250) services: mon: 3 daemons, quorum filler001,filler002,bezavrdat-master01 mgr: bezavrdat-master01(active), standbys: filler002, filler001 mds: cephfs-1/1/1 up {0=filler002=up:replay}, 1 up:standby osd: 108 osds: 32 up, 80 in; 16 remapped pgs data: pools: 2 pools, 2176 pgs objects: 2.73 M objects, 1.7 TiB usage: 2.3 TiB used, 580 TiB / 582 TiB avail pgs: 94.347% pgs not active 18473/27200783 objects degraded (0.068%) 1951 down 79 active+undersized+degraded 76 stale+down 23 stale+active+undersized+degraded 14 down+remapped 14 stale+active+clean 6 stale+peering 3 active+clean 3 stale+active+recovery_wait+degraded 2 incomplete 2 stale+down+remapped 1 stale+incomplete 1 stale+remapped+peering 1 active+recovering+undersized+degraded+remapped Thank you in advance! Regards, [Atos logo] Andrea Del Monaco HPC Consultant – Big Data & Security M: +31 612031174 Burgemeester Rijnderslaan 30 – 1185 MC Amstelveen – The Netherlands atos.net<https://atos.net/> [LinkedIn icon]<https://www.linkedin.com/company/1259/> [Twitter icon] <https://twitter.com/atos> [Facebook icon] <https://www.facebook.com/Atos/> [Youtube icon] <https://www.youtube.com/user/Atos> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.

1 1

moving EC pool from HDD to SSD without downtime
by Frank Schilder 01 Oct '19

01 Oct '19

I need to move a 6+2 EC pool from HDDs to SSDs while storage must remain accessible. All SSDs and HDDs are within the same failure domains. The crush rule in question is rule sr-rbd-data-one { id 5 type erasure min_size 3 max_size 8 step set_chooseleaf_tries 50 step set_choose_tries 1000 step take ServerRoom class hdd step chooseleaf indep 0 type host step emit } and I would be inclined just to change the entry "step take ServerRoom class hdd" to "step take ServerRoom class ssd" and wait for the dust to settle. However, this will almost certainly lead to all PGs being undersized and inaccessible as all objects are in the wrong place. I noticed that this is not an issue with PGs created by replicated rules as they can contain more OSDs than the replication factor while objects are moved. The same does not apply to EC rules. I suspect this is due to the setting "max_size 8", which does not allow for more than 6+2=8 OSDs being a member of a PG. What is the correct way to do what I need to do? Can I just set "max_size 16" and go? Will this work with EC rules? If not, what are my options? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

CephFS metadata: Large omap object found
by Eugen Block 01 Oct '19

01 Oct '19

Hi all, we have a new issue in our Nautilus cluster. The large omap warning seems to be more common for RGW usage, but we currently only use CephFS and RBD. I found one thread [1] regarding metadata pool, but it doesn't really help in our case. The deep-scrub of PG 36.6 brought up this message (deep-scrub finished with "ok"): 2019-09-30 20:18:22.548401 osd.9 (osd.9) 275 : cluster [WRN] Large omap object found. Object: 36:654134d2:::mds0_openfiles.0:head Key count: 238621 Size (bytes): 9994510 I checked xattr (none) and omapheader: ceph01:~ # rados -p cephfs-metadata listxattr mds0_openfiles.0 ceph01:~ # rados -p cephfs-metadata getomapheader mds0_openfiles.0 header (42 bytes) : 00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| 00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 74 c3 12 |me v011......t..| 00000020 00 00 00 00 00 01 00 00 00 00 |..........| 0000002a ceph01:~ # ceph fs volume ls [ { "name": "cephfs" } ] The respective OSD has default thresholds regarding large_omap: ceph02:~ # ceph daemon osd.9 config show | grep large_omap "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", Can anyone point me to a solution for this? Best regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033813.html

2 2

Ceph pool capacity question...
by Ilmir Mulyukov 01 Oct '19

01 Oct '19

Hello! We have a small proxmox farm with ceph consisting of three nodes. Each node has 6 disks each with a capacity of 4 TB. A only one pool has been created on these disks. Size 2/1. In theory, this pool should have a capacity: 32.74 TB But the ceph df command returns only: 22.4 TB (USED + MAX AVAIL)(16.7 + 5.7) How to explain this difference? *ceph version is:* 12.2.12-pve1 *ceph df command out:* POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED ala01vf01p01 7 N/A N/A 16.7TiB 74.53 5.70TiB 4411119 4.41M 2.62GiB 887MiB 33.4TiB *crush map:* host n01vf01 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -18 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.0 weight 3.669 item osd.13 weight 3.669 item osd.14 weight 3.669 item osd.15 weight 3.669 item osd.16 weight 3.669 item osd.17 weight 3.669 } host n02vf01 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily id -19 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.1 weight 3.669 item osd.8 weight 3.669 item osd.9 weight 3.669 item osd.10 weight 3.669 item osd.11 weight 3.669 item osd.12 weight 3.669 } host n04vf01 { id -34 # do not change unnecessarily id -35 class hdd # do not change unnecessarily id -36 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.7 weight 3.669 item osd.27 weight 3.669 item osd.24 weight 3.669 item osd.25 weight 3.669 item osd.26 weight 3.669 item osd.28 weight 3.669 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -21 class nvme # do not change unnecessarily # weight 66.042 alg straw2 hash 0 # rjenkins1 item n01vf01 weight 22.014 item n02vf01 weight 22.014 item n04vf01 weight 22.014 } rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit }

2 2