February 2021 - ceph-users

after update to 14.2.16 osd daemons begin to crash

by Boris Behrens

Hi, currently we experience osd daemon crashes and I can't pin the issue. I hope someone can help me with it. * We operate multiple cluster (440 SSD - 1PB, 36 SSD - 126TB, 40SSD 100TB, 84HDD - 680TB) * All clusters were updated around the same time (2021-02-03) * We restarted ALL ceph daemons (systemctl restart ceph.target) on 2021-02-11 after we added OOMScoreAdjust=-900 the all service files. now in our main cluster (440SSD with 1PB) the OSD daemons begin to crash: # ceph crash ls ID ENTITY NEW 2020-03-06_17:37:54.031675Z_0bbbb807-ff2f-46df-9508-58d319b89bd6 osd.397 2020-05-28_12:23:27.677741Z_061f2449-9a36-4747-a2f8-624e72cd1ad0 osd.410 2021-02-05_07:03:35.943384Z_dffab245-4788-4de2-a677-76b735d5fc01 osd.403 2021-02-15_15:41:27.934194Z_97b57f8f-58f2-4390-9d3e-993874e0e000 osd.395 2021-02-15_18:01:19.774879Z_18160e65-4659-451f-8aae-def2984f1f29 osd.178 2021-02-17_04:51:05.101052Z_9f04c6e8-d0c7-442c-9a38-33d5164d2a83 osd.384 osd.384 and osd.395 are on the same node, which had some memory issues we fixed 2021-02-16_12:00:00 osd.384 was marked as out for >24h when the daemon crashed, and there no more misplaced objects in the cluster. Here is the latest crash dump --- begin dump of recent events --- -9999> 2021-02-17 03:31:31.305 7fcf7e136700 1 do_command 'perf dump' 'result is 30067 bytes -9998> 2021-02-17 03:31:31.626 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9997> 2021-02-17 03:31:32.634 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9996> 2021-02-17 03:31:33.639 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9995> 2021-02-17 03:31:34.647 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9994> 2021-02-17 03:31:35.651 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9993> 2021-02-17 03:31:36.654 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9992> 2021-02-17 03:31:37.657 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9991> 2021-02-17 03:31:38.676 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9990> 2021-02-17 03:31:39.680 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9989> 2021-02-17 03:31:40.684 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9988> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' ' -9987> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' 'result is 30067 bytes <snip> -31> 2021-02-17 05:50:41.158 7fcf7e136700 1 do_command 'perf dump' ' -30> 2021-02-17 05:50:41.159 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -29> 2021-02-17 05:50:41.804 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -28> 2021-02-17 05:50:42.813 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -27> 2021-02-17 05:50:43.820 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -26> 2021-02-17 05:50:44.825 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -25> 2021-02-17 05:50:45.831 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -24> 2021-02-17 05:50:46.837 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -23> 2021-02-17 05:50:47.840 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -22> 2021-02-17 05:50:48.843 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -21> 2021-02-17 05:50:49.847 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -20> 2021-02-17 05:50:50.853 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -19> 2021-02-17 05:50:51.524 7fcf7e136700 1 do_command 'perf dump' ' -18> 2021-02-17 05:50:51.525 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -17> 2021-02-17 05:50:51.859 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -16> 2021-02-17 05:50:52.862 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -15> 2021-02-17 05:50:53.871 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -14> 2021-02-17 05:50:54.875 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -13> 2021-02-17 05:50:55.886 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -12> 2021-02-17 05:50:56.891 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -11> 2021-02-17 05:50:57.905 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -10> 2021-02-17 05:50:58.911 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9> 2021-02-17 05:50:59.917 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -8> 2021-02-17 05:51:00.929 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -7> 2021-02-17 05:51:01.566 7fcf7e136700 1 do_command 'perf dump' ' -6> 2021-02-17 05:51:01.567 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -5> 2021-02-17 05:51:01.935 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -4> 2021-02-17 05:51:02.943 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -3> 2021-02-17 05:51:03.949 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -2> 2021-02-17 05:51:04.967 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -1> 2021-02-17 05:51:05.091 7fcf743e7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h: In function 'uint64_t AllocatorLevel02<T>::claim_free_to_right(uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fcf743e7700 time 2021-02-17 05:51:04.998475 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h: 572: FAILED ceph_assert(available >= allocated) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x561c84cc2c7d] 2: (()+0x4d8e45) [0x561c84cc2e45] 3: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e) [0x561c853167de] 4: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x60) [0x561c85310b20] 5: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x3a) [0x561c853143ca] 6: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f) [0x561c851ee83f] 7: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be) [0x561c8522f4ae] 8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa) [0x561c8522fe9a] 9: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4] 10: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed] 11: (()+0x7ea5) [0x7fcf840a2ea5] 12: (clone()+0x6d) [0x7fcf82f6596d] 0> 2021-02-17 05:51:05.145 7fcf743e7700 -1 *** Caught signal (Aborted) ** in thread 7fcf743e7700 thread_name:bstore_kv_final ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (()+0xf630) [0x7fcf840aa630] 2: (gsignal()+0x37) [0x7fcf82e9d387] 3: (abort()+0x148) [0x7fcf82e9ea78] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x561c84cc2ccc] 5: (()+0x4d8e45) [0x561c84cc2e45] 6: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e) [0x561c853167de] 7: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x60) [0x561c85310b20] 8: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x3a) [0x561c853143ca] 9: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f) [0x561c851ee83f] 10: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be) [0x561c8522f4ae] 11: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa) [0x561c8522fe9a] 12: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4] 13: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed] 14: (()+0x7ea5) [0x7fcf840a2ea5] 15: (clone()+0x6d) [0x7fcf82f6596d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 years, 9 months

2
3
0 0

Ceph @ DevConf.CZ

by Mike Perez

Hi everyone, Ceph will be present at DevConf.CZ, February 18-20 in a joint booth with the Rook Community! https://www.devconf.cz If you're interested in more information about being present at the booth to provide expertise/content/presentations to our audience, please let me know privately. -- Mike Perez

2 years, 9 months

1
0
0 0

best use of NVMe drives

by Magnus HAGDORN

Hi there, we are in the process of growing our Nautilus ceph cluster. Currently, we have 6 nodes, 3 nodes with 2×5.5TB, 6x11TB disks and 8x186GB SSD and 3 nodes with 6×5.5TB and 6×7.5TB disks. All with dual link 10GE NICs. The SSDs are used for the CephFS metadata pool, the hard drives are used for the CephFS data pool. All OSD journals are kept on the drives themselves. Replication level is 3 for both data and metadata pools. The new servers have 12x12TB disks and 1 1.5TB NVMe drive. We expect to get another 3 similar nodes in the near future. My question is what is the most sensible thing to do with the NVMe drives. I would like to increase the replication level of the metadata pool. So my idea was to split the NVMes into say 4 partitions and add them to the metadata pool. Given the size of the drives and the metadata pool usage (~35GB) that seems overkill. Would it make sense to partition the drives further and stick the OSD journals on the NVMEs? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 years, 9 months

2
1
0 0

SUSE POC - Dead in the water

by Schweiss, Chip

For the past several months I had been building a sizable Ceph cluster that will be up to 10PB with between 20 and 40 OSD servers this year. A few weeks ago I was informed that SUSE is shutting down SES and will no longer be selling it. We haven't licensed our proof of concept cluster that is currently at 14 OSD nodes, but it looks like SUSE is not going to be the answer here. I'm seeking recommendations for consulting help on this project since SUSE has let me down. I have Ceph installed and operating, however, I've been struggling with getting the pool configured properly for CephFS and getting very poor performance. The OSD servers have TLC NVMe for DB, and Optane NVMe for WAL, so I should be seeing decent performance with the current cluster. I'm not opposed to completely switching OS distributions. Ceph on SUSE was our first SUSE installation. Almost everything else we run is on CentOS, but that may change thanks to IBM cannibalizing CentOS. Please reach out to me if you can recommend someone to sell us consulting hours and/or a support contract. -Chip Schweiss chip.schweiss(a)wustl.edu Washington University School of Medicine

2 years, 9 months

4
6
0 0

share haproxy config for radosgw

by Marc

I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.

2 years, 9 months

8
10
0 0

Consequences of setting bluestore_fsck_quick_fix_on_mount to false?

by Matthew Vernon

Hi, Looking at the Octopus upgrade instructions, I see "the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data)." and that I can disable this by setting bluestore_fsck_quick_fix_on_mount to false. A couple of questions about this: i) what are the consequences of turning off this "quick fix"? Is it possible to have it run in the background or similar? ii) is there any way to narrow down the time estimate? Our production cluster has 3060 OSDs on hdd (with block.db on NVME), and obviously 3000 lots of "a few hours" is an awful lot of time... I'll be doing some testing on our test cluster (by putting 10M objects into an S3 bucket before trying the upgrade), but it'd be useful to have some idea of how this is likely to work at scale... Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

2 years, 9 months

2
2
0 0

Upgrading Ceph luminous to mimic on debian-buster

by Jean-Marc FONTANA

Hello everyone, We just installed a Ceph cluster version luminous (12.2.11) on servers working with Debian buster (10.8) using ceph-deploy and we are trying to upgrade it to mimic but can't find a way to do it. We tried ceph-deploy install --release mimic mon1 mon2 mon3 (after having modified /etc/apt/sources.list.d/ceph.list) but this does nothing because the packets are said to be up to date. Could someone help us, please ? Best regards

2 years, 9 months

2
1
0 0

osds processes shutdown during outage

by Marcel Kuiper

Hi, (sorry if this gets posted twice. I forgot a subject in the first mail) We expereinced an outage this morning on a jewel cluster with 1559 osds. It appeared that a switch uplink in a rack misbehaved and once shutting that interface ceph health restored quickly. I have some questions though on osd behaviour that I hope someone can answer 1 - In a lot of osd logs I saw that neighbours reported the osd down (while the process was still running and obviously logging). Then after a while the logs shows * Got signal Interrupt * prepare_to_stop starting shutdown and the osd process stops Why does the osd proces stop? Is it instructed to do so by the monitor because neighbours reported it down and ceph wants to avoid flapping? 2 - The osds reported a lot of * heartbeat_check: no reply from #ip:#port When I telnet to the ip and port I get a connection just fine. Is there a way to run a heartbeat_check from the commandline so that we can try capture the traffic to determine why it fails Thanks Marcel

2 years, 9 months

1
0
0 0

struggling to achieve high bandwidth on Ceph dev cluster - HELP

by Bobby

Hi, Hello I am using rados bench tool. Currently I am using this tool on the development cluster after running vstart.sh script. It is working fine and I am interested in benchmarking the cluster. However I am struggling to achieve a good bandwidth i.e. bandwidth (MB/sec). My target throughput is at least 50 MB/sec and more. But mostly I am achieving is around 15-20 MB/sec. So, very poor. I am quite sure I am missing something. Either I have to change my cluster through vstart.sh script or I am not fully utilizing the rados bench tool. Or may be both. i.e. not the right cluster and also not using the rados bench tool correctly. Some of the shell examples I have been using to build the cluster are bellow: MDS=0 RGW=1 ../src/vstart.sh -d -l -n --bluestore MDS=0 RGW=1 MON=1 OSD=4../src/vstart.sh -d -l -n --bluestore While using rados bench tool I have been trying with different block sizes 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K. And I have also been changing the -t parameter in the shell to increase concurrent IOs. Looking forward to help. Bobby

2 years, 9 months

3
9
0 0

POC Hardware questions

by Oliver Weinmann

Dear All, A questions that probalby has been asked by many other users before. I want to do a POC. For the POC I can use old decomissioned hardware. Currently I have 3 x IBM X3550 M5 with: 1 Dualport 10G NIC Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz 64GB RAM the other two have a slower CPU but more RAM: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz 512GB RAM Of course I can re-arrange the RAM. The switches are not LACP capable, so I'm planning to use bonding in active-active. For the disks I'm planning on buying 12 x Samsung PM883 1.9TB and use them in an EC pool. My questions are: 1. Which bonding mode should I choose? balance-alb? 2. Are the disks ok for a POC? Or should I rather go with more smaller disks (960GB) e.g. 24 in total? 3. Are there any drawbacks when using EC pools? Workload will be mostly VMs (Vsphere / Openstack), but also CephFS with Samba Gateway. Many thanks

2 years, 9 months

2
1
0 0

2023

2022

2021

2020

2019

ceph-users February 2021