- ceph-users - lists.ceph.io

RadosGW - Performance Expectations
by Shawn Weeks 10 Feb '23

10 Feb '23

Good morning everyone, been running a small Ceph cluster with Proxmox for a while now and I’ve finally run across an issue I can’t find any information on. I have a 3 node cluster with 9 Samsung PM983 960GB NVME drives running on a dedicated 10gb network. RBD and CephFS performance have been great, most of the time I see over 500MBs writes and a rados benchmark shows 951 MB/s write and 1140 MB/s read bandwidth. The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at around 25MBs with the official AWS CLI. Using s3cmd is slightly better at around 45MB/s. I’m going directly to the RadosGW instance with no load balancers in between and no ssl enabled. Just trying to figure out if this is normal. I’m not expecting it to be as fast as writing directly to a RBD but I was kinda hoping for more than this. So what should I expect in performance from the RadosGW? Here are some rados bench results and my ceph report https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5 https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438 Thanks Shawn

4 5

No such file or directory when issuing "rbd du"
by Mehmet 10 Feb '23

10 Feb '23

Hello Friends, i have a strange output when issuing following command root@node35:~# rbd du -p cephhdd-001-mypool NAME PROVISIONED USED ... vm-99936587-disk-0@H202302091535 400 GiB 5.2 GiB vm-99936587-disk-0@H202302091635 400 GiB 1.2 GiB vm-99936587-disk-0 400 GiB 732 MiB vm-9999104-cloudinit 4 MiB 4 MiB vm-9999104-disk-0 600 GiB 586 GiB <TOTAL> 49 TiB 44 TiB rbd: du failed: (2) No such file or directory root@node35:~# I do not know why i receive "rbd: du failed: (2) No such file or directory". How can i find the origin for this? My Ceph-Version 17.2.3 installed with "cephadm". Cluster is "HEALTH_OK" with 108 OSDs distributed over 3 Nodes where mgr/mon also resides. Hope you can help Mehmet

2 2

Yet another question about OSD memory usage ...
by Ulrich Klein 10 Feb '23

10 Feb '23

Hi, Yet another question about OSD memory usage ... I have a test cluster running. When I do a ceph orch ps I see for my osd.11: ceph orch ps --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.11 ceph01 running (2h) 97s ago 2h 23.0G 13.1G 17.2.5 cc65afd6173a 5d1062e8d392 When I chek via top on the machine I see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 39807 ceph 20 0 6254956 3.7g 9228 S 31.2 3.0 846:21.63 /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-lo+ Now, where does ceph orch ps get those 23.0G from, when top just shows 3.7G resident and 6.2G virtual for osd.11? (I do understand that the MEM LIM n the ceph orch ps list is not really the limit) Anyone know where that discrepancy comes from? Ciao, Uli

1 0

OSD fail to authenticate after node outage
by tsmgeek＠gmail.com 10 Feb '23

10 Feb '23

Release: 16.2.7 (pacific) Infra: 4 x Nodes (4xOSD HDD), 3 x Nodes (mon/mds, 1 x OSD NVMe) We recently had a couple of node which went offline unexpectedly triggering a rebalance which is still ongoing. The OSDs on the restarted node are marked as down and they keep showing in the log `authenticated timed out`, after a period of time they get marked `autoout`. We tried setting `noout` on the cluster which has stopped them being marked out but they still never authenticate. We can access all the ceph tooling from those nodes which indicates connection to mons. The node keyring/time are both in sync. We are at a loss to why we can not get the OSDs to authenticate. Any help would be apreciated. ``` cluster: id: d5126e5a-882e-11ec-954e-90e2baec3d2c health: HEALTH_WARN 7 failed cephadm daemon(s) 2 stray daemon(s) not managed by cephadm insufficient standby MDS daemons available nodown,noout flag(s) set 8 osds down 2 hosts (8 osds) down Degraded data redundancy: 195930251/392039621 objects degraded (49.977%), 160 pgs degraded, 160 pgs undersized 2 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph5,ceph7,ceph6 (age 38h) mgr: ceph2.tofizp(active, since 9M), standbys: ceph1.vnkagp mds: 3/3 daemons up osd: 19 osds: 11 up (since 38h), 19 in (since 45h); 5 remapped pgs flags nodown,noout data: volumes: 1/1 healthy pools: 6 pools, 257 pgs objects: 102.94M objects, 67 TiB usage: 68 TiB used, 50 TiB / 118 TiB avail pgs: 195930251/392039621 objects degraded (49.977%) 3205811/392039621 objects misplaced (0.818%) 155 active+undersized+degraded 97 active+clean 3 active+undersized+degraded+remapped+backfill_wait 2 active+undersized+degraded+remapped+backfilling io: client: 511 B/s rd, 102 KiB/s wr, 0 op/s rd, 2 op/s wr recovery: 13 MiB/s, 16 objects/s ```

2 1

Corrupt bluestore after sudden reboot (17.2.5)
by Peter van Heusden 09 Feb '23

09 Feb '23

Due to the ongoing South African energy crisis <https://en.wikipedia.org/wiki/South_African_energy_crisis> our datacenter experienced sudden power loss. We are running ceph 17.2.5 deployed with cephadm. Two of our OSDs did not start correctly, with the error: # ceph-bluestore-tool fsck --path /var/lib/ceph/ed7b2c16-b053-45e2-a1fe-bf3474f90508/osd.27/ 2023-01-15T08:38:04.289+0200 7f2a2a03c540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f2a2a03c540 time 2023-01-15T08:39:31.304968+0200 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: 18968: FAILED ceph_assert(collection_ref) 2023-01-15T08:39:31.298+0200 7f2a2a03c540 -1 bluestore::NCB::read_allocation_from_onodes::stray object 2#55:ffffffff:::2000055f327.00002287:head# not owned by any collection ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f2a2acc07c6] 2: /usr/lib/ceph/libceph-common.so.2(+0x27c9d8) [0x7f2a2acc09d8] 3: (BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0xa24) [0x560d6baf5754] 4: (BlueStore::reconstruct_allocations(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0x5f) [0x560d6baf66ff] 5: (BlueStore::read_allocation_from_drive_on_startup()+0x99) [0x560d6baf68b9] 6: (BlueStore::_init_alloc(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >*)+0xaca) [0x560d6bb0c15a] 7: (BlueStore::_open_db_and_around(bool, bool)+0x35c) [0x560d6bb380dc] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x250) [0x560d6bb3a8c0] 9: main() 10: __libc_start_main() 11: _start() *** Caught signal (Aborted) ** in thread 7f2a2a03c540 thread_name:ceph-bluestore- 2023-01-15T08:39:31.306+0200 7f2a2a03c540 -1 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f2a2a03c540 time 2023-01-15T08:39:31.304968+0200 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: 18968: FAILED ceph_assert(collection_ref) (complete log https://gist.github.com/pvanheus/5c57455cacdc91afc9ce27fd489cae25) Is there a way to recover from this? Or should I accept the OSDs as lost and rebuild them? Thanks, Peter

2 2

Throttle down rebalance with Quincy
by Victor Rodriguez 09 Feb '23

09 Feb '23

Hello, I'm adding OSDs to a 5 node cluster using Quincy 17.2.5. The network is a bonded 2x10G link. The issue I'm having is that the rebalance operation seems to impact client I/O and running VMs do not . OSDs are big 6'4TB NVMe drives, so there will be a lot of data to move. With previous versions it was easy to throttle down the rebalance with "ceph config set osd osd_max_backfills", but as Quincy uses MClock, those values are not used. In fact, default values are overridden with 1000. If I'm understanding the MClock behavior it will use the estimated osd_mclock_max_capacity_iops_ssd (benchmarked at OSD deploy time) and allow client/rebalance/backfill/trims/scrubs I/O to fill the drive with IOPs up to what is defined in osd_mclock_profile (default value is high_client_ops). Am I correct? How could I throttle down the rebalance so it gives more headroom for client I/O? Many thanks in advance. --

1 0

OSD logs missing from Centralised Logging
by Peter van Heusden 09 Feb '23

09 Feb '23

Hi there I am running Ceph version 17.2.5 and have deployed centralised logging as per this guide: https://ceph.io/en/news/blog/2022/centralized_logging/ The logs from the OSDs are not, however, showing up in the Grafana dashboard, as per this screenshot: [image: image.png] The Promtail daemons are running on each node including the OSDs. The Loki server and Grafana are running on one of our monitor nodes. Thanks for any clarifications you can provide. Peter

2 2

Deep scrub debug option
by Broccoli Bob 09 Feb '23

09 Feb '23

3 3

Permanently ignore some warning classes
by Nicola Mori 09 Feb '23

09 Feb '23

Dear Ceph users, my cluster is build with old hardware on a gigabit network, so I often experience warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers alert mails too often, forcing me to disable alerts which is not sustainable. So my question is: is it possible to tell Ceph to ignore (or at least to not send alerts for) a given class of warnings? Thank you, Nicola

2 5

Is autoscaler doing the right thing?
by Kyriazis, George 08 Feb '23

08 Feb '23

Hello ceph community, I have some questions about the pg autoscaler. I have a cluster with several pools. One of them is a cephfs pool which is behaving in an expected / sane way, and another is a RBD pool with an ec profile of k=2, m=2. The cluster has about 60 drives across across about 10 failure domains. (Failure domain is set to “chassis”, and there are some chassis with 4 blades per chassis, and the rest with 1 host per chassis). The rbd ec pool has 66TiB stored with 128 PGs. Each PG has about 500k objects in them, which seems like quite a lot. When rebalancing, this EC pool is always the longpole. The confusing part is that I am getting inconsistent output on the status on the autoscaler. For example: root@vis-mgmt:~# ceph osd pool autoscale-status | grep rbd_ec rbd_ec 67241G 2.0 856.7T 0.1533 1.0 64 on False Which tells me I have 64 PG_NUM (a lie). root@vis-mgmt:~# ceph osd pool ls detail | grep rbd_ec pool 4 'rbd_ec' erasure profile ec22 size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 120 pg_num_target 64 pgp_num_target 64 autoscale_mode on last_change 83396 lfor 0/83395/83393 flags hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 8192 application rbd Tells me I have 128 PGs (correct), but with a pgp_num which is not a power of 2 (120 pgs). Also, I am not sure what the pg_num_target and pgp_num_raget are and why they are different from pg_num and pgp_num. Is there anything I can look into to find out is the autoscaler is working correctly for this pool? Any other tweaks I need to do? Seems to me that with that capacity it ought to have more than 128 PGs… Thank you! George

1 0