February 2023 - ceph-users

RadosGW - Performance Expectations
by Shawn Weeks 10 Feb '23

10 Feb '23

Good morning everyone, been running a small Ceph cluster with Proxmox for a while now and I’ve finally run across an issue I can’t find any information on. I have a 3 node cluster with 9 Samsung PM983 960GB NVME drives running on a dedicated 10gb network. RBD and CephFS performance have been great, most of the time I see over 500MBs writes and a rados benchmark shows 951 MB/s write and 1140 MB/s read bandwidth. The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at around 25MBs with the official AWS CLI. Using s3cmd is slightly better at around 45MB/s. I’m going directly to the RadosGW instance with no load balancers in between and no ssl enabled. Just trying to figure out if this is normal. I’m not expecting it to be as fast as writing directly to a RBD but I was kinda hoping for more than this. So what should I expect in performance from the RadosGW? Here are some rados bench results and my ceph report https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5 https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438 Thanks Shawn

4 5

No such file or directory when issuing "rbd du"
by Mehmet 10 Feb '23

10 Feb '23

Hello Friends, i have a strange output when issuing following command root@node35:~# rbd du -p cephhdd-001-mypool NAME PROVISIONED USED ... vm-99936587-disk-0@H202302091535 400 GiB 5.2 GiB vm-99936587-disk-0@H202302091635 400 GiB 1.2 GiB vm-99936587-disk-0 400 GiB 732 MiB vm-9999104-cloudinit 4 MiB 4 MiB vm-9999104-disk-0 600 GiB 586 GiB <TOTAL> 49 TiB 44 TiB rbd: du failed: (2) No such file or directory root@node35:~# I do not know why i receive "rbd: du failed: (2) No such file or directory". How can i find the origin for this? My Ceph-Version 17.2.3 installed with "cephadm". Cluster is "HEALTH_OK" with 108 OSDs distributed over 3 Nodes where mgr/mon also resides. Hope you can help Mehmet

2 2

Yet another question about OSD memory usage ...
by Ulrich Klein 10 Feb '23

10 Feb '23

Hi, Yet another question about OSD memory usage ... I have a test cluster running. When I do a ceph orch ps I see for my osd.11: ceph orch ps --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.11 ceph01 running (2h) 97s ago 2h 23.0G 13.1G 17.2.5 cc65afd6173a 5d1062e8d392 When I chek via top on the machine I see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 39807 ceph 20 0 6254956 3.7g 9228 S 31.2 3.0 846:21.63 /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-lo+ Now, where does ceph orch ps get those 23.0G from, when top just shows 3.7G resident and 6.2G virtual for osd.11? (I do understand that the MEM LIM n the ceph orch ps list is not really the limit) Anyone know where that discrepancy comes from? Ciao, Uli

1 0

OSD fail to authenticate after node outage
by tsmgeek＠gmail.com 10 Feb '23

10 Feb '23

Release: 16.2.7 (pacific) Infra: 4 x Nodes (4xOSD HDD), 3 x Nodes (mon/mds, 1 x OSD NVMe) We recently had a couple of node which went offline unexpectedly triggering a rebalance which is still ongoing. The OSDs on the restarted node are marked as down and they keep showing in the log `authenticated timed out`, after a period of time they get marked `autoout`. We tried setting `noout` on the cluster which has stopped them being marked out but they still never authenticate. We can access all the ceph tooling from those nodes which indicates connection to mons. The node keyring/time are both in sync. We are at a loss to why we can not get the OSDs to authenticate. Any help would be apreciated. ``` cluster: id: d5126e5a-882e-11ec-954e-90e2baec3d2c health: HEALTH_WARN 7 failed cephadm daemon(s) 2 stray daemon(s) not managed by cephadm insufficient standby MDS daemons available nodown,noout flag(s) set 8 osds down 2 hosts (8 osds) down Degraded data redundancy: 195930251/392039621 objects degraded (49.977%), 160 pgs degraded, 160 pgs undersized 2 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph5,ceph7,ceph6 (age 38h) mgr: ceph2.tofizp(active, since 9M), standbys: ceph1.vnkagp mds: 3/3 daemons up osd: 19 osds: 11 up (since 38h), 19 in (since 45h); 5 remapped pgs flags nodown,noout data: volumes: 1/1 healthy pools: 6 pools, 257 pgs objects: 102.94M objects, 67 TiB usage: 68 TiB used, 50 TiB / 118 TiB avail pgs: 195930251/392039621 objects degraded (49.977%) 3205811/392039621 objects misplaced (0.818%) 155 active+undersized+degraded 97 active+clean 3 active+undersized+degraded+remapped+backfill_wait 2 active+undersized+degraded+remapped+backfilling io: client: 511 B/s rd, 102 KiB/s wr, 0 op/s rd, 2 op/s wr recovery: 13 MiB/s, 16 objects/s ```

2 1

Corrupt bluestore after sudden reboot (17.2.5)
by Peter van Heusden 09 Feb '23

09 Feb '23

Due to the ongoing South African energy crisis <https://en.wikipedia.org/wiki/South_African_energy_crisis> our datacenter experienced sudden power loss. We are running ceph 17.2.5 deployed with cephadm. Two of our OSDs did not start correctly, with the error: # ceph-bluestore-tool fsck --path /var/lib/ceph/ed7b2c16-b053-45e2-a1fe-bf3474f90508/osd.27/ 2023-01-15T08:38:04.289+0200 7f2a2a03c540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f2a2a03c540 time 2023-01-15T08:39:31.304968+0200 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: 18968: FAILED ceph_assert(collection_ref) 2023-01-15T08:39:31.298+0200 7f2a2a03c540 -1 bluestore::NCB::read_allocation_from_onodes::stray object 2#55:ffffffff:::2000055f327.00002287:head# not owned by any collection ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f2a2acc07c6] 2: /usr/lib/ceph/libceph-common.so.2(+0x27c9d8) [0x7f2a2acc09d8] 3: (BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0xa24) [0x560d6baf5754] 4: (BlueStore::reconstruct_allocations(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0x5f) [0x560d6baf66ff] 5: (BlueStore::read_allocation_from_drive_on_startup()+0x99) [0x560d6baf68b9] 6: (BlueStore::_init_alloc(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >*)+0xaca) [0x560d6bb0c15a] 7: (BlueStore::_open_db_and_around(bool, bool)+0x35c) [0x560d6bb380dc] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x250) [0x560d6bb3a8c0] 9: main() 10: __libc_start_main() 11: _start() *** Caught signal (Aborted) ** in thread 7f2a2a03c540 thread_name:ceph-bluestore- 2023-01-15T08:39:31.306+0200 7f2a2a03c540 -1 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f2a2a03c540 time 2023-01-15T08:39:31.304968+0200 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: 18968: FAILED ceph_assert(collection_ref) (complete log https://gist.github.com/pvanheus/5c57455cacdc91afc9ce27fd489cae25) Is there a way to recover from this? Or should I accept the OSDs as lost and rebuild them? Thanks, Peter

2 2

Throttle down rebalance with Quincy
by Victor Rodriguez 09 Feb '23

09 Feb '23

Hello, I'm adding OSDs to a 5 node cluster using Quincy 17.2.5. The network is a bonded 2x10G link. The issue I'm having is that the rebalance operation seems to impact client I/O and running VMs do not . OSDs are big 6'4TB NVMe drives, so there will be a lot of data to move. With previous versions it was easy to throttle down the rebalance with "ceph config set osd osd_max_backfills", but as Quincy uses MClock, those values are not used. In fact, default values are overridden with 1000. If I'm understanding the MClock behavior it will use the estimated osd_mclock_max_capacity_iops_ssd (benchmarked at OSD deploy time) and allow client/rebalance/backfill/trims/scrubs I/O to fill the drive with IOPs up to what is defined in osd_mclock_profile (default value is high_client_ops). Am I correct? How could I throttle down the rebalance so it gives more headroom for client I/O? Many thanks in advance. --

1 0

OSD logs missing from Centralised Logging
by Peter van Heusden 09 Feb '23

09 Feb '23

Hi there I am running Ceph version 17.2.5 and have deployed centralised logging as per this guide: https://ceph.io/en/news/blog/2022/centralized_logging/ The logs from the OSDs are not, however, showing up in the Grafana dashboard, as per this screenshot: [image: image.png] The Promtail daemons are running on each node including the OSDs. The Loki server and Grafana are running on one of our monitor nodes. Thanks for any clarifications you can provide. Peter

2 2

Deep scrub debug option
by Broccoli Bob 09 Feb '23

09 Feb '23

3 3

Permanently ignore some warning classes
by Nicola Mori 09 Feb '23

09 Feb '23

Dear Ceph users, my cluster is build with old hardware on a gigabit network, so I often experience warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers alert mails too often, forcing me to disable alerts which is not sustainable. So my question is: is it possible to tell Ceph to ignore (or at least to not send alerts for) a given class of warnings? Thank you, Nicola

2 5

Is autoscaler doing the right thing?
by Kyriazis, George 09 Feb '23

09 Feb '23

Hello ceph community, I have some questions about the pg autoscaler. I have a cluster with several pools. One of them is a cephfs pool which is behaving in an expected / sane way, and another is a RBD pool with an ec profile of k=2, m=2. The cluster has about 60 drives across across about 10 failure domains. (Failure domain is set to “chassis”, and there are some chassis with 4 blades per chassis, and the rest with 1 host per chassis). The rbd ec pool has 66TiB stored with 128 PGs. Each PG has about 500k objects in them, which seems like quite a lot. When rebalancing, this EC pool is always the longpole. The confusing part is that I am getting inconsistent output on the status on the autoscaler. For example: root@vis-mgmt:~# ceph osd pool autoscale-status | grep rbd_ec rbd_ec 67241G 2.0 856.7T 0.1533 1.0 64 on False Which tells me I have 64 PG_NUM (a lie). root@vis-mgmt:~# ceph osd pool ls detail | grep rbd_ec pool 4 'rbd_ec' erasure profile ec22 size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 120 pg_num_target 64 pgp_num_target 64 autoscale_mode on last_change 83396 lfor 0/83395/83393 flags hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 8192 application rbd Tells me I have 128 PGs (correct), but with a pgp_num which is not a power of 2 (120 pgs). Also, I am not sure what the pg_num_target and pgp_num_raget are and why they are different from pg_num and pgp_num. Is there anything I can look into to find out is the autoscaler is working correctly for this pool? Any other tweaks I need to do? Seems to me that with that capacity it ought to have more than 128 PGs… Thank you! George

1 0

Adding osds to each nodes
by Szabo, Istvan (Agoda) 08 Feb '23

08 Feb '23

Hi, What is the safest way to add disk(s) to each of the node in the cluster? Should it be done 1 by 1 or can add all of them at once and let it rebalance? My concern is that if add all in one due to host based EC code it will block all the host. The other side if I add 1 by 1, one node will have more performance and more osds than the others which is also not a good setup, so wonder which is the safer way? (have 9 nodes with host based EC 4:2, 1 disk is going to have 4osds) Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

RGW archive zone lifecycle
by Ondřej Kukla 08 Feb '23

08 Feb '23

Hi, I have two Ceph clusters in a multi-zone setup. The first one (master zone) would be accessible to users for their interaction using RGW. The second one is set to sync from the master zone with the tier type of the zone set as an archive (to version all files). My question here is. Is there an option to set a lifecycle for the version files saved on the archive zone? For example, keep only 5 versions per file or delete version files older than one year? Thanks a lot.

2 1

Replacing OSD with containerized deployment
by mailing-lists 08 Feb '23

08 Feb '23

Dear Ceph-Users, i am struggling to replace a disk. My ceph-cluster is not replacing the old OSD even though I did: ceph orch osd rm 232 --replace The OSD 232 is still shown in the osd list, but the new hdd will be placed as a new OSD. This wouldnt mind me much, if the OSD was also placed on the bluestoreDB / NVME, but it doesn't. My steps: "ceph orch osd rm 232 --replace" remove the failed hdd. add the new one. Convert the disk within the servers bios, so that the node can have direct access on it. It shows up as /dev/sdt, enter maintenance mode reboot server drive is now /dev/sdm (which the old drive had) "ceph orch device zap node-x /dev/sdm " A new OSD is placed on the cluster. Can you give me a hint, where did I take a wrong turn? Why is the disk not being used as OSD 232? Best Ken

3 16

Cephalocon 2023 Amsterdam CFP ENDS in Less Than Five Days
by Mike Perez 07 Feb '23

07 Feb '23

Upcoming Events ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ [image: Logo] <https://ceph.io> Upcoming Events [image: Ceph23-LaunchAssets-v1_snackable-1200x627-A.png] <https://ceph.io/en/community/events/2023/cephalocon-amsterdam/> Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members worldwide to learn more about Ceph and the project's future from the developers writing the code and the operators deploying it at scale. You can take advantage of early bird registration <https://events.linuxfoundation.org/cephalocon/register/> pricing rates before *February 26th*. Does your company want to show support for the Ceph community, be recognized, and reach an audience of passionate Ceph users? You can review the sponsorship prospectus <https://events.linuxfoundation.org/cephalocon/sponsor/> and contact sponsorships(a)ceph.io to discuss your custom sponsorship package. *The CFP ends in 5 days - February 12th at 23:59 PST.* *All talks are in-person.* Cephalocon Event Page <https://ceph.io/en/community/events/2023/cephalocon-amsterdam/> Upcoming Ceph Days [image: ceph-days-socal.png] <https://ceph.io/en/community/events/2023/ceph-days-socal/> Ceph Days Southern California <https://ceph.io/en/community/events/2023/ceph-days-socal/> Ceph is coming to Pasadena, CA, co-located with SCALE, on *March 9th*. Registration is open, and the CFP has been extended to *February 9th | 2 DAYS LEFT.* [image: Ceph Days Webpage Banner.png] <https://ceph.io/en/community/events/2023/ceph-days-nyc/> Ceph Days New York City <https://ceph.io/en/community/events/2023/ceph-days-nyc/> Join us in the big city for a big event you want to attend on *February 21st*. Hear from Ceph users providing massively scalable storage clusters and how they manage them. Thank you to our sponsor Tech At Bloomberg <https://www.bloomberg.com/company/values/tech-at-bloomberg/>! [image: Link icon] <https://ceph.io> [image: Twitter icon] <https://twitter.com/ceph> [image: YouTube icon] <https://youtube.com/c/cephstorage> [image: LinkedIn icon] <https://www.linkedin.com/company/ceph/> [image: Facebook icon] <https://www.facebook.com/cephstorage/> z7bx26gewrsc

1 0

Nautilus to Octopus when RGW already on Octopus
by r.burrowes＠auckland.ac.nz 07 Feb '23

07 Feb '23

We are finally going to upgrade our Ceph from Nautilus to Octopus, before looking at moving onward. We are still on Ubuntu 18.04, so once on Octopus, we will then upgrade the OS to 20.04, ready for the next upgrade. Unfortunately, we have already upgraded our rados gateways to Ubuntu 20.04, last Sept, which had the side effect of upgrading the RGWs to Octopus. So I'm looking to downgrade the rados gateways, back to Nautilus, just to be safe. We can then do the upgrade in the right order. I have no idea if the newer Octopus rados gateways will have altered any metadata, that would affect a downgrade back to Nautilus. Any advise.

3 3

ceph-fuse in infinite loop reading objects without client requests
by Andras Pataki 07 Feb '23

07 Feb '23

We've been running into a strange issue with ceph-fuse on some nodes lately. After some job runs on the node (and finishes or gets killed), ceph-fuse gets stuck busy requesting objects from the OSDs without any processes on the node using cephfs. When this happens, ceph-fuse uses 2-3 cores, spinning in what seems like an infinite loop making objecter requests to various OSDs of files that were perhaps requested by some process that is long gone. The same object gets requested every few seconds cycling through a list of objects. This close to saturates the traffic in the 25Gbps NIC of the node. I have a one minute log file with debug_objecter/objectcacher/client/context/finisher set to 20 that traces this, but I don't see why this is happening exactly. Can someone with a better understanding on the client->cache->OSD flow share some insights into what is going wrong here? Here is an example: 2023-02-03T14:42:47.593-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42777727 osd.4349 2023-02-03T14:43:00.056-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42778921 osd.4349 2023-02-03T14:43:02.048-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42779104 osd.4349 2023-02-03T14:43:05.392-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42779408 osd.4349 2023-02-03T14:43:10.076-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42779847 osd.4349 2023-02-03T14:43:13.288-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42780156 osd.4349 2023-02-03T14:43:18.908-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42780697 osd.4349 2023-02-03T14:43:29.056-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42781660 osd.4349 2023-02-03T14:43:33.707-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42782079 osd.4349 2023-02-03T14:42:41.609-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42777194 osd.3251 2023-02-03T14:42:49.809-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42777954 osd.3251 2023-02-03T14:43:07.884-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42779646 osd.3251 2023-02-03T14:43:16.736-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42780500 osd.3251 2023-02-03T14:43:22.160-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42781009 osd.3251 2023-02-03T14:43:31.603-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42781892 osd.3251 2023-02-03T14:43:35.503-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42782258 osd.3251 Taking a specific object as an example: 100dbad3fce.000000b3 2023-02-03T14:42:46.293-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [729808896,25165824]) 2023-02-03T14:42:46.293-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:46.293-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read miss 25165824 left, bh[ 0x7fffb0f461a0 0~25165824 0x7fffb0e4b720 (0) v 0 missing] waiters = {} ... a few times the above ... ... then an OSD read ... 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [50331648,25165824]) 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read miss 25165824 left, bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 missing] waiters = {} 2023-02-03T14:42:48.557-0500 7fffdbfff700 7 objectcacher bh_read on bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 missing] waiters = {} outstanding reads 170 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 client.151672236.objecter _op_submit op 0x7fffb12de170 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter _calc_target epoch 2609365 base 100dbad3fce.000000b3 @9 precalc_pgid 0 pgid 0.0 is_read 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter _calc_target target 100dbad3fce.000000b3 @9 -> pgid 9.502eec40 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 client.151672236.objecter _calc_target raw pgid 9.502eec40 -> actual 9.6c40s0 acting [975,3712,536,3462,2276,1365,2726,1648,4053] primary 975 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter _get_session s=0x7fff9824d360 osd=975 3 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 100dbad3fce.000000b3 '@9' '@9' [read 0~25165824] tid 42777830 osd.975 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter get_session s=0x7fff9824d360 osd=975 3 2023-02-03T14:42:48.557-0500 7fffdbfff700 15 client.151672236.objecter _session_op_assign 975 42777830 2023-02-03T14:42:48.557-0500 7fffdbfff700 15 client.151672236.objecter _send_op 42777830 to 9.6c40s0 on osd.975 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter put_session s=0x7fff9824d360 osd=975 4 2023-02-03T14:42:48.557-0500 7fffdbfff700 5 client.151672236.objecter 170 in flight ... 2023-02-03T14:42:50.045-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [729808896,25165824]) 2023-02-03T14:42:50.045-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:50.045-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read rx bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 rx] waiters = { 0->[0x7fffb0798310, ]} ... read reply ... 2023-02-03T14:42:51.653-0500 7fffeb7fe700 10 client.151672236.objecter ms_dispatch 0x55555e305810 osd_op_reply(42777830 100dbad3fce.000000b3 [read 0~25165824 out=25165824b] v0'0 uv471599 ondisk = 0) v8 2023-02-03T14:42:51.653-0500 7fffeb7fe700 10 client.151672236.objecter in handle_osd_op_reply 2023-02-03T14:42:51.653-0500 7fffeb7fe700 7 client.151672236.objecter handle_osd_op_reply 42777830 ondisk uv 471599 in 9.6c40 attempt 0 2023-02-03T14:42:51.653-0500 7fffeb7fe700 10 client.151672236.objecter op 0 rval 0 len 25165824 2023-02-03T14:42:51.653-0500 7fffeb7fe700 15 client.151672236.objecter handle_osd_op_reply completed tid 42777830 2023-02-03T14:42:51.653-0500 7fffeb7fe700 15 client.151672236.objecter _finish_op 42777830 2023-02-03T14:42:51.653-0500 7fffeb7fe700 20 client.151672236.objecter put_session s=0x7fff9824d360 osd=975 5 2023-02-03T14:42:51.653-0500 7fffeb7fe700 15 client.151672236.objecter _session_op_remove 975 42777830 2023-02-03T14:42:51.653-0500 7fffeb7fe700 5 client.151672236.objecter 169 in flight 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 finisher(0x55555e43bf90) finisher_thread doing [0x7fffb0f2ad60,0] 2023-02-03T14:42:51.657-0500 7fffdbfff700 7 objectcacher bh_read_finish 100dbad3fce.000000b3/head tid 42770553 0~25165824 (bl is 25165824) returned 0 outstanding reads 170 2023-02-03T14:42:51.657-0500 7fffdbfff700 20 objectcacher checking bh bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 rx] waiters = { 0->[0x7fffb0798310, ]} 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher bh_read_finish read bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (25165824) v 0 clean firstbyte=5] waiters = {} 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) try_merge_bh bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (25165824) v 0 clean firstbyte=5] waiters = {} 2023-02-03T14:42:51.657-0500 7fffdbfff700 20 objectcacher finishing waiters 0x7fffb0798310 ... 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [50331648,25165824]) 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:51.657-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read hit bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (25165824) v 0 clean firstbyte=5] waiters = {} ... a few times the above ... 2023-02-03T14:42:55.085-0500 7fffacff9700 10 objectcacher trim trimming object[100dbad3fce.000000b3/head oset 0x7fffd2895870 wr 0/0] 2023-02-03T14:42:55.085-0500 7fffacff9700 10 objectcacher close_object object[100dbad3fce.000000b3/head oset 0x7fffd2895870 wr 0/0] ... 2023-02-03T14:42:55.293-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [50331648,25165824]) 2023-02-03T14:42:55.293-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:55.293-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read miss 25165824 left, bh[ 0x7fffb1c36480 0~25165824 0x7fffb0dfefa0 (0) v 0 missing] waiters = {} ... some more times ... ... then eventually another OSD read for the same object - and the cycle repeats ... The client is ceph-fuse Pacific 16.2.10, the back end ceph is Octopus 15.2.7. The cluster is healthy during this time, no down OSDs, recoveries, etc. Here is the tracker: https://tracker.ceph.com/issues/58640 Any insight/help would be appreciated, Andras

2 1

Re: Removing Rados Gateway in ceph cluster
by Eugen Block 07 Feb '23

07 Feb '23

Please send responses to the mailing-list. If the orchestrator is available, please share also this output (mask sensitive data): ceph orch ls rgw --export --format yaml Which ceph version is this? The command 'ceph dashboard get-rgw-api-host' was removed between Octopus and Pacific, that's why I asked for your ceph version. I also forgot that mgr/dashboard/RGW_API_HOST was used until Octopus, in Pacific it's not applied anymore. I'll need to check how it is determined now. Zitat von Michel Niyoyita <micou12(a)gmail.com>: > Hello Eugen, > > Thanks for your reply , > > I am trying the shared command but no output . > > root@ceph-mon1:~# ceph config dump | grep mgr/dashboard/RGW_API_HOST > root@ceph-mon1:~# ceph config dump | grep mgr/dashboard/ > mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST > http://10.10.110.196:9093 > * > mgr advanced mgr/dashboard/GRAFANA_API_PASSWORD XXXX > > * > mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false > > * > mgr advanced mgr/dashboard/GRAFANA_API_URL > https://10.10.110.198:3000 > * > mgr advanced mgr/dashboard/GRAFANA_API_USERNAME admin > > * > mgr advanced mgr/dashboard/PROMETHEUS_API_HOST > http://10.10.110.196:9092 > * > mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY > XXXX > * > mgr advanced mgr/dashboard/RGW_API_SECRET_KEY > XXXX > * > mgr advanced mgr/dashboard/RGW_API_SSL_VERIFY false > > * > mgr advanced mgr/dashboard/ceph-mon1/server_addr 10.10.110.196 > > * > mgr advanced mgr/dashboard/ceph-mon2/server_addr 10.10.110.197 > > * > mgr advanced mgr/dashboard/ceph-mon3/server_addr 10.10.110.198 > > * > mgr advanced mgr/dashboard/motd {"message": > "WELCOME TO AOS ZONE 3 STORAGE CLUSTER", "md5": > "87149a6798ce42a7e990bc8584a232cd", "severity": "info", "expires": ""} * > mgr advanced mgr/dashboard/server_port 8443 > > * > mgr advanced mgr/dashboard/ssl true > > * > mgr advanced mgr/dashboard/ssl_server_port 8443 > > > for the second one it seems is not valid > > root@ceph-mon1:~# ceph dashboard get-rgw-api-host > no valid command found; 10 closest matches: > dashboard set-jwt-token-ttl <seconds:int> > dashboard get-jwt-token-ttl > dashboard create-self-signed-cert > dashboard grafana dashboards update > dashboard get-account-lockout-attempts > dashboard set-account-lockout-attempts <value> > dashboard reset-account-lockout-attempts > dashboard get-alertmanager-api-host > dashboard set-alertmanager-api-host <value> > dashboard reset-alertmanager-api-host > Error EINVAL: invalid command > root@ceph-mon1:~# > > > Kindly check the output . > > Best Regards > > Michel > * > > On Mon, Feb 6, 2023 at 2:06 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Hi, >> >> can you paste the output of: >> >> ceph config dump | grep mgr/dashboard/RGW_API_HOST >> >> Does it match your desired setup? Depending on the ceph version (and >> how ceph-ansible deploys the services) you could also check: >> >> ceph dashboard get-rgw-api-host >> >> I'm not familiar with ceph-ansible, but if you shared your rgw >> definitions and the respective ceph output we might be able to assist >> resolving this. >> >> Regards, >> Eugen >> >> Zitat von Michel Niyoyita <micou12(a)gmail.com>: >> >> > Hello team, >> > >> > I have a ceph cluster deployed using ceph-ansible , running on ubuntu >> 20.04 >> > OS which have 6 hosts , 3 hosts for OSD and 3 hosts used as monitors and >> > managers , I have deployed RGW on all those hosts and RGWLOADBALENCER on >> > top of them , for testing purpose , I have switched off one OSD , to >> check >> > if the rest can work properly , The test went well as expected, >> > unfortunately while coming back an OSD , the RGW failed to connect >> through >> > the dashboard. below is the message : >> > The Object Gateway Service is not configuredError connecting to Object >> > GatewayPlease consult the documentation >> > < >> https://docs.ceph.com/en/latest/mgr/dashboard/#enabling-the-object-gateway-… >> > >> > on >> > how to configure and enable the Object Gateway management functionality. >> > >> > would like to ask how to solve that issue or how can I proceed to remove >> > completely RGW and redeploy it after . >> > >> > >> > root@ceph-mon1:~# ceph -s >> > cluster: >> > id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 >> > health: HEALTH_OK >> > >> > services: >> > mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 72m) >> > mgr: ceph-mon2(active, since 71m), standbys: ceph-mon3, ceph-mon1 >> > osd: 48 osds: 48 up (since 79m), 48 in (since 3d) >> > rgw: 6 daemons active (6 hosts, 1 zones) >> > >> > data: >> > pools: 9 pools, 257 pgs >> > objects: 59.49k objects, 314 GiB >> > usage: 85 TiB used, 348 TiB / 433 TiB avail >> > pgs: 257 active+clean >> > >> > io: >> > client: 2.0 KiB/s wr, 0 op/s rd, 0 op/s wr >> > >> > Kindly help >> > >> > Best Regards >> > >> > Michel >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

4 9

Rotate lockbox keyring
by Zhongzhou Cai 07 Feb '23

07 Feb '23

Hi, I'm on Ceph 16.2.10, and I'm trying to rotate the ceph lockbox keyring. I used ceph-authtool to create a new keyring, and used `ceph auth import -i <new-keyring>` to update the lockbox keyring. I also updated the keyring file, which is /var/lib/ceph/osd/ceph-<osd-id>/lockbox.keyring. I tried `systemctl restart ceph-volume@lvm-<osd-id>-<osd-fsid>.service`, the command succeeded. Then I rebooted the node, ceph-volume failed because the lockbox.keyring file was overwritten with the old key, which doesn't match the lockbox keyring in `ceph auth get`. Does anyone know where it gets the lockbox.keyring during reboot? Thanks, Zhongzhou Cai

1 1

Rotate lockbox keyring
by Zhongzhou Cai 06 Feb '23

06 Feb '23

Hi, I'm on Ceph 16.2.10, and I'm trying to rotate the ceph lockbox keyring. I used ceph-authtool to create a new keyring, and used `ceph auth import -i <new-keyring>` to update the lockbox keyring. I also updated the keyring file, which is /var/lib/ceph/osd/ceph-<osd-id>/lockbox.keyring. Then I ran `systemctl restart ceph-volume@lvm-<osd-id>-<osd-fsid>.service`, the command succeeded. Then I rebooted the node, ceph-volume failed because the lockbox.keyring file was overwritten with the old key. Does anyone know where it gets the lockbox.keyring during reboot? Thanks, Zhongzhou Cai

1 0

Ceph Pacific 16.2.11 : ceph-volume does not like LV with the same name in different VG
by Gilles Mocellin 06 Feb '23

06 Feb '23

Hello ! It seems like ceph-volume from Ceph Pacific 16.2.11 has a problem with same LV names in different VG. I use ceph-ansible (stable-6), with a pre-existing LVM configuration. Here's the error : TASK [ceph-osd : include_tasks scenarios/lvm.yml] ************************************************************************************************************************************** Monday 06 February 2023 16:13:55 +0100 (0:00:00.065) 0:03:41.576 ******* included: /home/cephadmin/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml for fidcllabs-sto-01.labs.fidcl.cloud, fidcllabs-sto-02.labs.fidcl.cloud TASK [ceph-osd : use ceph-volume to create bluestore osds] ***************************************************************************************************************************** Monday 06 February 2023 16:13:55 +0100 (0:00:00.121) 0:03:41.698 ******* failed: [fidcllabs-sto-01.labs.fidcl.cloud] (item={'data': 'data-lv1', 'data_vg': 'data-vg1', 'crush_device_class': 'sas15k'}) => changed=false ansible_loop_var: item item: crush_device_class: sas15k data: data-lv1 data_vg: data-vg1 msg: 'Could not decode json output: from the command [''ceph-volume'', ''--cluster'', ''ceph'', ''lvm'', ''list'', ''data-vg1/data-lv1'', ''--format=json'']' rc: 1 If I execute the ceph-volume command myself on one target host, I get : fcadmin@fidcllabs-sto-01:~$ sudo ceph-volume --cluster ceph lvm list data-vg1/data-lv1 --format=json --> RuntimeError: Filters {'lv_name': 'data-lv1'} matched more than 1 LV present on this host. My LVs : 1 fcadmin@fidcllabs-sto-01:~$ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data-lv1 data-vg1 -wi-ao---- <1024.00g data-lv1 data-vg2 -wi-ao---- <1024.00g data-lv1 data-vg3 -wi-ao---- <2.00t data-lv1 data-vg4 -wi-ao---- <2.00t logs sys -wi-ao---- 3.81g root sys -wi-ao---- <15.26g swap sys -wi-ao---- <7.63g unused sys -wi-a----- <43.30g Yes, all data LVs have the same name, but under a different VG. I look at the tracker but don't find a clear corresponding issue. I will certainly open one if there's nothing known around here.

1 0

Re: Telemetry service is temporarily down
by Yaarit Hatuka 06 Feb '23

06 Feb '23

Hi everyone, Our telemetry service is up and running again. Thanks Adam Kraitman and Dan Mick for restoring the service. We thank you for your patience and appreciate your contribution to the project! Thanks, Yaarit On Tue, Jan 3, 2023 at 3:14 PM Yaarit Hatuka <yhatuka(a)redhat.com> wrote: > Hi everyone, > > We are having some infrastructure issues with our telemetry backend, and > we are working on fixing it. > Thanks Jan Horacek for opening this issue > <https://tracker.ceph.com/issues/58371> [1]. We will update once the > service is back up. > We are sorry for any inconvenience you may be experiencing, and appreciate > your patience. > > Thanks, > Yaarit > > [1] https://tracker.ceph.com/issues/58371 >

1 2

Inconsistency in rados ls
by Ramin Najjarbashi 06 Feb '23

06 Feb '23

Hi I hope this email finds you well. I am reaching out to you because I have encountered an issue with my CEPH Bluestore cluster and I am seeking your assistance. I have a cluster with approximately one billion objects and when I run a PG query, it shows that I have 27,000 objects per PG. I have run the following command: "rados -p <mypool> ls | wc -l" which returns the correct number of one billion objects. However, when I run the same command per pg, the results are much less, with only 20 million objects being reported. For example, "rados -p <mypool> --pgid 1.xx ls | wc -l" shows only three objects in the specified PG. This is a significant discrepancy and I am concerned about the integrity of my data. Do you have any idea about this discrepancy? p.s: I have a total of 30 million objects in a single bucket and versioning has not been enabled for this particular bucket. Thank you for your time and I look forward to your response. Best regards, Ramin

3 5

Any ceph constants available?
by Thomas Cannon 06 Feb '23

06 Feb '23

Hello Ceph community. The company that recently hired me has a 3 mode ceph cluster that has been running and stable. I am the new lone administrator here and do not know ceph and this is my first experience with it. The issue was that it is/was running out of space, which is why I made a 4th node and attempted to add it into the cluster. Along the way, things have begun to break. The manager daemon on boreal-01 failed to boreal-02 along the way and I tried to get it to fail back to boreal-01, but was unable, and realized while working on it yesterday I realized that the nodes in the cluster are all running different versions of the software. I suspect that might be a huge part of why things aren’t working as expected. Boreal-01 - the host - 17.2.5: root@boreal-01:/home/kadmin# ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) root@boreal-01:/home/kadmin# Boreal-01 - the admin docker instance running on the host 17.2.1: root@boreal-01:/home/kadmin# cephadm shell Inferring fsid 951fa730-0228-11ed-b1ef-f925f77b75d3 Inferring config /var/lib/ceph/951fa730-0228-11ed-b1ef-f925f77b75d3/mon.boreal-01/config Using ceph image with id 'e5af760fa1c1' and tag 'v17' created on 2022-06-23 19:49:45 +0000 UTC quay.io/ceph/ceph@sha256:d3f3e1b59a304a280a3a81641ca730982da141dad41e942631e4c5d88711a66b <http://quay.io/ceph/ceph@sha256:d3f3e1b59a304a280a3a81641ca730982da141dad41…> root@boreal-01:/# ceph -v ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable) root@boreal-01:/# Boreal-02 - 15.2.6: root@boreal-02:/home/kadmin# ceph -v ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable) root@boreal-02:/home/kadmin# Boreal-03 - 15.2.8: root@boreal-03:/home/kadmin# ceph -v ceph version 15.2.18 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable) root@boreal-03:/home/kadmin# And the host I added - Boreal-04 - 17.2.5: root@boreal-04:/home/kadmin# ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) root@boreal-04:/home/kadmin# The cluster ins’t rebalancing data, and drives are filling up unevenly, despite auto balancing being on. I can run a df and see that it isn’t working. However it says it is: root@boreal-01:/# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.011905", "last_optimize_started": "Fri Feb 3 18:39:02 2023", "mode": "upmap", "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } root@boreal-01:/# root@boreal-01:/# ceph -s cluster: id: 951fa730-0228-11ed-b1ef-f925f77b75d3 health: HEALTH_WARN There are daemons running an older version of ceph 6 nearfull osd(s) 3 pgs not deep-scrubbed in time 3 pgs not scrubbed in time 4 pool(s) nearfull 1 daemons have recently crashed services: mon: 4 daemons, quorum boreal-01,boreal-02,boreal-03,boreal-04 (age 22h) mgr: boreal-02.lqxcvk(active, since 19h), standbys: boreal-03.vxhpad, boreal-01.ejaggu mds: 2/2 daemons up, 2 standby osd: 89 osds: 89 up (since 5d), 89 in (since 45h) data: volumes: 2/2 healthy pools: 7 pools, 549 pgs objects: 227.23M objects, 193 TiB usage: 581 TiB used, 356 TiB / 937 TiB avail pgs: 533 active+clean 16 active+clean+scrubbing+deep io: client: 55 MiB/s rd, 330 KiB/s wr, 21 op/s rd, 45 op/s wr root@boreal-01:/# Part of me suspects that I exacerbated the problems by trying to monkey with boreal-04 for several days, trying to get the drives inside the machine turned into OSDs so that they would be used. One thing I did was attempt to upgrade the code on that machine, and I could have triggered a cluster-wide upgrade that failed outside of 1 and 4. With 2 and 3 not even running the same major release, if I did make that mistake, I can see why instead of an upgrade, things would be worse. According to the documentation, I should be able to upgrade the entire cluster by running a single command on the admin node, but when I go to run commands I get errors that even google can’t solve: root@boreal-01:/# ceph orch host ls Error ENOENT: Module not found root@boreal-01:/# Consequently, I have very little faith that running commands to upgrade everything so that it’s all running the same code will work. I think each host could be upgraded and fix things, but do not feel confident doing so and risking our data. Hopefully that gives a better idea of the problems I am facing. I am hoping for some professional services hours with someone who is a true expert with this software, to get us to a stable and sane deployment that can be managed without it being a terrifying guessing game, trying to get it to work. If that is you, or if you know someone who can help — please contact me! Thank you! Thomas

4 5

Removing Rados Gateway in ceph cluster
by Michel Niyoyita 06 Feb '23

06 Feb '23

Hello team, I have a ceph cluster deployed using ceph-ansible , running on ubuntu 20.04 OS which have 6 hosts , 3 hosts for OSD and 3 hosts used as monitors and managers , I have deployed RGW on all those hosts and RGWLOADBALENCER on top of them , for testing purpose , I have switched off one OSD , to check if the rest can work properly , The test went well as expected, unfortunately while coming back an OSD , the RGW failed to connect through the dashboard. below is the message : The Object Gateway Service is not configuredError connecting to Object GatewayPlease consult the documentation <https://docs.ceph.com/en/latest/mgr/dashboard/#enabling-the-object-gateway-…> on how to configure and enable the Object Gateway management functionality. would like to ask how to solve that issue or how can I proceed to remove completely RGW and redeploy it after . root@ceph-mon1:~# ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 72m) mgr: ceph-mon2(active, since 71m), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 79m), 48 in (since 3d) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 9 pools, 257 pgs objects: 59.49k objects, 314 GiB usage: 85 TiB used, 348 TiB / 433 TiB avail pgs: 257 active+clean io: client: 2.0 KiB/s wr, 0 op/s rd, 0 op/s wr Kindly help Best Regards Michel

2 1

PG increase / data movement fine tuning
by Szabo, Istvan (Agoda) 06 Feb '23

06 Feb '23

Hi, I've increased the placement group in my octopus cluster firstly in the index pool and I gave almost 2.5 hours bad performance for the user. I'm planning to increase the data pool also, but first I'd like to know is there any way to make it smoother or not. At the moment I have these values: osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 But seems like this still generates slow ops. Should I turn off scrubbing or any other way to make it even smoother? Some information about the setup: * I have 9 nodes, each of them has 2x nvme drives with 4osd on those and this is where the index pool lives. * Currently has 2048 pg-s for the index pool Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

cephadm and the future
by Christopher Durham 04 Feb '23

04 Feb '23

Question: What does the future hold with regard to cephadm vs rpm/deb packages? If it is now suggested to use cephadm and thus containers to deploy new clusters, what does the future hold? Is there an intent, at sometime in the future, to no longer support rpm/deb packages for Linux systems, and only support the cephadm container method? I am not asking to argue containers vs traditional bare metal installs. I am just trying to plan for the future. Thanks -Chris

2 1

'ceph orch upgrade...' causes an rbd outage on a proxmox cluster
by Pierre BELLEMAIN 03 Feb '23

03 Feb '23

Hi everyone, (sorry for the spam, apparently I was not subscribed to the ml) I have a ceph test cluster and a proxmox test cluster (for try upgrade in test before the prod). My ceph cluster is made up of three servers running debian 11, with two separate networks (cluster_network and public_network, in VLANs). In ceph version 16.2.10 (cephadm with docker). Each server has one MGR, one MON and 8 OSDs. cluster: id: xxx health: HEALTH_OK services: mon: 3 daemons, quorum ceph01,ceph03,ceph02 (age 2h) mgr: ceph03(active, since 77m), standbys: ceph01, ceph02 osd: 24 osds: 24 up (since 7w), 24 in (since 6M) data: pools: 3 pools, 65 pgs objects: 29.13k objects, 113 GiB usage: 344 GiB used, 52 TiB / 52 TiB avail pgs: 65 active+clean io: client: 1.3 KiB/s wr, 0 op/s rd, 0 op/s wr The proxmox cluster is also made up of 3 servers running proxmox 7.2-7 (with proxmox ceph pacific which is on 16.2.9 version). The ceph storage used is RBD (on the ceph public_network). I added the RBD datastores simply via the GUI. So far so good. I have several VMs, on each of the proxmox. When I update ceph to 16.2.11, that's where things go wrong. I don't like when the update does everything for me without control, so I did a "staggered upgrade", following the official procedure (https://docs.ceph.com/en/pacific/cephadm/upgrade/#staggered-upgrade) As the version I'm starting from doesn't support staggered upgrade, I follow the procedure (https://docs.ceph.com/en/pacific/cephadm/upgrade/#upgrading-to-a-version-th…) When I do the "ceph orch redeploy" of the two standby MGRs, everything is fine. I do the "sudo ceph mgr fail", everything is fine (it switches well to an mgr which was standby, so I get an MGR 16.2.11). However, when I do the "sudo ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.11 --daemon-types mgr", it updates me the last MGR which was not updated (so far everything is still fine), but it does a last restart of all the MGRs to finish, and there, the proxmox visibly loses the RBD and turns off all my VMs. Here is the message in the proxmox syslog: Feb 2 16:20:52 pmox01 QEMU[436706]: terminate called after throwing an instance of 'std::system_error' Feb 2 16:20:52 pmox01 QEMU[436706]: what(): Resource deadlock avoided Feb 2 16:20:52 pmox01 kernel: [17038607.686686] vmbr0: port 2(tap102i0) entered disabled state Feb 2 16:20:52 pmox01 kernel: [17038607.779049] vmbr0: port 2(tap102i0) entered disabled state Feb 2 16:20:52 pmox01 systemd[1]: 102.scope: Succeeded. Feb 2 16:20:52 pmox01 systemd[1]: 102.scope: Consumed 43.136s CPU time. Feb 2 16:20:53 pmox01 qmeventd[446872]: Starting cleanup for 102 Feb 2 16:20:53 pmox01 qmeventd[446872]: Finished cleanup for 102 For ceph, everything is fine, it does the update, and tells me everything is OK in the end. Ceph is now on 16.2.11 and the health is OK. When I redo a downgrade of the MGRs, I have the problem again and when I start the procedure again, I still have the problem. It's very reproducible. According to my tests, the "sudo ceph orch upgrade" command always gives me trouble, even when trying a real staggered upgrade from and to version 16.2.11 with the command: sudo ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.11 --daemon-types mgr --hosts ceph01 --limit 1 Does anyone have an idea? Thank you everyone ! Pierre.

1 1

Ceph Upgrade path
by Iztok Gregori 02 Feb '23

02 Feb '23

Hi to all! We are running a Ceph cluster (Octopus) on (99%) CentOS 7 (deployed at the time with ceph-deploy) and we would like to upgrade it. As far as I know for Pacific (and later releases) there aren't packages for CentOS 7 distribution (at least not on download.ceph.com) so we need to upgrade (change) not only Ceph but also the distribution. What is the raccomended path to do so? We could upgrade (reinstall) all the nodes to Rocky 8 and then upgrade Ceph to Quincy, but we will "stuck" with "not the latest" distribution and probably we will have to upgrade (reinstall) again in the near future. Our second idea is to leverage cephadm (which we would like to implement) and switch from rpms to containers, but I don't have a clear vision of how to do it. I was thinking to: 1. install a new monitor/manager with Rocky 9. 2. prepare the node for cephadm. 3. start the manager/monitor containers on that node. 4. repeat for the other monitors. 5. repeat for the OSD servers. I'm not sure how to execute the point 2 and 3. The documentation says how to bootstrap a NEW cluster and how to ADOPT an existing one, but our situation is a hybrid (or in my mind it is...). I cannot also adopt my current cluster to cephadm because we have 30% of our OSD still on filestore. My intention was to drain them, reinstall them and then adopt them. But I would like to avoid (if not necessary) multiple reinstallations. In my mind all the OSD servers will be drained before been reinstalled, just to be sure to have a "fresh" start). Have you any ideas and/or advice to give us? Thanks a lot! Iztok P.S. I saw that the script cephadm doesn't support Rocky, I can modify it to do so and it should work, but is there a plan to officially support it? -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. Telephone: +39 040 3758948 http://www.elettra.eu

3 2

'ceph orch upgrade...' causes an rbd outage on a proxmox cluster
by Pierre BELLEMAIN 02 Feb '23

02 Feb '23

Hi everyone, I have a ceph test cluster and a proxmox test cluster (for try upgrade in test before the prod). My ceph cluster is made up of three servers running debian 11, with two separate networks (cluster_network and public_network, in VLANs). In ceph version 16.2.10 (cephadm with docker). Each server has one MGR, one MON and 8 OSDs. cluster: id: xxx health: HEALTH_OK services: mon: 3 daemons, quorum ceph01,ceph03,ceph02 (age 2h) mgr: ceph03(active, since 77m), standbys: ceph01, ceph02 osd: 24 osds: 24 up (since 7w), 24 in (since 6M) data: pools: 3 pools, 65 pgs objects: 29.13k objects, 113 GiB usage: 344 GiB used, 52 TiB / 52 TiB avail pgs: 65 active+clean io: client: 1.3 KiB/s wr, 0 op/s rd, 0 op/s wr The proxmox cluster is also made up of 3 servers running proxmox 7.2-7. The ceph storage used is RBD (on the ceph public_network). I added the RBD datastores simply via the GUI. So far so good. I have several VMs, on each of the proxmox. When I update ceph to 16.2.11, that's where things go wrong. I don't like when the update does everything for me without control, so I did a "staggered upgrade", following the official procedure (https://docs.ceph.com/en/pacific/cephadm/upgrade/#staggered-upgrade) As the version I'm starting from doesn't support staggered upgrade, I follow the procedure (https://docs.ceph.com/en/pacific/cephadm/upgrade/#upgrading-to-a-version-th…) When I do the "ceph orch redeploy" of the two standby MGRs, everything is fine. I do the "sudo ceph mgr fail", everything is fine (it switches well to an mgr which was standby, so I get an MGR 16.2.11). However, when I do the "sudo ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.11 --daemon-types mgr", it updates me the last MGR which was not updated (so far everything is still fine), but it does a last restart of all the MGRs to finish, and there, the proxmox visibly loses the RBD and turns off all my VMs. Here is the message in the proxmox syslog: Feb 2 16:20:52 pmox01 QEMU[436706]: terminate called after throwing an instance of 'std::system_error' Feb 2 16:20:52 pmox01 QEMU[436706]: what(): Resource deadlock avoided Feb 2 16:20:52 pmox01 kernel: [17038607.686686] vmbr0: port 2(tap102i0) entered disabled state Feb 2 16:20:52 pmox01 kernel: [17038607.779049] vmbr0: port 2(tap102i0) entered disabled state Feb 2 16:20:52 pmox01 systemd[1]: 102.scope: Succeeded. Feb 2 16:20:52 pmox01 systemd[1]: 102.scope: Consumed 43.136s CPU time. Feb 2 16:20:53 pmox01 qmeventd[446872]: Starting cleanup for 102 Feb 2 16:20:53 pmox01 qmeventd[446872]: Finished cleanup for 102 For ceph, everything is fine, it does the update, and tells me everything is OK in the end. Ceph is now on 16.2.11 and the health is OK. When I redo a downgrade of the MGRs, I have the problem again and when I start the procedure again, I still have the problem. It's very reproducible. According to my tests, the "sudo ceph orch upgrade" command always gives me trouble, even when trying a real staggered upgrade from and to version 16.2.11 with the command: sudo ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.11 --daemon-types mgr --hosts ceph01 --limit 1 Does anyone have an idea? Thank you everyone ! Pierre.

1 0

January Ceph Science Virtual User Group
by Kevin Hrpcek 02 Feb '23

02 Feb '23

Hey all, We will be having a Ceph science/research/big cluster call on Tuesday January 31st. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20230131 Ceph calendar event details: January 31, 2023 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <http://bjn.vc> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

4 3

ceph quincy cannot change osd_recovery_max_active, please help
by 辣条➀号 02 Feb '23

02 Feb '23

Greetings from the enthusiastic ceph official team! Recently, when our company was using ceph quincy (stable), it was found that the initial value of osd_recovery_max_active could not be changed. 1. When I try to modify osd_mclock_override_recovery_settings to true, I get an error. It seems that there is no such option? How should I modify the initial value of osd_recovery_max_active root@pve-ceph01:~# ceph config set osd osd_mclock_override_recovery_settings true Error EINVAL: unrecognized config option 'osd_mclock_override_recovery_settings'

1 0

CEPHADM_STRAY_DAEMON does not exist, how do I remove knowledge of it from ceph?
by ceph＠mikesoffice.com 01 Feb '23

01 Feb '23

Hi All, I'm getting this error while setting up a ceph cluster. I'm relatively new to ceph, so there is no telling what kind of mistakes I've been making. I'm using cephadm, ceph v16 and I apparently have a stray daemon. But it also doesn't seem to exist and I can't get ceph to forget about it. $ ceph health detail [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.cmon01 on host cmgmt01 not managed by cephadm mon.cmon01 also shows up in dashboard->hosts as running on cmgmt01. It does not show up in the monitors section though. But, there isn't a monitor daemon running on that machine at all (no podman container, not in process list, not listening on a port). On that host in cephadm shell, # ceph orch daemon rm mon.cmon01 --force Error EINVAL: Unable to find daemon(s) ['mon.cmon01'] I don't currently have any real data on the cluster, so I've also tried deleting the existing pools (except device_health_metrics) in case ceph was connecting that monitor to one of the pools. I'm not sure what to try next in order to get ceph to forget about that daemon.

3 2

How to get RBD client log?
by Jinhao Hu 01 Feb '23

01 Feb '23

Hi, How can I collect the logs of the RBD client?

3 3

CLT meeting summary 2023-02-01
by Casey Bodley 01 Feb '23

01 Feb '23

distro testing for reef * https://github.com/ceph/ceph/pull/49443 adds centos9 and ubuntu22 to supported distros * centos9 blocked by teuthology bug https://tracker.ceph.com/issues/58491 - lsb_release command no longer exists, use /etc/os-release instead - ceph stopped depending on lsb_release in 2021 with https://github.com/ceph/ceph/pull/42770 * ubuntu22 not blocked by teuthology, but the new python version breaks most of the rgw tests can we drop centos8 or ubuntu20 support for reef? * we usually support the latest centos and two ubuntu LTSs * users need an upgrade path that doesn't require OS and ceph upgrade at the same time * we might be able to drop centos8 support for Reef by adding centos9 support to Quincy * python versioning issues make longer-term support of older distros problematic. related work: - https://github.com/ceph/ceph/pull/41979 - https://github.com/ceph/ceph/pull/47501 ondisk format changes in minor releases * https://github.com/ceph/ceph/pull/48915 introduced some BlueFS log changes in 16.2.11 which makes it incompatible with previous Pacific releases. Hence no downgrade is permitted any more. - doc text tracked in https://tracker.ceph.com/issues/58625 * how do we prevent these issues in the future? - better testing of mixed-version rgw/mds/mgr/etc infrastructure update * a planned network outage yesterday still affecting LRC

1 0

Adding Labels Section to Perf Counters Output
by Ali Maredia 01 Feb '23

01 Feb '23

Hi Ceph Developers and Users, Various upstream developers and I are working on adding labels to perf counters (https://github.com/ceph/ceph/pull/48657) We would like to understand the ramifications of changing the format of the json dumped by the `perf dump` command for the Reef Release on users and components of Ceph. As an example given in the PR, currently unlabeled counters are dumped like this in comparison with their new labeled counterparts. "some unlabeled_counter": { "put_b": 1048576, }, "some labeled_counter": { "labels": { "Bucket: "bkt1", "User: "user1", }, "counters": { "put_b": 1048576, }, }, Here is an example given in the PR of the old style unlabeled counters being dumped in the same format as the labeled counters: "some unlabeled": { "labels": { }, "counters": { "put_b": 1048576, }, }, "some labeled": { "labels": { "Bucket: "bkt1", "User: "user1", }, "counters": { "put_b": 1048576, }, }, Would users/consumers of these counters be opposed to changing the format? Why is this the case? As far as I know there are ceph-mgr modules related to Prometheus and telemetry that are consuming the current unlabeled counters. Also this topic will be discussed at the upcoming Ceph Developer Monthly EMEA as well. Best, Ali

2 1

Documentation - February 2023 - Request for Comments
by John Zachary Dover 01 Feb '23

01 Feb '23

This is an email thread for reporting documentation concerns. I won't guarantee that our documentation team will drop everything and immediately service your documentation request, but I do guarantee that your request will go into the queue of things we will eventually tackle. https://docs.ceph.com Zac Dover Upstream Documentation Ceph Foundation

1 0