- ceph-users - lists.ceph.io

Multisite s3 website slow period update
by Ondřej Kukla 15 Aug '23

15 Aug '23

Hello, I’ve had a quite unpleasant experience today that I would like to share. In our setup we use two set’s of RGW one that has only s3 and admin API and a second set with s3website and admin API. I was changing the global quota setting which means that I’ve then needed to commit the updated period. The first set of s3 RGWs updated without issue, but the s3website RGWs not. They were somehow stuck at the period update took minutes in some cases meaning service disruption for MINUTES! When I’ve looked at the RGW logs I’ve found this line that seems to show the issue. rgw realm reloader: Pausing frontends for realm update... req 3935991300378050219 61.212696075s s3:get_obj iterate_obj() failed with -104 We are running on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Did someone had a similar issue? Is there a something we can do about it? Is there an option to update the period machine by machine or some other way that would let us to update the period and not disrupt the service? Your help is much appreciated. Kind regards, Ondrej

1 0

Announcing go-ceph v0.23.0
by Sven Anderson 15 Aug '23

15 Aug '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.23.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

OSD containers lose connectivity after change from Rocky 8.7->9.2
by Dan O'Brien 15 Aug '23

15 Aug '23

I recently updated one of the hosts (an older Dell PowerEdge R515) in my Ceph Quincy (17.2.6) cluster. I needed to change the IP address, so I removed the host from the cluster (gracefully removed OSDs and daemons, then removed the host). I also took the opportunity to upgrade the host from Rocky 8.7 to 9.2 before re-joining it to the cluster with cephadm. I zapped the storage, so for all intents and purposes, it should have been a completely clean instance and went smoothly. I have 2 other hosts (new Dell PowerEdge R450s) using Rocky 9.2 with no problems. Before the upgrade, the R515 host was well-behaved and unremarkable. Our cluster is connected to our internal network, and has a 10G private network used for interconnect between the nodes. Since the upgrade, the OSDs on the R515 host regularly, after a period of minutes to hours (usually a few hours), drop out of the cluster. I can restart the OSDs and they immediately reconnect and join the cluster, which returns to a HEALTHY state after a short period. The OSD logs show Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: log_channel(cluster) log [WRN] : Monitor daemon marked osd.9 down, but it is still running Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: log_channel(cluster) log [DBG] : map e17993 wrongly marked me down at e17988 Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 17993 start_waiting_for_healthy Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 pg_epoch: 17988 pg[16.f( v 16315'2765702 (15197'2757704,16315'2765702] local-lis/les=17902/17903 n=188 ec=211/211 lis/c=17902/17902 les/c/f=17903/17903/0 sis=17988 pruub=8.000660896s) [23,18] r=-1 lpr=1798> Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 17993 is_healthy false -- only 0/10 up peers (less than 33%) Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 17993 not healthy; waiting to boot The MON logs show Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.9 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.9 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.10 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.10 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.9 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.10 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 reported immediately failed by osd.3 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: mon.os-storage-1(a)1(peon).osd e17989 e17989: 26 total, 23 up, 26 in Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: mon.os-storage-1(a)1(peon).osd e17989 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: 15.13 scrub starts Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Health check failed: 4 osds down (OSD_DOWN) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Health check failed: 2 hosts (4 osds) down (OSD_HOST_DOWN) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osdmap e17988: 26 total, 22 up, 26 in Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 marked itself dead as of e17988 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: from='mgr.16700599 10.192.126.85:0/2567473893' entity='mgr.os-storage.cecnet.gmu.edu.mouglb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: pgmap v121406: 374 pgs: 3 peering, 20 stale+active+clean, 3 active+remapped+backfilling, 348 active+clean; 2.1 TiB data, 4.1 TiB used, 76 TiB / 80 TiB avail; 102 B/s rd, 338 KiB/s wr, 6 op/s; 36715/1268774 objects > Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Health check cleared: OSD_HOST_DOWN (was: 2 hosts (4 osds) down) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: from='mgr.16700599 10.192.126.85:0/2567473893' entity='mgr.os-storage.cecnet.gmu.edu.mouglb' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 [v2:10.192.126.76:6808/3768579449,v1:10.192.126.76:6809/3768579449] boot Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osdmap e17989: 26 total, 23 up, 26 in Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 marked itself dead as of e17989 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.24 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.24) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.5 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.3 Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: mon.os-storage-1(a)1(peon).osd e17990 e17990: 26 total, 23 up, 26 in Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Monitor daemon marked osd.11 down, but it is still running Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: map e17988 wrongly marked me down at e17988 Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: 25.16 continuing backfill to osd.22 from (17451'16681348,17987'16685608] 25:6977376b:::rbd_data.4040a64151e028.000000000000a3c3:head to 17987'16685608 Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Monitor daemon marked osd.12 down, but it is still running The system logs show no problems around the time of the drop. My best guess at the moment is that it's a networking issue with podman, but I've found no evidence of a problem. Output from ethtool doesn't show any problems [root@ceph99 ~]# ethtool -S enp3s0f0 | egrep 'error|drop|timeout' rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_timeout_count: 0 rx_length_errors: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_csum_offload_errors: 0 tx_hwtstamp_timeouts: 0 So far I have: - Updated the Intel X550-T NIC's firmware and driver (ixgbe) to the latest from Intel - Reverted the kernel, podman and NetworkManager packages to match the other Rocky 9.2 hosts that are working - Reverted the Intel driver to the ixgbe included in the kernel - sworn, cried and pleaded with the gods to spare me further anguish, to no avail. Other, possibly relevant, information: - There's no MON or MGR daemons on the host; just node exporter, crash, alerter and promtail. Is there anything else I should be looking at before I remove the host from the cluster and re-install Rocky 8.7 (and hope it works again)? This host was used while we were standing up the cluster and is due to be retired as we repurpose some of our other storage (standalone NFS servers with RAID6) and move them into the cluster.

1 0

Re: ceph orch upgrade stuck between 16.2.7 and 16.2.13
by Robert Sander 15 Aug '23

15 Aug '23

On 8/15/23 11:16, Curt wrote: > Probably not the issue, but do all your osd servers have internet > access? I've had a similar experience when one of our osd servers > default gateway got changed, so it was just waiting to download and took > a bit to timeout. Yes, all nodes can manually pull the image from quay.io. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: help, ceph fs status stuck with no response
by Patrick Donnelly 14 Aug '23

14 Aug '23

On Tue, Aug 8, 2023 at 1:18 AM Zhang Bao <lonsdale8734(a)gmail.com> wrote: > > Hi, thanks for your help. > > I am using ceph Pacific 16.2.7. > > Before my Ceph stuck at `ceph fs status fsname`, one of my cephfs became readonly. Probably the ceph-mgr is stuck (the "volumes" plugin) somehow talking to the read-only CephFS. That's not a scenario we've tested well. > The metadata pool of the readonly cephfs grew up from 10GB to 3TB. Then I shut down the readonly mds. Your metadata pool grew from 10GB to 3TB in read-only mode? That's unbelievable! We would need a lot more information to help figure the cause. Such as: `ceph tell mds.X perf dump`, `ceph tell mds.X status`, `ceph fs dump`, and `ceph tell mgr.X perf dump` while this is occurring. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

radosgw-admin command hangs out ,many hours
by nguyenvandiep＠baoviet.com.vn 14 Aug '23

14 Aug '23

We r using ceph 16.2.4 While init rados gw service, we run : " ceph orch apply rgw S3GW --placement="1 cephgw03uat" radosgw-admin user create --uid=rgwadmin --display-name=rgwadmin --system " but the command stuck, return nothing, hust HANGS. we add "--verbose" option and see log overlays: uncrognized mount option "volatile" or missing value how can we pass it

2 1

Lots of space allocated in completely empty OSDs
by Andras Pataki 14 Aug '23

14 Aug '23

Here is a strange problem that I don't seem to be able to figure out. Some of our OSDs that have zero weight, and no PGs have lots of allocated space: [root@cephosd0032 ~]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2925 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.32 0.15 0 up 2926 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.33 0.15 0 up 2927 ssd 0 1.00000 1.1 TiB 125 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.31 0.15 0 up 2928 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.32 0.15 0 up so 120GB+ allocated, but no PGs. The cluster is clean, all PGs are active+clean, no rebalancing/recovery/etc. happening. There has been no rebalancing/recovery for at least a couple of days (after having added some nodes to the cluster). Doing a perf dump on the OSD confirms the space usage - that it has 130GB or so allocated: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok perf dump .... "bluestore_allocated": 133277466624, "bluestore_stored": 133014427663, ... Bringing the OSD down - ceph-objectstore-tool claims that there are no objects stored in the OSD: [root@cephosd0032 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2925 --op list [root@cephosd0032 ~]# No fragmentation on the OSD either: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok bluestore allocator fragmentation block { "fragmentation_rating": 1.2165147867554808e-08 } BlueFS also claims not to use significant space: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok bluefs stats 1 : device size 0x11548000000 : own 0x[8518510000~b175d0000] = 0xb175d0000 : using 0xd1e0000(210 MiB) : bluestore has 0xeb3b630000(941 GiB) available wal_total:0, db_total:1131368205516, slow_total:0 So the puzzle is, what is stored in that 130GB allocated space? The cluster is Octopus 15.2.17. I came to looking at this by a mismatch in free space on our OSDs that are in service. After adding some nodes and waiting for the rebalance to finish, the free space didn't increase by the amount of space that was added, although the total space in the cluster did increase correctly (and data in pools didn't increase by a significant amount to explain this). So there is a bit of a mystery regarding what a bunch of space is allocated for. Any ideas/pointers would be appreciated. Andras

3 2

librbd 4k read/write?
by Murilo Morais 13 Aug '23

13 Aug '23

Good afternoon everybody! I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img If I increase the blocksize I can easily reach 1.5 GBps or more. But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read. If I use librbd's cache I get a considerable improvement in writing, but reading remains the same. I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either. Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result. I don't know exactly what to look for or configure to have any improvement.

9 15

Puzzle re 'ceph: mds0 session blocklisted"
by Harry G Coin 12 Aug '23

12 Aug '23

Can anyone help me understand seemingly contradictory cephfs error messages? I have a RHEL ceph client that mounts a cephfs file system via autofs. Very typical. After boot, when a user first uses the mount, for example 'ls /mountpoint' , all appears normal to the user. But on the system console I get, every time after first boot: ... [ 412.762310] Key type dns_resolver registered [ 412.912107] Key type ceph registered [ 412.925268] libceph: loaded (mon/osd proto 15/24) [ 413.110488] ceph: loaded (mds proto 32) [ 413.124870] libceph: mon3 (2)[fc00:1002:c7::44]:3300 session established [ 413.128298] libceph: client56471655 fsid 4067126d-01cb-40af-824a-881c130140f8 [ 413.355716] ceph: mds0 session blocklisted ... The autofs line is /mountpoint -fstype=ceph,fs=cephlibraryfs,fsid=really-big-number,name=cephuser,secretfile=/etc/ceph/secret1.key,ms_mode=crc,relatime,recover_session=clean,mount_timeout=15,fscontext="system_u:object_r:cephfs_t:s0" [fc00:1002:c7::41]:3300,[fc00:1002:c7::42]:3300,[fc00:1002:c7::43]:3300,[fc00:1002:c7::44]:3300:/ 'blocklisting' is, well, 'bad'... but there's no obvious user effect. Is there an 'unobvious' problem? What am I missing? Ceph pacific everywhere latest. Thanks Harry Coin

2 1

CEPHADM_STRAY_DAEMON
by Tyler Jurgens 11 Aug '23

11 Aug '23

Hey, sorry, I'm a ceph noob here, but I'm trying to figure out how to clear this up. I've got 8 servers (ceph-admin, ceph-mon(1->3), ceph-osd(1->4) as part of my ceph deployment (reef, upgraded from quincy), and when I originally deployed it Ceph used random servers for the mon daemon. Wanting to target the intended servers for ceph, I created a label (mon) and assigned that label to my three mon hosts (ceph orch host label add ceph-mon1 mon), etc. Then I assigned the mon daemon to my hosts with the mon label: ceph orch apply mon label:mon. However, now I have this health message: stray daemon mon.ceph-osd3 on host ceph-mon3 not managed by cephadm stray daemon mon.ceph-osd2 on host ceph-osd2 not managed by cephadm How do I clean that up? The docs ( https://docs.ceph.com/en/reef/cephadm/operations/#cephadm-stray-daemon) say to add the hosts with ceph orch host add <hostname> but those hosts with stray daemons are already added. I tried adding them again, but it didn't make a difference. All hosts are/were always managed by cephadm (edited) I also rebooted one of the hosts, that didn't make a difference. Any ideas on what I can try? Cheers, Tyler

3 2