- ceph-users - lists.ceph.io

RBD to fail fast/auto unmap in case of timeout
by Mathias Chapelain 20 Jan '23

20 Jan '23

Hello, We would like to run a RAID1 between a local storage and a RBD device. This would allow us to sustain network failures or Ceph failures and also give better read performance as we would set it up with write-mostly on RBD in mdadm. Basically we would like to implement https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme…. RAID1 is working well but if there is timeouts, the RBD volume won't fail and mdadm will not catch the broken device. Also the writes then hangs waiting for the network/RBD to come back. If we force unmap the RBD device then it fails as expected and writes can continue on other RAID1 device. We tried setting the `osd_request_timeout` to a small value (3 or 2 seconds) but it only gives us timeout in kernel logs: ``` libceph: tid 25792 on osd39 timeout rbd: rbd0: write at objno 602 0~512 result -110 rbd: rbd0: write result -110 print_req_error: 15 callbacks suppressed blk_update_request: timeout error, dev rbd0, sector 4931584 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 libceph: tid 25794 on osd39 timeout rbd: rbd0: write at objno 602 512~512 result -110 rbd: rbd0: write result -110 blk_update_request: timeout error, dev rbd0, sector 4931585 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 ``` Is there something that we missed or is it currently impossible with kRBD to kind of "fail fast" in case of timeout and unmap/remove associated RBD devices? Or is there another client that can do what we want (ceph-nbd or with librbd)? We found this rook issue that is not really helpful but give insight https://github.com/rook/rook/issues/376. Thanks! -- Mathias Chapelain Storage Engineer Proton AG

1 0

ceph quincy rgw openstack howto
by Shashi Dahal 19 Jan '23

19 Jan '23

Hi, How to set up values for rgw_keystone_url and other related fields that are not possible to change via the GUI under cluster configuration. ? ceph qunicy is deployed using cephadm. -- Cheers, Shashi

2 1

journal fills ...
by Michael Lipp 19 Jan '23

19 Jan '23

I'm running Quincy and my journal fills with messages that I consider "debug" level such as: * ceph-mgr[1615]: [volumes INFO mgr_util] scanning for idle connections.. * ceph-mon[1617]: pgmap v1176995: 145 pgs: 145 active+clean; ... * ceph-mgr[1615]: [dashboard INFO request] ... * ceph-mgr[1615]: [pg_autoscaler INFO root] effective_target_ratio 0.0 0.0 0 1649254858752 I've never changed levels after installation and e.g. `ceph config show osd.0 | fgrep debug` does not show anything (and `ceph config show mgr.0`results in "Error ENOENT: no config state for daemon mgr.0". I have no ideas left, any hints? - Michael

1 0

journal fills ...
by Michael Lipp 19 Jan '23

19 Jan '23

I'm running Quincy and my journal fills with messages that I consider "debug" level such as: * ceph-mgr[1615]: [volumes INFO mgr_util] scanning for idle connections.. * ceph-mon[1617]: pgmap v1176995: 145 pgs: 145 active+clean; ... * ceph-mgr[1615]: [dashboard INFO request] ... * ceph-mgr[1615]: [pg_autoscaler INFO root] effective_target_ratio 0.0 0.0 0 1649254858752 I've never changed levels after installation and e.g. `ceph config show osd.0 | fgrep debug` does not show anything (and `ceph config show mgr.0`results in "Error ENOENT: no config state for daemon mgr.0". I have no ideas left, any hints? - Michael

1 0

Ceph rbd clients surrender exclusive lock in critical situation
by Frank Schilder 19 Jan '23

19 Jan '23

Hi all, we are observing a problem on a libvirt virtualisation cluster that might come from ceph rbd clients. Something went wrong during execution of a live-migration operation and as a result we have two instances of the same VM running on 2 different hosts, the source- and the destination host. What we observe now is the the exclusive lock of the RBD disk image moves between these two clients periodically (every few minutes the owner flips). We are pretty sure that no virsh commands possibly having that effect are executed during this time. The client connections are not lost and the OSD blacklist is empty. I don't understand why a ceph rbd client would surrender an exclusive lock in such a split brain situation, its exactly when it needs to hold on to it. As a result, the affected virtual drives are corrupted. The questions we have in this context are: Under what conditions does a ceph rbd client surrender an exclusive lock? Could this be a bug in the client or a ceph config error? Is this a known problem with libceph and libvirtd? Anyone else making the same observation and having some guidance? The VM hosts are on alma8 and we use the advanced virtualisation repo providing very recent versions of qemu and libvirtd. We have seen this floating exclusive lock before on mimic. Now we are on octopus and I can't really blame it on the old ceph version any more. We use opennebula as a KVM front-end. Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

MDS crash in "inotablev == mds->inotable->get_version()"
by Kenny Van Alstyne 18 Jan '23

18 Jan '23

Hey all! I’ve run into an MDS crash on a cluster recently upgraded from Ceph 16.2.7 to 16.2.10. I’m hitting an assert nearly identical to this one gathered by the telemetry module: https://tracker.ceph.com/issues/54747 I have a new build compiling to test whether https://github.com/ceph/ceph/pull/43184/ makes a difference or not, when setting mds_inject_skip_replaying_inotable. Relevant logs are below, but I’m wondering if anyone has hit anything like this? Thanks in advance! === BEGIN LOG SNIPPET === -2> 2023-01-18T20:16:29.789+0000 7f6190243700 -1 log_channel(cluster) log [ERR] : journal replay alloc 0x10000000010 not in free [0x10000000011~0x3dc,0x100000003fb~0x1e8,0x100000005e5~0x2,0x100000009d4~0x2,0x1000005cc6d~0x4,0x10001c6b44e~0x4,0x10001cb91f4~0x1f4,0x10001cb93f4~0x3dd,0x10007582c15~0x279,0x10007582e90~0x1f4,0x10007583094~0xfff8a7cf6c] -1> 2023-01-18T20:16:29.789+0000 7f6190243700 -1 /builds/66321/e7c73776/ceph/-build//WORKDIR/ceph-16.2.10/src/mds/journal.cc: In function 'void EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)' thread 7f6190243700 time 2023-01-18T20:16:29.794189+0000 /WORKDIR/ceph-16.2.10/src/mds/journal.cc: 1577: FAILED ceph_assert(inotablev == mds->inotable->get_version()) ceph version 16.2.10 (e7c73776b3136f6d18a35febeb38f5fdd41be364) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14c) [0x7f619d548645] 2: /usr/lib/ceph/libceph-common.so.2(+0x27182f) [0x7f619d54882f] 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5815) [0x560bfd1c6935] 4: (EUpdate::replay(MDSRank*)+0x3c) [0x560bfd1c7ecc] 5: (MDLog::_replay_thread()+0xca9) [0x560bfd153de9] 6: (MDLog::ReplayThread::entry()+0xd) [0x560bfce78fdd] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f619cf29fa3] 8: clone() 0> 2023-01-18T20:16:29.793+0000 7f6190243700 -1 *** Caught signal (Aborted) ** in thread 7f6190243700 thread_name:md_log_replay ceph version 16.2.10 (e7c73776b3136f6d18a35febeb38f5fdd41be364) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f619cf34730] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19d) [0x7f619d548696] 5: /usr/lib/ceph/libceph-common.so.2(+0x27182f) [0x7f619d54882f] 6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5815) [0x560bfd1c6935] 7: (EUpdate::replay(MDSRank*)+0x3c) [0x560bfd1c7ecc] 8: (MDLog::_replay_thread()+0xca9) [0x560bfd153de9] 9: (MDLog::ReplayThread::entry()+0xd) [0x560bfce78fdd] 10: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f619cf29fa3] 11: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. === END LOG SNIPPET ===

1 0

Flapping OSDs on pacific 16.2.10
by J-P Methot 18 Jan '23

18 Jan '23

Hi, We have a full SSD production cluster running on Pacific 16.2.10 and deployed with cephadm that is experiencing OSD flapping issues. Essentially, random OSDs will get kicked out of the cluster and then automatically brought back in a few times a day. As an example, let's take the case of OSD.184 : -It flapped 9 times between January 15th and 17th with the following log message each time : 2023-01-15T16:33:19.903+0000 prepare_failure osd.184 from osd.49 is reporting failure:1 -On January 17th, it complains that there are slow ops and spam its logs with the following line : heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s The storage node itself has over 30 GB of ram still available in cache and the drives themselves only seldom peak at 100% usage and that never lasts more than a few seconds. CPU usage is also constantly around 5%. Considering there is no other error messages in any of the regular logs, including the systemd logs, why would this OSD not reply to heartbeats? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

4 6

ceph orch osd spec questions
by Wyll Ingersoll 18 Jan '23

18 Jan '23

Ceph Pacific 16.2.9 We have a storage server with multiple 1.7TB SSDs dedicated to the bluestore DB usage. The osd spec originally was misconfigured slightly and had set the "limit" parameter on the db_devices to 5 (there are 8 SSDs available) and did not specify a block_db_size. ceph layed out the original 40 OSDs and put 8 DBs across 5 of the SSDs (because of limit param). Ceph seems to have auto-sized the bluestore DB partitions to be about 45GB, which is far less than the recommended 1-4% (using 10TB drives). How does ceph-volume determine the size of the bluestore DB/WAL partitions when it is not specified in the spec? We updated the spec and specified a block_db_size of 300G and removed the "limit" value. Now we can see in the cephadm.log that the ceph-volume command being issued is using the correct list of SSD devices (all 8) as options to the lvm batch (--db-devices ...), but it keeps failing to create the new OSD because we are asking for 300G and it thinks there is only 44G available even though the last 3 SSDs in the list are empty (1.7T). So, it appears that somehow the orchestrator is ignoring the last 3 SSDs. I have verified that these SSDs are wiped clean, have no partitions or LVM, and no label (sgdisk -Z, wipefs -a). They appear as available in the inventory and not locked or otherwise in use. Also, the "db_slots" spec parameter is ignored in pacific due to a bug so there is no way to tell the orchestrator to use "block_db_slots". Adding it to the spec like "block_db_size" fails since it is not recognized. Any help figuring out why these SSDs are being ignored would be much appreciated. Our spec for this host looks like this: --- spec: data_devices: rotational: 1 size: '3TB:' db_devices: rotational: 0 size: ':2T' vendor: 'SEAGATE' block_db_size: 300G ---

1 1

Ceph-ansible: add a new HDD to an already provisioned WAL device
by Len Kimms 18 Jan '23

18 Jan '23

Hello all, we’ve set up a new Ceph cluster with a number of nodes which are all identically configured. There is one device vda which should act as WAL device for all other devices. Additionally, there are four other devices vdb, vdc, vdd, vde which use vda as WAL. The whole cluster was set up using ceph-ansible (branch stable-7.0) and Ceph version 17.2.0. Device configuration in osds.yml looks as follows: devices: [/dev/vdb, /dev/vdc, /dev/vdd, /dev/vde] bluestore_wal_devices: [/dev/vda] As expected vda contains four logical volumes for WAL each 1/4 of the overall vda disk size (‘ceph-ansible/group_vars/all.yml’ has default ‘block_db_size: -1’). After the initial setup, we’ve added an additional device vdf which should become a new OSD. The new OSD should use vda for WAL as well. This means the previous four WAL LVs have to be resized down to 1/5 and a new LV has to be added. Is it possible to retroactively add a new device to an already provisioned WAL device? We suspect that this is not possible because the ceph-bluestore-tool does not provide any way to shrink an existing BlueFS device. Only expanding is currently possible (https://docs.ceph.com/en/quincy/man/8/ceph-bluestore-tool/) Simply adding the new device to the devices list and rerunning the playbook does nothing. And so does only setting “devices: [/dev/vdf]” and “bluestore_wal_devices: [/dev/vda]”. In both cases vda is rejected because “Insufficient space (<10 extents) on vgs” which makes sense because vda is already fully used by the previous four OSD WALs. Thanks for the help and kind regards. Additional notes: - We’re testing pre-production on an emulated cluster hence the device names vdx and unusually small device sizes. - The output of `lsblk` after the initial setup looks as follows: ``` vda 252:0 0 8G 0 disk ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--3677c354--8d7d--4db9--a2b7--68aeb8248d40 253:2 0 2G 0 lvm ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--52d71122--b573--4077--9633--968c178612fd 253:4 0 2G 0 lvm ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--2d7eb467--cfb1--4a00--8a45--273932036599 253:6 0 2G 0 lvm └─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--d7b13b79--219c--4002--9e92--370dff7a5376 253:8 0 2G 0 lvm vdb 252:16 0 8G 0 disk └─ceph--49ddaa8b--5d8f--4267--85f9--5cac608ce53d-osd--block--861a53c7--ee57--4c5f--9546--1dd7cb0185ef 253:1 0 8G 0 lvm vdc 252:32 0 5G 0 disk └─ceph--1ed9ee91--e071--4ea4--9703--d56d84d9ae0a-osd--block--8aacb66a--e29b--4b7a--8ad5--a9fb1f81c6d6 253:3 0 5G 0 lvm vdd 252:48 0 5G 0 disk └─ceph--554cdd8b--e722--41a9--8f64--c09c857cc0dc-osd--block--4dee3e1b--b50d--4154--b2ff--80cadb67e2a0 253:5 0 5G 0 lvm vde 252:64 0 5G 0 disk └─ceph--5d58de32--ca55--4895--8ac7--af94ee07672e-osd--block--3f563f40--0c1e--4cca--9325--d9534cceb711 253:7 0 5G 0 lvm vdf 252:80 0 5G 0 disk ``` - Ceph status is happy and healthy: ``` cluster: id: ff043ce8-xxxx-xxxx-xxxx-e98d073c9d09 health: HEALTH_WARN mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum baloo-1,baloo-2,baloo-3 (age 13m) mgr: baloo-2(active, since 5m), standbys: baloo-3, baloo-1 mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 4m), 24 in (since 5m) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 7 pools, 177 pgs objects: 213 objects, 584 KiB usage: 98 MiB used, 138 GiB / 138 GiB avail pgs: 177 active+clean ```

2 2

[RFC] Detail view of OSD network I/O
by Nico Schottelius 18 Jan '23

18 Jan '23

Good morning ceph community, for quite some time I was wondering if it would not make sense to add an iftop alike interface to ceph that shows network traffic / iops on a per IP basis? I am aware of "rbd perf image iotop", however I am much more interested into a combined metric featuring 1) Which clients read/write to where and 2) inter OSDs traffic to see the total load on the cluster and being able to drill down. For example, the metric could look like this: -------------------------------------------------------------------------------- FROM TO Bytes/s Packets/s osd.0 [IP] -> [IP] osd.10 .. .. osd.0 [IP] -> [IP] client .. .. -------------------------------------------------------------------------------- Given that this table would be sortable by from/to/min-or-max bytes/min-or-max packets, this would allow spotting the And maybe a summarised view such as: -------------------------------------------------------------------------------- FROM IN Bytes/s OUT Bytes/s IN Packets/s OUT Packets/s osd.0 [IP] osd.10 [IP] -------------------------------------------------------------------------------- This way it would be nicely possible to identify high load. If it was combined with average/current latency, it would potentially also be able to find the bottlenecks in the cluster. From my perspective, easily combining client + intra cluster traffic would be very helpful. What do you think, does that make sense, does it already exist or how do you approach this? Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0