Yes, this is all set up. It was working fine until after the problem with the osd host that lost the cluster/sync network occured. There are a few other VMs that keep running along fine without this issue. I've restarted the problematic VM without success (that is, creating a file works, but overwriting it still hangs right away). fsck runs fine so reading the whole image works. I'm kind of stumped as to what can cause this. Because of the lengthy recovery, and then pg autoscaler currenty doing things there are currently lots of PGs that haven't been scrubbed, but I doubt that is an issue here. Den 2023-09-29 kl. 18:52, skrev Anthony D'Atri:
EC for RBD wasn't possible until Luminous IIRC, so I had to ask. You have a replicated metadata pool defined? Does proxmox know that this is an EC pool? When connecting it needs to know both the metata and data pools.
On Sep 29, 2023, at 12:49, peter.linder@fiberdirekt.se wrote:
(sorry for duplicate emails)
This turns out to be a good question actually.
The cluster is running Quincy, 17.2.6.
The compute node that is running the VM is proxmox, version 7.4-3. Supposedly this is fairly new, but the version of librbd1 claims to be 14.2.21 when I check with "apt list". We are not using proxmox's own ceph cluster release. However we haven't had any issues with this setup before, but we haven't been using neither erasure coded pools nor had the node-half-dead problem for such a long time.
The VM is configured using proxmox which is not libvirt but similar, and krbd is not enabled. I don't know for sure if proxmox has its own librbd linked in qemu/kvm.
"ceph features" looks like this:
{ "mon": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 5 } ], "osd": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 24 } ], "client": [ { "features": "0x3f01cfb87fecffff", "release": "luminous", "num": 4 }, { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 2 } ] }
Regards,
Peter
Den 2023-09-29 kl. 17:55, skrev Anthony D'Atri:
Which Ceph releases are installed on the VM and the back end? Is the VM using librbd through libvirt, or krbd?
On Sep 29, 2023, at 09:09, Peter Linder <peter.linder@fiberdirekt.se> wrote:
Dear all,
I have a problem that after an OSD host lost connection to the sync/cluster rear network for many hours (the public network was online), a test VM using RBD cant overwrite its files. I can create a new file inside it just fine, but not overwrite it, the process just hangs.
The VM's disk is on an erasure coded data pool and a replicated pool in front of it. EC overwrites is on for the pool.
The custer consists of 5 hosts and 4 OSDs on each, and separate hosts for compute. There is a public and separate cluster network, separated. In this case, the AOC cable to the cluster network went link down on a host and it had to be replaced and the host was rebooted. Recovery took about a week to complete. The host was half-down for about 12 hours like this.
I have some other VMs as well with images in the same pool (totally 4), and they seem to work fine, it is just this one that cant overwrite.
I'm thinking there is somehow something wrong with just this image?
Regards,
Peter _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io