Hi Torkil i would check the logs of the firewalls. First I would check the palo alto firewall logs. Joachim Am Di., 13. Aug. 2024 um 14:36 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi Torkil,
did anything change in the network setup? If those errors haven't popped up before, what changed? I'm not sure if I have seen this one yet...
Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
Ceph version 18.2.1.
We have a nightly backup job snapshotting and exporting all RBDs used for libvirt VMs. Since a couple of weeks ago we've seen one or more getting stuck like this on 3 occasions, so intermittently:
" zoperator@yggdrasil:~/backup$ cat /mnt/scratch/personal/zoperator/slurm-2888600.out Creating snap: 10% complete...2024-08-07T10:35:24.687+0200 7f5a2a44b640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59fc002060 0x7f59fc0094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2
2024-08-07T10:36:02.064+0200 7f5a2a44b640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc00a2b0 0x7f59fc015010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2
2024-08-07T10:37:38.191+0200 7f5a23fff640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a0c063ea0 0x7f5a0c066320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2
2024-08-07T10:38:16.677+0200 7f5a28c48640 -1 librbd::ImageWatcher: 0x7f5a100076d0 image watch failed: 140024590898336, (107) Transport endpoint is not connected
2024-08-07T10:38:16.677+0200 7f5a28c48640 -1 librbd::Watcher: 0x7f5a100076d0 handle_error: handle=140024590898336: (107) Transport endpoint is not connected "
The VM is also throwing stack traces from stuck I/O.
Every VM seen affected by this map the RBD through multiple firewalls so that is likely a factor.
Hypervisor <-> Palo Alto firewall <-> OpenBSD firewall <-> Ceph
Any ideas? I haven't found anything in the ceph logs yet.
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io