A quick update.
I have done a fresh install of the CloudStack host server running Ubuntu 18.04 with the
latest updates. I've installed ceph 12.x and connected it to Cloudstack which uses
kvm/libvirt/ceph/rbd. The rest of the ceph services (mon,mgr,osd,etc) are all running
15.2.3. Works like a charm.
As soon as I've updated the host server to version 15.2.3, Libvirt stopped working. It
just hangs without doing much it seems. Common commands like 'virsh pool-list' or
'virsh list' are just hanging. I've strace the process and it just doesn't
show any activity.
2020-07-06 18:18:36.930+0000: 3273: info : virEventPollUpdateTimeout:265 :
EVENT_POLL_UPDATE_TIMEOUT: timer=993 frequen
cy=5000
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollUpdateTimeout:282 : Set timer
freq=5000 expires=1594059521930
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollInterruptLocked:722 : Skip
interrupt, 1 140123172218240
2020-07-06 18:18:36.930+0000: 3273: info : virEventPollUpdateHandle:152 :
EVENT_POLL_UPDATE_HANDLE: watch=1004 events=1
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollInterruptLocked:722 : Skip
interrupt, 1 140123172218240
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollCleanupTimeouts:525 : Cleanup 8
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollCleanupHandles:574 : Cleanup 22
2020-07-06 18:18:36.930+0000: 3273: debug : virEventRunDefaultImpl:324 : running default
event implementation
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollCleanupTimeouts:525 : Cleanup 8
2020-07-06 18:18:36.930+0000: 3273: debug : virEventPollCleanupHandles:574 : Cleanup 22
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1,
f=5 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2,
f=7 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=2 w=3,
f=10 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=3 w=4,
f=11 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=4 w=5,
f=12 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=5 w=6,
f=13 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=6 w=7,
f=14 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=7 w=8,
f=15 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=8 w=9,
f=16 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=9
w=10, f=17 e=1 d=0
2020-07-06 18:18:36.931+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=10
w=11, f=18 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=11
w=12, f=19 e=0 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=12
w=13, f=19 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=13
w=14, f=24 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=14
w=15, f=25 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=15
w=19, f=26 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=16
w=82, f=79 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=17
w=944, f=22 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=18
w=993, f=82 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=19
w=1001, f=30 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=20
w=1002, f=33 e=1 d=0
2020-07-06 18:18:36.932+0000: 3273: debug : virEventPollMakePollFDs:401 : Prepare n=21
w=1004, f=83 e=1 d=0
2020-07-06 18:18:36.933+0000: 3273: debug : virEventPollCalculateTimeout:338 : Calculate
expiry of 8 timers
2020-07-06 18:18:36.933+0000: 3273: debug : virEventPollCalculateTimeout:346 : Got a
timeout scheduled for 1594059521930
2020-07-06 18:18:36.933+0000: 3273: debug : virEventPollCalculateTimeout:359 : Schedule
timeout then=1594059521930 now=1594059516933
2020-07-06 18:18:36.933+0000: 3273: debug : virEventPollCalculateTimeout:369 : Timeout at
1594059521930 due in 4997 ms
2020-07-06 18:18:36.933+0000: 3273: info : virEventPollRunOnce:640 : EVENT_POLL_RUN:
nhandles=21 timeout=4997
The ceph itself seems to work, i.e. I can execute ceph -s / rbd -p <pool> ls -l,
etc. That produces the output. It's just the libvirt seems to be no joy.
The version of libvirt installed is:
libvirt-bin 4.0.0-1ubuntu8.1
Any idea how I can make ceph Octopus to play nicely with libvirt?
Cheers
Andrei
----- Original Message -----
From: "Andrei Mikhailovsky"
<andrei(a)arhont.com>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Monday, 29 June, 2020 20:40:01
Subject: [ceph-users] Octopus upgrade breaks Ubuntu 18.04 libvirt
> Hello,
>
> I've upgraded ceph to Octopus (15.2.3 from repo) on one of the Ubuntu 18.04 host
> servers. The update caused problem with libvirtd which hangs when it tries to
> access the storage pools. The problem doesn't exist on Nautilus. The libvirtd
> process simply hangs. Nothing seem to happen. The log file for the libvirtd
> shows:
>
> 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 :
> dispatching to max 0 clients, called from event watch 11
> 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event
> not handled.
> 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 :
> dispatching to max 0 clients, called from event watch 11
> 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event
> not handled.
> 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:707 :
> dispatching to max 0 clients, called from event watch 11
> 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:720 : event
> not handled.
> 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:707 :
> dispatching to max 0 clients, called from event watch 11
> 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:720 : event
> not handled.
>
> Running strace on the libvirtd process shows:
>
> root@ais-cloudhost1:/home/andrei# strace -p 12040
> strace: Process 12040 attached
> restart_syscall(<... resuming interrupted poll ...>
>
>
> Nothing happens after that point.
>
> The same host server can get access to the ceph cluster and the pools by running
> ceph -s or rbd -p <pool> ls -l commands for example.
>
> Need some help to get the host servers working again with Octopus.
>
> Cheers
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io