So everything works, until virConnectGetAllDomainStats is called on the domain:
2016-05-18 06:01:15.922+0000: 3070: debug : virThreadJobSet:96 : Thread 3070 (virNetServerHandleJob) is now running job remoteDispatchConnectGetAllDomainStats 2016-05-18 06:01:15.922+0000: 3070: debug : virConnectGetAllDomainStats:11489 : conn=0x7f5a5c000ec0, stats=0x0, retStats=0x7f5a8124fad0, flags=0x0
The API wants to start a job...
2016-05-18 06:01:15.922+0000: 3070: debug : qemuDomainObjBeginJobInternal:2097 : Starting job: query (vm=0x7f5a64009bf0 name=a1, current job=query async=none) 2016-05-18 06:01:15.922+0000: 3070: debug : qemuDomainObjBeginJobInternal:2120 : Waiting for job (vm=0x7f5a64009bf0 name=a1)
... and keeps waiting for it. In the meantime, other APIs (which don't require a job) keep working just fine:
2016-05-18 06:01:45.920+0000: 3069: debug : virThreadJobSet:96 : Thread 3069 (virNetServerHandleJob) is now running job remoteDispatchDomainGetVcpus 2016-05-18 06:01:45.920+0000: 3069: debug : virDomainGetVcpus:7733 : dom=0x7f5a70004070, (VM: name=a1, uuid=048f8624-03fc-4729-8f4d-12cb4387f018), info=0x7f5a70002140, maxinfo=2, cpumaps=0x7f5a70002200, maplen=1 ... 2016-05-18 06:01:45.920+0000: 3069: debug : virDomainDispose:313 : release domain 0x7f5a70004070 a1 048f8624-03fc-4729-8f4d-12cb4387f018 2016-05-18 06:01:45.920+0000: 3069: debug : virThreadJobClear:121 : Thread 3069 (virNetServerHandleJob) finished job remoteDispatchDomainGetVcpus with ret=0
But when virConnectGetAllDomainStats times out on acquiring a job,
2016-05-18 06:01:45.922+0000: 3070: warning : qemuDomainObjBeginJobInternal:2180 : Cannot start job (query, none) for domain a1; current job is (query, none) owned by (3066 remoteDispatchDomainGetBlockIoTune, 0 <null>) for (30s, 0s) 2016-05-18 06:01:45.922+0000: 3070: error : qemuDomainObjBeginJobInternal:2192 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainGetBlockIoTune) 2016-05-18 06:01:45.922+0000: 3070: debug : virCgroupGetValueStr:814 : Get value /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1 \x2da1.scope/cpuacct.usage 2016-05-18 06:01:45.922+0000: 3070: debug : virCgroupGetValueStr:814 : Get value /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1 \x2da1.scope/cpuacct.stat 2016-05-18 06:01:45.922+0000: 3070: debug : qemuGetProcessInfo:1486 : Got status for 3500/3505 user=1507 sys=209 cpu=1 rss=531128 2016-05-18 06:01:45.922+0000: 3070: debug : qemuGetProcessInfo:1486 : Got status for 3500/3506 user=1150 sys=55 cpu=0 rss=531128
the domain object stays locked and all APIs will be blocked on trying to lock the domain...
2016-05-18 06:01:45.939+0000: 3068: debug : virThreadJobSet:96 : Thread 3068 (virNetServerHandleJob) is now running job remoteDispatchDomainGetControlInfo 2016-05-18 06:01:45.939+0000: 3068: debug : virDomainGetControlInfo:2526 : dom=0x7f5a6c002b50, (VM: name=a1, uuid=048f8624-03fc-4729-8f4d-12cb4387f018), info=0x7f5a79a50ad0, flags=0 ...
So everything works, until virConnectGetAl lDomainStats is called on the
domain:
2016-05-18 06:01:15.922+0000: 3070: debug : virThreadJobSet:96 : ndleJob) is now running job tchConnectGetAl lDomainStats lDomainStats: 11489 : 0x7f5a5c000ec0, stats=0x0, retStats= 0x7f5a8124fad0, flags=0x0
Thread 3070 (virNetServerHa
remoteDispa
2016-05-18 06:01:15.922+0000: 3070: debug : virConnectGetAl
conn=
The API wants to start a job...
2016-05-18 06:01:15.922+0000: 3070: debug : qemuDomainObjBe ginJobInternal: 2097 : ginJobInternal: 2120 :
Starting job: query (vm=0x7f5a64009bf0 name=a1, current job=query async=none)
2016-05-18 06:01:15.922+0000: 3070: debug : qemuDomainObjBe
Waiting for job (vm=0x7f5a64009bf0 name=a1)
... and keeps waiting for it. In the meantime, other APIs (which don't require
a job) keep working just fine:
2016-05-18 06:01:45.920+0000: 3069: debug : virThreadJobSet:96 : ndleJob) is now running job tchDomainGetVcp us us:7733 : 0x7f5a70004070, (VM: name=a1, uuid=048f8624- 03fc-4729- 8f4d-12cb4387f0 18), 0x7f5a70002140, maxinfo=2, cpumaps= 0x7f5a70002200, maplen=1 e:313 : 03fc-4729- 8f4d-12cb4387f0 18 ar:121 : ndleJob) finished job tchDomainGetVcp us with ret=0
Thread 3069 (virNetServerHa
remoteDispa
2016-05-18 06:01:45.920+0000: 3069: debug : virDomainGetVcp
dom=
info=
...
2016-05-18 06:01:45.920+0000: 3069: debug : virDomainDispos
release domain 0x7f5a70004070 a1 048f8624-
2016-05-18 06:01:45.920+0000: 3069: debug : virThreadJobCle
Thread 3069 (virNetServerHa
remoteDispa
But when virConnectGetAl lDomainStats times out on acquiring a job,
2016-05-18 06:01:45.922+0000: 3070: warning : qemuDomainObjBe ginJobInternal: 2180 : omainGetBlockIo Tune, 0 <null>) for (30s, 0s) ginJobInternal: 2192 : omainGetBlockIo Tune) ueStr:814 : cgroup/ cpu,cpuacct/ machine. slice/machine- qemu\x2d1 scope/cpuacct. usage ueStr:814 : cgroup/ cpu,cpuacct/ machine. slice/machine- qemu\x2d1 scope/cpuacct. stat nfo:1486 : nfo:1486 :
Cannot start job (query, none) for domain a1; current job is (query, none)
owned by (3066 remoteDispatchD
2016-05-18 06:01:45.922+0000: 3070: error : qemuDomainObjBe
Timed out during operation: cannot acquire state change lock
(held by remoteDispatchD
2016-05-18 06:01:45.922+0000: 3070: debug : virCgroupGetVal
Get value /sys/fs/
\x2da1.
2016-05-18 06:01:45.922+0000: 3070: debug : virCgroupGetVal
Get value /sys/fs/
\x2da1.
2016-05-18 06:01:45.922+0000: 3070: debug : qemuGetProcessI
Got status for 3500/3505 user=1507 sys=209 cpu=1 rss=531128
2016-05-18 06:01:45.922+0000: 3070: debug : qemuGetProcessI
Got status for 3500/3506 user=1150 sys=55 cpu=0 rss=531128
the domain object stays locked and all APIs will be blocked on trying to lock
the domain...
2016-05-18 06:01:45.939+0000: 3068: debug : virThreadJobSet:96 : ndleJob) is now running job tchDomainGetCon trolInfo trolInfo: 2526 : 0x7f5a6c002b50, (VM: name=a1, uuid=048f8624- 03fc-4729- 8f4d-12cb4387f0 18), 0x7f5a79a50ad0, flags=0
Thread 3068 (virNetServerHa
remoteDispa
2016-05-18 06:01:45.939+0000: 3068: debug : virDomainGetCon
dom=
info=
...