[fuel-devops] dos.py revert fails host dosen't support invariant TSC

Bug #1356104 reported by Andrew Woodward
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Opinion
Wishlist
Fuel QA Team
6.1.x
Won't Fix
Medium
Fuel QA Team
7.0.x
Invalid
Medium
Fuel QA Team

Bug Description

This appears to be due to the admin node having specific host CPU requirements, but the slave nodes don't have the same settings. However node starts fine when initially configured by the tests, but no longer functions after the reset.

Looking at initialization, both slave nodes and the admin have the same cpu settings

QEMU emulator version 2.1.0 (Debian 2.1+dfsg-2), Copyright (c) 2003-2008 Fabrice Bellard
(fuel)andreww@aw-linux:~$ libvirtd
Display all 136 possibilities? (y or n)
(fuel)andreww@aw-linux:~$ libvirtd --version
libvirtd (libvirt) 1.2.7

(fuel)andreww@aw-linux:~$ dos.py revert --snapshot-name empty 423
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
libvirt: QEMU Driver error : unsupported configuration: host doesn't support invariant TSC
Traceback (most recent call last):
  File "/home/andreww/.virtualenvs/fuel/bin/dos.py", line 10, in <module>
    execfile(__file__)
  File "/home/andreww/git/fuel-devops/bin/dos.py", line 22, in <module>
    Shell().execute()
  File "/home/andreww/git/fuel-devops/devops/shell.py", line 29, in execute
    self.commands.get(self.params.command)(self)
  File "/home/andreww/git/fuel-devops/devops/shell.py", line 66, in do_revert
    self.params.snapshot_name)
  File "/home/andreww/git/fuel-devops/devops/models.py", line 142, in revert
    node.revert(name, destroy=False)
  File "/home/andreww/git/fuel-devops/devops/models.py", line 372, in revert
    self.driver.node_revert_snapshot(node=self, name=name)
  File "/home/andreww/git/fuel-devops/devops/helpers/retry.py", line 28, in wrapper
    return func(*args, **kwargs)
  File "/home/andreww/git/fuel-devops/devops/driver/libvirt/libvirt_driver.py", line 352, in node_revert_snapshot
    domain.revertToSnapshot(snapshot, 0)
  File "/home/andreww/.virtualenvs/fuel/local/lib/python2.7/site-packages/libvirt.py", line 1851, in revertToSnapshot
    if ret == -1: raise libvirtError ('virDomainRevertToSnapshot() failed', dom=self)
libvirt.libvirtError: unsupported configuration: host doesn't support invariant TSC
(fuel)andreww@aw-linux:~$ virsh snapshot-list 423_admin
 Name Creation Time State
------------------------------------------------------------
 empty 2014-08-11 21:05:09 -0700 paused

(fuel)andreww@aw-linux:~$ virsh snapshot-revert 423_admin
error: --snapshotname or --current is required

(fuel)andreww@aw-linux:~$ virsh snapshot-revert 423_admin --current
error: Domain snapshot not found: the domain does not have a current snapshot

(fuel)andreww@aw-linux:~$ virsh snapshot-revert 423_admin --snapshotname empty
error: unsupported configuration: host doesn't support invariant TSC

(fuel)andreww@aw-linux:~$ virsh snapshot-revert 423_slave-02 --snapshotname empty

(fuel)andreww@aw-linux:~$ virsh dumpxml 423_admin
<domain type='kvm'>
  <name>423_admin</name>
  <uuid>5997f1e6-0d0d-4a7a-9ec0-d47013822617</uuid>
  <memory unit='KiB'>1572864</memory>
  <currentMemory unit='KiB'>1572864</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
  </os>
  <cpu mode='host-model'>
    <model fallback='forbid'/>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='smep'/>
    <feature policy='require' name='fsgsbase'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='unsafe'/>
      <source file='/var/lib/libvirt/images/423_admin-system'/>
      <target dev='sda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='unsafe'/>
      <source file='/var/lib/libvirt/images/423_admin-iso'/>
      <target dev='sdb' bus='ide' tray='open'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0'/>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='64:7e:32:ba:d3:3a'/>
      <source network='423_admin'/>
      <target dev='donet51'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:91:89:3d:67:4d'/>
      <source network='423_public'/>
      <target dev='donet52'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:d9:a5:76:0a:31'/>
      <source network='423_management'/>
      <target dev='donet53'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:f7:03:3b:02:32'/>
      <source network='423_private'/>
      <target dev='donet54'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:fc:a3:9f:94:09'/>
      <source network='423_storage'/>
      <target dev='donet55'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

(fuel)andreww@aw-linux:~$ virsh dumpxml 423_slave-01
<domain type='kvm'>
  <name>423_slave-01</name>
  <uuid>6e96b44a-025e-43a4-b468-45f734d4706d</uuid>
  <memory unit='KiB'>1572864</memory>
  <currentMemory unit='KiB'>1572864</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='network'/>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <cpu mode='host-model'>
    <model fallback='forbid'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='unsafe'/>
      <source file='/var/lib/libvirt/images/423_slave-01-system'/>
      <target dev='sda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='unsafe'/>
      <source file='/var/lib/libvirt/images/423_slave-01-cinder'/>
      <target dev='sdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='unsafe'/>
      <source file='/var/lib/libvirt/images/423_slave-01-swift'/>
      <target dev='sdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='64:a4:85:77:a3:47'/>
      <source network='423_admin'/>
      <target dev='donet56'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:59:bb:52:79:82'/>
      <source network='423_public'/>
      <target dev='donet57'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:a4:35:a9:13:dd'/>
      <source network='423_management'/>
      <target dev='donet58'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:a7:e6:6a:61:82'/>
      <source network='423_private'/>
      <target dev='donet59'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='64:05:56:4d:55:10'/>
      <source network='423_storage'/>
      <target dev='donet60'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Changed in fuel:
assignee: nobody → Fuel QA Team (fuel-qa)
milestone: none → 6.0
Igor Shishkin (teran)
tags: added: fuel-devops
removed: devops
summary: - devops, dos.py revert fails host dosen't support invariant TSC
+ fuel-devops' dos.py revert fails host dosen't support invariant TSC
Igor Shishkin (teran)
Changed in fuel:
status: New → Confirmed
Changed in fuel:
milestone: 6.0 → 6.1
summary: - fuel-devops' dos.py revert fails host dosen't support invariant TSC
+ [fuel-devops] dos.py revert fails host dosen't support invariant TSC
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

As medium issue, moved it 7.0.

Changed in fuel:
status: Confirmed → Won't Fix
tags: added: qa-agree-7.0
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Moving to 'Incomplete' because issue is not reproduced for a long time.

Please feel free to provide additional information and reopen the bug.

Revision history for this message
Roman Prykhodchenko (romcheg) wrote :

Since theres's no activity for a long time after the bug was set as an incomplete one, setting it to Invalid.

Changed in fuel:
status: Invalid → Confirmed
Revision history for this message
Aleksei Stepanov (penguinolog) wrote :

Reproduced:
Ubuntu 14.04/Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz/32G RAM
Linux version 3.13.0-68-generic (buildd@lgw01-46) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015
QEMU emulator version 2.1.2 (Debian 2.1.2+dfsg-0~14.04~ppa0), Copyright (c) 2003-2008 Fabrice Bellard

Command started: bvt_2 test set

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/8.0.x
Dmitry Pyzhov (dpyzhov)
tags: added: area-qa
Revision history for this message
Aleksei Stepanov (penguinolog) wrote :

After investigations:
1. Source of issue:
    a. Virtualization support is limited by BIOS - rarely.
    b. Corrupted XML of virtual machine - most cases.
2. How to fix:
    a. fix XML manually (fast-fix for single VM, but bad way to fix any new VM's).
    b. upgrade libvirt to actual version. Some versions of libvirt contains confirmed bug, which is a reason for this behavior.

Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Moved to 9.0, due to Medium status.

Changed in fuel:
milestone: 8.0 → 9.0
Revision history for this message
Aleksei Stepanov (penguinolog) wrote :
Changed in fuel:
status: Confirmed → Opinion
Changed in fuel:
importance: Medium → Wishlist
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.