Hardware problems / good replacements

[Phil Reynolds]

Since I upgraded my motherboard, processor and memory last summer I have
found that virtual machines (be they under kvm or virtualbox) perform
less well than expected. My host system is Debian squeeze, with some
backports including the kernel. The problem is most noticeable with kvm
running FreeBSD - lots of WRITE_DMA timeouts on the emulated disk.

The motherboard is an Asus M5A99X EVO, and I am using a 6 core Phenom
processor (1090T I think). I currently have 8GB of RAM, far more than I
am allowing VMs, and I am not, at least at present, trying to run more
than one at a time.

I have just updated the BIOS on the motherboard - the timeouts are still
happening, but more quickly, and there seems to be more processing by
the VM between them. (I do have the "SVT" option enabled in the BIOS)

I am wondering whether this problem is likely to be down to the
motherboard or RAM, or a defective processor. Or could it possibly be
the graphics card? I am running the VM with the -nographic switch when
using BSDs, but with graphics for other systems.

If it is likely to be the motherboard, is there a particularly good one
for Linux systems at the moment, with at least nine SATA ports?

I mention my graphics card because, in the last few weeks, I have had
several odd crashes where everything except X kept working, but the
display was frozen - even killing X left the display on. tvtime and
Anagram Genius (under wine) both appeared to cause some display
corruption shortly before this happened.

Is there a good graphics card on the market, preferably with
DFSG-compliant drivers available, that would be adequate for office use
and some multimedia, but no particularly intense games? If so, I might
try replacing it anyway. I have tended to avoid on-board graphics in the
past, but if things have improved, that would be acceptable.

Suggestions are welcome on these problems.

[James Courtier-Dutton]

Post by Phil Reynolds
Since I upgraded my motherboard, processor and memory last summer I have
found that virtual machines (be they under kvm or virtualbox) perform
less well than expected. My host system is Debian squeeze, with some
backports including the kernel. The problem is most noticeable with kvm
running FreeBSD - lots of WRITE_DMA timeouts on the emulated disk.
The motherboard is an Asus M5A99X EVO, and I am using a 6 core Phenom
processor (1090T I think). I currently have 8GB of RAM, far more than I
am allowing VMs, and I am not, at least at present, trying to run more
than one at a time.
I have just updated the BIOS on the motherboard - the timeouts are still
happening, but more quickly, and there seems to be more processing by
the VM between them. (I do have the "SVT" option enabled in the BIOS)
I am wondering whether this problem is likely to be down to the
motherboard or RAM, or a defective processor. Or could it possibly be
the graphics card? I am running the VM with the -nographic switch when
using BSDs, but with graphics for other systems.
If it is likely to be the motherboard, is there a particularly good one
for Linux systems at the moment, with at least nine SATA ports?
I mention my graphics card because, in the last few weeks, I have had
several odd crashes where everything except X kept working, but the
display was frozen - even killing X left the display on. tvtime and
Anagram Genius (under wine) both appeared to cause some display
corruption shortly before this happened.
Is there a good graphics card on the market, preferably with
DFSG-compliant drivers available, that would be adequate for office use
and some multimedia, but no particularly intense games? If so, I might
try replacing it anyway. I have tended to avoid on-board graphics in the
past, but if things have improved, that would be acceptable.
Suggestions are welcome on these problems.

In theory, stuff running in a VM should not mind what the actual hardware is.
The only thing I can think of that might be a problem is the system timer.
Maybe the VM is not keeping accurate time, so timeouts are happening
when they should not.

Kind Regards

James
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Phil Reynolds]

Post by James Courtier-Dutton
In theory, stuff running in a VM should not mind what the actual hardware is.
The only thing I can think of that might be a problem is the system timer.
Maybe the VM is not keeping accurate time, so timeouts are happening
when they should not.

Hmmm... how could I check for a system timer problem?

[James Courtier-Dutton]

Post by Phil Reynolds

Post by James Courtier-Dutton
In theory, stuff running in a VM should not mind what the actual hardware is.
The only thing I can think of that might be a problem is the system timer.
Maybe the VM is not keeping accurate time, so timeouts are happening
when they should not.

Hmmm... how could I check for a system timer problem?

How far does it get in the boot process? Does it get as far as the command line?

Start with a PC that is keeping accurate time that you trust, using ntpd.
One way to check the timer is to just type "date" at the command line
in the VM and compare it the other trusted machine. Note the
difference, to the nearest second or two.
Then, wait about 10 mins and try it again, and see how far out it is that time.
If the difference is the same both times, you don't have a timer
problem, if the time is out, then there is your problem.
On problem machines, I saw the time drifting out by more than 10
seconds over a 10 minute period, so it is easy to spot.
If the time is your problem, you need to look at virtualised
clock-source drivers for inside the VM.

Kind Regards

James
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Phil Reynolds]

Post by James Courtier-Dutton

Post by Phil Reynolds
Hmmm... how could I check for a system timer problem?

How far does it get in the boot process? Does it get as far as the command line?

Yes, always did even before the recent improvements.

Post by James Courtier-Dutton
Start with a PC that is keeping accurate time that you trust, using ntpd.
One way to check the timer is to just type "date" at the command line
in the VM and compare it the other trusted machine. Note the
difference, to the nearest second or two.
Then, wait about 10 mins and try it again, and see how far out it is that time.
If the difference is the same both times, you don't have a timer
problem, if the time is out, then there is your problem.
On problem machines, I saw the time drifting out by more than 10
seconds over a 10 minute period, so it is easy to spot.
If the time is your problem, you need to look at virtualised
clock-source drivers for inside the VM.

When I next actually see a problem, or when I finish my current round of
bringing my VMs up to date (they are all roughly six months out of date
at present), I will try this test. Thanks for the hint, though if it
does prove a problem, I am not sure how to deal with the fix.

[Phil Reynolds]

Post by James Courtier-Dutton

Post by Phil Reynolds
Hmmm... how could I check for a system timer problem?

Start with a PC that is keeping accurate time that you trust, using ntpd.
One way to check the timer is to just type "date" at the command line
in the VM and compare it the other trusted machine. Note the
difference, to the nearest second or two.
Then, wait about 10 mins and try it again, and see how far out it is that time.
If the difference is the same both times, you don't have a timer
problem, if the time is out, then there is your problem.
On problem machines, I saw the time drifting out by more than 10
seconds over a 10 minute period, so it is easy to spot.
If the time is your problem, you need to look at virtualised
clock-source drivers for inside the VM.

Well, I finally finished the task of bringing my VMs up to date. I've
also now tried this test. Only in NetBSD did it fail.

Guest system Time drift Messages Notes
============ ========== ======== =====

FreeBSD 9.0 0s in 10 min None None
NetBSD 5.1_STABLE 102s in 10 min None None
OpenBSD 5.0-stable 1s in 10 min None Note 1
Debian kFreeBSD 6.0 1s in 13 min Message 1 below None
Debian Linux wheezy 0s in 10 min None None
Reactos 0.3.14 0s in 10 min N/A None
Windows 7 2s in 2h10m N/A None

Message 1:
ad0: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=... (number varies)

(FreeBSD 8.x used to do the same)

Note 1: OpenBSD result above is with acpihpet* disabled as per the
suggestion at http://tinyurl.com/7ov4ezw - although with acpihpet*
enabled, the result is the same, so obviously not significant in this
case. As it defaults to enabled, I will leave it so.

These are all still on the qemu-kvm that came with Debian Linux
squeeze... I have not yet tried the backports, but I intend to do so.

[James Courtier-Dutton]

Post by Phil Reynolds

Post by James Courtier-Dutton

Post by Phil Reynolds
Hmmm... how could I check for a system timer problem?

Start with a PC that is keeping accurate time that you trust, using ntpd.
One way to check the timer is to just type "date" at the command line
in the VM and compare it the other trusted machine. Note the
difference, to the nearest second or two.
Then, wait about 10 mins and try it again, and see how far out it is that time.
If the difference is the same both times, you don't have a timer
problem, if the time is out, then there is your problem.
On problem machines, I saw the time drifting out by more than 10
seconds over a 10 minute period, so it is easy to spot.
If the time is your problem, you need to look at virtualised
clock-source drivers for inside the VM.

Well, I finally finished the task of bringing my VMs up to date. I've
also now tried this test. Only in NetBSD did it fail.
Guest system            Time drift      Messages                Notes
============            ==========      ========                =====
FreeBSD 9.0             0s in 10 min    None                    None
NetBSD 5.1_STABLE       102s in 10 min  None                    None
OpenBSD 5.0-stable      1s in 10 min    None                    Note 1
Debian kFreeBSD 6.0     1s in 13 min    Message 1 below         None
Debian Linux wheezy     0s in 10 min    None                    None
Reactos 0.3.14          0s in 10 min    N/A                     None
Windows 7               2s in 2h10m     N/A                     None
ad0: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=... (number varies)
(FreeBSD 8.x used to do the same)
Note 1: OpenBSD result above is with acpihpet* disabled as per the
suggestion at http://tinyurl.com/7ov4ezw - although with acpihpet*
enabled, the result is the same, so obviously not significant in this
case. As it defaults to enabled, I will leave it so.
These are all still on the qemu-kvm that came with Debian Linux
squeeze... I have not yet tried the backports, but I intend to do so.

Ok, so updating the VMs to more up to date kernels has cured your
problem on everything except BSD.
For information, my comments/experience regarding the timer were
around Linux and not BSD.
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Richard W.M. Jones]

Post by Phil Reynolds
Since I upgraded my motherboard, processor and memory last summer I have
found that virtual machines (be they under kvm or virtualbox) perform
less well than expected.

What test are you running to determine this?

Post by Phil Reynolds
My host system is Debian squeeze, with some backports including the
kernel. The problem is most noticeable with kvm running FreeBSD -
lots of WRITE_DMA timeouts on the emulated disk.

Does FreeBSD have a virtio driver? Google is a bit unclear as to
whether a stable, finished driver exists. In any case if you're using
IDE emulation, that would be your performance problem right there.
Use a guest that supports virtio block and network .. it'll be a lot
faster.

The second problem is Debian squeeze ... because it's old. Using the
latest kernel, qemu-kvm and libvirt should yield significantly better
results.

Post by Phil Reynolds
The motherboard is an Asus M5A99X EVO, and I am using a 6 core Phenom
processor (1090T I think). I currently have 8GB of RAM, far more than I
am allowing VMs, and I am not, at least at present, trying to run more
than one at a time.

A 6 core Phenom surely has enough oomph to run virtual machines fast
(although not as fast as if you're using virtio and the latest
software). To check, you could compare your cpu flags to this page:

http://virt-tools.org/learning/check-hardware-virt/

To check for a defective processor / cache etc. look at 'mcelog'.

Rich.

[Phil Reynolds]

Post by Richard W.M. Jones

Post by Phil Reynolds
Since I upgraded my motherboard, processor and memory last summer I have
found that virtual machines (be they under kvm or virtualbox) perform
less well than expected.

What test are you running to determine this?

Nothing very scientific - just that they are (or were, before the BIOS
upgrade, at least) noticeably much slower than on the previous 3 core
processor.

Post by Richard W.M. Jones
Does FreeBSD have a virtio driver? Google is a bit unclear as to
whether a stable, finished driver exists. In any case if you're using
IDE emulation, that would be your performance problem right there.
Use a guest that supports virtio block and network .. it'll be a lot
faster.

I have looked into this before and not really reached any conclusion. I
fully intend to look into this feature when I can but it will probably
need a complete "nuke and pave". One attempt I made proved difficult to
partition.

Post by Richard W.M. Jones
The second problem is Debian squeeze ... because it's old. Using the
latest kernel, qemu-kvm and libvirt should yield significantly better
results.

I tried a later qemu-kvm... it didn't seem to make much difference on
what I could actually get it to do at the time. I will try it again
fairly soon, if this problem persists.

Post by Richard W.M. Jones
A 6 core Phenom surely has enough oomph to run virtual machines fast
(although not as fast as if you're using virtio and the latest
http://virt-tools.org/learning/check-hardware-virt/

A pass, as ever when I checked.

Post by Richard W.M. Jones
To check for a defective processor / cache etc. look at 'mcelog'.

Installed - I'll see what happens.

[Richard W.M. Jones]

Post by Phil Reynolds

Post by Richard W.M. Jones

Post by Phil Reynolds
Since I upgraded my motherboard, processor and memory last summer I have
found that virtual machines (be they under kvm or virtualbox) perform
less well than expected.

What test are you running to determine this?

Nothing very scientific - just that they are (or were, before the BIOS
upgrade, at least) noticeably much slower than on the previous 3 core
processor.

I'm surprised that a BIOS update made any difference. I wasn't aware
that virtualization extensions could be disabled on AMD; on Intel it
is (or used to be) a problem that virtualization would be disabled by
the BIOS and couldn't be reenabled, requiring a BIOS update or even a
replacement motherboard to rectify. Perhaps the BIOS update just
fixed some clock speed or memory settings, making the machine
generally faster.

It's academic now, but before the BIOS update did you check the
obvious stuff; all cores enabled? core speed? BogoMIPS? memory
settings set for maximum performance ...?

Post by Phil Reynolds

Post by Richard W.M. Jones
The second problem is Debian squeeze ... because it's old. Using the
latest kernel, qemu-kvm and libvirt should yield significantly better
results.

I tried a later qemu-kvm... it didn't seem to make much difference on
what I could actually get it to do at the time. I will try it again
fairly soon, if this problem persists.

You need to upgrade the whole stack, from kernel up through userspace
tools. This needs coordination from all components.

Post by Phil Reynolds

Post by Richard W.M. Jones
A 6 core Phenom surely has enough oomph to run virtual machines fast
(although not as fast as if you're using virtio and the latest
http://virt-tools.org/learning/check-hardware-virt/

A pass, as ever when I checked.

Also check that KVM is actually being used. You should see the
'kvm_amd' module is loaded, and there should be no warnings or errors
in 'dmesg' about virtualization being disabled.

Rich.

[Phil Reynolds]

Post by Richard W.M. Jones
I'm surprised that a BIOS update made any difference. I wasn't aware
that virtualization extensions could be disabled on AMD; on Intel it
is (or used to be) a problem that virtualization would be disabled by
the BIOS and couldn't be reenabled, requiring a BIOS update or even a
replacement motherboard to rectify. Perhaps the BIOS update just
fixed some clock speed or memory settings, making the machine
generally faster.

Well, it could be switched off, but to my knowledge it was always on.

Post by Richard W.M. Jones
It's academic now, but before the BIOS update did you check the
obvious stuff; all cores enabled? core speed? BogoMIPS? memory
settings set for maximum performance ...?

Checked the lot - if anything, attempting to improve performance made it
even worse.

Post by Richard W.M. Jones

Post by Phil Reynolds

Post by Richard W.M. Jones
The second problem is Debian squeeze ... because it's old. Using the
latest kernel, qemu-kvm and libvirt should yield significantly better
results.

I tried a later qemu-kvm... it didn't seem to make much difference on
what I could actually get it to do at the time. I will try it again
fairly soon, if this problem persists.

You need to upgrade the whole stack, from kernel up through userspace
tools. This needs coordination from all components.

I will consider that before expecting miracles. If I recall correctly,
libvirt was the most daunting to deal with.

Post by Richard W.M. Jones
Also check that KVM is actually being used. You should see the
'kvm_amd' module is loaded, and there should be no warnings or errors
in 'dmesg' about virtualization being disabled.

That was the first thing I checked. Nothing ever appeared to be going
wrong with it.

[David L Neil]

Hi Rich,

Post by Richard W.M. Jones
I'm surprised that a BIOS update made any difference. I wasn't aware
that virtualization extensions could be disabled on AMD; on Intel it
is (or used to be) a problem that virtualization would be disabled by
the BIOS and couldn't be reenabled, requiring a BIOS update or even a
replacement motherboard to rectify. Perhaps the BIOS update just
fixed some clock speed or memory settings, making the machine
generally faster.

In my !extensive! experience (three (Intel) machines) VT was always
disabled. However I've not had to re-flash the BIOS to enable it.

Post by Richard W.M. Jones
It's academic now, but before the BIOS update did you check the
obvious stuff; all cores enabled? core speed? BogoMIPS? memory
settings set for maximum performance ...?

...

Post by Richard W.M. Jones
You need to upgrade the whole stack, from kernel up through userspace
tools. This needs coordination from all components.

...

Post by Richard W.M. Jones
Also check that KVM is actually being used. You should see the
'kvm_amd' module is loaded, and there should be no warnings or errors
in 'dmesg' about virtualization being disabled.

These are all useful pointers. Thank you.

Most of the 'advice' out-there seems of the order of:
- grep /proc/cpuinfo,
- yum kvm,
- reboot,
- start manager,
- build shiny new VMs
(and they all lived happily ever-after).

Without wishing to hi-jack the thread too far away from the OP's needs,
could you please suggest some 'homework reading' that offers the
user-level virt-admin solid (and cohesive) advice about
- setting-up the hardware [per "whole stack", "coordination from all"];
- configuring a minimal host to run KVM, the virt tools, Manager, etc;
- checking/optimising that["dmesg"];
- before some good-practices for configuring VMs? (both to co-exist and
for individual efficiency)

Is there any reliable (and up-to-date!?) material, falling somewhere
between the 'clickety-click and there you are' fairy story (which
plainly didn't work for the OP either) and the 'disappearing under the
bonnet covered in greasy source code' of a virt-Rich?
--
Regards,
=dn

PS [ref earlier posts] when my newly-recycled and specifically-purchased
h/w didn't seem to be enough for them (nor did it even work properly -
sigh, sob, snivel), I dropped-out of the RHEV beta. Perhaps some of
those docs would serve? However most of us are likely using a less
bleeding-edge distro lacking their (your?) neatly packaged management
features. (OP=Debian Squeeze, me=Fedora 16, ...)
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[James Roberts]

Post by David L Neil
Without wishing to hi-jack the thread too far away from the OP's needs,

This is not a suggestion of homework reading but something that you (and
perhaps others) might be interested in poking at (and getting ideas, or
dire warnings, from) as it seems, in my clearly-limited experience and
knowledge, to work rather well as a KVM solution +:

http://pve.proxmox.com/wiki/Main_Page

It is a Debian-based build, so probably not suitable if you prefer the
RH route. We have been running it for a couple of years with
satisfaction. It has most of the bits joined up nicely and working well
(IM,c-l,HO).
--
Stabilys Ltd www.stabilys.com
244 Kilburn Lane
LONDON
W10 4BA

0845 838 5370
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Richard W.M. Jones]

Post by David L Neil
Without wishing to hi-jack the thread too far away from the OP's
needs, could you please suggest some 'homework reading' that offers
the user-level virt-admin solid (and cohesive) advice about
- setting-up the hardware [per "whole stack", "coordination from all"];
- configuring a minimal host to run KVM, the virt tools, Manager, etc;
- checking/optimising that["dmesg"];
- before some good-practices for configuring VMs? (both to co-exist
and for individual efficiency)

Some is at http://virt-tools.org/ -- wanting to make this more
comprehensive when someone has the time.

We also have the RHEL 6 virtualization guide (actually 3 different
guides):

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/index.html

There is also talk of writing a RHEL 6 virt performance guide but I
don't know how that is going.

Rich.