the virtuozzo admin

Have a tricky VPS that vzmigrate just can’t move? Luckily, Virtuozzo (and OpenVZ for that matter) are pretty simple in terms of how the containers are set up. You’re mainly dealing with two locations that data for the container is stored:

/vz/private/$VEID
/etc/vz/conf/$VEID.conf

To manually move a container to another node, all you need to do is copy those folders:

rsync -av /vz/private/$VEID $newserver:/vz/private
rsync -av /etc/vz/conf/$VEID $newserver:/etc/vz/conf

Then on the server server, just start it right up. You do need to have the container’s OS template installed on the new node, as referenced in the container’s config file.

This same process also applies to cloning a VPS, which is particularly useful on an OpenVZ server where the vzmlocal command isn’t available. All you have to do is use similar commands to rsync the container’s private folder to a new VEID:

rsync -av /vz/private/$VEID /vz/private/$NEWVEID
rsync -av /etc/vz/conf/$VEID /etc/vz/conf/$NEWVEID

Then just change the hostname and ip variables in the new container’s config file.

If your server already has VZ 4.0, it’s very easy to upgrade to 4.6. You pretty much don’t have a choice because if you have other servers running 4.6, you won’t be able to move containers back and forth, and you won’t have many new kernel updates headed your way.

Note: This procedure is only for VZ 4.0 servers. If you run a previous version, you need to re-OS and reinstall VZ. You can’t upgrade to 4.6 from previous vesions!

1. Migrate all your containers off the server

If this is a live server, you’ll want to move off all your customers to avoid lengthy downtime.  If you plan on moving them back, you can speed up the reverse process by moving off the containers and retaining their private areas

for ve in `vzlist -H -o veid,diskspace |sort -rnk2 |awk '{print $1}'` ; do vzmigrate -r no 192.168.x.x $ve ; done

2. Run a Yum update

Take this opportunity to update your operating system, noting that Virtuozzo already has an exclude file (/etc/yum/swsoft-excludes) to prevent the updates of certain packages.

yum update -y

3) install VZ 4.6

cd /root
wget http://download.parallels.com/pvc/46/lin/rtm/vzinstall-linux-x86_64.bin
chmod 755 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin

4) Run the self update

After the installation, run vzup2date twice (one is a self-update, the other is the kernel update)

5) reboot

6) Reapply customizations

If you had any customizations to VZ scripts (such as files in /etc/sysconfig/vz-script), you may need to re-apply them

7) Re-populate the server

Start migrations back to the server, which will take a fraction of the time since the private areas were retained:

for ve in `vzlist -H -o veid,diskspace |sort -rnk2 |awk '{print $1}'` ; do vzmigrate -r yes 192.168.x.x $ve; done

When activating or updating a VZ license you need to have a HWID for your machine. In case you don’t know how to find that, simply run:

cat /proc/vz/hwid

If you see more than one listed, always use the first one

If you attempt to start a container after a hardware node crash, you may see an error like this:

Starting Container ...
vzquota : (warning) Incorrect quota shutdown for id 101, recalculating disk usage
vzquota : (error) quota check : lstat `somefile.ext': No such file or directory
Running vzquota on failed for Container 101 [1]

This means that the container’s quota system is crashed. You can restart it simply switching the quota off and on:

vzquota off 101
vzquota on 101

The container should now start up properly.

When troubleshooting load issues on a node, top is still an effective command for a quick glance at what’s going on. However, since the PIDs for processes in a container do not correlate with users on the parent node, it can sometimes be unclear as to what container is running the processes.

Virtuozzo has a built-in command called vzpid which identifies the container running a given process. For example, if I wanted to find out which container was running process ID #31299:

[root@server ~]# vzpid 31299
Pid    VEID    Name
31299    102    mysqld

When starting up a VPS, you may see an error like this:

vzquota : (error) Quota on syscall for 101: Device or resource busy

In more detail:

Starting Container ...
vzquota : (error) Quota on syscall for id 1: Device or resource busy
vzquota : (error)     Possible reasons:
vzquota : (error)     - Container's root is already mounted
vzquota : (error)     - there are opened files inside Container's private area
vzquota : (error)     - your current working directory is inside Container's
vzquota : (error)       private area
vzquota : (error)     Use -v option to see currently opened file(s).
Running vzquota on failed for Container 1 [3]
Container start failed

If this happens, run a grep to see if any processes are running from inside the container’s private folder (replace ’1′ with your VPS id)

lsof 2> /dev/null | egrep '/vz/root/1/|/vz/private/1/'

Kill of any processes this returns, and restart the container.

I came across a VPS that was recently moved to another node and noticed that Apache wasn’t starting due to missing modules.  When I saw that the modules folder in Apache was linked to /usr/lib64, I checked that folder to see that the httpd directory was there, but not really “there”. It looked like this:

?---------  ? ?    ?       ?            ? httpd

Question marks in the file name and descriptor, clearly indicating that the VPS container was missing something. Well, I went onto the parent node and into /vz/private/VEID/fs/root/usr/lib64 and the files were there, along with some strange broken symlink that looks like this:

lrwxrwxrwx   1 root root   72 Mar  8  2009 ._vzlnk_. -> ////centos/5/x86_64/httpd-2.2.3-11.el5_2.centos.4.x86_64/usr/lib64/httpd

To fix one instance, I removed the broken symlink and that fixed the problem. I also tried to fix this on another container by copying over the httpd folder in the lib64 directory and that did the trick as well, but I assume it’s the broken symlink that caused the problem.

Further troubleshooting indicated that the container was somehow using a different OS template than what it was on the previous server, which was causing Virtuozzo to be unable to find some files. If this is the case, you can edit /etc/vz/conf/$VEID.conf and change the ostemplate value to what it should be, then restart the container.

If that doesn’t fix it, you’ll probably still need to remove all those bad symlinks. Here’s a simple script stolen from here that should remove all these for you:

#!/bin/bash
VEID=101 # Replace with your container ID
for i in `find /vz/private/$VEID/fs/root/ -noleaf -type l -print | perl -nle ‘-e || print’|grep ._vzlnk_.`
do
ln -sf `ls -ga $i|grep ._vzlnk_.|awk ‘{print $10}’|sed ‘s/\/\/\//\/vz\/template/’` $i
done

When executing commands on a Redhat-family VPS (Such as Fedora or CentOS) you may get the following error when executing commands via sudo:

audit_log_user_command(): Connection refused

All this means is that a feature for audit logging is disabled in the Linux kernel, and it’s nothing to be concerned about. However, it seems that no one knows how to get rid of it…

If your service container isn’t working or you need to recreate it, follow these steps:

1) Download the autoinstaller for Virtuozzo:

wget http://download.parallels.com/virtuozzo/virtuozzo4.0/linux/vzinstall-linux.bin

2) Run the autoinstaller and select the download only option. This will download the files to /root/virtuozzo

chmod 755 vzinstall-linux.bin
./vzinstall-linux.bin

3) Extract the source to a temporary location:

mkdir -p /vz/temp
/root/virtuozzo/download/Linux/{arch}/virtuozzo-{version}-{arch}.sfx -d /vz/temp--extract

4) Now create the service container and assign it an IP:

vzsveinstall -v -D /vz/temp -s $ip

If you’re installing APF on a VPS, chances are you’ll need to make a couple changes to your APF’s config file. Here are a couple errors that are common:

eth0: error fetching interface information: Device not found

You need to change the interface values in apf.conf to use venet0 instead of eth0 since the VPS is on a virtual interface:

IFACE_IN="eth0"
IFACE_OUT="eth0"

unable to load iptables module (ip_tables), aborting.

Change the MONOKERN value from ’0′ to ’1′ .