Gandi Kitchen

Home > Hosting > How to Create a System Images for Your Server

How to Create a System Images for Your Server

Create a Data Disk

You need to create a data disk via the disk creation interface of your hosting account.  If you wish to make a copy of an existing disk, the data disk must be of sufficient capacity.

Simply attach your new disk to the target server ; the server that contains the virtual disk to copy, or the server used to perform the base installation.

Create a System Image

By making a copy of the data from a virtual disk.

Warning: You must have sufficient space on the destination disk.

Copy the data from the source disk to the destination with the 'tar' command:

tar cC /srv/disk1 . | tar xC /srv/disk2

If /srv/disk1 is the directory where the source disk is mounted and /srv/disk2 is the mount point of the destination disk.  Using 'tar' will only copy the data and not empty blocks, so the operation is relatively quick.

Warning:  if you want to copy the local system disk, you must add a few exclusions:

tar
--exclude=/proc --exclude=/sys --exclude=/srv/disk2 -c \ | tar xC /srv/disk2


By installing a base system

Preamble

Your new disk attached to a server is ready for use.  You can partition your disk to prepare a swap partition for the image, but we advise that you do not create  partitions and to create the file system directly on the disk.

In this way you have the advantage of flexibility if you intend to resize the disk later (no need to calculate the partition table), whilst being better adapted to the block storage on our physical filers, and to be able to decide later whether or not to use a swap file.

In the near future, all of the OS images provided by Gandi will be without partitions, and the Gandi hosting platform will automatically provide additional swap space.

Meanwhile, the virtual servers will still see the first disk as "xvda1" and the swap disk as "xvda2".

Whether or not you decide to partition, the Gandi hosting platform will be able to start a system disk with your image as source by detecting the disk boot sector and adapting the kernel boot options.  Gandi also provide a server containing copies of the distributions for which we provide system images.  You can use this server to speed up your installation : mirrors.gandi.net

Preparation

Initially, verify that the virtual disk is not already mounted (for example in /proc/mounts).  Unmount the disk if it is mounted.  Partition the disk if required and then format it.

We recommend that you format the disk directly and use the ext4 file system.  If, for example, your disk is xvdc :  mkfs.ext4 -j -m0 /dev/xvdc  and then mount the disk to a directory of your system :  mount -o rw /dev/xvdc /var/tmp

Bootstrap installation for distributions using .deb packages

The base system installation can be accomplished with debootstrap in a specific directory.  Once the installation is completed and a few modifications are made, the disk will contain a bootable and fully functional GNU/Linux system.  The system will have a basic set of applications and you will need to adapt the system to your requirements by installing any required applications with apt-get and  configuring the locales.

We highly recommend the installation of a system with the amd64/x86_64 architecture, as this will be faster on the Gandi platform.  When it comes to selecting the kernel, you should choose the appropriate x86_64 kernel (2.6.32-3831, for example).

An example installation for an Ubuntu Maverick 10.10 distribution on a previously prepared and mounted disk:

debootstrap
--arch=amd64 --verbose --components=main,universe,multiverse \
--include=openssh-server,openssh-client
maverick /var/tmp \
http://mirrors.gandi.net/ubuntu/

Should debootstrap happen to complain that no configuration file for maverick is accessible, you only need create a link in /usr/share/debootstrap/scripts from lucid to maverick.  The following step will modify the file containing the source package media locations.  Edit the file /etc/apt/sources.list in the directory by adding the distribution media and the Gandi package media.

Example :

# cat
/var/tmp/etc/apt/sources.list
deb
http://mirrors.gandi.net/ubuntu maverick main universe multiverse
deb
http://mirrors.gandi.net/ubuntu maverick-security main universe multiverse
deb
http://mirrors.gandi.net/ubuntu maverick-updates main universe multiverse
deb
http://mirrors.gandi.net/gandi/ubuntu maverick main

Once the debootstrap command and the configuration has been completed, you will be able to access the newly installed system via chroot to the directory.

For example: chroot /var/tmp

In this way, you can refresh the package media with apt-get update  and install the various applications and packages that you may require.  Beware, however, that the installation of packages in a chroot require a few corrections.  Notably, you will need to mount /proc and /dev/pts which are usually available in /etc/fstab :  mount -a  then complete the package configurations with dpkg --configure -a.  Certain packages attempt to automatically start the daemons and servers as they are installed. You will have to stop these services to umount your image later.

In order to complete the installation of these packages, you will need to modify the postinst files installed in /var/lib/dpkg/info by commenting out the calls to invoke-rc.d or start, such as with procps : #start procps in procps.postinst or for rsyslog : #invoke-rc.d in rsyslog.posinst.

For recent Ubuntu distributions, you will need to add an entry for /dev in the /etc/fstab file:  dev  /dev  tmpfs  rw  0 0  in order to boot correctly.

Next, copy the files /etc/hosts, /etc/resolv.conf and /etc/fstab of the server to the directory containing the newly installed system (/var/tmp in our example).  Change these by deleting the hostname of the current machine in /etc/hosts and adapting /etc/fstab to your new disk image.

We recommend installing the gandi-hosting-agent and gandi-hosting-vm packages available on the Gandi distribution media server.  To do so, add the Gandi maintainer key as follows:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D8EAC2F4DAFE3FA5

then :

apt-get install gandi-hosting-agent gandi-hosting-vm

 

The packages will install their required dependencies.  For a description of the functions of these packages, please refer to the previous article in the Gandi Kitchen.

If your virtual server uses python version 2.6, you will need to install the package gandi-hosting-agent-py2.6 instead of gandi-hosting-agent.

Basic Installation for Distributions using .rpm

The installation of a distribution based on .rpm packages follows the same method as for a .deb based distribution, but using rinse or a specific application for the chosen distribution.

For example, using the same directory :

rinse --arch=amd64 --directory=/var/tmp --distribution=centos-5

 

According to your package management system, you will need to then add the package media locations in the configuration in order to install the necessary applications.

Please refer to the debootstrap / .deb base system installation for the remaining step : Gandi specific package installation, copy of system configuration file.

Other Ways to Install a Base System

Other distributions (often more specific) have other methods of installing the base system.  Sometimes an image is already available, thus you need only mount it in 'loop' and copy the files from the image.

Taking the above example :

mkdir /srv/a
mount -o loop,ro /my/image/directory /srv/a
cp -raf /srv/a/ /var/tmp/


Yet a More Radical Method

If you have a source disk to copy, you could also copy the entire disk;  Ensure that the destination disk that was just attached to the server is not mounted, and verify that the disk nameis not present in /proc/mounts.

Warning : this method only works where the destination disk is the same size as the source.

To create your system image and copy it to your new data disk, you will use the 'dd' command.  You need to pay close attention to the syntax of the command.

Here is an example, which will be explained afterwards:

 

dd if=/dev/xvda of=/dev/xvdc conv=sync

The parameter if= refers to the source disk; of= refers to, of course, the destination disk.  Any data on the destination disk will be completed erased.  In order to distinguish between the disks, we recommend first using the 'df' command.

Note that the image will not be bootable.  You will need to reconfigure the network settings of your server using DHCP before proceeding with the creation of the image.  Copying using 'dd' takes significant time and resources as it copies byte-per-byte the source disk to the destination, without any distinction between a data block and an empty block.  The Linux kernel for the Gandi hosting platform boots by default on the first partition ( root=/dev/xvda1 on the boot command line).  If you copy an existing system disk, the partition table will be correct on the destination.  Otherwise, you should ensure that the first partition of the destination disk contains the system files and is flagged as bootable.  You can also additionally create a swap partition.

Requirements to Boot

In the case of a copy of an existing virtual disk, the source disk having already booted, you only need to check that the network configuration is using DHCP so that the produced images will be correctly bootable.  In the case of an image created by a base system installation, there are a few items to verify and/or modify:

All of the services or modules related to physical elements of the server must be deactivated (such as, the service associated with the system clock - hwclock)

Some services at boot-time, such as ureadahead, console-setup, ondemand, plymouth, must be deactivated.  Move away the init-scripts associated with these services.  In an Ubuntu distribution, these scripts are found in /etc/init.

Add the gandi-hosting-agent and gandi-hosting-vm packages.  The agent allows the installation of your server to complete.  The scripts contained in the gandi-hosting-vm package allow the hosting platform to automatically and dynamically manage the server resources.  For a system with python 2.6, you should install gandi-hosting-agent-py2.6 instead. For more details on these packages, please read the  previous post on the Gandi Kitchen.

Delete the files related to ssh keys generated during the installation of the sshd package to avoid having the same key present on all of the servers generated from the same source image.

Verify that xinetd/inetd is started at boot -- if not, then activate it.  The Gandi-agent is spawed via xinetd/inetd and will enable the final configuration of your server.  The package gandi-hosting-agent installs its configuration file in /etc/gandi/agent.yml.

Verify that the permissions of  your installation directory structure and the directories lib/ root/ and tmp/ of the installation directory.  If in doubt, apply the same permissions to these directories as your existing server outside of the chroot.

chmod 0755 /var/tmp/lib
chmod 1777 /var/tmp/tmp
chmod 0750 /var/tmp/root
chmod 0755 /var/tmp/

Create the base files in /dev of your installation for the first boot stages, for example:

[ -e "$chroot"/dev/xvc0 ] || mknod "$chroot"/dev/xvc0 c 204 191
[ -e "$chroot"/dev/console ] || mknod "$chroot"/dev/console c 5 1
[ -e "$chroot"/dev/null ] || mknod "$chroot"/dev/null c 1 3
[ -e "$chroot"/dev/ptmx ] || mknod "$chroot"/dev/ptmx c 5 2
[ -e "$chroot"/dev/zero ] || mknod "$chroot"/dev/zero c 1 5

Add the necessary kernel modules corresponding to the kernel version in /lib/modules of your installation directory.  The modules are available on mirrors.gandi.net/kernel/  (See the associated article on the wiki)

 

At the end of the configuration, if you are inside the chroot, leave it using 'exit' then unmount the various elements of your images in /proc/mounts.  Kill any processes and daemons that may have been started during package installation within the chroot/directory.  For example :  grep /var/tmp/proc/mounts and then umount /var/tmp/proc and any others that may be listed.  End by umount /var/tmp which should work without errors.

Detach the Disk

Return to your disk management interface in your Gandi hosting account, and detach the disk to which you have just copied, or on which you have just prepared the installation.

 

Make the disk bootable by associating a kernel

In the administration interface, you have the posibility to define a kernel for a disk.  Select the virtual disk and associate it with a kernel suitable for the image you have just created. Let's take a "data" disk as an example, that we want to transform into a bootable virtual disk:

Use the link to change the disk information, and at the bottom of the page you will find an option to change the disk type:

 

You have now three advanced options, as for a bootable virtual disk:

By associating a kernel, the disk becomes usable as a custom image and will be shown in the list of images, just like the Gandi AI or expert images provided by Gandi during the creation of a new service via the administration interface.

 

Create a server from this image

During the creation of a server via your administration interface on the website, you will find your new custom image in the list of available images.  The server creation will thus use this image as the source for the system disk.  The server will then normally boot from a disk which is a copy of your image.  As such, you have the possibility of creating serveral identical service from the same custom image.

 Troubleshooting the server during boot

The Gandi hosting platform gives you the ability to access the command line of your server via the emergency console, which is accessible via an ssh session, and provides access to the console commandline of your server.

If you configure a getty on the console by default (tty1, xvc0, hvc0, depending on the chosen kernel version), you will have an emergency shell in case of boot errors, or a login prompt in the case of a successful server boot.

This emergency console will allow you to view the boot messages, and more importantly, any errors that may occur and thereby allowing you to debug your image.

 

Correcting the Image

Later, if you notice errors, or have forgotten anything in your custom image, you only need re-attach your new disk image to one of your servers (ultimately, and preferably, a server which was created from the same source image).  Then you only need make the changes on the source image and then detach the disk.  The image will thus be available to create new servers without error.