Gandi Kitchen

SimpleHosting PaaS Platform

Thomas — Wed, 22 Feb 2012 15:17:00 -0800

We are pleased to write about our new Platform As A Service (PaaS), which we call SimpleHosting. We have been working on this for more than a year, and are integrating the latest versions of component software while maintaining homogeneity with the management framework of our existing Infrastructure As A Service (IaaS) hosting.

Our goal in offering this platform is full abstraction of the system configuration, using our specific knowledge and experience gained from thousands of existing hosting installations. This abstraction then allows you full latitude on the application side, within reasonable limits. We have chosen the limits to be as open as possible by default.

More details

Our first abstracted system image is LAMP hosting: Linux, Apache, MySQL, and PHP. This is the most used combination of applications, but also among the most complicated to handle on a system and security level. Simple Hosting addresses these concerns with its robust architecture and scaling features. The architecture of SimpleHosting contains load-balancers in front of the caching server and in front of your personal instance. The instance is only available using HTTP through the clustered load-balancers, and uses SFTP to store your files. There is no plan to allow direct console access.

What you get is an instance of a Linux system with Apache2, PHP5 and MySQL installed. The files and application resources you place on it are up to you. Remember, this is not shared hosting across multiple customers. Another resource hungry web server will not cripple yours. At the load-balancer and caching server levels, all the virtual hosts are served by a group of machines, but all the data is partitioned.

Your instance has only one user, so:

you do not have to do any user rights management
you have a dedicated disk where all your data and application error logs are stored

You can use your instance to share privately accessible data using SFTP (through the /private directory).

Your instance has a dedicated apache2 daemon, so:

you can get all the data on application state, running queries, error logs, and more using the administration interface

Your instance has a dedicated PHP daemon, which means:

very few functional limitations as compared to shared hosting
you can change the PHP configuration using ini_set(). As the PHP5 interpreter runs as a fastCGI daemon, it performs better and allows greater resource control. Note: Configuring php_value in .htaccess file is disabled, due to fastCGI use
We use APC, a PHP opcode cache of 64 MB for size S instance, which is doubled at each larger size.
APC and PHP statistics are accessible using the administration interface.

Your instance has a dedicated MySQL daemon:

a full daemon to which you have root access
data is stored on your disk
configuration can be managed using PHPMyAdmin, available in your administration interface
there is no direct access to the MySQL port from outside your instance
memory use depends on the configuration of the various instance sizes, which have been set up to avoid swapping. There is no feature difference between PHP/MySQL on the different size packs

PHP and System Details

The instance is based on the Debian Linux distribution with standard upstream packages for the PHP5 interpreter and its modules. To get an exact listing of the version and included modules, see our wiki article or, for the result of a PHPinfo() command, look here. SimpleHosting is quite similar to servers that are available with our IAAS hosting, but configured to support a specific kind of web site.

The module list and configuration are set up by our technical team; you cannot install different PHP packages or Apache modules on your instance. However, if we see a significant quantity of requests for a specific module on our wishlist, we may decide to include the requested module or package in subsequent SimpleHosting images.

At the security level, our technical team will regularly update versions of libraries and interpreters on each SimpleHosting image. For example, the patch for the latest PHP5 security alert has already been applied to the LAMP instance. In the case of regular security alerts on PHP or PHP applications supplied by Gandi, our technical team will react quickly and apply fixes as quickly as possible following the announcement. We are monitoring security alerts on the PHP interpreter, on PHPMyAdmin (available in your administration interface), on apache2, MySQL, and on the base system (Linux kernel, libc6, ...). Be advised: Security alerts concerning the applications you install are of course your responsibility.

Performance

The platform architecture is designed to provide the shortest possible response time at each level:

dedicated memory space : your data stay cached in memory or in MySQL
PHP cache: APC, which is really efficient at the large memory sizes we configure
HTTP cache: Varnish provides efficient caching of static content to improve response time: pages and images that can be cached are served directly from the cache.

Scaling

This is the harder part to handle. If your site is successful, it may be rendered unavailable to your visitors by virtue of a lot of queries. You will have to take two parameters into account:

HTTP request flow : our Varnish farm is regulating this flow to avoid saturating your instance. If your pages can be cached, this process should be efficient. The quantity of data sent is capped with a quota, to avoid abusive behaviors.
Site complexity: Varnish cannot cache every byte of data. For example, if your site contains a lot of PHP pages using cookies, you will have to increase the size of your SimpleHosting instance to handle the query load.

Each size of SimpleHosting instance will start a specific number of PHP process (two for S size, for example). Each process can handle the generation of one dynamic page, and send the result to the visitors browser. For example, you can serve the contents of dynamic pages to two distinct visitors at the same exact time using a size S instance. Static pages are served directly by apache2 processes.

As generation of dynamic pages consumes CPU and RAM resources, we set up a maximum execution time (see the PHPinfo() output). Once the timeout value is reached, the PHP process will be stopped, and information about why it was stopped will be written to the error log. Your visitor will get an error message as a reply. If this is happening, you will need a larger instance size (for more processes) and/or a longer timeout. If your pages contains errors, you can easily find the information in the PHP error log available in your administration interface at http://"login".dc0.gpaas.net ("login" is your SimpleHosting login, a short numeric string).

Caching

Generated web pages served up by your SimpleHosting instance are added to a cache by a pool of Varnish servers, to handle heavy traffic by serving data already in the cache to new visitors. Obviously, you have the possibility to completely disable caching on your instance for all pages or one page, or to empty the cache at any time.

Directly in front of cache server, a couple of load-balancers help insure that visitors can reach the website on your instance. These appliances are reachable by a pool of IPV4 addresses associated to a set of names like "gpaasX.dc0.gandi.net", where X is an integer. IPV6 access is not yet available, due to a lack of hardware support: we are waiting for a new version of a firmware to correct it and allow ful IPv6 access.

Quick comments to finish off this article:

storage is provided on a compressed filesystem. The 10 GB feature in the SimpleHosting offer might actually contain much more data. Be careful not to fill it up, nonetheless!
the LAMP instance is designed to serve web pages and simple PHP apps like Drupal or Wordpress. It's not set up for streaming or large file downloads. Use it for what it is good for, and you will be successful.

Mirror server and CentOS

aegiap — Fri, 30 Dec 2011 11:27:00 +0100

Information about the CentOS repository on Gandi mirror server.

Local CentOS mirror

In order to reconfigure our CentOS mirror repository the way CentOS use it, the default version '6' which is currently pointing to the 6.1 version will be switch to 6.2 version in the beginning of january (hopefully january the 5th). With this configuration change, our local CentOS repository will be compliant with upstream mirror and customer will be able to easily follow the version of the 6 branch with simple update operations.

CentOS 6 image

Following the release of the 6.2 version of CentOS, both 32 bits and 64 bits OS images on Paris and Baltimore datacenters are now updated with the new version.

Local mirror URL

The URL of the official mirror server which contains distributions tree is http://mirrors.gandi.net. The index page is listing all the distributions and systems availables.

With Debian, for example, the URL to add to /etc/apt/sources.list is :

deb http://mirrors.gandi.net/debian squeeze main experimental non-free

Older URL like http://centos.mirror.gandi.net/.../ or http://debian.mirror.gandi.net/.../ are now obsolete and will be removed soon. They were setup during the beta step of the IAAS hosting.

The mirror server is available for virtual server on IAAS hosting and is not reachable for servers outside Gandi network.

What's Happening at the Cloud Expo in Santa Clara?

Thomas — Mon, 14 Nov 2011 11:03:00 -0800

Gandi's Thomas Stocking went to the Cloud Expo 2011 West in Santa Clara this week, and found that marketing is alive and well in the Cloudsphere. Despite the fluff, there were some interesting ideas getting discussed, and companies developing strategies to leverage the evolving cloudscape...
What's Happening at the Cloud Expo?

Attending Cloud Expo 2011 West in Santa Clara requires a lot of stamina. The conference is a grueling 4 days long, and has a whopping 160 sessions and lectures. Quantity, however, does not always equal quality, and I was a bit underwhelmed by much of what was presented. Enough marketing fluff! I come to conferences to get to the core technical truths. Some interesting ideas were being discussed, of course, but weeding them out of the pitch and jargon was more challenging than I like to put up with. If I want a challenging search, I’ll go geocacheing!

Ok, so what were the interesting ideas? Well, for one thing, Open Source (capitalization is mine) is clearly in the forefront of innovation in the cloud space. I spoke with Krishnan Subramanian who contended in his talk that Stallman got it wrong and that cloud technology is not evil, or inimical to Open Source, and in fact that Open Source is a necessary counterpoint to the imminent consolidation of cloud vendors for the enterprise space. I thought it was interesting that Krishnan is admitting that companies (Rackspace for instance, with the release of OpenStack) are going to be developing Open Source, rather than relying on individual users to find and fix bugs. This means the Open Source software development model will change from the “with enough eyeballs, all bugs are shallow” to a company-sponsored community feedback model. That’s not the “free software” model Stallman is talking about, but it’s still a viable one, or at least innovators had better hope it is. Greenqloud CEO Eirikur Hrafnsson was there, and we talked a bit about the relative merits of CloudStack and OpenStack, which Greenqloud is enhancing for use in it's environmentally optimized data centers.

Speaking of community development, it seems like there is a fair amount of thought going into this as well, with cloud computing offering some unique opportunities to leverage data that users input, or at least the way they use the data. In the traditional model of client-deployed software, the poor lonesome developers really had no easy way to see what the users were actually doing with their software, whereas if the users access a hosted solution, common use cases become obvious in the usage patterns. A SaaS vendor who can detect and capture this data can quickly adjust a difficult workflow, for example. Dell Boomi, a hosted solution for mapping data exchanges and workflows between enterprise applications, leverages the fact that their SaaS is actually a shared model, and so the data for such things as field mappings that users input can be used to “suggest” field mappings based on work done by other users. Cloud vendors could do the same but would need to do more to harvest that data across multiple instances. It’s easy to see how this helps developers and product managers do more to select popular features for focused development. Gandi does that: we choose the packages for the Simple Hosting offering based on what our cloud hosting customers are choosing most often.

On the show floor there were many examples of what I’ll term epiphytic businesses. Players like Teremark and PhoenixNap offer added value on top of VMware hosting technology. They pay VMware for that in license fees, of course. Terremark scored big by getting GSA listing and tapping into the billions pledged by the federal government to Cloud Computing, which is probably one good reason why Verizon Business bought them. Lest you be an American taxpayer and think this is yet another boondoggle, the case is really good for consolidation of many, many redundant federal data centers.

Apparently the hunt for differentiation on the basic value proposition of cloud computing is still going on, and some vendors are striking pay dirt. Other epiphytic businesses tackle areas that are key to cloud adoption, but that the main cloud technology providers don’t really want to mess with, like enhanced security, or end user support. CloudPassage has a way of dealing with security using agents deployed to instances. Perspecsys has a middleman security scanner that aims to protect data in the cloud from sloppy or compromised user systems. Appriver provides desperately needed end-user support and front-end sales to Microsoft’s nacent Office 360 packages.

Some of these business are dependent on the “mother” technology vendor, and others, trying to defend themselves from this risk are building interfaces to all the providers they judge to have staying power. One such is a cloud management platform called Abiquo that gives you a nice gui (of course) as well as an API on top of “all the major cloud vendors” offerings. This app lets you build, migrate, and monitor virtual data centers with your installed private and public clouds (where have I heard this before?).

Good ideas are being bought: Gluster’s cloud-based storage clustering solution was acquired by Redhat, and this made many sit up and take notice, I’m sure. Redhat has also entered the PaaS fray with OpenShift, which will presumably allow them to take advantage of the great work done to manage linux packages and, of course the JBOSS Java EE framework. Perhaps Gluster’s sauce will make that meat more savory?

What’s Still Missing?

Cloudbursting remains elusive, that is, the true, automated ability to augment private cloud capacity with public cloud resources to achieve higher scale for short periods.
Standards are lacking. APIs abound, like Amazon’s API for EC2, and despite efforts of well-intentioned people in places like the Open Data Center Alliance to come up with shared and complete models for us to work with, it will be a while before we settle on one (or two, or three) standards.

What's Next for Gandi in the US?

We are going to keep developing out cloud hosting offering at Gandi SAS, and Gandi US is going to make sure the American customers get excellent support. We will be expanding our US presence, so look for us in your town! See you at the LISA conference in Boston in December! Happy computing.

When Null0 and BGP May Cause Problems

Leland Vandervort — Mon, 20 Jun 2011 09:10:00 +0200

If you read any networking textbook or study guide on the subject of BGP and route stability, very often you will find a mention or even a suggestion to tie your aggregated prefix to null0 to ensure that the prefix is remains in the routing table thereby increasing the stability of your BGP advertisements.

While this is a good thing, to a certain extent, there are situations where such a configuration may lead to service interruption in case of an outage. This quick article talks a little about internet routing using BGP and some "common practices".

Let us take the following simplified pair of autonomous systems as an example. On the left, AS 1 and on the right AS 2; These two networks have connectivity to the internet as well as a direct peering relation between them. Under normal circumstances, the traffic between the client in AS1 and the server in AS2 will follow the shortest AS-Path which in this case will be over the peering session.

In this example, the routers have static routes to null0 to 'pin' up the prefixes that they advertise in BGP for their respective ASNs. The only significant difference in this example between the two networks, is that the connectivity between R2 in AS1 and the rest of the AS1 network has a single connection. After all, it is just a peering router, and less critical than central core and internet transit, right? Well, let's have a look for a moment.

Let us imagine that the connectivity between R2 and the rest of the AS1 network is severed. The routes to the network in AS 2 will then only be seen via the internet across the transit connection, as seen in the BGP advertisements. Under correct configuration, traffic between the two networks should now be flowing via the Internet across their respective transit connections, as in the following diagram.

In this example, however, R2's connection to the peering session between the two ASNs is still active, and as such the BGP session is still active between the two. Since R2 has statically 'pinned' the network prefixes via a route to null0, these prefixes are still advertised to R3 in AS 2.

As a result, the return traffic, following the shortest AS-path, will be sent via R3 to R2. Unfortunately, due to the break in connectivity between R2 and the rest of its network, the client destination is unreachable, resulting in a partial "black hole" instigated by the break in connectivity and exacerbated by what is likely to be either an administrative "oversight", or a lack of understanding of the network topology at the time of configuration.

So how can we mitigate this problem?

There are three primary ways in which this problem can be mitigated. The first method would involve adding a separate redundant connection from R2 to the rest of the AS 1 network (similar to that visible on R3 in AS 2 in the above diagrams), thereby mitigating (or at least reducing, while not completely eliminating) the risk of R2 becoming isolated and generating the partial black hole in the first place. Depending on the distance and geographic location of R2, however, this may prove to be costly.

An alternative would be to tie the aggregated prefixes only in the core (which itself should already have redundant connectivity between the critical equipment), and advertise these with an internal routing protocol, such as OSPF or IS-IS, to the rest of the the border devices.

As a result here, if R2 loses connectivity to the rest of the AS1 network, it no longer receives the internal protocol advertisements for the aggregated prefixes, and since these prefixes are no longer in the routing table of R2, it will cease to advertise them to any peers in BGP. As a result, the return traffic from AS 2 to AS 1 would follow the only remaining path via the internet across the transit connections.

The third alternative involves more complex configuration, and of course assumes a dynamic internal routing protocol within the AS. Under this scenario, the BGP advertisements would be conditioned on the existence (or non-existence, according to preferences) of another prefix in the routing table; this other prefix being received via the internal routing protocol. This is sometimes referred to as "BGP Conditional Route Injection". On Cisco equipment, this involves using advertise-maps within the BGP configuration, and there is documentation on this available on Cisco's website.

Aside from these methods, a fourth possibility would be to bind the static routes to null0 against a track object which in turn is associated with an IP SLA monitor. For example, the router can be configured to regularly poll an IP address elsewhere within the network either via PING or another protocol. By binding the this IP SLA monitor and track object against the static route to null0, this static route will remain in the routing table only if the IP SLA monitor returns successful. (Thresholds may be configured as well to guard against transient false negatives...) If the SLA monitor fails, then the route is withdrawn from the routing table, and thereby suppressing the prefix from being advertised in BGP.

Storage Migration

Leland Vandervort — Thu, 05 May 2011 15:09:00 +0200

For the past few months perhaps you might have made use of the servers in the US. The changes in the storage technology was one of the strong points in the hosting infrastructure. Prior to this, we had to adapt the infrastructure so that it could understand "n" datacenters. The implementation of this new storage platform was not as complicated as it seems since it is completely independent to the architecture in France. With a new datacenter, it was therefore quite a trivial matter to build from scratch, and all of the new servers in the US made use of this new platform from day one.

On the French side, an inevitable migration was required in order to arrive at a standardised platform to utilise the new functionalities. Here, however, the problem is different and a little more complicated, with the coexistence of two different storage solutions. In reality, we were confronted with a number of challenges so that the machines hosting the servers could happily play ball with two different storage platforms at the same time. The opening of our US datacenter was already a few months ago, and so a large proportion of our efforts have been dedicated to this migration. This is, of course, proceeding and certainly takes [a lot of] time, and will soon be available to all of our customers. We will thus be in a position to run the two storage platforms together in order to perform the migrations efficiently.

Once active, all new disk creations will take place on the new storage platform. At this point we will enter the migration phase, and specifically the phase which directly impacts you since it also means the migration of your disks.

There are a number of ways to perform the migration:

Create a new disk:

The creation of this new disk will be on the new platform.

Next, simply attach this new disk to your server and copy the relevant data to the new disk. This would be a good occasion to do some housekeeping and get rid of any old data that you no longer need. Such commands as 'cp' or 'rsync' would do the job nicely.

Create a new disk from the image of an existing one:

This function has been available through the API for several months: disk.create_from(apikey, disk_spec, src_disk_id) -- see the API documentation for more details at http://doc.rpc.gandi.net/hosting/reference.html#disk.create_from

The next Gandi Website update will include the ability to create a new disk from and existing one to make life easier for you if you are not a user of our API. This method, nevertheless, requires either that the server that has the original disk attached be stopped, or that the disk be detached from the server.

Please note that the time take to make the copy will be directly dependent upon the size of the disk, so you should have some patience and/or a good coffee break, if you decide to employ this method on a comparatively large disk.

If the new disk is to be a system disk, then you will need to define the disk as a boot-disk either via the web interface ("Boot Disk" in the server details) or using the API : vm.disk_attach(apikey, vm_id, disk_id {'position' : 0} ) ( see http://doc.rpc.gandi.net/hosting/reference.html#vm.disk_attach )

This migration will enable you, among other things, to make use of the new storage features such as snapshots, resizing, and rapid-copy, which will be available in the coming weeks. Aside from the platform which is seeing new features every month, we hope to soon be able to talk to you about these new functions which may change the way you use your servers.

New storage infrastructure

William — Mon, 24 Jan 2011 23:42:00 +0100

Our new storage infrastructure is being deployed.

Disk Management is now handled through virtualization... and we are excited to share it with you!

As you will be seeing, the features are very interesting and we will be integrating them over time in your management interface as well as in the new API.

Improvements

Besides the disks, still using RAID for redundancy and a write cache resistant to a power failure, the storage servers are now organized in clusters: in case of controller failure, a standby controller is ready to take over the management of disks.

The storage capacity has been improved as well as disk I/O. Moreover, you will be soon able to create larger disks.

Creating, copying and resizing disks will be faster and more automated.

Snapshot

This mechanism enables you to create a snapshot of your disk that is either in use or detached. The snapshot can be used to achieve a consistent backup of your disk. Within a few weeks, a snapshot on demand will be available. We are also working on periodic management of snapshots. For example the ability to create a snapshot every day and only the last seven.

Copying disk

Thanks to the snapshots, it will be possible to create a fast and live copy of your disk, even with activity on it. This feature will facilitate fast deployment (clone) of servers.

Rollback

From a snapshot you will be able to create a disk, getting back your lost data or apply a rollback operation on your disk.

Automatic backup drive

A backup system, disk to disk, will be available in the future; we are still working on the specifications. For most of you, the snapshot mechanism will probably fulfill your needs.

Disks and Partitioning

Gandi disks are logical drives, meaning that we provide an access to disk already optimized, secured; it is easy to create new one, resize or delete them.

The partitions, or RAID algorithms are for your virtual machine, at best an unnecessary overhead, and at worst a reorganization of disk accesses that can result in performance degradation (see below). In addition, you can deprive yourself of convenient features: a snapshot made on a disk that is part of a RAID has no interest, resizing a partitioned drive (our old system) is complex even dangerous if not made carefuly.

Like with SSD and new hard disks, blocks are not 512 bytes long anymore but 4KB. If you are using partitions, access to your data may not be aligned, and performance divided by three.

Partitions are helpful for physical disks. This complicates the management of your disks for nothing: consolidate your data on a single disk or create multiple data disks is the right solution.

The disks created by Gandi will no longer use partitions. Currently only the system disks have a default partition (system and swap): it will disappear. The system disk will be managed like any other disk, which will facilitate management. To keep the same level of service, we will provide a temporary disk for swap. To remain consistent with the current configurations, we will emulate virtual partitions to present the system disk and swap: the system will be seen as xvda1, the swap as xvda2, like on the current system but it will be two different disks.

Note that the new storage infrastructure is available form the beginning in our Baltimore data center; features will be along the time.

Operation Dragonfly - Next Generation Gandi Network

Leland Vandervort — Sat, 22 Jan 2011 22:21:00 +0100

As most of you may have already noticed over the past 18 months, there have been several periods of scheduled maintenance on the Gandi network. Some of these have been fairly intrusive, while others have taken place quietly behind the scenes. I have made a number of innuendos here on the Gandi Kitchen and also on the Gandi Bar over the past year dropping hints about some of the things to come, so I decided to take this opportunity to reveal a little more about what our network operations team here at Gandi is up to and what it means for you over the coming months.

Why are we calling this "Operation Dragonfly" ?? well.. that is for you to try to guess! We will be giving a Gandi T-Shirt to the first five people who find the link/significance of the name of this project, and send us a short one paragraph description of this significance by email to dragonfly AT gandi DOT net :) Contest ends on 14 February 2011 when the first five people to send us a correct answer will be sent their Gandi T-Shirts! (Please include your size and address in your email so that we know where to send it to, and to make sure we don't send you one for a Barbie doll!) Oh .. and before I forget, there are a few subtle hints embedded throughout this article to help you, and after the end of the competition we will also reveal the clues for everyone !

A Bit of History

The current Gandi network has grown organically over the course of the past ten years, with everything starting from a relatively simple "flat" design. Over time, as more and more services and features were added to the Gandi product lineup, the network has had to be extended to keep up with the product innovations. Most notable in this gradual expansion has been the extension of the network across multiple datacenters, and the desire to maintain resilience across multiple sites.

Because of the relatively "flat" nature of the original network, and the desire for layer-2 adjacency across multiple datacenters, one of the simplest ways to provide this cross-site adjacency was to simply "span" or "trunk" the relevant LAN segments cross the different sites. While this is relatively efficient for a small network, with only a couple of sites to worry about, it becomes rapidly less and less efficient and drastically more cumbersome to continue this approach for very long. This is perhaps where we went astray several years ago.

In 2008 we realised that although the various pieces of routing and switching equipment in the network had "names" as if they were part of a structured hierarchical network model, this in fact turned out not to be the case because we found ourselves with a network infrastructure with six core switch/routers and numerous distribution aggregation routers, across four different sites -- all with the same network segments spanned/trunked through the core to all layers of the network. The result of this is a nightmare to manage and diagnose troubles related to spanning tree and odd routing behaviour.

The Origins of Operation Dragonfly

In addition to simply the network architecture itself, much of the equipment was starting to show signs of age and was not capable of features that were now required in order to further expand and improve Gandi's services. Therefore, starting in 2009, we set out to modernise and restructure the network to meet not only the current needs of the Gandi services, but also our future needs for the next three to five years, both here and France and internationally.

The first step was to consolidate inter-site connectivity, which until that point was using multiple trunked VLANs across dark-fibre connections. We retained the dark fibre, of course, but migrated to wavelength multiplexing to provide different point to point connectivity between equipment across the dark fibre links.

The challenge that we have with any major network restructuring activities is making the changes without drastically impacting the Gandi services at the same time. Certain services require a near 100% uptime to ensure correct operations and synchronisation and simply going around ripping out cables en-masse and reinstalling new equipment is not really feasible in an environment where high-availablity is required. This is why this work has been taking so long to accomplish, and we have been doing it piece by piece over the course of nearly two years.

Nevertheless, we must continue to salvage and maintain the services, whilst we move forward at full speed towards the exciting new features that will be implemented over the course of the next year. For the sailors among you, we just had to overhaul the outboards !

In September 2009, we upgraded the core switch processors from the Cisco Sup32 platform to the Sup720 3BXL platform.

The next step in November 2009 was the rejuvenation of the old Cisco 6500 platforms we had in the distribution layers of the network at our datacenters in St. Denis, migrating these from the old Supervisor-2A platforms to the Cisco Virtual Switching System 1440 (VSS). We also began deploying MultiProtocol Label Switching (MPLS) technology across the core of our network in order to ultimately provide traffic engineering and eventually innovative ways of interconnecting remote network segments together across our network core.

International Goals

With the expansion of Gandi's network, not only here in France, but also internationally, we had to overcome a challenge of how to adapt the legacy network infrastructure to something scalable and capable of handling our international requirements both in terms of internet connectivity and for eventually datacenters on the other side of the Atlantic.

Any network architect will tell you that a scalable network is modular and hierarchical. This has been one of the guiding principles of network infrastructure design for many years, and still holds true today. Cisco calls this the "three-layer network model". Over the past few years, however, this has become a little more complex with virtualisation and service-oriented architectures where the service becomes part of the network itself. Nevertheless, practically all large scale network infrastructures in operation today follow a hierarchical and modular architecture model.

With the expansion of Gandi to the north-american continent, we needed to ensure resilient connectivity between the datacenters. A simple rule of thumb applies here -- each datacenter is connected to at least two other datacenters via diverse connectivity paths.

This past year, we built the datacenter in Baltimore, and this datacenter is now fully operational for the Gandi Hosting platform. Over the course of the coming months, a number of the other Gandi systems will be available in Baltimore as well -- most notably DNS and the Gandi Website.

As of the end of 2010, our international connectivity also included additional peering capacity in London, Amsterdam, and Ashburn Virginia/Washington DC -- more to come over the course of the next couple of years!

What's Next?

We have accelerated the network engineering activities since the beginning of 2011 and over the next few months there will be a number of highly-intrusive maintenance activities undertaken with the objective of finalising Operation Dragonfly before the end of the first half of 2011.

The first part of this final phase took place last weekend, at St. Denis, where we finally and definitively removed the "spanned/trunked" LANs between the distribution/aggregation and core layers of the network in St. Denis. As a result of this move, any "issue" in the access or distribution layers of the St. Denis network segments will have no impact on the routing core of the Gandi network in that location. Next we have to do the same thing at Telehouse2, which is a little more complex!

Over the course of the next few weeks, you will see a number of scheduled maintenance periods during which we will perform pretty much the rest of the necessary activities in support of this three-year project:

Closure of our point of presence at Paris Telehouse-1 (Jeuneurs)
Updating of a number of critical Gandi services to make use of new hardware and software technologies.
Implementation of Anycast and Content Delivery technologies for a number of services to include DNS and the Gandi Website across all of our datatenters both in Europe and in the US.
Complete upgrade and re-installation of the Gandi services at Paris Telehouse-1 (Voltaire) with inter-site resilience and interoperability.

What Will it Look Like?

The new Gandi network infrastructure will resemble something like the diagram below (though this is somewhat simplified). The key elements here is that the architecture is designed and build for resilience, modularity, and ultimate expansion… it's pretty elementary after all… As Mickey Mouse once said… "Arithmetic is being able to count to twenty without taking off your shoes". ;)

New kernels available for your server.

aegiap — Mon, 17 Jan 2011 16:32:00 +0100

The kernel list on Gandi hosting grows. You have now the choice to use a 2.6.36 or a 2.6.32 with the grsecurity patch.

Gandi hosting allow customer to choose from a pool of patched and built kernels from Gandi team. Three new kernels have been added to the available kernel list :

2.6.36 in x86_32 and x86_64

The Linux kernel is in active development : new versions containing features and bugfixes are published on a regular basis. We have prepared new build in 32 bits and 64 bits of the 2.6.36 kernel with D.Kipper patch about additional memory managment. Moreover the feature of resizing an online disk attached to a server is now merged into the kernel. The feature will arrive soon in the hostinig interface.

2.6.32 in x86_64 with grsecurity patch

Previous kernels allowed customers to setup security solution in the server system (thanks to selinux for example). When choosing this new 2.6.32 kernel you should be able to setup an advanced security policy thanks to grsecurity in version 2.1.14-201005151340. The kernel allow disk replication thanks to DRBD in version 8.3.8 which was not yet merged in the upstream source tree (integrated since the 2.6.33).

You can check the changelog of Gandi kernels and the process to update kernel modules according to your selected kernel.

Gandi Hosting : US and France Datacenters FAQ

Leland Vandervort — Thu, 23 Dec 2010 14:33:00 +0100

With the opening of our Baltimore datacenter, we have decided to provide a quick FAQ to respond to a few of the commonly asked questions about the hosting product and how it will work with the two datacenters, as well as some of the other features that we are putting in place.

If you have any questions not addressed here, then please feel free to let us know!

Q: Can I migrate my server automatically from Paris to Baltimore, or vice-versa?
A: No; there is no automatic solution to migrate a virtual server between the Paris and Baltimore datacenters due to technical limitations (IP addressing, platform differences, etc.)

Q: Okay, but I really want to migrate my server... how can I do it ?

A: If you are really sure, then there are a couple of ways to do it.
Firstly you will need to create a new server through the admin interface (or public API). You will then need to transfer the data from the old server to the new one.

For Gandi AI, you will need to save the source files for your website and export the database, for example, with PHPMyAdmin, and then import these on the new server.

If your server is running in expert mode, it is possible to transfer these files directly over SSH. Another method, again in expert mode, involves creating a new server with a data disk in the target datacenter, and transfer the system disk from the old server to the newly attached data disk of the new server (for example, with 'dd' but please note that this may take some time).

Finally you can then change the disk type from data to system via the admin interface in the advanced configuration for the disk, shut down the new VM, remove the old system disk and attach this new image as the new system disk, then start the VM.

There are a few methods to achieve this:

With 'dd/gzip':

dd iflag=direct bs=8k if=/dev/xvdYX | gzip -9 | ssh baltimore-server "gzip -d | dd bs=8k of=/dev/xvdYx"

With 'mbuffer':

1. First start the target on the Baltimore server:

mbuffer -q -s 128k -m 8M -I <baltimore_server>:<port> | gzip -d | dd bs=8k of=/dev/xvdYX

2. Start the sender on the Paris server:

dd iflag=direct bs=8k if=/dev/xvdYX | gzip -9 | mbuffer -q -s 128k -m 2M -0 <baltimore_server>:<port>

With 'rsync':
You can simply copy the files that you need.

    rsync -arvz src baltimore_server:/dst

Q: Can I detach the IP address from my Paris server and attach it to my Baltimore server?

A: No; the IP address allocations and the server provisioning systems for the two datacenters are completely independent.

Q: Can I attach a disk in the Paris datacenter onto my server in the Baltimore, for example, to make an off-site backup?

A: No; for the same reasons that an automatic migration between datacenters is not possible, the two platforms are completely independent.

Q: I see that Gandi will be deploying Anycast DNS servers, will this allow me to do geolocalization for my servers?

A: No; the anycast DNS servers simply give a physical presence for the Gandi DNS servers in multiple locations so that DNS lookup requests can be answered by the closest server. The DNS service does not provide any geolocalization capability.

Q: Can I have a server in Paris and another one in Baltimore and use anycast to allow the closest server to serve the content?

A: No, because the hosting platforms are independent of each other and use separate IP allocations.

Q: I have my own ASN and assigned /24 for Anycast purposes, can I use a Gandi VPS to host my anycast service, assuming that I use other providers to expand the geographic and network scope?

A: It is technically possible to do this under certain conditions; We have already tested such a solution, but it is entirely bespoke and has certain limitations concerning how your own IP block can be used on your VM. You will need to discuss your requirements with our technical team to determine if we can accommodate your requirement.

Q: Why when I traceroute from my Paris to my Baltimore server, do I have a 92ms latency all of a sudden?

A: Latency is defined as the one-way path delay between two points on a network.; What you see in a traceroute is the round-trip delay for the hop in question (with a special exception for hops contained within an MPLS label-switched-path -- see the next question)

Transatlantic latency plus the physical fiber cable distances incur a one-way path delay of roughly 45ms (plus or minus a little). While it is true that at the speed of light the delay to cross the Atlantic is only in the order of 28ms in a straight line, you need to add in the factors of ocean depth, and then the land-based cable distances between the two end locations. This translates to a round-trip delay of at least 90ms, thus this is normal.

(This is a simplified explanation, as there are actually a number of factors that determine the delay observed, and we would be happy to discuss these with those who are really interested in the nitty gritty details...)

Q: When I traceroute from my Paris to my Baltimore server, I reach a router which is still in Paris but shows a 92ms round-trip time, even though the hop just before it is only 1ms. Doesn't this point to a problem on your network?

A: Our network is running MPLS (MultiProtocol Label Switching) in the core, with Traffic Engineering (TE) enabled. As such, when a packet crosses the network and enters a known end-to-end pathway between two endpoints (known as a Label-Switched-Path, or LSP), the IP packets remain within that pathway until they egress at the remote end if there is an existing TE tunnel along that path.

As a result, the intermediate routers within the LSP will still respond to the traceroute with the ICMP TTL-Exceeded packets as normal, but these packets are forwarded along the full path and back before being sent back to the originating source of the traceroute.

In consequence, the round-trip-time that you see for the Paris-side router in this instance is actually the round-trip-time for the full MPLS path, plus the time between the source and the router itself. Again, this is perfectly normal behavior.

Q: Why do you run MPLS on your network?

A: MPLS allows for a number of different solutions, not least of which is providing traffic engineering and flow control based on certain criteria. We can specify certain traffic based on its characteristics to follow a given MPLS path under normal circumstances, while still allowing for alternative paths in case of link failures.

We can also make use of other features of MPLS to enable layer2 and layer3 VPN connectivity across different portions of the network for some of the Gandi services, whilst keeping the size of the routing tables as low as possible. (The DNS servers being deployed in anycast use some of the features provided by MPLS across the core of the network, for example).

Q: I have a server in Baltimore and one in Paris. Can I have a second interface each with a private VLAN to connect the two servers with a back-end private network?

A: We are working on a private VLAN solution to be deployed later in 2011 and we are looking at ways to provide private VLAN connectivity for customers between their servers in different datacenters.

Q: I'm not convinced! This is a new datacenter, and I am not running any services yet, but even between my server and its next hop router I get weird and inconsistent ping results. Why?

A: This is due to hardware buffer sizes on physical ethernet interfaces. In order to transmit data physically "onto the wire" the interface needs to fill the buffer. If there is little or no traffic, small amounts of data will be "stored" in the buffer until there is enough data to transmit. This will result in false delay/round-trip and variance/jitter readings. Under normal data usage, this phenomenon is much less evident because the buffers will fill more quickly. It is, nevertheless, completely normal behavior for any network device or server NIC.

We hope that these help answer some of the questions that you have expressed. We will additionally be putting this FAQ along with any new questions and answers over time on Gandi's online knowledge center (http://wiki.gandi.net), so please check back from time to time for updates!

How to Create a System Images for Your Server

Leland Vandervort — Tue, 16 Nov 2010 16:46:00 +0100

There are many reasons to create a system image for your servers: to build a custom system with your preferred applications pre-installed, to create an image of a game server that can be easily deployed, to simply duplicate a custom server, or simply to backup one's system...

The procedure is relatively simple and can be performed by anybody, as long as you pay careful attention to the detail.

Create a Data Disk

You need to create a data disk via the disk creation interface of your hosting account. If you wish to make a copy of an existing disk, the data disk must be of sufficient capacity.

Simply attach your new disk to the target server ; the server that contains the virtual disk to copy, or the server used to perform the base installation.

Create a System Image

By making a copy of the data from a virtual disk.

Warning: You must have sufficient space on the destination disk.

Copy the data from the source disk to the destination with the 'tar' command:

tar cC /srv/disk1 . | tar xC /srv/disk2

If /srv/disk1 is the directory where the source disk is mounted and /srv/disk2 is the mount point of the destination disk. Using 'tar' will only copy the data and not empty blocks, so the operation is relatively quick.

Warning: if you want to copy the local system disk, you must add a few exclusions:

tar
--exclude=/proc --exclude=/sys --exclude=/srv/disk2 -c \ | tar xC /srv/disk2

By installing a base system

Preamble

Your new disk attached to a server is ready for use. You can partition your disk to prepare a swap partition for the image, but we advise that you do not create partitions and to create the file system directly on the disk.

In this way you have the advantage of flexibility if you intend to resize the disk later (no need to calculate the partition table), whilst being better adapted to the block storage on our physical filers, and to be able to decide later whether or not to use a swap file.

In the near future, all of the OS images provided by Gandi will be without partitions, and the Gandi hosting platform will automatically provide additional swap space.

Meanwhile, the virtual servers will still see the first disk as "xvda1" and the swap disk as "xvda2".

Whether or not you decide to partition, the Gandi hosting platform will be able to start a system disk with your image as source by detecting the disk boot sector and adapting the kernel boot options. Gandi also provide a server containing copies of the distributions for which we provide system images. You can use this server to speed up your installation : mirrors.gandi.net

Preparation

Initially, verify that the virtual disk is not already mounted (for example in /proc/mounts). Unmount the disk if it is mounted. Partition the disk if required and then format it.

We recommend that you format the disk directly and use the ext4 file system. If, for example, your disk is xvdc : mkfs.ext4 -j -m0 /dev/xvdc and then mount the disk to a directory of your system : mount -o rw /dev/xvdc /var/tmp

Bootstrap installation for distributions using .deb packages

The base system installation can be accomplished with debootstrap in a specific directory. Once the installation is completed and a few modifications are made, the disk will contain a bootable and fully functional GNU/Linux system. The system will have a basic set of applications and you will need to adapt the system to your requirements by installing any required applications with apt-get and configuring the locales.

We highly recommend the installation of a system with the amd64/x86_64 architecture, as this will be faster on the Gandi platform. When it comes to selecting the kernel, you should choose the appropriate x86_64 kernel (2.6.32-3831, for example).

An example installation for an Ubuntu Maverick 10.10 distribution on a previously prepared and mounted disk:

debootstrap
--arch=amd64 --verbose --components=main,universe,multiverse \

--include=openssh-server,openssh-client
maverick /var/tmp \

http://mirrors.gandi.net/ubuntu/

Should debootstrap happen to complain that no configuration file for maverick is accessible, you only need create a link in /usr/share/debootstrap/scripts from lucid to maverick. The following step will modify the file containing the source package media locations. Edit the file /etc/apt/sources.list in the directory by adding the distribution media and the Gandi package media.

Example :

# cat
/var/tmp/etc/apt/sources.list

deb
http://mirrors.gandi.net/ubuntu maverick main universe multiverse

deb
http://mirrors.gandi.net/ubuntu maverick-security main universe multiverse

deb
http://mirrors.gandi.net/ubuntu maverick-updates main universe multiverse

deb
http://mirrors.gandi.net/gandi/ubuntu maverick main

Once the debootstrap command and the configuration has been completed, you will be able to access the newly installed system via chroot to the directory.

For example: chroot /var/tmp

In this way, you can refresh the package media with apt-get update and install the various applications and packages that you may require. Beware, however, that the installation of packages in a chroot require a few corrections. Notably, you will need to mount /proc and /dev/pts which are usually available in /etc/fstab : mount -a then complete the package configurations with dpkg --configure -a. Certain packages attempt to automatically start the daemons and servers as they are installed. You will have to stop these services to umount your image later.

In order to complete the installation of these packages, you will need to modify the postinst files installed in /var/lib/dpkg/info by commenting out the calls to invoke-rc.d or start, such as with procps : #start procps in procps.postinst or for rsyslog : #invoke-rc.d in rsyslog.posinst.

For recent Ubuntu distributions, you will need to add an entry for /dev in the /etc/fstab file: dev /dev tmpfs rw 0 0 in order to boot correctly.

Next, copy the files /etc/hosts, /etc/resolv.conf and /etc/fstab of the server to the directory containing the newly installed system (/var/tmp in our example). Change these by deleting the hostname of the current machine in /etc/hosts and adapting /etc/fstab to your new disk image.

We recommend installing the gandi-hosting-agent and gandi-hosting-vm packages available on the Gandi distribution media server. To do so, add the Gandi maintainer key as follows:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D8EAC2F4DAFE3FA5

then :

apt-get install gandi-hosting-agent gandi-hosting-vm

The packages will install their required dependencies. For a description of the functions of these packages, please refer to the previous article in the Gandi Kitchen.

If your virtual server uses python version 2.6, you will need to install the package gandi-hosting-agent-py2.6 instead of gandi-hosting-agent.

Basic Installation for Distributions using .rpm

The installation of a distribution based on .rpm packages follows the same method as for a .deb based distribution, but using rinse or a specific application for the chosen distribution.

For example, using the same directory :

rinse --arch=amd64 --directory=/var/tmp --distribution=centos-5

According to your package management system, you will need to then add the package media locations in the configuration in order to install the necessary applications.

Please refer to the debootstrap / .deb base system installation for the remaining step : Gandi specific package installation, copy of system configuration file.

Other Ways to Install a Base System

Other distributions (often more specific) have other methods of installing the base system. Sometimes an image is already available, thus you need only mount it in 'loop' and copy the files from the image.

Taking the above example :

mkdir /srv/a

mount -o loop,ro /my/image/directory /srv/a

cp -raf /srv/a/ /var/tmp/

Yet a More Radical Method

If you have a source disk to copy, you could also copy the entire disk; Ensure that the destination disk that was just attached to the server is not mounted, and verify that the disk nameis not present in /proc/mounts.

Warning : this method only works where the destination disk is the same size as the source.

To create your system image and copy it to your new data disk, you will use the 'dd' command. You need to pay close attention to the syntax of the command.

Here is an example, which will be explained afterwards:

dd if=/dev/xvda of=/dev/xvdc conv=sync

The parameter if= refers to the source disk; of= refers to, of course, the destination disk. Any data on the destination disk will be completed erased. In order to distinguish between the disks, we recommend first using the 'df' command.

Note that the image will not be bootable. You will need to reconfigure the network settings of your server using DHCP before proceeding with the creation of the image. Copying using 'dd' takes significant time and resources as it copies byte-per-byte the source disk to the destination, without any distinction between a data block and an empty block. The Linux kernel for the Gandi hosting platform boots by default on the first partition ( root=/dev/xvda1 on the boot command line). If you copy an existing system disk, the partition table will be correct on the destination. Otherwise, you should ensure that the first partition of the destination disk contains the system files and is flagged as bootable. You can also additionally create a swap partition.

Requirements to Boot

In the case of a copy of an existing virtual disk, the source disk having already booted, you only need to check that the network configuration is using DHCP so that the produced images will be correctly bootable. In the case of an image created by a base system installation, there are a few items to verify and/or modify:

All of the services or modules related to physical elements of the server must be deactivated (such as, the service associated with the system clock - hwclock)

Some services at boot-time, such as ureadahead, console-setup, ondemand, plymouth, must be deactivated. Move away the init-scripts associated with these services. In an Ubuntu distribution, these scripts are found in /etc/init.

Add the gandi-hosting-agent and gandi-hosting-vm packages. The agent allows the installation of your server to complete. The scripts contained in the gandi-hosting-vm package allow the hosting platform to automatically and dynamically manage the server resources. For a system with python 2.6, you should install gandi-hosting-agent-py2.6 instead. For more details on these packages, please read the previous post on the Gandi Kitchen.

Delete the files related to ssh keys generated during the installation of the sshd package to avoid having the same key present on all of the servers generated from the same source image.

Verify that xinetd/inetd is started at boot -- if not, then activate it. The Gandi-agent is spawed via xinetd/inetd and will enable the final configuration of your server. The package gandi-hosting-agent installs its configuration file in /etc/gandi/agent.yml.

Verify that the permissions of your installation directory structure and the directories lib/ root/ and tmp/ of the installation directory. If in doubt, apply the same permissions to these directories as your existing server outside of the chroot.

chmod 0755 /var/tmp/lib

chmod 1777 /var/tmp/tmp

chmod 0750 /var/tmp/root

chmod 0755 /var/tmp/

Create the base files in /dev of your installation for the first boot stages, for example:

[ -e "$chroot"/dev/xvc0 ] || mknod "$chroot"/dev/xvc0 c 204 191

[ -e "$chroot"/dev/console ] || mknod "$chroot"/dev/console c 5 1

[ -e "$chroot"/dev/null ] || mknod "$chroot"/dev/null c 1 3

[ -e "$chroot"/dev/ptmx ] || mknod "$chroot"/dev/ptmx c 5 2

[ -e "$chroot"/dev/zero ] || mknod "$chroot"/dev/zero c 1 5

Add the necessary kernel modules corresponding to the kernel version in /lib/modules of your installation directory. The modules are available on mirrors.gandi.net/kernel/ (See the associated article on the wiki)

At the end of the configuration, if you are inside the chroot, leave it using 'exit' then unmount the various elements of your images in /proc/mounts. Kill any processes and daemons that may have been started during package installation within the chroot/directory. For example : grep /var/tmp/proc/mounts and then umount /var/tmp/proc and any others that may be listed. End by umount /var/tmp which should work without errors.

Detach the Disk

Return to your disk management interface in your Gandi hosting account, and detach the disk to which you have just copied, or on which you have just prepared the installation.

Make the disk bootable by associating a kernel

In the administration interface, you have the posibility to define a kernel for a disk. Select the virtual disk and associate it with a kernel suitable for the image you have just created. Let's take a "data" disk as an example, that we want to transform into a bootable virtual disk:

Use the link to change the disk information, and at the bottom of the page you will find an option to change the disk type:

You have now three advanced options, as for a bootable virtual disk:

By associating a kernel, the disk becomes usable as a custom image and will be shown in the list of images, just like the Gandi AI or expert images provided by Gandi during the creation of a new service via the administration interface.

Create a server from this image

During the creation of a server via your administration interface on the website, you will find your new custom image in the list of available images. The server creation will thus use this image as the source for the system disk. The server will then normally boot from a disk which is a copy of your image. As such, you have the possibility of creating serveral identical service from the same custom image.

Troubleshooting the server during boot

The Gandi hosting platform gives you the ability to access the command line of your server via the emergency console, which is accessible via an ssh session, and provides access to the console commandline of your server.

If you configure a getty on the console by default (tty1, xvc0, hvc0, depending on the chosen kernel version), you will have an emergency shell in case of boot errors, or a login prompt in the case of a successful server boot.

This emergency console will allow you to view the boot messages, and more importantly, any errors that may occur and thereby allowing you to debug your image.

Correcting the Image

Later, if you notice errors, or have forgotten anything in your custom image, you only need re-attach your new disk image to one of your servers (ultimately, and preferably, a server which was created from the same source image). Then you only need make the changes on the source image and then detach the disk. The image will thus be available to create new servers without error.

Gandi modification on standard OS

aegiap — Wed, 27 Oct 2010 18:03:00 +0200

What are the modification that Gandi staff makes on standard installed OS to be used on Gandi hosting?

Local modifications

Gandi hosting infrastructure is using Xen virtualization in paravirtualization mode (for the moment). As such we have to build a Linux kernel with specific options to allow it to boot your virtual server. Moreover as we allow customers to dynamically add or remove resources, the hotplug system in the kernel was patched by our team to allow a correct use of this features (mainly correct udev call). All kernel modules are available at each new kernel release on http://mirrors.gandi.net/kernel/. As of the 2.6.32, we now use upstream kernel source for building the xenU kernel and you should find the buildconfig file in /proc/config.gz on your virtual server. We add external patches such as drbd (before upstream integration).

Each Linux base system that we provide on Gandi hosting contains modifications by our team. For example, we removed services based on hardware clock as Xen does not provide direct access to it. On some distribution we had to disable boot features such as ureadhead or plymouth to allow a flawless boot of the virtual server. The main configuration is done during the boot process, especially the first boot process.

On a side note, on x86_32 architecture, to use the hardware capability of Xen, the libc could use the nosegneg hwcap with the correct libc-xen package.

Package gandi-hosting-vm

The idea which triggered this article was the release of a new version of gandi-hosting-vm. The package contains a collection of scripts to setup the local system of your virtual server at each boot and when specific events about hosting resources happen.

Changing hosting resources

When you add or remove resources dynamically to your virtual server, the Linux kernel receives information from Xen - the system managing all the virtualization. Each of these events are passed to the udevd daemon which apply configured rules to these events. It mainly creates files in directory /dev to allow access to the newly discovered resources.

For resources that Gandi hosting allows you to dynamically change, we wrote some udev rules (located in /etc/udev/rules.d/86-gandi.rules) to start a script when a virtual disk, a virtual interface or even a virtual cpu is attached (or removed) to the virtual server.

On a more detailed level, when a virtual interface is attached, the script /etc/gandi/manage_iface.sh is called by udev and a DHCP request is sent for this interface. A couple of other scripts setup the default route (/etc/gandi/dhcp-postconf) and store network configuration (/etc/dchp-hostname) in a tmpfs directory for further configuration at the end of the boot process. When the virtual interface is removed, the script simply removes the local network interface.

When a virtual disk is attached a similar script (/etc/gandi/manage_data_disk.py) is called. It tries to check the file system on the device or in its partitions (only in GandiAI mode) and mounts the file system in a specific mountpoint using the file system label /srv/<FS label>. If no label is setup on the file system, it uses the device or partition name as mount point (/srv/xvdc1 for example). To change the default mount options, please edit the variable mount_options in the beginning of the Python script.

During the boot process

The gandi-hosting-vm package provide two services called on boot : gandi-mount and gandi-config. The first one mounts already attached virtual disks in the local system in the /srv directory (see the description of /etc/gandi/manage_data_disk.py). You can start the service again once your server is booted and it will mount attached disks to the server (if you remove udev packages for example).

The second service starts a couple of specific plugins to setup your local system. Some of these configurations are optional and a configuration file is available for you to choose to setup each of these optional features /etc/default/gandi. Each configuration variable contains a short description in the default config file. These plugins are configuring the default local console for the hosting emergency console, configure the hostname and dns resolver, change the timezone to Europe/Paris, change the hwcap nosegneg according of your kernel version, change the motd to the default and so on.

For example, when the plugin 11-config_ssh is called, it creates SSH key for the local system if the keys are not already present. Then, depending on your configuration, it could add the Gandi SSH management key to the root user keyring (variable CONFIG_SSHMGMT) and reconfigure your sshd server by disabling password access for root, disabling empty password and enabling compression (variable CONFIG_SSHD).

Package gandi-hosting-agent

Gandi agent is used to setup the virtual machine according to customer information. In case of expert mode server, the setup of the local system is limited to setting the root password and creating the administrator user (as chosen by the customer) to avoid ssh-ing the server as root. In case of a GandiAI mode server, the agent uses specific modules to setup applications on the local system.

Once your expert server is setup after creation, you can remove gandi-hosting-agent packages. For example : dpkg -P $(dpkg -l | awk '/gandi-hosting-agent/ { print $2 }' | xargs) in deb based package system or rpm -e gandi-hosting-agent in rpm backed package system.

Hosting Public API 1.0 beta

Leland Vandervort — Wed, 20 Oct 2010 12:56:00 +0200

As you are probably already aware, we have been beavering away to offer you a public API to manage your resources on our Cloud hosting platform. In order to ease the work, several things have been reorganised so as to provide an interface which is easy to use, and allow a real management of your hosting resources at Gandi.

By way of this introduction, this article will be deliberately less technical, and will only succinctly present the elements that will be developed more in-depth when the official release of the API is launched.

API ?

For most human beings, the term API doesn't really mean a whole lot. Therefore, I suggest that we begin with a quick explanation of what an API is.

An Application Programming Interface (API) is an interface provided by a program or system. This interface allows other programs to use a standard machine to machine interface which in turn opens the possibility to create an appropriate human to machine interface. From the technical perspective, an API is a collection of functions, procedures, or classes exposed by a software library, operating system, or service. Knowledge of these APIs is paramount to the interoperability of various different software.

You can see the complete definition here.

Concretely, the Hosting API allows a user to manipulate and manage his various Gandi Hosting resources. For example, he may create new virtual machines, new disks, release IP addresses, etc.

Our API has been designed to be as simple as possible to use.

Use of the API

The API will allow you to manipulate all of your hosting "objects", from entire virtual machines, to components such as disks, network interfaces, IP addresses, etc., all in a logical and programmatic fashion. You will be able to create, delete, modify, and list the various objects.

One of the main objectives, however, is that it used by other software or programs in order to create something directly associated with us.

Let's look at two small examples:

One useful for everyone, thanks to the "vm.update" method, if you happen to detect in your log files that your server is in need of more memory, you could automatically add this memory to the virtual machine "on the fly".
An example for a company that produces software, thanks to "vm.create", you could potentially automatically deploy a server quickly from a disk that has previously been prepared with your application, and offer it directly to your customers.

The first set of methods will allow manipulation of the following objects:

datacenter - list , info

image - list, info

vm - list, info, count, create, update, start, stop, reboot, disk_attach, disk_detach, iface_attach, iface_detach

disk - list, info, count, create, create_from, update, delete

iface - list, info, count, create, delete

As well as management of operations:

operation - list, info, count, delete

New Features to Come

Taking advantage of the release of the API, several new features will also see the light of day. These are features for which you have been asking, for the most part, and which initially will be managed by our support team. Certain functions have been revised and move to areas where they would allow us to offer something new.

The choice of kernel and command-line options, for example, is now made on each disk. The disks, as a result, now have a more important role to play. It will also be possible to detach a former "system" disk, and to re-attach it to another virtual machine in order to, for example, recover the system following an unsuccessful upgrade. You could also choose from which disk a given virtual machine will boot.

How Do I Take Part?

The private beta phase has already begun. Simply send an email to beta-hosting AT gandi.net with your user ID (handle) and we will contact you with the details to allow access to the API.

Test Our Solution!

All European business customers at Gandi (and, initially, those subscribed to our newsletter) will find a coupon in their account allowing a free test-drive of our solution for a month. For everyone else, a request form for test shares is also available.

Be on the lookout though, as more information and updates will be made available in the coming weeks!

Kernel and cmdline

Ryan — Thu, 03 Jun 2010 11:00:00 +0200

We are pleased to announce that you can (finally!) choose the version of your kernel that you want (from a list which will be continually expanded), and associated boot options (cmdline). 2.6.18 and 2.6.27 are "base" versions supplied by Xen (backport of Xen patches for 2.6.27). Version 2.6.32, which is is currently available, uses paravirt_ops and the "Linux" implementation of Xen patches.

We will show you the new kernels here. They can be found in the "advanced mode" within the server's management page:

And you will then be able to access:

Concerning cmdline, you may now deactivate selinux at boot, boot as a single user, change the disk and the boot partition (which is practical for working with "images"), or choose the most appropriate console for your needs. In short, everything that you need to manage your updates in a more friendly environment, or to repair your server in the most autonomous manner.

If you feel that an option is missing, please let us know.

Mandriva 2010 image in alpha (updated)

aegiap — Fri, 21 May 2010 17:35:00 +0200

Server hosting by Gandi allow customers to choose from a selection of OS images available during the creation process of the virtual server. After the creation of the image by Gandi and internal testing, a new distribution is released to a specific group of hosting customers called 'alpha'. These clients can create server using these release candidate images. This allows Gandi to increase the types of testing and usage, and to find more bugs and problems by working with a small group of its customers.

Today - May the 21th - the Mandriva 2010.0 image has been released . This new version of the Mandriva distribution boots with a 2.6.27 kernel by default. It is currently only available for the 'alpha' customer group but will shortly be available for all customers. Please contact us if you wish to participate in our alpha testing phase.

16th August 2010 : Image is now available for everybody

Cherokee Arrives at Gandi

Leland Vandervort — Thu, 29 Apr 2010 13:48:00 +0200

(Translator's note: Taskigi Sequoyah would be proud! ;) )

A friend had explained to me that he had just finished installing a web server. Having decided on a change and not to use the mammoth Apache that everyone knows and loves, my friend seemed to have found a decent alternative. Bearing the name Cherokee, it appeared much leaner and with greater performance than Apache. See the benchmarks from the project website at the end of this article.

Personally, I had no knowledge of Cherokee, though I had certainly heard the name pandered around once or twice, but I never really paid it any attention at the time. So to give it a whirl, I decided to install it locally, which to my surprise exceeded my expectations. On a debian (or derived) distribution, installation is achieved by a simple apt-get install cherokee (or 'aptitude'...)

The web management interface provided is clean and intuitive. The basic idea is quite interesting; It is possible to activate the PHP interpreter with a simple mouse click; same for activating different virtual hosts as well as other options. There are also wizards to assist with the installation of such applications and frameworks as Django, Rails or Wordpress, etc.

I thought that it would be interesting to present this project during a developers meeting at Gandi, especially for those who use GandiAI. The presentation was well received. After a few pow-wow, and a glance through the source code, there was nothing really unusual or risky. This doesn't mean, of course, that there aren't any bugs in it, but at any rate clean code is a good sign! The core is written in C with the administrator interface written in Python.

Thus is the way in which Cherokee made its appearance at Gandi, and a server image was produced. In this way you can rapidly test a server with Cherokee pre-installed.

We have also added an "expert" mode distribution with Cherokee pre-installed... You need only to select this distribution during the steps to create your server. And, just as a reminder, in case you haven't yet tried our service, you can always request a free trial using our online form!

For further information about Cherokee, you can visit the project website. Additionally, you can find a comprehensive article about Cherokee in the Gnu/Linux magazine France, written [in French] by Carl Chenet in issue number 125.

E-Mail 'Goes Postal' - Gandi Mail Version 2

Leland Vandervort — Fri, 26 Mar 2010 14:54:00 +0100

Okay, excuse the corny punch line, but for those who haven't heard the expression, to "go postal" simply means to go crazy. Without delving into the origins of the phrase, suffice it to say that the term these days can also mean to go crazy or emphatic in a positive sense as well.. (or so some would like to believe ;-) )

As many may know by now, the Gandi Mail platform in the past couple of years has seen a few recurring instances of performance degradation every couple of months. As mentioned on the Gandi Bar, our teams have been working to bring you a new and more robust mail platform. This new platform is the culmination of a considerable amount of resource investment over many months.

I am pleased to say that the new platform is fully operational and all Gandi Mail customers are fully migrated to the new system, with the migration itself taking about six weeks or so. We decided to stage the migration gradually over a number of weeks to minimise any impact to our customers, and for the most part (with a very small handful of exceptions), the bulk of the migration was completed without anyone even noticing ;)

Anyway, without further ado, let's take a brief tour of the new mail platform and what has changed in Gandi Mail version 2. Oh.. and before we go on, this article will be at times a little bit technical with the use of some acronyms and other geek-speak. Don't worry, you'll soon get the gist of what we bearded technophiles blabber on about on a daily basis! ;)

In the Beginning, There Was the Word...

... and the word was written.. and ever since the dawn of the internet, the written word has become a critical part of our lives... yes.. that's right... e-mail. So, as a domain registrar, Gandi provides email services for use with the domains registered with us. So far so good...

The original platform was designed to be horizontally scalable to support several tens or even hundreds of thousands of mailboxes. One of the challenges faced was an architecture that had scalable storage capability whilst at the same time allowing the user to access his or her mailbox irrespective of which storage system or access server he or she connected to. Sounds like a job for the good old Network File System (NFS). As a result, the original platform was based on an NFS storage infrastructure for the mailboxes.

Incoming mail was received on any one of a number of inbound spool servers running Postfix. Once the mail was received and passed through a number of anti-spam and other filters, the spool would then identify which storage filer contained the mailbox in question, and forward the mail using SMTP to the back-end storage server (itself also running Postfix) for local delivery.

When the user wanted to access his or her mailbox, he would access a front-end server running Dovecot. The access servers would have "local" access to all of the mailboxes through the use of NFS mounts. The user would simply use a POP or IMAP client to connect to the server, to access his or her mailbox.

Outgoing mail is quite simple; basically an SMTP relay using SASL authentication.

The following diagram shows a very high level overview of the original version of the mail platform.

So What Has Changed?

Okay - before we get to the "what" has changed - let's first look at "why" we had to change it.

The original platform was great and the architecture works very well for moderate traffic levels, irrespective of the number of mailboxes. The risk with scaling a platform based on the number of mailboxes is that it is easy to overlook, or in some cases, misinterpret the knock-on effects of that increased scale. As the amount of traffic began to increase over the years, from time to time the front-end access servers would start having to contend for access to the NFS filesystems, which uses a system of locks to avoid corruption that may occur when there are multiple read/write operations on the same file or block.

This diagram outlines the average volumes for the past year. (note that the graph is not "stacked", so the elements are cumulative.) The vertical axis is "messages per minute", while the horizontal axis is by month.

As these locks increased over time (and remember that all of the servers had access across all of the filesystems), the result was a snowball effect that caused severe performance degradation of the whole platform -- and not only for the mailboxes on the storage server with the lock in place. During this time, users would attempt to connect to their mailboxes at which time the server would accept the connection and simply wait for the lock to free in order to access the mailbox.

So the challenges were simple:

How to eliminate the need for NFS and still allow horizontal scalability.
How to avoid impacting the entire mail platform in case of a difficulty on just one storage server, and how to minimise the impact to customers in this case.
How to maximise the performance of the platform to allow vertical scalability as well as horizontal.

Since there is very little change on the incoming SMTP spool elements, and the majority of the load was associated with NFS, let's look straight as the access elements.

Where Is My Mailbox?

Okay, so the user connects to his mailbox with his mail client (Thunderbird, Outlook Express, Mail.App, Evolution, or anything else for that matter...). The client connection arrives on one of a number of front-end mail access servers running Dovecot. How then, does the server know where to look to find the mailbox? Originally, the mailbox was "local" because it was mounted via NFS. Dovecot made a simple database lookup to determine the filesystem path that the mailbox was mounted under. With the new system, there is no NFS, so there is no "local" filesystem for Dovecot to look under.

This is where a very useful feature of Dovecot come into play -- the proxy function. Using this, the front end server performs the authentication of the user, checks which storage server the mailbox is located on, and then initiates a proxy "client" connection directly to the storage server which itself is running Dovecot. If the client connects using IMAP, then the proxy connection is also IMAP. Similarly if the client is using POP3, then the proxy is also POP3. The storage server does not need to re-authenticate the connection.

There are a few benefits of this architecture:

Elimination of NFS also eliminates the side-effect of NFS locks.
Since the back-end storage server actually has the mailbox physically locally attached, there is no contention on the filesystem, and no need for locks. Plus, since the storage arrays are high performance anyway, access to the mailbox is much faster.
The front-end servers no longer have to perform local disk I/O operations, and thus consume considerably less CPU. (In fact, technically, there is no real reason for the front-end servers to even have disks of their own -- this could enable lower cost horizontal access scaling by being able to use diskless servers...)

What Happens if a Filer Breaks?

To respond to the other part of the challenge, and to limit the impact in case of component failure to as few customers as possible, the original concept of scaling storage to hold as many mailboxes as possible had to be discarded. After all, if a filer happened to fall over, all the mailboxes on that filer would also be offline.

So the idea here is to increase the number of storage servers, and spread the mailboxes more thinly across them. In this way, in case of a failure of the storage server, fewer mailboxes are affected.

The second aspect to minimising the impact of a component failure is fairly simple as well. With the previous version of the platform (yes, remember the NFS locks?), a client connection to a mailbox would be answered by the access server and simply wait for the filesystem access. The effect for the user is that his client would just "sit there" and eventually time out.

Using the IMAP/POP3 proxy arrangement, if the actual storage server is down, the front-end server will reply immediately to the mail client with a "Temporarily Unavailable" message, and the TCP connection is closed.

The disks arrays themselves are, of course, redundant. The only real potential single point of failure is the server that controls the disk arrays since due to technical limitations of the disk arrays, it is not possible to have dual controllers if using split and mirrored RAID volumes across two disk arrays. It would have been possible if the volumes weren't mirrored across arrays, but this would have been more risky as there would be no "backup" copy of the data volume in case of an array failure... we thus considered that the single controller server is an acceptable risk provided a spare is available and can be easily swapped in. The following image depicts the mail storage solution.

But.. I Didn't Notice the Migration

If you are in this category, then all I can say is "super!" -- that's what we intended. Though, we did have one or two hiccups along the way, and a very small minority of customers noticed, at no point was data lost, jeopardised, or otherwise endangered :)

So, how did we do the migration? Well, over a course of a number of weeks, and mostly during off-peak times, our admins worked on one filer at a time, migrating all customer mailboxes to the new filer structure using rsync... several iterations of it, in fact. At the last iteration to fully synchronise, the database was immediately updated to reflect the new filer as the storage location of the mailbox. All new deliveries, access requests, etc., were then made to the new filer.

This process went on filer by filer over the course of a number of weeks. An interesting side-effect of this migration and the gradual removal of NFS from the architecture is the gradual reduction of average CPU load of the access servers over the migration period, as can be seen in the graphs below. Of course now the CPU load is pretty much negligible since NFS has been eliminated. The two graphs are relative scale based on the average of the time period. The first graph is the past six months, while the second graph is the average over the past five weeks.

Some Interesting Figures

Here are just a few interesting facts about the Gandi Mail platform.

Average 60 million emails per day via the SMTP incoming spools and outgoing relays.

8 outgoing SMTP relays
8 incoming SMTP spools
10 mail filters (anti-spam, etc.)
7 front-end access servers (POP3 / IMAP)
16 mailbox storage filers
4 database servers (2 read plus 2 master with replication)
Hardware distributed among multiple datacentres

Just a Quick Stats Update Three Days Later

Just wanted to add a quick update to the IO-Wait CPU load for the front-end servers now three days on. In the following graph showing the CPU load on the front-end access servers for the past seven days, you can see the significant difference before and after the final migration ;)

I hope that this article has given a useful insight into the new Gandi Mail platform.

Gandi 10th Anniversary - The Experience

Leland Vandervort — Wed, 17 Mar 2010 19:33:00 +0100

To celebrate Gandi's 10th anniversary, this hair-brained idea to give away, in ten days, 55000 domains, raise a very practical question. How, once we open the floodgates on such an operation, to maintain the highest quality of service on the site? The festive spirit could well have transformed into a nightmare for our customers if they were suddenly unable to access their management interface.

So we took the decision to host the event on a dedicated site. This was a hitherto dreamed of occasion to put ourselves into our customers' shoes, and use our hosting infrastructure for this event. We defined the rules of play: Using only the tools provided to our customers, build an architecture which was easily scalable and didn't break the bank, and to demonstrate our renowned flexibility.

Keep it Simple, Stupid.

We had the "luxury" of one week from design to implementation. As a result, the charming idea to demonstrate our "cloudlike" site based on modern technology was summarily put out of our minds. To be perfectly honest, given the time scale, the lucky chosen developer had a nifty precept to select the technology: "You have full choice of the technology, but you have one week." It would be PHP/MySQL, which isn't exactly everybody's favourite! It would, nevertheless, allow us to release a tested site within the tight time frame.

To adequately sustain the load generated by such an event, several servers would be needed. We then hit our first stumbling block: Gandi does not [yet] have a load balancing solution for the hosting solution! Nevermind, we'll use the old yet faithful round-robin DNS method, with a low TTL to be able to quickly remove a front-end server from production in case of an incident.

Our Linux distribution for this occasion would be Ubuntu 9.10 - because it is reasonably up-to-date, with the 2.6.27 kernel.

Small Servers - Go Forth and Multiply!

The best way to sustain the high loads for our solution is to split the functionality among several small servers. This way we would maintain a minimum level of "vertical" scalability (you can dynamically increase the memory and CPU allocated to a server), and the architecture provides "horizontal" scalability.

In this way we could easily add resources if we started to feel the pinch, and add or migrate shares from one server to another as load requirements necessitate. There are numerous advantages of using multiple small servers:

each server gets a minimum of one core, burstable, of the CPU (yes, one whole core, even with one share -- that's new, by the way!)
assured resilience, with shares spread somewhat randomly across a few hundred different physical servers.
specifically in a virtualised environment, the memory performance is best with less than 1GB of memory.
if you have 4 servers of one share each, rather than one big server of 4 shares, they can dynamically increase to 8x4 shares, or 24x4 shares with a reboot, and all of this without modification to the architecture. A big server, however, would only scale to 8 shares without a reboot, or 24 after reboot.
resources may be easily moved towards servers that need it most

We commence with a simple architecture:

24 (!) servers of one share each, to manage the PHP website: 10 for the English site, 10 for the French site, and an additional 2 of each for IPv6.
2 servers of 4 shares each for replicated memcached to reduce database load and manage sessions.
1 MySQL server of 4 shares, which contains the pre-generated promotion codes (they actually all fit within memory, so the database itself should really be pretty bored doing nothing...)

After a couple of lovely overloads and a bit of code review, the database would finally be greatly spared by memcached (see the section "lightweight coding...").

One of our administrators would put his fingers to work on the site to create and configure 24 servers -- at the same time! Obviously the release of the hosting API or an admin interface function would have been welcome. (Thanks to cssh in this case!)

Lock Down (somewhat) the Machines

A default installation always needs a few finishing touches. The very fact of opening a MySQL database on the "public" network made us a little edgy. So, swooping 'netstat' and shutting down non-critical services listening on public ports. With the help of tcp wrappers (hosts.allow, hosts.deny), all of the "private" interfaces are also locked down (sshd, mysql accessibly only from the web farm).

Finally it behooved us to pay close attention to the PHP code and MySQL queries; The safest way to avoid php code injections is to bind all the parameters after a prepare(). This also helps reduce load on the database when several execute() are called.

One important detail: since the site should allow a user to send an email to any "arbitrary" address, it was absolutely critical to limit its potential for abuse by some clever black-hat as much as possible. At the very minimum, the number of sent emails per promotion code was limited, in addition to very close monitoring.

Setup the Development and Deployment Environment

The sharing of data between the sites in effect adds a single point of failure, as well as a potential architectural bottleneck. As such, we decided to deploy the content of the site locally on each of the servers. We would use one server for developing and staging, and ultimately for the development and testing of updates. A quick script and some 'rsync' would allow rapid deployment across the entire front-end architecture. Simple! (some would say ;) )

Resource Monitoring

A few moments before the operation, more as a precaution rather than a cure, all of the virtual machines from one to two shares. Using the statistics interface, from day one, one can see that the the virtual machines were essentially sitting "twiddling their thumbs" from boredom ;) :

It would have been cool, at this very moment, to reduce back to a single share per server, or make use of Gandi "Autoflex", or even given the actual load observed, set up scheduled flex for each hour to hand out the promotion codes! Unfortunately, with all hands on deck, we missed this opportunity to demonstrate this [econono-techno-ecological ;)] feature.

Lightweight Code is Worth More than a Thousand Beefy CPUs

Even though we physically had several thousand CPUs and a few Terabytes of RAM at our fingertips, Tuesday turned out to be somewhat chaotic and worthy of note here. After Monday, which managed the load very well, the "smooth" execution of our one and only SELECT COUNT brutally altered and became excruciatingly slow (300ms). We had naively thought that this "only" query, on a table held exclusively in memory, wouldn't be an issue. As such, it was executed on every page of the site. The multiple simultaneous accesses to the database, coupled with the UPDATE operations for the promotion codes, resulted in the database, despite the near-idle system performance, started causing database lock contention.

The usual knee-jerk reaction to such a situation is to increase the number of shares to support the load. It's great for a quick-fix temporary solution, but it's not enough!

A new analysis of the system, questions about the code, and the use (or salvation) of memcached resulted in recovering the optimal performance. Equally, a modification of the database queries used probably would have been prudent.

The moral of the story: the code, indexes, architecture (etc.) are the cornerstones of your ability to support usage load, and if they are "CPU friendly", they will save the day. Otherwise a catastrophe could be lurking, or at the very least, the unnecessary purchase of additional shares.

Also, as we said earlier somewhat tongue-in-cheek -- it's eco-friendly!

Some Numbers

36 shares total, but we could have done it with less (*sniff*)
5% CPU usage at peak
4000 requests per front-end web server in the first minute of each hour (roughly 1400 requests/second total)
a minimum of 11 seconds to hand out 1000 promotion codes.
a maximum of 40 minutes to hand out the same number of promotion codes, during the Tuesday incident described above.

What to do if your server stops responding?

Ryan — Wed, 13 Jan 2010 13:26:00 +0100

As you probably already know, our platform protects you from hardware failures that might occur on your server.

In the event of a problem on the machine, or if we suspect that a problem might occur (abnormal temperature, corrupted memory, etc.), your “server” will automatically be migrated to another machine. However, if you have an internal problem on your "server" that is not to due to the physical machine, and if it no longer responds, then you will need to take action.
The first action to take is to be sure that your server’s status is shown as “Running” on your Gandi interface. This is because the status may also be “Stopped” or “Paused” if, for example, it has not been renewed. So given that it is "Running" and non-responding, here's what to do.

There are three different cases:

1. You can still connect to your server via SSH. In this case, the following commands will help you analyze the situation:

"uptime" will give you the current load of the machine,
"free", will show you the amount of memory used by your applications in the “used” column,
"top" (we recommend that you install “htop") will show you the ranking of applications in realtime ordered by their use of resources (memory, CPU),
"dmesg" shows you messages from your linux server’s kernel,
Consulting logs such as /var/log/messages or /var/log/daemons with the command, “tail”, for example (tail /var/log/daemons) will also provide you with precious information.
"df –h" shows you the amount of disk space available on your disks.

The most frequent causes of error are:

No space left on your system disk: this situation is often caused by a lack of appropriate log management on the server, or by a database which fills up too quickly. The solution is to clean up wasted space, or enlarge the disk (see the guide on this).
Not enough RAM on the server, or too much memory used: the simple solution is to add more RAM by adding additional shares. If you have an expert server, you can also try to modify the behavior of the Linux’s available memory by using the command 'sysctl -w vm.overcommit_memory = 2'. Warning: so that the modification can be maintained following a reboot, you must also add "vm.overcommit_memory = 2" to the "/etc/sysctl.conf" and "/etc/gandi/sysctl.conf" files.
Too may processes are running simultaneously on your machine: you will need to lower the values in the configuration files of your applications (the number of simultaneous connections on Apache, for example) or increasing the power of your server by adding shares.

2. You can no longer connect to your server under SSH, and it does not respond to ping or it has a slow response time.

The virtual console which you may activate from your account (guide available), gives you direct access to your machine as if a monitor and a keyboard (still virtual) were directly attached to the server.

In this case, you can stop all the applications that are causing problems, and once again get access to the server.

The 'sysreq' shortcuts are available from the console of your server. The commands can be given by pressing Ctrl and “o” (as Oscar) in order to enter the sysreq mode, and then you can enter the command. By doing this, you can stop all the processes by: Ctrl+o +i (to kill). Ctrl+o +h gives you quick help to all of the available sysreq commands.

3. Your server may be “unreachable” but may nonetheless be working normally, and without any technical problem.

This may happen if your server is the victim of a DDoS attack for example. Your server will therefore be isolated from the rest of the network in order to protect our infrastructure and the quality of service for other customers.

You may verify whether or not your server is in this state by performing a “traceroute” command on the IP address of your server. If it stops at its arrival at Gandi, on one of our routers for example, it is likely that your server has been isolated from the network. You may then connect to your server via the console, though you will need to contact the support team to correct the problem (they will usually contact you first). This sort of isolation happens only very rarely.

After that, if you are still blocked, then this means that you are in the 4th case: your very own case ;-). Please send an email to our support team indicating that your server has been blocked, and you will get an answer as soon as possible.

How to turn your website into a "Web Infrastructure"

Ryan — Wed, 13 Jan 2010 13:25:00 +0100

Many websites start with a single server solution, a box acting as a web server and database server all in one. Simply, easy, cheap. The problem comes when traffic gets too high (a victim of their own success!). Many customers want a bigger box, but the answer is actually changing your architecture from "web server" to "web infrastructure". You can duplicate web servers, use the DNS to load balance them and ramp up your capacity very fast and very far.

For simplicity's sake, let's take a real example. As some of you already know, we support the Millenium association in their promotion of online video games.

The increasing success of the website, millenium.org, made us re-design the architecture of the website so that it could handle the numerous videos shown on the website to its 17,000 unique visitors per day far more easily and efficiently.

Following a major update of one of the games served by Millenium (WOW patch 3.1), we increased the server power to 16 shares in anticipation of an increase in load. Our expectations were quickly exceeded with over 50,000 unique visitors the first day, and just as many over the following days.

The website received between 500 and 1,000 simultaneous visitors and a large number of videos, which is not viable for a single LAMP server. We immediately changed the infrastructure, by moving from a unique-server model (which is often the starting choice) to an infrastructure-based model. We went from a vertical system (more power) to a horizontal system (more servers):

As you can see in the diagram, we moved the database to a separate server and duplicated the web server to two machines. The load was therefore split by the domain's DNS (using a simple DNS round robin technique). We could also move to dedicating a 1-share server to load distribution, though such a solution would take a bit longer to implement. In our case it took 2 minutes to add the shares to the account, 6 minutes to create the 2 servers, 10 minutes to transfer the data, and 2 hours to configure the services.

Today, the platform easily handles 1 million unique visitors per month, for over 3 million pages viewed - a good thing! Best of all the platform is now capable of evolving. If the database suffers, all we need to do is to add more shares to the database server. If the web front starts to become saturated, we just need to add another.

Because our cloud infrastructure allows you to create as many servers as you want from the hosting resources in your account, you can always add more, and always change your web architecture without replacing physical servers. The VPS hosting system is flexible and allows you to increase from a 1-share server (1/64 of a machine + 1/64 as reserve, 256MB of Ram) to a 16-share server (1/4 of a machine + 1/4 as a reserve, 4GB de Ram) at any time and as often as you want.

If you find yourself confronted by this type of problem, please feel free to contact us, as we would be more than happy to help you.

Gandi Hosting Team