Gandi Kitchen

Kernel and cmdline

Ryan — Thu, 03 Jun 2010 11:00:00 +0200

We are pleased to announce that you can (finally!) choose the version of your kernel that you want (from a list which will be continually expanded), and associated boot options (cmdline). 2.6.18 and 2.6.27 are "base" versions supplied by Xen (backport of Xen patches for 2.6.27). Version 2.6.32, which is is currently available, uses paravirt_ops and the "Linux" implementation of Xen patches.

We will show you the new kernels here. They can be found in the "advanced mode" within the server's management page:

And you will then be able to access:

Concerning cmdline, you may now deactivate selinux at boot, boot as a single user, change the disk and the boot partition (which is practical for working with "images"), or choose the most appropriate console for your needs. In short, everything that you need to manage your updates in a more friendly environment, or to repair your server in the most autonomous manner.

If you feel that an option is missing, please let us know.

Mandriva 2010 image in alpha

aegiap — Fri, 21 May 2010 17:35:00 +0200

Server hosting by Gandi allow customers to choose from a selection of OS images available during the creation process of the virtual server. After the creation of the image by Gandi and internal testing, a new distribution is released to a specific group of hosting customers called 'alpha'. These clients can create server using these release candidate images. This allows Gandi to increase the types of testing and usage, and to find more bugs and problems by working with a small group of its customers.

Today - May the 21th - the Mandriva 2010.0 image has been released . This new version of the Mandriva distribution boots with a 2.6.27 kernel by default. It is currently only available for the 'alpha' customer group but will shortly be available for all customers. Please contact us if you wish to participate in our alpha testing phase.

Cherokee Arrives at Gandi

Leland Vandervort — Thu, 29 Apr 2010 13:48:00 +0200

(Translator's note: Taskigi Sequoyah would be proud! ;) )

A friend had explained to me that he had just finished installing a web server. Having decided on a change and not to use the mammoth Apache that everyone knows and loves, my friend seemed to have found a decent alternative. Bearing the name Cherokee, it appeared much leaner and with greater performance than Apache. See the benchmarks from the project website at the end of this article.

Personally, I had no knowledge of Cherokee, though I had certainly heard the name pandered around once or twice, but I never really paid it any attention at the time. So to give it a whirl, I decided to install it locally, which to my surprise exceeded my expectations. On a debian (or derived) distribution, installation is achieved by a simple apt-get install cherokee (or 'aptitude'...)

The web management interface provided is clean and intuitive. The basic idea is quite interesting; It is possible to activate the PHP interpreter with a simple mouse click; same for activating different virtual hosts as well as other options. There are also wizards to assist with the installation of such applications and frameworks as Django, Rails or Wordpress, etc.

I thought that it would be interesting to present this project during a developers meeting at Gandi, especially for those who use GandiAI. The presentation was well received. After a few pow-wow, and a glance through the source code, there was nothing really unusual or risky. This doesn't mean, of course, that there aren't any bugs in it, but at any rate clean code is a good sign! The core is written in C with the administrator interface written in Python.

Thus is the way in which Cherokee made its appearance at Gandi, and a server image was produced. In this way you can rapidly test a server with Cherokee pre-installed.

We have also added an "expert" mode distribution with Cherokee pre-installed... You need only to select this distribution during the steps to create your server. And, just as a reminder, in case you haven't yet tried our service, you can always request a free trial using our online form!

For further information about Cherokee, you can visit the project website. Additionally, you can find a comprehensive article about Cherokee in the Gnu/Linux magazine France, written [in French] by Carl Chenet in issue number 125.

E-Mail 'Goes Postal' - Gandi Mail Version 2

Leland Vandervort — Fri, 26 Mar 2010 14:54:00 +0100

Okay, excuse the corny punch line, but for those who haven't heard the expression, to "go postal" simply means to go crazy. Without delving into the origins of the phrase, suffice it to say that the term these days can also mean to go crazy or emphatic in a positive sense as well.. (or so some would like to believe ;-) )

As many may know by now, the Gandi Mail platform in the past couple of years has seen a few recurring instances of performance degradation every couple of months. As mentioned on the Gandi Bar, our teams have been working to bring you a new and more robust mail platform. This new platform is the culmination of a considerable amount of resource investment over many months.

I am pleased to say that the new platform is fully operational and all Gandi Mail customers are fully migrated to the new system, with the migration itself taking about six weeks or so. We decided to stage the migration gradually over a number of weeks to minimise any impact to our customers, and for the most part (with a very small handful of exceptions), the bulk of the migration was completed without anyone even noticing ;)

Anyway, without further ado, let's take a brief tour of the new mail platform and what has changed in Gandi Mail version 2. Oh.. and before we go on, this article will be at times a little bit technical with the use of some acronyms and other geek-speak. Don't worry, you'll soon get the gist of what we bearded technophiles blabber on about on a daily basis! ;)

In the Beginning, There Was the Word...

... and the word was written.. and ever since the dawn of the internet, the written word has become a critical part of our lives... yes.. that's right... e-mail. So, as a domain registrar, Gandi provides email services for use with the domains registered with us. So far so good...

The original platform was designed to be horizontally scalable to support several tens or even hundreds of thousands of mailboxes. One of the challenges faced was an architecture that had scalable storage capability whilst at the same time allowing the user to access his or her mailbox irrespective of which storage system or access server he or she connected to. Sounds like a job for the good old Network File System (NFS). As a result, the original platform was based on an NFS storage infrastructure for the mailboxes.

Incoming mail was received on any one of a number of inbound spool servers running Postfix. Once the mail was received and passed through a number of anti-spam and other filters, the spool would then identify which storage filer contained the mailbox in question, and forward the mail using SMTP to the back-end storage server (itself also running Postfix) for local delivery.

When the user wanted to access his or her mailbox, he would access a front-end server running Dovecot. The access servers would have "local" access to all of the mailboxes through the use of NFS mounts. The user would simply use a POP or IMAP client to connect to the server, to access his or her mailbox.

Outgoing mail is quite simple; basically an SMTP relay using SASL authentication.

The following diagram shows a very high level overview of the original version of the mail platform.

So What Has Changed?

Okay - before we get to the "what" has changed - let's first look at "why" we had to change it.

The original platform was great and the architecture works very well for moderate traffic levels, irrespective of the number of mailboxes. The risk with scaling a platform based on the number of mailboxes is that it is easy to overlook, or in some cases, misinterpret the knock-on effects of that increased scale. As the amount of traffic began to increase over the years, from time to time the front-end access servers would start having to contend for access to the NFS filesystems, which uses a system of locks to avoid corruption that may occur when there are multiple read/write operations on the same file or block.

This diagram outlines the average volumes for the past year. (note that the graph is not "stacked", so the elements are cumulative.) The vertical axis is "messages per minute", while the horizontal axis is by month.

As these locks increased over time (and remember that all of the servers had access across all of the filesystems), the result was a snowball effect that caused severe performance degradation of the whole platform -- and not only for the mailboxes on the storage server with the lock in place. During this time, users would attempt to connect to their mailboxes at which time the server would accept the connection and simply wait for the lock to free in order to access the mailbox.

So the challenges were simple:

How to eliminate the need for NFS and still allow horizontal scalability.
How to avoid impacting the entire mail platform in case of a difficulty on just one storage server, and how to minimise the impact to customers in this case.
How to maximise the performance of the platform to allow vertical scalability as well as horizontal.

Since there is very little change on the incoming SMTP spool elements, and the majority of the load was associated with NFS, let's look straight as the access elements.

Where Is My Mailbox?

Okay, so the user connects to his mailbox with his mail client (Thunderbird, Outlook Express, Mail.App, Evolution, or anything else for that matter...). The client connection arrives on one of a number of front-end mail access servers running Dovecot. How then, does the server know where to look to find the mailbox? Originally, the mailbox was "local" because it was mounted via NFS. Dovecot made a simple database lookup to determine the filesystem path that the mailbox was mounted under. With the new system, there is no NFS, so there is no "local" filesystem for Dovecot to look under.

This is where a very useful feature of Dovecot come into play -- the proxy function. Using this, the front end server performs the authentication of the user, checks which storage server the mailbox is located on, and then initiates a proxy "client" connection directly to the storage server which itself is running Dovecot. If the client connects using IMAP, then the proxy connection is also IMAP. Similarly if the client is using POP3, then the proxy is also POP3. The storage server does not need to re-authenticate the connection.

There are a few benefits of this architecture:

Elimination of NFS also eliminates the side-effect of NFS locks.
Since the back-end storage server actually has the mailbox physically locally attached, there is no contention on the filesystem, and no need for locks. Plus, since the storage arrays are high performance anyway, access to the mailbox is much faster.
The front-end servers no longer have to perform local disk I/O operations, and thus consume considerably less CPU. (In fact, technically, there is no real reason for the front-end servers to even have disks of their own -- this could enable lower cost horizontal access scaling by being able to use diskless servers...)

What Happens if a Filer Breaks?

To respond to the other part of the challenge, and to limit the impact in case of component failure to as few customers as possible, the original concept of scaling storage to hold as many mailboxes as possible had to be discarded. After all, if a filer happened to fall over, all the mailboxes on that filer would also be offline.

So the idea here is to increase the number of storage servers, and spread the mailboxes more thinly across them. In this way, in case of a failure of the storage server, fewer mailboxes are affected.

The second aspect to minimising the impact of a component failure is fairly simple as well. With the previous version of the platform (yes, remember the NFS locks?), a client connection to a mailbox would be answered by the access server and simply wait for the filesystem access. The effect for the user is that his client would just "sit there" and eventually time out.

Using the IMAP/POP3 proxy arrangement, if the actual storage server is down, the front-end server will reply immediately to the mail client with a "Temporarily Unavailable" message, and the TCP connection is closed.

The disks arrays themselves are, of course, redundant. The only real potential single point of failure is the server that controls the disk arrays since due to technical limitations of the disk arrays, it is not possible to have dual controllers if using split and mirrored RAID volumes across two disk arrays. It would have been possible if the volumes weren't mirrored across arrays, but this would have been more risky as there would be no "backup" copy of the data volume in case of an array failure... we thus considered that the single controller server is an acceptable risk provided a spare is available and can be easily swapped in. The following image depicts the mail storage solution.

But.. I Didn't Notice the Migration

If you are in this category, then all I can say is "super!" -- that's what we intended. Though, we did have one or two hiccups along the way, and a very small minority of customers noticed, at no point was data lost, jeopardised, or otherwise endangered :)

So, how did we do the migration? Well, over a course of a number of weeks, and mostly during off-peak times, our admins worked on one filer at a time, migrating all customer mailboxes to the new filer structure using rsync... several iterations of it, in fact. At the last iteration to fully synchronise, the database was immediately updated to reflect the new filer as the storage location of the mailbox. All new deliveries, access requests, etc., were then made to the new filer.

This process went on filer by filer over the course of a number of weeks. An interesting side-effect of this migration and the gradual removal of NFS from the architecture is the gradual reduction of average CPU load of the access servers over the migration period, as can be seen in the graphs below. Of course now the CPU load is pretty much negligible since NFS has been eliminated. The two graphs are relative scale based on the average of the time period. The first graph is the past six months, while the second graph is the average over the past five weeks.

Some Interesting Figures

Here are just a few interesting facts about the Gandi Mail platform.

Average 60 million emails per day via the SMTP incoming spools and outgoing relays.

8 outgoing SMTP relays
8 incoming SMTP spools
10 mail filters (anti-spam, etc.)
7 front-end access servers (POP3 / IMAP)
16 mailbox storage filers
4 database servers (2 read plus 2 master with replication)
Hardware distributed among multiple datacentres

Just a Quick Stats Update Three Days Later

Just wanted to add a quick update to the IO-Wait CPU load for the front-end servers now three days on. In the following graph showing the CPU load on the front-end access servers for the past seven days, you can see the significant difference before and after the final migration ;)

I hope that this article has given a useful insight into the new Gandi Mail platform.

Gandi 10th Anniversary - The Experience

Leland Vandervort — Wed, 17 Mar 2010 19:33:00 +0100

To celebrate Gandi's 10th anniversary, this hair-brained idea to give away, in ten days, 55000 domains, raise a very practical question. How, once we open the floodgates on such an operation, to maintain the highest quality of service on the site? The festive spirit could well have transformed into a nightmare for our customers if they were suddenly unable to access their management interface.

So we took the decision to host the event on a dedicated site. This was a hitherto dreamed of occasion to put ourselves into our customers' shoes, and use our hosting infrastructure for this event. We defined the rules of play: Using only the tools provided to our customers, build an architecture which was easily scalable and didn't break the bank, and to demonstrate our renowned flexibility.

Keep it Simple, Stupid.

We had the "luxury" of one week from design to implementation. As a result, the charming idea to demonstrate our "cloudlike" site based on modern technology was summarily put out of our minds. To be perfectly honest, given the time scale, the lucky chosen developer had a nifty precept to select the technology: "You have full choice of the technology, but you have one week." It would be PHP/MySQL, which isn't exactly everybody's favourite! It would, nevertheless, allow us to release a tested site within the tight time frame.

To adequately sustain the load generated by such an event, several servers would be needed. We then hit our first stumbling block: Gandi does not [yet] have a load balancing solution for the hosting solution! Nevermind, we'll use the old yet faithful round-robin DNS method, with a low TTL to be able to quickly remove a front-end server from production in case of an incident.

Our Linux distribution for this occasion would be Ubuntu 9.10 - because it is reasonably up-to-date, with the 2.6.27 kernel.

Small Servers - Go Forth and Multiply!

The best way to sustain the high loads for our solution is to split the functionality among several small servers. This way we would maintain a minimum level of "vertical" scalability (you can dynamically increase the memory and CPU allocated to a server), and the architecture provides "horizontal" scalability.

In this way we could easily add resources if we started to feel the pinch, and add or migrate shares from one server to another as load requirements necessitate. There are numerous advantages of using multiple small servers:

each server gets a minimum of one core, burstable, of the CPU (yes, one whole core, even with one share -- that's new, by the way!)
assured resilience, with shares spread somewhat randomly across a few hundred different physical servers.
specifically in a virtualised environment, the memory performance is best with less than 1GB of memory.
if you have 4 servers of one share each, rather than one big server of 4 shares, they can dynamically increase to 8x4 shares, or 24x4 shares with a reboot, and all of this without modification to the architecture. A big server, however, would only scale to 8 shares without a reboot, or 24 after reboot.
resources may be easily moved towards servers that need it most

We commence with a simple architecture:

24 (!) servers of one share each, to manage the PHP website: 10 for the English site, 10 for the French site, and an additional 2 of each for IPv6.
2 servers of 4 shares each for replicated memcached to reduce database load and manage sessions.
1 MySQL server of 4 shares, which contains the pre-generated promotion codes (they actually all fit within memory, so the database itself should really be pretty bored doing nothing...)

After a couple of lovely overloads and a bit of code review, the database would finally be greatly spared by memcached (see the section "lightweight coding...").

One of our administrators would put his fingers to work on the site to create and configure 24 servers -- at the same time! Obviously the release of the hosting API or an admin interface function would have been welcome. (Thanks to cssh in this case!)

Lock Down (somewhat) the Machines

A default installation always needs a few finishing touches. The very fact of opening a MySQL database on the "public" network made us a little edgy. So, swooping 'netstat' and shutting down non-critical services listening on public ports. With the help of tcp wrappers (hosts.allow, hosts.deny), all of the "private" interfaces are also locked down (sshd, mysql accessibly only from the web farm).

Finally it behooved us to pay close attention to the PHP code and MySQL queries; The safest way to avoid php code injections is to bind all the parameters after a prepare(). This also helps reduce load on the database when several execute() are called.

One important detail: since the site should allow a user to send an email to any "arbitrary" address, it was absolutely critical to limit its potential for abuse by some clever black-hat as much as possible. At the very minimum, the number of sent emails per promotion code was limited, in addition to very close monitoring.

Setup the Development and Deployment Environment

The sharing of data between the sites in effect adds a single point of failure, as well as a potential architectural bottleneck. As such, we decided to deploy the content of the site locally on each of the servers. We would use one server for developing and staging, and ultimately for the development and testing of updates. A quick script and some 'rsync' would allow rapid deployment across the entire front-end architecture. Simple! (some would say ;) )

Resource Monitoring

A few moments before the operation, more as a precaution rather than a cure, all of the virtual machines from one to two shares. Using the statistics interface, from day one, one can see that the the virtual machines were essentially sitting "twiddling their thumbs" from boredom ;) :

It would have been cool, at this very moment, to reduce back to a single share per server, or make use of Gandi "Autoflex", or even given the actual load observed, set up scheduled flex for each hour to hand out the promotion codes! Unfortunately, with all hands on deck, we missed this opportunity to demonstrate this [econono-techno-ecological ;)] feature.

Lightweight Code is Worth More than a Thousand Beefy CPUs

Even though we physically had several thousand CPUs and a few Terabytes of RAM at our fingertips, Tuesday turned out to be somewhat chaotic and worthy of note here. After Monday, which managed the load very well, the "smooth" execution of our one and only SELECT COUNT brutally altered and became excruciatingly slow (300ms). We had naively thought that this "only" query, on a table held exclusively in memory, wouldn't be an issue. As such, it was executed on every page of the site. The multiple simultaneous accesses to the database, coupled with the UPDATE operations for the promotion codes, resulted in the database, despite the near-idle system performance, started causing database lock contention.

The usual knee-jerk reaction to such a situation is to increase the number of shares to support the load. It's great for a quick-fix temporary solution, but it's not enough!

A new analysis of the system, questions about the code, and the use (or salvation) of memcached resulted in recovering the optimal performance. Equally, a modification of the database queries used probably would have been prudent.

The moral of the story: the code, indexes, architecture (etc.) are the cornerstones of your ability to support usage load, and if they are "CPU friendly", they will save the day. Otherwise a catastrophe could be lurking, or at the very least, the unnecessary purchase of additional shares.

Also, as we said earlier somewhat tongue-in-cheek -- it's eco-friendly!

Some Numbers

36 shares total, but we could have done it with less (*sniff*)
5% CPU usage at peak
4000 requests per front-end web server in the first minute of each hour (roughly 1400 requests/second total)
a minimum of 11 seconds to hand out 1000 promotion codes.
a maximum of 40 minutes to hand out the same number of promotion codes, during the Tuesday incident described above.

What to do if your server stops responding?

Ryan — Wed, 13 Jan 2010 13:26:00 +0100

As you probably already know, our platform protects you from hardware failures that might occur on your server.

In the event of a problem on the machine, or if we suspect that a problem might occur (abnormal temperature, corrupted memory, etc.), your “server” will automatically be migrated to another machine. However, if you have an internal problem on your "server" that is not to due to the physical machine, and if it no longer responds, then you will need to take action.
The first action to take is to be sure that your server’s status is shown as “Running” on your Gandi interface. This is because the status may also be “Stopped” or “Paused” if, for example, it has not been renewed. So given that it is "Running" and non-responding, here's what to do.

There are three different cases:

1. You can still connect to your server via SSH. In this case, the following commands will help you analyze the situation:

"uptime" will give you the current load of the machine,
"free", will show you the amount of memory used by your applications in the “used” column,
"top" (we recommend that you install “htop") will show you the ranking of applications in realtime ordered by their use of resources (memory, CPU),
"dmesg" shows you messages from your linux server’s kernel,
Consulting logs such as /var/log/messages or /var/log/daemons with the command, “tail”, for example (tail /var/log/daemons) will also provide you with precious information.
"df –h" shows you the amount of disk space available on your disks.

The most frequent causes of error are:

No space left on your system disk: this situation is often caused by a lack of appropriate log management on the server, or by a database which fills up too quickly. The solution is to clean up wasted space, or enlarge the disk (see the guide on this).
Not enough RAM on the server, or too much memory used: the simple solution is to add more RAM by adding additional shares. If you have an expert server, you can also try to modify the behavior of the Linux’s available memory by using the command 'sysctl -w vm.overcommit_memory = 2'. Warning: so that the modification can be maintained following a reboot, you must also add "vm.overcommit_memory = 2" to the "/etc/sysctl.conf" and "/etc/gandi/sysctl.conf" files.
Too may processes are running simultaneously on your machine: you will need to lower the values in the configuration files of your applications (the number of simultaneous connections on Apache, for example) or increasing the power of your server by adding shares.

2. You can no longer connect to your server under SSH, and it does not respond to ping or it has a slow response time.

The virtual console which you may activate from your account (guide available), gives you direct access to your machine as if a monitor and a keyboard (still virtual) were directly attached to the server.

In this case, you can stop all the applications that are causing problems, and once again get access to the server.

The 'sysreq' shortcuts are available from the console of your server. The commands can be given by pressing Ctrl and “o” (as Oscar) in order to enter the sysreq mode, and then you can enter the command. By doing this, you can stop all the processes by: Ctrl+o +i (to kill). Ctrl+o +h gives you quick help to all of the available sysreq commands.

3. Your server may be “unreachable” but may nonetheless be working normally, and without any technical problem.

This may happen if your server is the victim of a DDoS attack for example. Your server will therefore be isolated from the rest of the network in order to protect our infrastructure and the quality of service for other customers.

You may verify whether or not your server is in this state by performing a “traceroute” command on the IP address of your server. If it stops at its arrival at Gandi, on one of our routers for example, it is likely that your server has been isolated from the network. You may then connect to your server via the console, though you will need to contact the support team to correct the problem (they will usually contact you first). This sort of isolation happens only very rarely.

After that, if you are still blocked, then this means that you are in the 4th case: your very own case ;-). Please send an email to our support team indicating that your server has been blocked, and you will get an answer as soon as possible.

How to turn your website into a "Web Infrastructure"

Ryan — Wed, 13 Jan 2010 13:25:00 +0100

Many websites start with a single server solution, a box acting as a web server and database server all in one. Simply, easy, cheap. The problem comes when traffic gets too high (a victim of their own success!). Many customers want a bigger box, but the answer is actually changing your architecture from "web server" to "web infrastructure". You can duplicate web servers, use the DNS to load balance them and ramp up your capacity very fast and very far.

For simplicity's sake, let's take a real example. As some of you already know, we support the Millenium association in their promotion of online video games.

The increasing success of the website, millenium.org, made us re-design the architecture of the website so that it could handle the numerous videos shown on the website to its 17,000 unique visitors per day far more easily and efficiently.

Following a major update of one of the games served by Millenium (WOW patch 3.1), we increased the server power to 16 shares in anticipation of an increase in load. Our expectations were quickly exceeded with over 50,000 unique visitors the first day, and just as many over the following days.

The website received between 500 and 1,000 simultaneous visitors and a large number of videos, which is not viable for a single LAMP server. We immediately changed the infrastructure, by moving from a unique-server model (which is often the starting choice) to an infrastructure-based model. We went from a vertical system (more power) to a horizontal system (more servers):

As you can see in the diagram, we moved the database to a separate server and duplicated the web server to two machines. The load was therefore split by the domain's DNS (using a simple DNS round robin technique). We could also move to dedicating a 1-share server to load distribution, though such a solution would take a bit longer to implement. In our case it took 2 minutes to add the shares to the account, 6 minutes to create the 2 servers, 10 minutes to transfer the data, and 2 hours to configure the services.

Today, the platform easily handles 1 million unique visitors per month, for over 3 million pages viewed - a good thing! Best of all the platform is now capable of evolving. If the database suffers, all we need to do is to add more shares to the database server. If the web front starts to become saturated, we just need to add another.

Because our cloud infrastructure allows you to create as many servers as you want from the hosting resources in your account, you can always add more, and always change your web architecture without replacing physical servers. The VPS hosting system is flexible and allows you to increase from a 1-share server (1/64 of a machine + 1/64 as reserve, 256MB of Ram) to a 16-share server (1/4 of a machine + 1/4 as a reserve, 4GB de Ram) at any time and as often as you want.

If you find yourself confronted by this type of problem, please feel free to contact us, as we would be more than happy to help you.

Gandi Hosting Team