Gandi Kitchen - Hosting

Kernel and cmdline

Ryan — Thu, 03 Jun 2010 11:00:00 +0200

We are pleased to announce that you can (finally!) choose the version of your kernel that you want (from a list which will be continually expanded), and associated boot options (cmdline). 2.6.18 and 2.6.27 are "base" versions supplied by Xen (backport of Xen patches for 2.6.27). Version 2.6.32, which is is currently available, uses paravirt_ops and the "Linux" implementation of Xen patches.

We will show you the new kernels here. They can be found in the "advanced mode" within the server's management page:

And you will then be able to access:

Concerning cmdline, you may now deactivate selinux at boot, boot as a single user, change the disk and the boot partition (which is practical for working with "images"), or choose the most appropriate console for your needs. In short, everything that you need to manage your updates in a more friendly environment, or to repair your server in the most autonomous manner.

If you feel that an option is missing, please let us know.

Mandriva 2010 image in alpha (updated)

aegiap — Fri, 21 May 2010 17:35:00 +0200

Server hosting by Gandi allow customers to choose from a selection of OS images available during the creation process of the virtual server. After the creation of the image by Gandi and internal testing, a new distribution is released to a specific group of hosting customers called 'alpha'. These clients can create server using these release candidate images. This allows Gandi to increase the types of testing and usage, and to find more bugs and problems by working with a small group of its customers.

Today - May the 21th - the Mandriva 2010.0 image has been released . This new version of the Mandriva distribution boots with a 2.6.27 kernel by default. It is currently only available for the 'alpha' customer group but will shortly be available for all customers. Please contact us if you wish to participate in our alpha testing phase.

16th August 2010 : Image is now available for everybody

Cherokee Arrives at Gandi

Leland Vandervort — Thu, 29 Apr 2010 13:48:00 +0200

(Translator's note: Taskigi Sequoyah would be proud! ;) )

A friend had explained to me that he had just finished installing a web server. Having decided on a change and not to use the mammoth Apache that everyone knows and loves, my friend seemed to have found a decent alternative. Bearing the name Cherokee, it appeared much leaner and with greater performance than Apache. See the benchmarks from the project website at the end of this article.

Personally, I had no knowledge of Cherokee, though I had certainly heard the name pandered around once or twice, but I never really paid it any attention at the time. So to give it a whirl, I decided to install it locally, which to my surprise exceeded my expectations. On a debian (or derived) distribution, installation is achieved by a simple apt-get install cherokee (or 'aptitude'...)

The web management interface provided is clean and intuitive. The basic idea is quite interesting; It is possible to activate the PHP interpreter with a simple mouse click; same for activating different virtual hosts as well as other options. There are also wizards to assist with the installation of such applications and frameworks as Django, Rails or Wordpress, etc.

I thought that it would be interesting to present this project during a developers meeting at Gandi, especially for those who use GandiAI. The presentation was well received. After a few pow-wow, and a glance through the source code, there was nothing really unusual or risky. This doesn't mean, of course, that there aren't any bugs in it, but at any rate clean code is a good sign! The core is written in C with the administrator interface written in Python.

Thus is the way in which Cherokee made its appearance at Gandi, and a server image was produced. In this way you can rapidly test a server with Cherokee pre-installed.

We have also added an "expert" mode distribution with Cherokee pre-installed... You need only to select this distribution during the steps to create your server. And, just as a reminder, in case you haven't yet tried our service, you can always request a free trial using our online form!

For further information about Cherokee, you can visit the project website. Additionally, you can find a comprehensive article about Cherokee in the Gnu/Linux magazine France, written [in French] by Carl Chenet in issue number 125.

Gandi 10th Anniversary - The Experience

Leland Vandervort — Wed, 17 Mar 2010 19:33:00 +0100

To celebrate Gandi's 10th anniversary, this hair-brained idea to give away, in ten days, 55000 domains, raise a very practical question. How, once we open the floodgates on such an operation, to maintain the highest quality of service on the site? The festive spirit could well have transformed into a nightmare for our customers if they were suddenly unable to access their management interface.

So we took the decision to host the event on a dedicated site. This was a hitherto dreamed of occasion to put ourselves into our customers' shoes, and use our hosting infrastructure for this event. We defined the rules of play: Using only the tools provided to our customers, build an architecture which was easily scalable and didn't break the bank, and to demonstrate our renowned flexibility.

Keep it Simple, Stupid.

We had the "luxury" of one week from design to implementation. As a result, the charming idea to demonstrate our "cloudlike" site based on modern technology was summarily put out of our minds. To be perfectly honest, given the time scale, the lucky chosen developer had a nifty precept to select the technology: "You have full choice of the technology, but you have one week." It would be PHP/MySQL, which isn't exactly everybody's favourite! It would, nevertheless, allow us to release a tested site within the tight time frame.

To adequately sustain the load generated by such an event, several servers would be needed. We then hit our first stumbling block: Gandi does not [yet] have a load balancing solution for the hosting solution! Nevermind, we'll use the old yet faithful round-robin DNS method, with a low TTL to be able to quickly remove a front-end server from production in case of an incident.

Our Linux distribution for this occasion would be Ubuntu 9.10 - because it is reasonably up-to-date, with the 2.6.27 kernel.

Small Servers - Go Forth and Multiply!

The best way to sustain the high loads for our solution is to split the functionality among several small servers. This way we would maintain a minimum level of "vertical" scalability (you can dynamically increase the memory and CPU allocated to a server), and the architecture provides "horizontal" scalability.

In this way we could easily add resources if we started to feel the pinch, and add or migrate shares from one server to another as load requirements necessitate. There are numerous advantages of using multiple small servers:

each server gets a minimum of one core, burstable, of the CPU (yes, one whole core, even with one share -- that's new, by the way!)
assured resilience, with shares spread somewhat randomly across a few hundred different physical servers.
specifically in a virtualised environment, the memory performance is best with less than 1GB of memory.
if you have 4 servers of one share each, rather than one big server of 4 shares, they can dynamically increase to 8x4 shares, or 24x4 shares with a reboot, and all of this without modification to the architecture. A big server, however, would only scale to 8 shares without a reboot, or 24 after reboot.
resources may be easily moved towards servers that need it most

We commence with a simple architecture:

24 (!) servers of one share each, to manage the PHP website: 10 for the English site, 10 for the French site, and an additional 2 of each for IPv6.
2 servers of 4 shares each for replicated memcached to reduce database load and manage sessions.
1 MySQL server of 4 shares, which contains the pre-generated promotion codes (they actually all fit within memory, so the database itself should really be pretty bored doing nothing...)

After a couple of lovely overloads and a bit of code review, the database would finally be greatly spared by memcached (see the section "lightweight coding...").

One of our administrators would put his fingers to work on the site to create and configure 24 servers -- at the same time! Obviously the release of the hosting API or an admin interface function would have been welcome. (Thanks to cssh in this case!)

Lock Down (somewhat) the Machines

A default installation always needs a few finishing touches. The very fact of opening a MySQL database on the "public" network made us a little edgy. So, swooping 'netstat' and shutting down non-critical services listening on public ports. With the help of tcp wrappers (hosts.allow, hosts.deny), all of the "private" interfaces are also locked down (sshd, mysql accessibly only from the web farm).

Finally it behooved us to pay close attention to the PHP code and MySQL queries; The safest way to avoid php code injections is to bind all the parameters after a prepare(). This also helps reduce load on the database when several execute() are called.

One important detail: since the site should allow a user to send an email to any "arbitrary" address, it was absolutely critical to limit its potential for abuse by some clever black-hat as much as possible. At the very minimum, the number of sent emails per promotion code was limited, in addition to very close monitoring.

Setup the Development and Deployment Environment

The sharing of data between the sites in effect adds a single point of failure, as well as a potential architectural bottleneck. As such, we decided to deploy the content of the site locally on each of the servers. We would use one server for developing and staging, and ultimately for the development and testing of updates. A quick script and some 'rsync' would allow rapid deployment across the entire front-end architecture. Simple! (some would say ;) )

Resource Monitoring

A few moments before the operation, more as a precaution rather than a cure, all of the virtual machines from one to two shares. Using the statistics interface, from day one, one can see that the the virtual machines were essentially sitting "twiddling their thumbs" from boredom ;) :

It would have been cool, at this very moment, to reduce back to a single share per server, or make use of Gandi "Autoflex", or even given the actual load observed, set up scheduled flex for each hour to hand out the promotion codes! Unfortunately, with all hands on deck, we missed this opportunity to demonstrate this [econono-techno-ecological ;)] feature.

Lightweight Code is Worth More than a Thousand Beefy CPUs

Even though we physically had several thousand CPUs and a few Terabytes of RAM at our fingertips, Tuesday turned out to be somewhat chaotic and worthy of note here. After Monday, which managed the load very well, the "smooth" execution of our one and only SELECT COUNT brutally altered and became excruciatingly slow (300ms). We had naively thought that this "only" query, on a table held exclusively in memory, wouldn't be an issue. As such, it was executed on every page of the site. The multiple simultaneous accesses to the database, coupled with the UPDATE operations for the promotion codes, resulted in the database, despite the near-idle system performance, started causing database lock contention.

The usual knee-jerk reaction to such a situation is to increase the number of shares to support the load. It's great for a quick-fix temporary solution, but it's not enough!

A new analysis of the system, questions about the code, and the use (or salvation) of memcached resulted in recovering the optimal performance. Equally, a modification of the database queries used probably would have been prudent.

The moral of the story: the code, indexes, architecture (etc.) are the cornerstones of your ability to support usage load, and if they are "CPU friendly", they will save the day. Otherwise a catastrophe could be lurking, or at the very least, the unnecessary purchase of additional shares.

Also, as we said earlier somewhat tongue-in-cheek -- it's eco-friendly!

Some Numbers

36 shares total, but we could have done it with less (*sniff*)
5% CPU usage at peak
4000 requests per front-end web server in the first minute of each hour (roughly 1400 requests/second total)
a minimum of 11 seconds to hand out 1000 promotion codes.
a maximum of 40 minutes to hand out the same number of promotion codes, during the Tuesday incident described above.

What to do if your server stops responding?

Ryan — Wed, 13 Jan 2010 13:26:00 +0100

As you probably already know, our platform protects you from hardware failures that might occur on your server.

In the event of a problem on the machine, or if we suspect that a problem might occur (abnormal temperature, corrupted memory, etc.), your “server” will automatically be migrated to another machine. However, if you have an internal problem on your "server" that is not to due to the physical machine, and if it no longer responds, then you will need to take action.
The first action to take is to be sure that your server’s status is shown as “Running” on your Gandi interface. This is because the status may also be “Stopped” or “Paused” if, for example, it has not been renewed. So given that it is "Running" and non-responding, here's what to do.

There are three different cases:

1. You can still connect to your server via SSH. In this case, the following commands will help you analyze the situation:

"uptime" will give you the current load of the machine,
"free", will show you the amount of memory used by your applications in the “used” column,
"top" (we recommend that you install “htop") will show you the ranking of applications in realtime ordered by their use of resources (memory, CPU),
"dmesg" shows you messages from your linux server’s kernel,
Consulting logs such as /var/log/messages or /var/log/daemons with the command, “tail”, for example (tail /var/log/daemons) will also provide you with precious information.
"df –h" shows you the amount of disk space available on your disks.

The most frequent causes of error are:

No space left on your system disk: this situation is often caused by a lack of appropriate log management on the server, or by a database which fills up too quickly. The solution is to clean up wasted space, or enlarge the disk (see the guide on this).
Not enough RAM on the server, or too much memory used: the simple solution is to add more RAM by adding additional shares. If you have an expert server, you can also try to modify the behavior of the Linux’s available memory by using the command 'sysctl -w vm.overcommit_memory = 2'. Warning: so that the modification can be maintained following a reboot, you must also add "vm.overcommit_memory = 2" to the "/etc/sysctl.conf" and "/etc/gandi/sysctl.conf" files.
Too may processes are running simultaneously on your machine: you will need to lower the values in the configuration files of your applications (the number of simultaneous connections on Apache, for example) or increasing the power of your server by adding shares.

2. You can no longer connect to your server under SSH, and it does not respond to ping or it has a slow response time.

The virtual console which you may activate from your account (guide available), gives you direct access to your machine as if a monitor and a keyboard (still virtual) were directly attached to the server.

In this case, you can stop all the applications that are causing problems, and once again get access to the server.

The 'sysreq' shortcuts are available from the console of your server. The commands can be given by pressing Ctrl and “o” (as Oscar) in order to enter the sysreq mode, and then you can enter the command. By doing this, you can stop all the processes by: Ctrl+o +i (to kill). Ctrl+o +h gives you quick help to all of the available sysreq commands.

3. Your server may be “unreachable” but may nonetheless be working normally, and without any technical problem.

This may happen if your server is the victim of a DDoS attack for example. Your server will therefore be isolated from the rest of the network in order to protect our infrastructure and the quality of service for other customers.

You may verify whether or not your server is in this state by performing a “traceroute” command on the IP address of your server. If it stops at its arrival at Gandi, on one of our routers for example, it is likely that your server has been isolated from the network. You may then connect to your server via the console, though you will need to contact the support team to correct the problem (they will usually contact you first). This sort of isolation happens only very rarely.

After that, if you are still blocked, then this means that you are in the 4th case: your very own case ;-). Please send an email to our support team indicating that your server has been blocked, and you will get an answer as soon as possible.

How to turn your website into a "Web Infrastructure"

Ryan — Wed, 13 Jan 2010 13:25:00 +0100

Many websites start with a single server solution, a box acting as a web server and database server all in one. Simply, easy, cheap. The problem comes when traffic gets too high (a victim of their own success!). Many customers want a bigger box, but the answer is actually changing your architecture from "web server" to "web infrastructure". You can duplicate web servers, use the DNS to load balance them and ramp up your capacity very fast and very far.

For simplicity's sake, let's take a real example. As some of you already know, we support the Millenium association in their promotion of online video games.

The increasing success of the website, millenium.org, made us re-design the architecture of the website so that it could handle the numerous videos shown on the website to its 17,000 unique visitors per day far more easily and efficiently.

Following a major update of one of the games served by Millenium (WOW patch 3.1), we increased the server power to 16 shares in anticipation of an increase in load. Our expectations were quickly exceeded with over 50,000 unique visitors the first day, and just as many over the following days.

The website received between 500 and 1,000 simultaneous visitors and a large number of videos, which is not viable for a single LAMP server. We immediately changed the infrastructure, by moving from a unique-server model (which is often the starting choice) to an infrastructure-based model. We went from a vertical system (more power) to a horizontal system (more servers):

As you can see in the diagram, we moved the database to a separate server and duplicated the web server to two machines. The load was therefore split by the domain's DNS (using a simple DNS round robin technique). We could also move to dedicating a 1-share server to load distribution, though such a solution would take a bit longer to implement. In our case it took 2 minutes to add the shares to the account, 6 minutes to create the 2 servers, 10 minutes to transfer the data, and 2 hours to configure the services.

Today, the platform easily handles 1 million unique visitors per month, for over 3 million pages viewed - a good thing! Best of all the platform is now capable of evolving. If the database suffers, all we need to do is to add more shares to the database server. If the web front starts to become saturated, we just need to add another.

Because our cloud infrastructure allows you to create as many servers as you want from the hosting resources in your account, you can always add more, and always change your web architecture without replacing physical servers. The VPS hosting system is flexible and allows you to increase from a 1-share server (1/64 of a machine + 1/64 as reserve, 256MB of Ram) to a 16-share server (1/4 of a machine + 1/4 as a reserve, 4GB de Ram) at any time and as often as you want.

If you find yourself confronted by this type of problem, please feel free to contact us, as we would be more than happy to help you.

Gandi Hosting Team