“ I want to learn about how it works under the hood”
Hybrid Web Cluster is a distributed web cluster which guarantees ease-of-management for those using it by providing automatic load-balancing, instant failure-recovery and automatic scaling. It runs every standard LAMP stack web application you can throw at it.
Hybrid Web Cluster is primarily built with the Python programming language, the Twisted networking framework and the excellent Spread Toolkit messaging service.
Hybrid Web Cluster is actually a number of separate pieces of software which work together to provide a web hosting platform
Under the hood, there are a number of distinct jobs that need to be done in a distributed web cluster environment in order to provide the unique load balancing, redundancy, scalability and system management capabilities that Hybrid Web Cluster offers. The entire Hybrid Web Cluster system is comprised of a number of individual software components created by us to solve these specific problems, each component designed to work together as a whole to provide you with rock-solid, easy-to-use cloud-based web hosting.
State Machine
HCFS
YakD
Apache
MySQL
AwesomeProxy
Control Panel
Developer UI
StateMachine
StateMachine is the algorithmic brain of the cluster – it runs on every node communicating with XML over Spread to all the other nodes in the group. Each machine is constantly updating the rest of the cluster about its state via Spread so that all machines in the cluster have a consistent view of the system.
StateMachine is responsible for making high-level load-balancing decisions using distributed algorithms, leadership election etc. StateMachine therefore has a pivotal role in providing the load-balancing and scalability features of the cluster. For example, StateMachine is responsible for deciding when a site should be moved from one node to another in order to more evenly balance load. Decisions are made locally on each node, based on global state information.
HCFS – Hybrid Cluster File System
HCFS is our name for our revolutionary ZFS-based cluster file system.
Each website and database is stored on its own ZFS filesystem, each filesystem is only ever mounted on one server at a time, so it is not a true distributed filesystem – but in order to meet our redundancy and scalability goals it does not need to be. HCFS ensures that near-live backups of each filesystem are stored on any number of servers you choose (from 2 to the total number of servers), this allows sites to be quickly transferred between servers for load balancing, and ensures that data is kept in multiple places for redundancy.
What it doesn’t do is allow you to scale beyond a setup where a single website is requiring more than two dedicated nodes all to itself (one for website and one for database), although with minor modifications to the web application code to ensure it stores all state information in the database, we are able to scale beyond this limitation via the use of GenieDB’s multi-master MySQL-compatible database server.
The HCFS filesystem daemon (FSDaemon) monitors filesystem change events on each node, and when a change occurs it creates a ZFS snapshot and sends an incremental binary diff of the filesystem to whichever nodes StateMachine has elected to hold redundant copies. FSDaemon handles “catching up” for nodes which need bringing up-to-date either due to having been offline, or due to network outage events.
Because HCFS works by creating incremental snapshots of each filesystem, this gives us the capability of allowing users to roll back time and restore their data to a previous point in history, recovering files or database entries that may have been completely deleted in the current version of their data. This means not only is the data protected from hardware failure by being kept on multiple servers, it’s also protected from the most common error – human error: if you accidentally delete something, you can get it back.
Network partitions (or netsplits) in which both sides of the split are live and serving during the split are handled by “most changed data wins” – meaning the filesystem with the most changed data will be chosen as the current copy when the network re-joins. However the system will automatically notify users when out-of-sync timestamps between different servers are detected and give them the option to recover data from a restored snapshot. However, while netsplits are common, they are most often caused by a complete network outage on one side of the split, in which case no recovery will be necessary.
YakD
YakD can be thought of as the automated-system administration daemon. YakD is responsible for performing functions like configuring the Apache web server, Bind DNS server, MySQL server, creating accounts on the system and rolling out system updates to all cluster nodes. Because all nodes in a Hybrid Web Cluster are homogenous, this means exactly the same software stack runs on every machine – every node is a web server, every node is a DNS server, every node is an e-mail server, every node is a database server etc. YakD therefore runs on every node and is in charge of keeping the system configuration in line with settings made in the web hosting control panel.
Apache
Apache on each node is dynamically reconfigured to act as a reverse proxy for any site which is not currently hosted on that node. This means you can direct any request for any site to any server and it will be transparently proxied to the appropriate current live server for that site. This makes DNS as simple as setting up multiple NS records across the cluster and multiple A records for each site (primary is returned first, with round-robin disabled in BIND) – of course all this is handled for you by the cluster, all you have to do is set the nameservers for your domains to point to cluster nodes (ideally in different data centres).
Despite being reconfigured on the fly as the cluster is load-balanced, the Apache installation provided is as standard as possible and provides all the expected modules and features. On top of Apache we run suPHP to allow PHP scripts to run as-the-user (which solves most permissions problems), and we chroot each user to their home directory so that there is no risk of users viewing or modifying each other’s files. From years of experience in the web hosting business, we find that this is the most secure, compatible and standard way of providing the LAMP stack to web hosting customers. This setup allows most standard LAMP applications to run unmodified.
Hybrid Web Cluster provides a standard LAMP stack, but does some clever things underneath it to provide scalability and redundancy
MySQL
Each database is stored on its own ZFS filesystem, which is symlinked from the MySQL data dir. As databases are transferred between servers they are swapped in and out by FSDaemon and MySQL is notified accordingly. FLUSH TABLES WITH READ LOCK allows us to make consistent snapshots of the databases and replicate them around the cluster just like websites.
AwesomeProxy
AwesomeProxy runs on every node and listens for MySQL connections to ‘localhost’, it then proxies the connection on to whichever node is the live database server for a particular database. AwesomeProxy is also responsible for pausing new database connections so that they don’t fail as sites move between servers – instead of a database error, you get a slightly slower page-load.
AwesomeProxy also sits on top of Apache providing the same pausing of connections as websites are moved around. This means that during normal operation the cluster can happily juggle sites between nodes, and each website or database move only results in a momentary pause in page loads for that particular site. This is crucial in allowing the cluster to effectively balance load and meet its scalability requirement, without ever having any downtime.
Control Panel
On top of the standard LAMP stack runs our brand-new custom web hosting control panel. The control panel runs just like any other load-balanced website on the cluster, it talks to StateMachine via a JSON API and triggers node reconfigurations based on settings made in the control panel. The control panel is used to add new accounts to the system, set up new domain names, configure e-mail addresses, create new databases etc.
The Hybrid Web Cluster control panel is a modern AJAX-based web application including full reseller support with whitelabelling, internationalisation and a high level of configurability. Cluster administrators and resellers have numerous options for managing and gathering business intelligence on their users or customers. Graphical displays that run in your web browser will give cluster administrators a view of the status and health of their cluster. All web cluster users get access to a number of simple one-click installs for common applications such as WordPress and Drupal, and an easy-to-use configuration panel for managing their domains and e-mail accounts.
The Control Panel uses the same JSON API (connecting to the StateMachine via HTTP) that we will eventually expose for developers wanting to integrate custom applications with our platform.
Developer UI
A distributed system like a web cluster is a complex beast to develop and debug for, our developers quickly found that staring at pages of debugging output from each cluster node was not a viable way to resolve issues with the cluster. We therefore built a graphical development user interface which allows a visual representation of the state of the cluster (as seen by a particular cluster node). This allows an in-depth view into state of the cluster without staring at fast-scrolling white-on-black text.
The development UI includes a number of features to assist development and administration across a number of different machines, including a multiple console client allowing the same command to be sent to the console of many nodes at once. The UI also supports snapshotting the entire state of the cluster to replay it later, very useful for rerunning the same conditions over and over again for debugging.
Many of the graphics from the developer UI will be ported to run in a web browser and available to cluster administrators within the cluster’s web control panel, giving a fascinating live visual representation of the state of the cluster.
