Hybrid Cluster — self-healing, auto-scaling & very forgiving

December 28th, 2011

You won’t have heard much from Hybrid Logic recently — now with an early stage tech company this can mean one of two things: either they’ve given up and gone home, or they’re mad busy innovating, building and shipping their product.

I’m pleased to report that in our case it’s the latter ;-)

Hybrid Cluster has had an extraordinary year of development and we’re on the cusp of releasing some very exciting new features for the world to get to grips with. What we’ve done is nothing short of revolutionary — we’re changing the fundamental assumptions about how your servers can co-operate together, how applications and databases can scale, and how companies do business continuity planning across data centres.

In the “old world”, a server is seen as a single entity; one which has its own specific configuration, and which hosts a set of applications and databases. If you’re staying up-to-date with the industry, you’ll have virtualized that server and put its storage in a centralized storage system (a SAN, for example) — now that’s all very well, but the virtual server is still conceptually a single server and can still suffer from these three problems:

  1. Hardware and networks fail
  2. Servers get over-loaded when there are spikes in demand
  3. Users make mistakes

At Hybrid Logic it’s our mission to solve all three of these problems for your existing LAMP applications, and our software — available for license today — solves them by employing a fundamental paradigm shift in industry thinking.

Individual servers and storage systems should not be the unit of concern for you, the developer or administrator. Applications, databases and mailboxes should be — the servers should look after themselves.

Now, if you look a little further down the road, this is the way the industry’s moving — in cloud, the move from IaaS to PaaS is exactly this — developers and sysadmins should not have to think about individual server instances ever again. Their servers should form a cognizant co-operative group on their own. This is exactly what our software does — it transforms a bunch of dumb, commodity machines, connected by slow and unreliable network connections, into a loosely-coupled distributed cluster where the failure of an individual server or even an entire data centre is automatically healed so that the cluster carries on working — keeping your applications, databases and mailboxes online even in the face of catastrophic failure of an entire region.

I’m Luke, the CTO here at Hybrid Logic, and in the next few blog posts I’m going to give you a bit of insight into how we do it ;-)

Happy New Year!

Cheers,
Luke

Announcement: txHybridCluster client released

September 12th, 2011

From a blustery grey Monday morning in Bristol, England, comes the announcement of the immediate availability of a Twisted client for interacting with the Hybrid Cluster API.

https://github.com/hybridlogic/txHybridCluster

http://www.hybrid-cluster.com/
http://www.hybrid-cluster.com/api/

The Hybrid Cluster API allows you to set up websites, databases, mailboxes, add SSL certificates, configure custom DNS, and more (over 100 API functions are documented).

Hybrid Cluster multi-region branch

August 18th, 2011

We’re very excited to announce the imminent release of our multi-region branch. In our labs we now have one cluster spanning East & West coast USA on cloud infrastructure and dedicated hardware in Europe, with densities of over 1,000 websites per node.

More to come ;-)

Announcing txMySQL – native async Twisted MySQL protocol

February 8th, 2011

Hybrid Logic is pleased to announce the release of txMySQL, a native Twisted MySQL protocol implementation at https://github.com/hybridlogic/txMySQL

The bulk of this code is courtesy of _habnabit (thank you!), we just added authentication support and fixed a couple of bugs which were stopping the MySQL protocol parser working.

This works well enough to .fetchall() basic results sets and .query() any other MySQL statements you care to run. See example.py.

Feel free to fork, tweak, fix, use, report issues, etc.

Hybrid HTTP API Tester

January 30th, 2011

During the development of our Hybrid Web Cluster API it became necessary to test a large number of our own API commands in a rigorous way. This testing regime includes both fully-automatic and manual testing. The manual testing of each API command requires sending HTTP requests to the API server with a specific set of parameters for each API command, trying various valid and invalid data, inspecting the responses to ensure our API responds in the way it’s supposed to. As the number of commands in the API is nearing 100, it made sense to build some kind of test harness to make the process of generating these HTTP requests easier. Hybrid HTTP API Tester was born – a simple Windows GUI program designed to make the process of manually testing an API much easier.

Since we thought this tool might be useful to other developers, both in developing their own APIs and to aid in performing integrations with 3rd party APIs – we decided to make the software and its source code available for free to anyone who wants to use it. The software is now available for download under the new BSD license, check the project website or its SourceForge page for more information.

Here’s a couple of screenshots so you can get an idea of how it works:

Main Form Screenshot

API Setup Screen

More performance and stability improvements

December 5th, 2010

Hello everyone,

We’ve been working hard over the weekend and have some good core cluster stability and performance improvements to show for it, new internal performance testing tools and a sneak peek of our Hybrid Sites project:

First, a bug in Twisted which was causing the distributed proxying layer to sometimes stop accepting new requests has been worked around. This means you shouldn’t see database connection errors any more. If you do, please report them by posting on the forums!

Second, we’ve now got a new internal performance testing tool which shows us a scatter graph of a cluster’s response time and stability:

This plot shows, for example, an average response time of around 400ms for a WordPress blog and our Control Panel (the CP in red, WordPress in blue). The few outliers show latencies of up to 10 seconds when a server fails! Much better than the usual hours or days of downtime!!

Next, I’ve got a couple of sneak peaks of our Hybrid Sites website and Control Panel. This will be our flagship cloud web hosting platform, perfect for developers, designers and publishers alike.

And here’s the control panel, showing off our whitelabel features:

A massive amount of under-the-hood work has gone on with the Control Panel in readying our incredibly powerful reseller system for Hybrid Site’s go-live in a couple of weeks.

We are also happy to announce that Hybrid Sites will be launched in association with ElasticHosts, giving their customers access to powerful, simple cloud web hosting — a much easier option than setting up their own Linux box over SSH. Hybrid Sites will in fact be launched across multiple cloud providers, including CloudSigma, to provide impressive cross-cloud redundancy.

We also have a reseller API which presently has 35 commands and growing. This will allow you to set up reseller accounts, take payments, set up websites, databases and purchase domains all through a powerful REST API. This will come hand-in-hand with the WordPress plugin which runs our — or your — frontend web hosting company page. Hybrid Sites will be the first to prove this technology tool-chain :-)

Lots more to come this week, but for now, please try to thrash the knackers off your beta clusters, and get in touch if you want to test drive it and you’re not on the beta yet!

Cheers!

Luke Marsden, CTO

Running FreeBSD 8.1 as a Xen HVM DomU on Flexiant

November 26th, 2010

Just thought I’d share the incantations which were necessary to get FreeBSD 8.1 XENHVM kernel to work well on Flexiant, with paravirtualised network and disk:

Just before the kernel boots (which you have to be quick to catch with Flexiant’s VNC client) hit F6 on the bootloader and type:

set hw.clflush_disable=1
boot

This will allow you to boot even a GENERIC kernel. Once you’re booted, chuck hw.clflush_disable="1" into /boot/loader.conf to make this permanent.

You’ll then want to build your own kernel for paravirtualised network and disk drivers. Edit the XENHVM kernel config (see the FreeBSD handbook on compiling your own kernel) – comment out the MODULES_OVERRIDE line which disables building all the modules (assuming you want ZFS support) and also comment out the whole section about WITNESS and INVARIANT, as having this enabled will slow down your kernel quite significantly.

Then you’ll need to patch the network driver as per this post (manually, since the code has changed a bit), else you get a lot of dropped packets:

http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00598.html

Then just (as root):

cd /usr/src
make buildkernel KERNCONF=XENHVM
make installkernel KERNCONF=XENHVM
shutdown -r now

And enjoy your speedy FreeBSD 8.1 VM in the cloud!

Hybrid Web Cluster now runs on Xen HVM IaaS providers

November 26th, 2010

The proof:

Lots more IaaS providers, here we come!

FTP support now in Hybrid Web Cluster

November 23rd, 2010

Update: This has now been deployed to your beta web clusters! And check out the new video below…

We now have clustered FTP support! This means you can open an FTP connection to any node on the cluster at any time, and you’ll get internally redirected to the correct backend server for that site, and authenticated against the password you’ve set up in your Control Panel.

FTP is one of the most annoying and broken protocols on the planet (a separate data connection, whyyy?), but it’s also crucial for uploading your website — so it’s rather good that we support it now :-)

This inherits the same nice properties which HTTP and MySQL requests enjoy while traveling through our distributed proxying layer, such as stopping site-juggling from occurring while an FTP connection is happening (so your site won’t get load-balanced if you’re uploading to it). It also means that if your site is being juggled at the same moment you connect to it, you’ll get a slight delay before your FTP connection is established, rather than an error message.

We are nearly feature complete!

Stability, performance and rendering improvements

November 19th, 2010

It’s been a busy first week of the beta here at Hybrid Logic HQ, and we’re very pleased by the response we’ve had to the start of the beta — thank you! There’s been a buzz of activity on the forums and we love it when you give us feedback, so please carry on experimenting with our software and tell us what you think.

Along with the awesome feedback from yourselves, which we taking careful note of, we’ve also been doing some improvements of our own. Here’s a quick breakdown of the fixes and improvements which have now been deployed across all your clusters:

  • At the deployment level, we can now add new instances to an existing cluster. This means we can unintrusively upgrade a cluster to include new or better spec machines (irrespective of physical location) so that you can scale your hosting operation seamlessly.
  • The “God Pod” has had some significant responsiveness improvements. When we first launched, it wasn’t the most responsive user experience in the world. It’s much quicker and more accurate now, so give it a go!
  • We’ve made significant improvements to the stability of the core web hosting platform. We’ve solved several problems which were causing “Default site on X” error messages where your websites should have been. Another bug was causing databases to sometimes become inaccessible, and we’ve solved that too. Stability is looking a lot better.
  • We’ve improved the intelligence of the core load balancing algorithms, meaning that the decisions to move a site from one server to another (due to load) is now a fair bit smarter, and you should see fewer unnecessary load balancing events. As ever, there’s still room for improvement.
  • We’ve enabled swap on all your machines, so that if your 1.4GB memory does ever get fully used up, your instances will just become slow for a few minutes as they recover, rather than falling over or crashing completely.
  • When a site is about to be moved from one server to another, what happens internally is that requests for that site get “paused” by the distributed proxying layer which runs on top of the web and database servers. This pausing happens so that during the transfer of the site or database from one server to another, none of the requests return error messages — rather, the user just experiences a slow page load. The Load Balancing Diagram in the God Pod now shows a dotted line around a site when it is paused. This gives you a better insight into what’s happening within the cluster during the process of moving sites from one server to another to keep your servers healthy and balanced.
  • Performance has been improved massively. Previously, load balancing events caused sites to be blocked for up to 20 seconds. We’ve managed to get this down to 3-6 seconds in most cases, resulting in fewer requests building up. We’ve also made some code changes which have made everything feel a lot snappier. We will be continuing to optimise for performance over the coming weeks and months — this is only the start!
  • Numerous tweaks and improvements to functionality in the Control Panel have also been deployed (more details on this will be posted to our forum in due course).

We can’t wait to see how much better we can make it next week!