Hosting and support is what we’re proudest of at Fat Beehive. Most of our clients are non-profits whose users rely on their websites every day for help and advice, so we take security and availability seriously. Some problems start outside our control, though. When they do, clear communication with clients and close work with our technical partners is what matters.
On Tuesday morning, a power interruption at Google’s London data centre (europe-west2) hit UK banks and financial services, and three of our server shards at WP Engine. It made the BBC and Sky News and disrupted a lot of people’s days, ours included.
The problems started at 7am UK time. We got straight on to WP Engine to find the cause and work out our options. Moving clients to new infrastructure is slow and needs care. Backups have to be collated and decrypted (every backup we take overnight is encrypted), new infrastructure has to be prepared, and each client needs DNS validation on the new setup. So we treated it as a last resort, to be used only if the outage ran into late afternoon.
It didn’t come to that. Google had their systems back by midday, and services returned soon after. The downtime was unwelcome and disruptive, but there was no data loss, and clients weren’t put through a move and DNS changes on top of it.
Then, late on Tuesday night, PHP announced a critical vulnerability in the language that needed immediate patching. We worked with WP Engine and Positive Internet, our Cambridge-based partners who host our Drupal clients among others, to put mitigations in place straight away through Cloudflare and upload restrictions. The patch was released at 2.15pm UK time, and we rolled it out across the whole estate over the next few hours, keeping client sites protected throughout.
Direct access to our partners’ technical teams, through several channels and not just support tickets, is what makes the difference. It means we get real answers and working fixes in place quickly. Throughout, our developers and account managers were on hand to give clients the information and support they needed.
We have key relationships with WP Engine and Positive Internet, and every one of our clients benefits from them. They also get the full weight of our own team, who know their sites and respond when it counts. Put those together and a client has two layers of expertise on their side, not one.
Find out more about our charity and non-profit website hosting
