Yesterday we were among hundreds of businesses impacted by the AWS S3 outage for more than three hours. A major outage to Amazon’s S3 service in the U.S resulted in numerous AWS services to fail or become degraded. This in turn caused a total outage of Xero, due to numerous impacts across our own platform.
Like all things in tech, we learn from it.
Xero has a strong internal focus on application reliability. We run reviews on incidents like this to learn where things went wrong. Our engineering team have already implemented a solution to minimise the impact on our customers if there was to be any further issues of this nature in the short term, and in the coming weeks will make it fully redundant. We know we can take from what we learned yesterday and build our applications so they are resilient to factors they are dependent on.
The ability to move this quickly is a feature of being hosted on AWS. Admittedly it was caused by an AWS outage in the first place, but yesterday’s incident showed us our ability to scale back up quickly once service was restored. We went from 0 to 18,000 users online in a matter of minutes without any problems on coping with the load — that’s an important feature of AWS’ scalability we didn’t previously have.
Since we completed our migration to AWS, we have quickly built up our uptime to more than 99.9%, which is what our customers have come to expect from us. We are excited about the capabilities AWS has given us to scale quickly and develop features using artificial intelligence and machine learning, which we will soon release at pace.
We know the frustration these types of incidents cause our customers and the impact it has on their own businesses particularly at month-end, and we’re working hard to build resilience into our own platform to reduce any risk of it having the same impact.
As we head into an era of technology development like we’ve never seen (or used) before, it is imperative that we are on a platform like AWS to offer the scalability and features that our customers have come to deserve and expect. I am truly excited about the features we’ll be able to unleash on small businesses and our accountants and bookkeeper advisors to help them to thrive.
The post Reviewing the Amazon outage – what we learned appeared first on Xero Blog.
Source: Xero
