What happened

During a routine automated deployment the load balancers that sit in front of Aura’s Neo4j databases were inadvertently replaced. The new load balancers were assigned different IP addresses to the old ones. The DNS entries for the databases were automatically updated to point to the new load balancers.

How the service was affected

All databases experienced an outage while the new IP addresses propagated through the DNS infrastructure to client applications. The length of the outage varied depending on the DNS caching characteristics of the infrastructure and applications. (The 50th percentile outage was approximately 5 minutes, the 95th percentile approximately 15 minutes.)

Some client applications that sit behind firewalls with explicit allowlisting of database IP addresses experienced a longer outage while those firewalls were reconfigured.

What we are doing now

We don’t believe that an outage of up to 15 minutes is acceptable for a DBaaS. We’ve carried out a thorough analysis of what went wrong in this situation. The actions that we’re carrying out to ensure that nothing like this can happen again fall into three areas.

• Infrastructure management. We will explicitly reserve specific IP addresses for each of our load balancers so that changes to our load balancers, intentional or otherwise, don’t result in changes to database addresses.
• Deployment. We will improve our deployment process so that changes are rolled out more gradually and with improved monitoring to ensure that any problem caused by the deployment is limited in scope.
• Testing. We will improve our testing to ensure that database IP addresses remain unchanged after system updates.