Network issue with accessing databases
Incident Report for Neo4j Aura
Postmortem

What happened

During a routine automated deployment the load balancers that sit in front of Aura’s Neo4j databases were inadvertently replaced. The new load balancers were assigned different IP addresses to the old ones. The DNS entries for the databases were automatically updated to point to the new load balancers.

How the service was affected

All databases experienced an outage while the new IP addresses propagated through the DNS infrastructure to client applications. The length of the outage varied depending on the DNS caching characteristics of the infrastructure and applications. (The 50th percentile outage was approximately 5 minutes, the 95th percentile approximately 15 minutes.)

Some client applications that sit behind firewalls with explicit allowlisting of database IP addresses experienced a longer outage while those firewalls were reconfigured.

What we are doing now

We don’t believe that an outage of up to 15 minutes is acceptable for a DBaaS. We’ve carried out a thorough analysis of what went wrong in this situation. The actions that we’re carrying out to ensure that nothing like this can happen again fall into three areas.

  • Infrastructure management. We will explicitly reserve specific IP addresses for each of our load balancers so that changes to our load balancers, intentional or otherwise, don’t result in changes to database addresses.
  • Deployment. We will improve our deployment process so that changes are rolled out more gradually and with improved monitoring to ensure that any problem caused by the deployment is limited in scope.
  • Testing. We will improve our testing to ensure that database IP addresses remain unchanged after system updates.
Posted Feb 05, 2021 - 14:33 UTC

Resolved
During the recovery of this networking incident our public static IP addresses have been renewed.
We apologise for the disruption and for anyone needing to re-configure your firewall we have now updated the list of addresses for each regions.

Please see the full details here : https://aura.support.neo4j.com/hc/en-us/articles/360050504254-What-are-the-public-IP-addresses-to-provision-in-a-firewall-configuration-to-allow-Aura-use-
Posted Jan 21, 2021 - 16:53 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jan 21, 2021 - 15:43 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jan 21, 2021 - 15:42 UTC
Identified
We are currently investigating remedial steps to this issue.
Customer restricting access to specific public IP addresses may suffer delays in queries.
Posted Jan 21, 2021 - 15:41 UTC
This incident affected: Aura Console (console.neo4j.io), Aura Databases (*.databases.neo4j.io), and Aura Support Portal (aura.support.neo4j.com).