What happened

On 2024-04-29 at 08:57:35 UTC, Neo4j implemented a code update that had unintended consequences for our infrastructure management.

The aforementioned code introduced a bug in our cluster management tool, which became apparent when an update to our Kubernetes clusters resulted in the termination of VMs hosting certain databases. While the system self-healed, this led to instances becoming temporarily unavailable for some AWS customers.

Since our code rollout occurs progressively across environments, the impact was staggered across AWS service tiers.

How the service was affected

Some of our AWS customers (running instances up to 32GB of RAM) experienced a <10 minute service interruption between 2024-04-29 08:57:35 UTC and 2024-04-29 12:33:40 UTC when the last batch of impacted databases completed the self-healing process.

The impact occurred during multiple time intervals. Each interval affected a different group of instances, resulting in shorter recovery times for individual instances.

This occurred in the following time intervals as the roll out was staggered:

• 08:57:35 - 09:09:50 UTC
• 09:11:55 - 09:25:05 UTC
• 09:37:05 - 09:48:55 UTC
• 10:08:35 - 10:20:50 UTC
• 11:13:00 - 11:27:10 UTC
• 12:10:20 - 12:23:40 UTC
• 12:19:30 - 12:33:40 UTC

What we are doing now

After conducting a comprehensive analysis of the situation, we are taking decisive actions to prevent such incidents from occurring in the future. Our efforts fall into two key areas:

Immediate Actions:

• Enhancing monitoring and alert systems for the underlying Aura infrastructure, with a focus on detecting and responding to individual (or small groups of) database unavailability promptly.
• Strengthening testing protocols and code review processes to identify and address bugs in the components managing Cloud Infrastructure at an earlier stage.

These measures are aimed at ensuring the reliability and stability of our services moving forward.