AuraDB Pro, AuraDS Pro and Aura BC instances on GCP may experience creation delays

Incident Report for Neo4j Aura

Postmortem

What happened

On March 13, 2025, some of our customers experienced delays in the creation of AuraDB Pro, AuraDS Pro, and Aura BC databases on our platform. This was due to an unexpected issue with our database management system, which caused some databases to become stuck during the creation process. Our team quickly identified the problem and worked diligently to resolve it. By March 14, all affected databases were successfully created and operational. We are committed to ensuring the reliability of our services and have taken steps to prevent similar issues in the future. We apologize for any inconvenience this may have caused and appreciate your understanding and patience.

Root Cause

Recently, we implemented a new security feature to enhance the way we manage access to our database systems. This involved setting up additional security permissions in the IAM Policy for each of our paid database services. While we were careful to stay within known limits for these permissions, we encountered an unexpected issue due to an undocumented restriction on the size of these permissions. This caused a problem where our system couldn't create new permissions for databases in certain areas, leading to delays in processing and updates. 

Customer Impact

  • AuraDB Pro, AuraDS Pro, and Aura BC Databases experienced issues where databases would get stuck during creation.
  • A total of 8 AuraDB Pro databases were directly affected by this issue.

Resolution

We promptly addressed the issue by reversing recent IAM changes to ensure the stability of our services.

What we are doing now

Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future.

New mitigations being deployed:

  • Modifying the access management method to eliminate the possibility of hitting IAM limits
  • Implementing additional automated checks to prevent service disruptions
  • Improving our service resilience to handle similar scenarios
Posted Apr 14, 2025 - 15:49 UTC

Resolved

We have confirmed that the fix has addressed issues and all operations fully recovered.
Posted Mar 14, 2025 - 10:41 UTC

Monitoring

The fix for this issue has been successfully rolled out across all impacted tiers.
We are currently monitoring the situation to ensure stability.
Posted Mar 14, 2025 - 06:48 UTC

Update

We are in the process of releasing a fix for this issue and would update with a status when it has been rolled out across all impacted tiers
Posted Mar 14, 2025 - 00:59 UTC

Update

we are continuing work on a fix for this issue.
Posted Mar 14, 2025 - 00:59 UTC

Identified

We have identified an issue in GCP causing some new AuraDB Pro, AuraDS Pro and Aura BC instances to get stuck in creation for an extended period of time. Updating instances in this cohort may also experience delay. We are already preparing a fix.
Posted Mar 13, 2025 - 21:27 UTC
This incident affected: AuraDB Professional (*.databases.neo4j.io) (AuraDB Professional on GCP (*.databases.neo4j.io)), AuraDS (*.databases.neo4j.io) (AuraDS on GCP (*.databases.neo4j.io)), and AuraDB Business Critical (*.databases.neo4j.io) (AuraDB Business Critical (*.databases.neo4j.io) on GCP).