Some issues with backups

Incident Report for Neo4j Aura

Postmortem

What happened

On August 18, 2025, at 19:18 UTC, the release of a faulty component in Neo4j Aura impacted the ability to perform backups of customer instances. The problem was identified when a critical component was updated without aligning with another dependent component, leading to failed backup attempts. This resulted in Neo4j not being able to take backups of customer instances for a period of approximately 17 hours. Our team quickly identified the root cause and began mitigation efforts early on August 19, 2025. By 11:54 UTC, the issue was fully resolved, and normal backup operations resumed. We apologize for any inconvenience this may have caused and are taking steps to prevent similar issues in the future.

The issue arose because the deployment of the neo4j-backup system was not properly synchronized with the deployment of the neo4j-operator. This misalignment led to critical changes being introduced that impacted the ability to create new backups effectively. Essentially, the two components were not updated in harmony, which caused disruptions in the backup process.

How customers were affected

Customers were unable to  get successful backups of their instances for 17 hours. We promptly addressed the issue by reverting the recent backup changes that caused disruptions and ensured the stability of our service by cleaning up old, failing backups.

What we are doing now

Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future.

New mitigations being deployed:

  • Enhancing our monitoring systems to detect similar issues faster
  • Implementing additional automated checks to prevent service disruptions
  • Improving our service resilience to handle similar scenarios
Posted Sep 12, 2025 - 19:42 UTC

Resolved

This incident has been resolved.
Posted Aug 19, 2025 - 12:04 UTC

Monitoring

The fix has been deployed and we will continue to monitor this incident.
Posted Aug 19, 2025 - 11:30 UTC

Update

We continue to investigate and have identified the issue more clearly.

Beyond backups, other operations are also affected: pause and clone.

Operations may fail or take a long time to complete.
Posted Aug 19, 2025 - 10:51 UTC

Update

We are continuing to investigate this issue.
Posted Aug 19, 2025 - 10:34 UTC

Update

We are continuing to investigate the issue with backups. New backups after 0:00 last night are failing to complete.
Existing backups are unaffected and remain available.
Posted Aug 19, 2025 - 10:05 UTC

Investigating

We are currently investigating an issue with backups. We will update shortly as we get more details
Posted Aug 19, 2025 - 09:19 UTC
This incident affected: AuraDB Virtual Dedicated Cloud (*.databases.neo4j.io) (AuraDB Virtual Dedicated Cloud on AWS (*.databases.neo4j.io), AuraDB Virtual Dedicated Cloud on Azure (*.databases.neo4j.io), AuraDB Virtual Dedicated Cloud on GCP (*.databases.neo4j.io)), AuraDB Professional (*.databases.neo4j.io) (AuraDB Professional on AWS (*.databases.neo4j.io), AuraDB Professional on Azure (*.databases.neo4j.io), AuraDB Professional on GCP (*.databases.neo4j.io)), AuraDS (*.databases.neo4j.io) (AuraDS on AWS (*.databases.neo4j.io), AuraDS on Azure (*.databases.neo4j.io), AuraDS on GCP (*.databases.neo4j.io)), AuraDS Enterprise (*.databases.neo4j.io) (AuraDS Enterprise on AWS (*.databases.neo4j.io), AuraDS Enterprise on Azure (*.databases.neo4j.io), AuraDS Enterprise on GCP (*.databases.neo4j.io)), AuraDB Business Critical (*.databases.neo4j.io) (AuraDB Business Critical (*.databases.neo4j.io) on AWS, AuraDB Business Critical (*.databases.neo4j.io) on Azure, AuraDB Business Critical (*.databases.neo4j.io) on GCP), and Aura Graph Analytics (Aura Graph Analytics on AWS, Aura Graph Analytics on Azure, Aura Graph Analytics on GCP).