GDS functions missing from AuraDS Enterprise and Professional instances

Incident Report for Neo4j Aura

Postmortem

What happened

On January 10, 2025, following a system update, some AuraDS instances experienced an issue where Graph Data Science (GDS) functionality became temporarily unavailable resulting in users receiving “unknown function” error messages while attempting to use GDS functions. Users with AuraDS Self-serve and AuraDS Enterprise instances were affected.

Timeline:

2025-01-10 15:39:00 - Code change merged and rolled out to production
2025-01-10 17:26:59 - Incident started - Customers begin reporting GDS function unavailability
2025-01-10 19:49:00 - Status Page updated with incident once initial scope determined
2025-01-11 00:09:00 - Fix merged and deployed to production. Begin process of restoring plugins to missing instances
2025-01-14 10:45:15 - Incident resolved - Fix fully deployed and verified

Root Cause

Neo4j rolled out an update which affects how GDS plugin assignments are managed for AuraDS instances. Incident-1966 unexpectedly resulted in the removal of the GDS library from AuraDS Self-Serve and AuraDS Enterprise instances.

Customer Impact

AuraDS instances lost access to GDS functionalities
A number of projections were lost due to the remediation activities to restore GDS functionality, affecting a number of user instances
A number of users needed to restart their GDS workflows, once a fix was put in place

Resolution

Neo4j worked to identify the root cause, affected instances, and deploy a fix to restore the GDS functionality as the first step. After step one, Neo4j updated affected instances to load the GDS plugins and verify the fix across all impacted customers’ instances.

What we are doing now

Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future.

New mitigations being deployed:

Improve on our Console reconciler testing methodologies
Increased observability metrics intended to detect and fix plugin-related issues more quickly
Improve deployment procedures by adding more effective guardrails against unintended plugin changes

Posted Jan 31, 2025 - 14:39 UTC

Resolved

We have monitored the service and this incident is now resolved.

Posted Jan 13, 2025 - 13:41 UTC

Monitoring

We have identified the issue causing the GDS plugin to be missing from the impacted instances and have rolled out a fix. If your GDS plugins are missing from your instance, please pause and resume your instance as a workaround until all remaining impacted instances are remediated.

Posted Jan 11, 2025 - 03:54 UTC

Update

We have identified the issue causing the GDS plugin to be missing from the impacted instances and we are working on releasing the fix.

Posted Jan 11, 2025 - 00:45 UTC

Identified

We have identified the issue causing the GDS plugin to be missing from the impacted instances and are working towards restoring the plugin to all missing instances.

Posted Jan 10, 2025 - 22:02 UTC

Investigating

We are investigating an issue where GDS functions are missing from AuraDS Enterprise and AuraDS instances.

Posted Jan 10, 2025 - 19:49 UTC

This incident affected: AuraDS (*.databases.neo4j.io) (AuraDS on AWS (*.databases.neo4j.io), AuraDS on Azure (*.databases.neo4j.io), AuraDS on GCP (*.databases.neo4j.io)) and AuraDS Enterprise (*.databases.neo4j.io) (AuraDS Enterprise on AWS (*.databases.neo4j.io), AuraDS Enterprise on Azure (*.databases.neo4j.io), AuraDS Enterprise on GCP (*.databases.neo4j.io)).