console.neo4j.io and Aura API are down

Incident Report for Neo4j Aura

Postmortem

What happened

Customers started to report errors accessing the Console and the Aura API on 2023-12-28 at 13:50 UTC. An incident was raised and our engineering teams identified requests to GCP Datastore timing out as the cause of the unavailability. We were particularly affected due to our usage of the python Datastore drivers version selected. We raised a support ticket with our cloud service provider (GCP) and in the meantime our SREs identified the issue and mitigated it. Service availability was restored around 2023-12-28 at 17:30 UTC.

How the service was affected

Both the Aura Console and API make use of the GCP Datastore service for user management.

Console access authentication was successful but loading the Aura tenant information was affected and blocked the display of the Aura Console UI. Aura API also operates at tenant level and was impacted. Requests started timing out causing the unavailability.

What we are doing now

We have taken steps to update our GCP Datastore’s driver version according to GCP’s recommendation as well as making sure we better handle an outage and timeout on some queries to prevent blocking. We will also implement a circuit breaker in our logic to reduce the impact of an outage. Finally we will be looking into improving our detection and alerting in case of a GCP Datastore service outage.

Posted Jan 16, 2024 - 22:20 UTC

Resolved

The incident impacting Aura Console and Aura API has been resolved, and stable for ~3 hours. We are considering this incident resolved, and systems are fully operational. We will update this incident with details regarding the outage after thorough investigation and review.

Posted Dec 28, 2023 - 20:35 UTC

Monitoring

The component identified to be causing the outage has been remedied. Aura Console and Aura API are operational and we are currently monitoring the health of both.

Posted Dec 28, 2023 - 17:30 UTC

Identified

We have identified an issue with a service component and are working to resolve that as quickly as possible to restore service to Aura Console and Aura API.

Posted Dec 28, 2023 - 17:02 UTC

Update

We are aware of an outage continuing to effect both the Aura Console and Aura API. We are continuing to investigate and have ruled out some components, and are taking ongoing action to restore service.

Posted Dec 28, 2023 - 16:20 UTC

Update

We are aware of an outage effecting both the Aura Console and Aura API. We are continuing to investigate and are taking action to restore service.

Posted Dec 28, 2023 - 15:18 UTC

Investigating

We are currently aware of an outage effecting both the Aura Console and Aura API. We are currently investigating and taking action to restore service.

Posted Dec 28, 2023 - 14:17 UTC

This incident affected: Aura Console (console.neo4j.io) and Aura API (api.neo4j.io).