Capacity issue in GCP us-east1 region

Incident Report for Neo4j Aura

Postmortem

What happened

AuraDB Professional tier in the GCP us-east1 (South Carolina) region ran out of the provisioned capacity impacting new and existing customers wanting to run databases in this region.

How the service was affected

During the incident, new databases could not be created and any existing paused databases could not be resumed based on the availability of capacity at the specific time.

In the early stages of the incident any database creation that was initiated would not complete. As Neo4j became aware of the issue, the capability to create new databases in that region was removed. Neo4j also removed non-customer-facing (internal) databases to free up capacity which allowed pause and resume operations to succeed while work to provision additional capacity was completed.

An attempt to provision more capacity in the region failed due to unexpected deployment issues. Neo4j overcame the new issues which allowed the provisioning to be successful. Once additional capacity was added Neo4j re-enabled the creation of AuraDB instances in the affected region. 

What we are doing now

Neo4j has taken steps to enhance the capacity monitoring and alerts of all regions, and modified internal processes designed to ensure the capability to create new databases, in the affected region(s), are temporarily disabled in situations where new capacity must be provisioned.

Posted Jul 28, 2022 - 17:05 UTC

Resolved

A fix has been applied to the capacity problem. The creation and pausing of databases are once again functional for Professional tier customers using us-east1
Posted Jul 21, 2022 - 18:42 UTC

Update

We are still working to address the capacity shortage in the us-east1 region, which is affecting our Professional tier. Database creation and pausing are still affected. All other database operations are functioning normally. We will keep you updated as new information becomes available.
Posted Jul 21, 2022 - 17:54 UTC

Update

We are still working to address the capacity shortage in the us-east1 region, which is affecting our Professional tier. Database creation and pausing are still affected. All other database operations are functioning normally. We will keep you updated as new information becomes available.
Posted Jul 21, 2022 - 15:05 UTC

Update

We are still working to address the capacity shortage in the us-east1 region, which is affecting our Professional tier. Database creation and pausing are still affected. All other database operations are functioning normally. We will keep you updated as new information becomes available.
Posted Jul 21, 2022 - 13:03 UTC

Update

Our cloud provider, GCP is continuing to work on the issue causing the inability to create or pause databases in the us-east1 region. This is only affecting Professional users. All other operations for existing running databases continue to work fine. We will update you as more information becomes available.
Posted Jul 21, 2022 - 10:46 UTC

Update

Our cloud provider, GCP is continuing to work on the issue causing the inability to create or pause databases in the us-east1 region. This is only affecting Professional users. All other operations for existing running databases continue to work fine. We will update you as more information becomes available.
Posted Jul 21, 2022 - 08:07 UTC

Update

Our cloud provider, GCP is continuing to work on the issue causing the inability to create or pause databases in the us-east1 region. We will continue to update regularly on the progress. This is only affecting Professional users. All other operations for existing running databases continue to work fine.
Posted Jul 21, 2022 - 05:34 UTC

Update

We are still working to address the capacity shortage in the us-east1 region, which is affecting our Professional tier. Database creation and pausing are still affected. All other database operations are functioning normally. We will keep you updated as new information becomes available.
Posted Jul 21, 2022 - 03:26 UTC

Update

We are still working to address the capacity shortage in the us-east1 region, which is affecting our Professional tier. Database creation and pausing are still affected. All other database operations are functioning normally. We will keep you updated as new information becomes available.
Posted Jul 21, 2022 - 01:22 UTC

Update

We are continuing to address the lack of capacity in the us-east1 region affecting our Professional tier. Creating and pausing of databases remain affected. All other operations for running databases are working as expected.
Posted Jul 20, 2022 - 22:59 UTC

Update

Our cloud provider, GCP is continuing to work on the issue causing the inability to create or pause databases in the us-east1 region. This is only affecting Professional users. All other operations for existing running databases continue to work fine. We will update you as more information becomes available.
Posted Jul 20, 2022 - 20:33 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 20, 2022 - 20:31 UTC

Update

We continue to work on a fix for the issue. We will continue our regular updates as the situation progresses.
Posted Jul 20, 2022 - 19:32 UTC

Update

Our team continues to look for a fix for the issue. We will continue status updates as more information becomes available.
Posted Jul 20, 2022 - 17:24 UTC

Update

We continue to work towards delivering a fix.
We will continue to update regularly on the progress.
Posted Jul 20, 2022 - 16:12 UTC

Update

Our teams continue to work on a fix for this issue and we have contacted all affected customers.
Posted Jul 20, 2022 - 14:35 UTC

Update

We continue to work on a fix for this issue.
To mitigate the effect we are temporarily disabling the creation of new databases in us-east1.
Posted Jul 20, 2022 - 12:16 UTC

Update

Updating the components for the impact of this incident as the AuraDS tier isn't affected.
We continue to work on remedial steps and a solution to this issue.
Posted Jul 20, 2022 - 11:08 UTC

Identified

We have identified a lack of available capacity in the GCP us-east1 region affecting our Professional tier.
Currently it is not possible to create new or resume any paused databases instances in this region but all other operations for existing running databases instances are working fine.
Posted Jul 20, 2022 - 11:04 UTC
This incident affected: AuraDB Professional (*.databases.neo4j.io) (AuraDB Professional on AWS (*.databases.neo4j.io)) and AuraDS (*.databases.neo4j.io) (AuraDS on AWS (*.databases.neo4j.io)).
← Current Status Powered by Atlassian Statuspage