APOC procedure availability impacted
Incident Report for Neo4j Aura
Postmortem

What happened

Aura rolled out a version of the Neo4j core product with missing java classes (Guava) that are needed by some apoc procedures and functions at runtime.

This was a bug in the build process itself in which the runtime classes were only included in test scope, meaning all apoc-related tests passed and the problem then surfaced in production.

How the service was affected

The impact to Aura users was that the 'apoc.coll.toSet' function that was previously present and available in Aura became unavailable as the release was rolled out, due to the specific version of Guava library not being present in the classpath any longer.

Customer impact was mitigated by Aura Engineering adding the missing Guava java library dependency to the classpath for impacted users via a patch.

The issue was fully resolved by deploying a version of the Neo4j core product that included a fixed apoc build that correctly loaded the specific version of the Guava java library into the classpath allowing all dependent apoc procedures to load at runtime.

What we are doing now

We don’t believe that the issues our customers experienced are acceptable for a DBaaS. We’ve carried out a thorough analysis of what went wrong in this situation. The actions that we’re carrying out to ensure that nothing like this can happen again fall into two areas:

  • Testing - We will improve our testing by producing a list of Cypher procedures/functions available from source and compare it with the runtime information available from a running Neo4j container, configured with APOC. Any difference will trigger a failure. This check will happen before we roll out any databases in production.
  • Deployment - We will stagger our roll out of these types of changes, releasing to less critical areas of Aura first, then monitoring, so that we can gain confidence before we roll fully to the wider platform.
Posted Jul 01, 2021 - 20:50 UTC

Resolved
We have identified all affected customers and have applied a fix to resolve the issue. Supported APOC procedures should be available and working again. If you continue to see errors about an unsupported APOC procedure that was working previously please reach out to aura-support@neo4j.com.
Posted May 27, 2021 - 21:10 UTC
Update
We continue to proactively address those customers who are affected by this issue. If you believe you are being affected by this issue, please reach out to aura-support@neo4j.com so that we can work on getting it fixed.
Posted May 27, 2021 - 19:45 UTC
Update
Aura Engineering and SRE teams have identified the source of the problem and are actively working to restore availability of the small number of missing apoc functions for affected customers.
Posted May 27, 2021 - 15:58 UTC
Update
We have identified remedial steps and we are continuing to work on deploying a fix.
Posted May 27, 2021 - 14:19 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted May 27, 2021 - 11:47 UTC
Investigating
Some APOC procedures are currently unavailable
Posted May 27, 2021 - 11:47 UTC
This incident affected: Aura Databases (*.databases.neo4j.io).