What happened

Aura rolled out a version of the Neo4j core product with missing java classes (Guava) that are needed by some apoc procedures and functions at runtime.

This was a bug in the build process itself in which the runtime classes were only included in test scope, meaning all apoc-related tests passed and the problem then surfaced in production.

How the service was affected

The impact to Aura users was that the 'apoc.coll.toSet' function that was previously present and available in Aura became unavailable as the release was rolled out, due to the specific version of Guava library not being present in the classpath any longer.

Customer impact was mitigated by Aura Engineering adding the missing Guava java library dependency to the classpath for impacted users via a patch.

The issue was fully resolved by deploying a version of the Neo4j core product that included a fixed apoc build that correctly loaded the specific version of the Guava java library into the classpath allowing all dependent apoc procedures to load at runtime.

What we are doing now

We don’t believe that the issues our customers experienced are acceptable for a DBaaS. We’ve carried out a thorough analysis of what went wrong in this situation. The actions that we’re carrying out to ensure that nothing like this can happen again fall into two areas:

• Testing - We will improve our testing by producing a list of Cypher procedures/functions available from source and compare it with the runtime information available from a running Neo4j container, configured with APOC. Any difference will trigger a failure. This check will happen before we roll out any databases in production.
• Deployment - We will stagger our roll out of these types of changes, releasing to less critical areas of Aura first, then monitoring, so that we can gain confidence before we roll fully to the wider platform.