What happened

On 2024-06-28 10:46 (UTC) we released a new version of the database (v. 5.21). This release of the database contained an improvement to fix a problem of queries returning an entity ordered by an indexed-backed property that could come out in the wrong order due to concurrent writes (eg with an index-backed property:  MATCH (n:L) WHERE n.x IS NOT NULL RETURN n ORDER BY n.x) Unfortunately this introduced two undetected regressions:

• key not found: VariableSlotKey(...) where the runtime would try to access a cached variable that was not in scope 
• NullCheckReferenceProperty cannot be cast to class ASTCachedProperty  a class cast exception in the slotted runtime happening in some cases when retrieving cached properties

Contrary to the usual process, a human error meant it was rolled-out directly to the AuraDB Enterprise tier without having been exposed first to the AuraDB Free tier and Professional tier.

How the service was affected

Customers running certain types of queries where the whole node would get cached as part of the optimization were seeing failures key not found: VariableSlotKey(...) OR  NullCheckReferenceProperty cannot be cast to class ASTCachedProperty with no simple workarounds. 

We fixed the regression and rolled out a new version 5.21.1 (2024-06-29 01:10) and then a second 5.21.2 to address both of these regressions.

What we are doing now

We consider this incident as severe and have committed resources to deliver on the following actions:

• Rolling out

  • Prevent rolling out to the Enterprise tier at the end of the working week
  • Release roll-out process and tooling to enforce the sequence of tiers to receive a new release
  • Automate the release and roll-out process to remove risks of further human error
  • Improve on the time it takes to release an emergency fix

• Database release

  • Add further tests around the release of improvements to Cypher

• Detection

  • Build a monitoring dashboard specific to detect spikes of errors due to Cypher queries