Incident postmortem - Database clone feature impacted

What happened

A provisioning pipeline deployment caused an incomplete configuration map to be formed, resulting in a couple of Google cloud regions being omitted, thereby making some control components unaware of these two Kubernetes cluster environments.

How the service was affected

The impact to Aura customers using Google Cloud was felt in the inability across all tiers (Free, Professional and Enterprise) to clone database instances. Also, users of missing Google cloud regions lost the ability to create / delete / suspend database instances or access the console.

The issue was resolved when engineers from multiple Aura teams applied fixes to various back-end components in order to remove bugs from the code.

What we are doing now

We don’t believe that the issues our customers experienced are acceptable for a DBaaS. We’ve carried out a thorough analysis of what went wrong in this situation. The actions that we’re carrying out to ensure that nothing like this can happen again fall into two areas:

• Preventative

  • Add guardrails to the logic that is updating the configuration map. 
  • Ensure that when the configuration map is updated, orchestras with running databases can’t be removed.
  • Change the data type of one of the fields related to previous identifier of the configuration map to be an integer.
  • Consider adopting a policy of not using bash in high-impact components.
  • Investigate if Aura relies on other large configuration maps that are mounted as environment variables.
  • Investigate ways of storing configurations without using configuration maps due to the 1 MiB hard limit.
  • Consider adopting a policy of requiring components to be mergeable & deployable within x minutes.

• Detection

  • Monitor the correctness of the configuration maps.
  • Monitor and alert based on the size of the configuration map.