A provisioning pipeline deployment caused an incomplete configuration map to be formed, resulting in a couple of Google cloud regions being omitted, thereby making some control components unaware of these two Kubernetes cluster environments.
The impact to Aura customers using Google Cloud was felt in the inability across all tiers (Free, Professional and Enterprise) to clone database instances. Also, users of missing Google cloud regions lost the ability to create / delete / suspend database instances or access the console.
The issue was resolved when engineers from multiple Aura teams applied fixes to various back-end components in order to remove bugs from the code.
We don’t believe that the issues our customers experienced are acceptable for a DBaaS. We’ve carried out a thorough analysis of what went wrong in this situation. The actions that we’re carrying out to ensure that nothing like this can happen again fall into two areas: