Systems are back online and operational at this point. We will be writing up a post-mortem to summarize our **top priorities** and next steps to address these database issues in the immediate term.
Posted Aug 08, 2017 - 17:51 EDT
Monitoring
We've brought our primary database back online after restarting and are monitoring performance closely until we get systems back to normal.
Posted Aug 08, 2017 - 17:39 EDT
Update
We're currently pulling the logs to ensure we have a record of details behind the errors that caused our primary database to erroneously cycle. Upon doing this, we'll restart our database servers in attempt to bring all databases back online, which takes approximately 5-10 minutes. We'll keep monitoring and will update this incident upon completion of the restarts.
Posted Aug 08, 2017 - 17:32 EDT
Identified
Our primary database is having an issue based upon the number of connections attempting to connect to our replicas. We're currently cycling through restarting the databases and should have an update on their success or failure shortly.
Posted Aug 08, 2017 - 17:23 EDT
Investigating
We're currently seeing some intermittent degraded performance on our primary database. This may be causing issues logging into the application and will cause issues in creating & signing requests.