Degraded Performance on Government ID Verification
Incident Report for Synapse
Postmortem

What happened: Recently we migrated all our Newton servers to a new environment with improved security features. Over the weekend, one of our hosts went down in this new environment. This was a result of poor logging which didn’t allow for immediate detection, which is our company standard.

When it happened: We were notified about the issue on Monday, June 18th at 5:34am and the host was restored Monday, June 18th at 6:17am. Later that day, a residual problem with outdated service images was discovered at 11:20am and fixed at 12:46pm the same day.

How we fixed it: We regenerated the host as soon as we discovered the failure. We also re-ran all end users that were affected.

What we are doing to ensure it will not happen again: We have added more logging and notification systems to all hosts in our new environment to ensure this doesn't happen again.

Who was affected: Only traffic that was going through this particular host was affected.

Posted Jun 20, 2018 - 11:17 PDT

Resolved
This incident has been resolved.
Posted Jun 18, 2018 - 05:34 PDT