On 7/31/21 from 10:28 UTC to 11:22 UTC MDW region ingest and playback was down or intermittent. During this time, users whose nearest datacenter was MDW, or who had chosen to stream into MDW, would have needed to manually reroute their streams to an alternative datacenter in order to successfully stream.
Although a precise root cause has not been identified, the team believes that the incident was related to our DNS. We’re currently working on changes that would increase resiliency of our systems should this issue happen in the future, including addressing single points of failures and increasing observability.