MDW and NYC Ingest Down
Incident Report for

What Happened

On 6/6 from 10:31 UTC to 10:43 UTC, MDW and NYC ingest were down, and users were unable to log in to the dashboard. During this time, users whose nearest datacenter was NYC or MDW, or who had chosen to stream into NYC or MDW, would have needed to manually reroute their streams to an alternative datacenter. Additionally, users were unable to access the dashboard at this time.

What We Are Doing About This

We’ve been experiencing similar outages recently related to this issue and are currently investigating. No root cause has been identified yet and we are working quickly to fix the issue. We will update this postmortem once we have identified the root cause and provide transparency around what additional steps we’ll take to ensure this service disruption does not happen in the future.

Posted Jun 08, 2021 - 17:12 UTC

We had received and confirmed reports that ingest and playback in NYC and MDW were down, along with issues with dashboard login page. It was shortly fixed by restarting the stuck API servers in the MDW datacenter.

This incident lasted from ~10:30 UTC to ~10:45 UTC.
Posted Jun 06, 2021 - 10:30 UTC