Degraded service for NYC Ingest and Playback
Incident Report for

What happened

On 4/24/21 from 4:20 UTC to 4:50 UTC, customers who were using NYC ingest were unable to stream. Customers using global ingest but were near NYC may have also experienced issues streaming. This was due to DNS failing within the Kubernetes cluster, causing a failure of a variety of crucial services. The DNS failed because an associated server had failed, due to planned maintenance.

What We Are Doing About This

We’re adding additional DNS servers in this cluster to make our systems more dependable in the region and prevent an issue like this from happening in the future. Additionally, we’ve corrected some process gaps that caused us to miss a notification about the pre-planned maintenance.

Posted Apr 30, 2021 - 22:06 UTC

The issue was identified to be maintenance work in of our datacenter vendors. We've re-routed our infrastructure to not use the under-maintenance nodes. The issue is now resolved.
Posted Apr 25, 2021 - 05:07 UTC
A fix has been implemented and we are monitoring the results.
Posted Apr 25, 2021 - 05:01 UTC
The issue has been identified and a fix is being implemented.
Posted Apr 25, 2021 - 04:50 UTC
We have received and confirmed reports that ingest and playback in NYC region is down. As a workaround, redirect new streams to another ingest datacenter.

If you're currently experiencing this issue or need additional assistance, please reach out to
Posted Apr 25, 2021 - 04:36 UTC
This incident affected: API and Ingest and Playback (New York (NYC) ingest and playback).