Degraded service for NYC Ingest and Playback
Incident Report for Livepeer.com
Postmortem

What happened

On 4/24/21 from 4:20 UTC to 4:50 UTC, customers who were using NYC ingest were unable to stream. Customers using global ingest but were near NYC may have also experienced issues streaming. This was due to DNS failing within the Kubernetes cluster, causing a failure of a variety of crucial services. The DNS failed because an associated server had failed, due to planned maintenance.

What We Are Doing About This

We’re adding additional DNS servers in this cluster to make our systems more dependable in the region and prevent an issue like this from happening in the future. Additionally, we’ve corrected some process gaps that caused us to miss a notification about the pre-planned maintenance.

Posted Apr 30, 2021 - 22:06 UTC

Resolved
The issue was identified to be maintenance work in of our datacenter vendors. We've re-routed our infrastructure to not use the under-maintenance nodes. The issue is now resolved.
Posted Apr 25, 2021 - 05:07 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 25, 2021 - 05:01 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Apr 25, 2021 - 04:50 UTC
Investigating
We have received and confirmed reports that ingest and playback in NYC region is down. As a workaround, redirect new streams to another Livepeer.com ingest datacenter.

If you're currently experiencing this issue or need additional assistance, please reach out to help@livepeer.com.
Posted Apr 25, 2021 - 04:36 UTC
This incident affected: Livepeer.com API and Livepeer.com Ingest and Playback (New York (NYC) ingest and playback).