Certain streams are being erroneously rejected
Incident Report for Livepeer.com
Postmortem

What Happened

On 4/27 from 20:23 UTC to 20:45 UTC all streams that were in progress failed to be transcoded. Customers who had ongoing streams needed to be restarted. This was due to a bug in recently deployed software that caused RTMP streams to be rejected.

What We Are Doing About This

Once discovered, we immediately rolled back the changes that caused this bug. We are adding additional alerting to our staging environment to improve testing so that we’re more confident that our deployments are bug-free. Additionally, we’re making it easier and faster to roll back changes so that if we ever encounter this issue again, we’ll be up and running faster.

Posted May 03, 2021 - 22:11 UTC

Resolved
We have confirmed that no streams are being erroneously rejected.
Posted Apr 27, 2021 - 21:12 UTC
Monitoring
The fix has been rolled out and streams are no longer being erroneously rejected. We will continue to monitor to be sure.
Posted Apr 27, 2021 - 21:07 UTC
Identified
We are rolling out a fix that should stop streams from being erroneously rejected.
Posted Apr 27, 2021 - 21:00 UTC
Update
We have identified the issue.
Posted Apr 27, 2021 - 20:55 UTC
Investigating
We are currently investigating this issue. Only a select few streams are being erroneously rejected.
Posted Apr 27, 2021 - 20:42 UTC
This incident affected: Livepeer.com API, Legacy Livepeer.com API, Media server transcoding via Livepeer.com (Media server transcoding via Livepeer - US, Media server transcoding via Livepeer - EU), and Livepeer.com Ingest and Playback (Chicago (MDW) ingest and playback, Frankfurt (FRA) ingest and playback, London (LON) ingest and playback, New York (NYC) ingest and playback, Singapore (SIN) Ingest and playback).