Transcoding outage
Incident Report for

What happened

On 3/25/21 from 19:34 UTC to 19:54 UTC, a misconfigured Livepeer node was returning a response with missing data. Due to a bug, our systems were unable to verify that there was missing data, crashing all nodes. During this time, all users were unable to transcode. Once the team implemented a fix for the bug, customers were able to transcode.

What We Are Doing About This

In addition to fixing the bug that caused this particular issue, we’re increasing our monitoring and alerts so that we can get better visibility into the performance of our systems so we can anticipate these issues before they happen and react to them faster when they do happen.

Posted Apr 01, 2021 - 21:08 UTC

A fix has been deployed. Transcoding is now back online
Posted Mar 25, 2021 - 19:54 UTC
Issue has been identified. The team is working on a hotfix.
Posted Mar 25, 2021 - 19:43 UTC
Transcoding disruption detected
Posted Mar 25, 2021 - 19:34 UTC
This incident affected: API and Media server transcoding via (Media server transcoding via Livepeer - US, Media server transcoding via Livepeer - EU).