Transcoding outage
Incident Report for Livepeer.com
Postmortem

What happened

On 3/25/21 from 19:34 UTC to 19:54 UTC, a misconfigured Livepeer node was returning a response with missing data. Due to a bug, our systems were unable to verify that there was missing data, crashing all nodes. During this time, all users were unable to transcode. Once the team implemented a fix for the bug, customers were able to transcode.

What We Are Doing About This

In addition to fixing the bug that caused this particular issue, we’re increasing our monitoring and alerts so that we can get better visibility into the performance of our systems so we can anticipate these issues before they happen and react to them faster when they do happen.

Posted Apr 01, 2021 - 21:08 UTC

Resolved
A fix has been deployed. Transcoding is now back online
Posted Mar 25, 2021 - 19:54 UTC
Identified
Issue has been identified. The team is working on a hotfix.
Posted Mar 25, 2021 - 19:43 UTC
Investigating
Transcoding disruption detected
Posted Mar 25, 2021 - 19:34 UTC
This incident affected: Livepeer.com API and Media server transcoding via Livepeer.com (Media server transcoding via Livepeer - US, Media server transcoding via Livepeer - EU).