On 3/25/21 from 19:34 UTC to 19:54 UTC, a misconfigured Livepeer node was returning a response with missing data. Due to a bug, our systems were unable to verify that there was missing data, crashing all nodes. During this time, all users were unable to transcode. Once the team implemented a fix for the bug, customers were able to transcode.
In addition to fixing the bug that caused this particular issue, we’re increasing our monitoring and alerts so that we can get better visibility into the performance of our systems so we can anticipate these issues before they happen and react to them faster when they do happen.