LON regional ingest/playback down
Incident Report for Livepeer Studio
Postmortem

What Happened

On 5/10 from 20:51 UTC to 21:23 UTC LON regional ingest/playback was down for users who were using regional URLs. During this time, users would have needed to manually reroute streams to a different datacenter or use the global ingest/playback URL. Users who were streaming using ingest/playback URLs were unaffected by this issue.

What We Are Doing About This

Once identified, we began reprovisioning the Kubernetes cluster that was the cause of this issue. Reprovisioning took time, and we’ve identified ways that we could have fixed the issue faster.

Additionally, we plan to run a simulation of this test in our stage environment to ensure that we understand all the causes of this incident and work to prevent this issue in the future.

Posted May 12, 2021 - 22:35 UTC

Resolved
The issue has been resolved.
Posted May 10, 2021 - 21:31 UTC
Update
We are continuing to monitor for any further issues.
Posted May 10, 2021 - 21:23 UTC
Monitoring
LON regional ingest/playback URLs are back up after the server issue was fixed. Ingest and playback in the region should both be working now. We are continuing to monitor.
Posted May 10, 2021 - 21:23 UTC
Identified
We have identified the reason for the downtime as a problem with one of the servers in the LON region. We're working on resolving the problem to bring the region back up.
Posted May 10, 2021 - 20:56 UTC
Update
We are continuing to investigate this issue.
Posted May 10, 2021 - 20:51 UTC
Investigating
LON regional ingest/playback URLs for livepeer.com currently are down. Users can manually re-route streams by manually using different regional ingest/playback URLs.
Posted May 10, 2021 - 20:51 UTC
This incident affected: Livepeer Studio Ingest and Playback (London (LON) ingest and playback).