Ingest for MDW, LAX, NYC down

Incident Report for Livepeer Studio

Postmortem

What Happened

On 6/8 from 16:13 UTC to 16:24 UTC, MDX, LAX, and NYC region ingest/playback was down. During this time, users whose nearest datacenter was NYC, MDW, or LAX, or who had chosen to stream into NYC, MDW or LAX, would have needed to manually reroute their streams to an alternative datacenter.

What We Are Doing About This

A root cause related to Postgres replication system setting has been identified and changed. Additionally, we’ve identified some systemic changes to prevent this issue from happening in the future, including adding improved monitoring and alerting so we’re able to identify these issues faster.

Posted Jun 14, 2021 - 17:38 UTC

Resolved

This incident has been resolved.

Posted Jun 08, 2021 - 16:31 UTC

Update

We believe that this issue is fixed for all customers, but are continuing to monitor this issue.

Posted Jun 08, 2021 - 16:24 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 08, 2021 - 16:23 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 08, 2021 - 16:19 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 08, 2021 - 16:18 UTC

Identified

We have received and confirmed reports that ingest and playback in MDW is down. As a workaround, redirect new streams to another Livepeer.com ingest datacenter.

If you're currently experiencing this issue or need additional assistance, please reach out to help@livepeer.com.

Posted Jun 08, 2021 - 16:13 UTC

This incident affected: Livepeer Streaming API, Livepeer Studio dashboard and Livepeer Studio Ingest and Playback (Chicago (MDW) ingest and playback, New York (NYC) ingest and playback, Los Angeles (LAX) ingest and playback).