Summary

This is a follow-up report describing the incident that occurred on 03/29/24 https://status.livepeer.studio/incidents/qtl6n8ksdtv7

Description

Several users reported difficulties with new livestreams failing to ingest and existing livestreams either unable to playback or experiencing buffering issues. Following an investigation, the Livepeer Studio team determined that the root cause of these issues was outdated DNS configurations resulting from decommissioning nodes in certain North American regions for maintenance. This maintenance occurred during the early morning hours in the North American timezone of the same day, with users in specific regions experiencing later in the day.

Impact

• Livestreams:

  • New livestreams were not able to be ingested in certain North American regions

• Viewers:

  • Playback for certain North American regions was not playing back livestreams while some were playing but experienced buffering issues.

Regions:

• North American Regions

  • New York
  • Miami
  • Seattle
  • Los Angeles

Current status

The service has been fully restored

https://status.livepeer.studio/

Timeline

• 12:16 PM EST - Reports from multiple users indicating livestreams were having issues with broadcasting and playback
• 12:26 PM EST - The team from Livepeer Studio acknowledged this incident and started an investigation
• 1:04 PM EST - The investigation from the Livepeer team led to a recent modification to the DNS configuration due to maintenance carried out on the nodes earlier that day
• 1:13 PM EST - After monitoring the fix for the incident, the Livepeer Studio team concluded that the issue was resolved

Prevention

• Livepeer Studio is now running the DNS configurations hourly to make sure it is consistently up to date
• Livepeer Studio is improving alerting systems for catching ingest failures