On 5/10 from 20:51 UTC to 21:23 UTC LON regional ingest/playback was down for users who were using regional URLs. During this time, users would have needed to manually reroute streams to a different datacenter or use the global ingest/playback URL. Users who were streaming using ingest/playback URLs were unaffected by this issue.
Once identified, we began reprovisioning the Kubernetes cluster that was the cause of this issue. Reprovisioning took time, and we’ve identified ways that we could have fixed the issue faster.
Additionally, we plan to run a simulation of this test in our stage environment to ensure that we understand all the causes of this incident and work to prevent this issue in the future.