On 4/24/21 from 4:20 UTC to 4:50 UTC, customers who were using NYC ingest were unable to stream. Customers using global ingest but were near NYC may have also experienced issues streaming. This was due to DNS failing within the Kubernetes cluster, causing a failure of a variety of crucial services. The DNS failed because an associated server had failed, due to planned maintenance.
We’re adding additional DNS servers in this cluster to make our systems more dependable in the region and prevent an issue like this from happening in the future. Additionally, we’ve corrected some process gaps that caused us to miss a notification about the pre-planned maintenance.