Problem Description, Impact, and Resolution

At 6:10 PM UTC on June 8, 2022 we were alerted that requests to all services began failing, which resulted in requests not resolving to the Basis Theory hostnames resulting in a DNS Resolution error for all connections to Basis Theory services. A global DNS outage at our service provider caused the incident. While the incident was ongoing, we were unable to route traffic around the affected service and the provider resolved the incident before we were able to implement the mitigation. The service was restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC.

Mitigation Steps and Future Preventative Measures

To ensure this issue does not occur again we have created a process and run book to route around this service and limit the impact on customers. By end of Q2 2022, we will have a long-term plan to eliminate service impacts to be completed by end of Q3 2022.