Problem Description, Impact, and Resolution

At 18:08 UTC on November 29th customers in our US region started experiencing higher than normal latencies, at points reaching as high as 5 seconds, for all requests and some requests were failing with 504 or 499 status codes. During this incident, 7.1% of all requests failed. The immediate cause of the incident was a drastic increase in CPU usage on our API’s primary database in the US region because there was an issue with data indexing...

Problem Description, Impact, and Resolution

At 19:18 UTC on Nov 17, 2023 we observed networking issues on the Reactors service in the US region, which resulted in intermittent 404 errors on token retrieves. The issue was caused by an internal and private DNS route update that temporarily broke the traffic route for the bt instantiated Basis Theory SDK in Reactors. We immediately updated the DNS record to route all traffic to the correct destination and the issue was fully resolved at 2...