Problem Description, Impact, and Resolution

At 18:08 UTC on November 29th customers in our US region started experiencing higher than normal latencies, at points reaching as high as 5 seconds, for all requests and some requests were failing with 504 or 499 status codes. During this incident, 7.1% of all requests failed. The immediate cause of the incident was a drastic increase in CPU usage on our API’s primary database in the US region because there was an issue with data indexing...

Problem Description, Impact, and Resolution

At 19:18 UTC on Nov 17, 2023 we observed networking issues on the Reactors service in the US region, which resulted in intermittent 404 errors on token retrieves. The issue was caused by an internal and private DNS route update that temporarily broke the traffic route for the bt instantiated Basis Theory SDK in Reactors. We immediately updated the DNS record to route all traffic to the correct destination and the issue was fully resolved at 2...

Basis Theory will be performing a scheduled system upgrade on Monday, November 13, 2023, from 01:00 UTC to 04:00 UTC. (Sunday, November 12th, 2023 from 8:00 PM ET to 11:00 PM ET)

We do not anticipate any outage as a part of this upgrade, although a brief increase in latency and errors may occur for a short number of seconds.