Incidents reported on status page for Basis Theory https://status.basistheory.com/ https://d1lppblt9t2x15.cloudfront.net/logos/f35dd65f3c7cbd534672a7f536bd197b.png https://status.basistheory.com/ en https://status.basistheory.com/incident/534944 Wed, 26 Mar 2025 21:57:00 -0000 https://status.basistheory.com/incident/534944#0c3f734a8fd468d4a7ca2a9f70f64d77677b347d25302fbae5919f9f9c9d9c66 # Problem Description, Impact, and Resolution At 14:22 UTC on March 26th, 2025, the API in our primary region across both the US and EU services became unavailable. After a few seconds of unavailability, our services correctly redirected traffic to another region and successfully served traffic until 14:24 UTC, when these services also became unavailable. This resulted in customers receiving a `503 Service Unavailable` status code, preventing them from interacting with APIs. The issue was caused by a policy change that removed access to an asset before it had been entirely de-provisioned from the health check of the underlying service; this caused the health check to fail and our system to remove it from rotation, causing a cascading error to our API. We rolled back the policy change at 14:25 UTC, and the issue was resolved at 14:32 UTC. # Mitigation Steps and Future Preventative Measures We have fixed the issue with the policy change and successfully released the platform update. But, to ensure a similar issue does not occur again, we are actively working to solve the following as immediate fixes within our platform: 1. In addition to our existing automated smoke testing release process, we will ensure these policy changes have specific smoke tests in our secondary region before being promoted to our primary region. We will also review our entire portfolio of services and ensure any missing smoke tests are added to remove the chance of complete regional cascading failures in the future. 2. We are reducing our primary dependency on this service to reduce the impact and blast radius it can expose to our API. 3. In addition to the automated steps above, we will review and update our code review processes and procedures to ensure manual checks are in place for policy changes. We will also be investigating and prioritizing the following changes over the next few months: 1. Introducing a new environment will further stage changes within our deployment process and create another smoke test gate within our release lifecycle. 2. Simplifying coordination of service and policy deployment into a single deployable https://status.basistheory.com/incident/534944 Wed, 26 Mar 2025 21:57:00 -0000 https://status.basistheory.com/incident/534944#0c3f734a8fd468d4a7ca2a9f70f64d77677b347d25302fbae5919f9f9c9d9c66 # Problem Description, Impact, and Resolution At 14:22 UTC on March 26th, 2025, the API in our primary region across both the US and EU services became unavailable. After a few seconds of unavailability, our services correctly redirected traffic to another region and successfully served traffic until 14:24 UTC, when these services also became unavailable. This resulted in customers receiving a `503 Service Unavailable` status code, preventing them from interacting with APIs. The issue was caused by a policy change that removed access to an asset before it had been entirely de-provisioned from the health check of the underlying service; this caused the health check to fail and our system to remove it from rotation, causing a cascading error to our API. We rolled back the policy change at 14:25 UTC, and the issue was resolved at 14:32 UTC. # Mitigation Steps and Future Preventative Measures We have fixed the issue with the policy change and successfully released the platform update. But, to ensure a similar issue does not occur again, we are actively working to solve the following as immediate fixes within our platform: 1. In addition to our existing automated smoke testing release process, we will ensure these policy changes have specific smoke tests in our secondary region before being promoted to our primary region. We will also review our entire portfolio of services and ensure any missing smoke tests are added to remove the chance of complete regional cascading failures in the future. 2. We are reducing our primary dependency on this service to reduce the impact and blast radius it can expose to our API. 3. In addition to the automated steps above, we will review and update our code review processes and procedures to ensure manual checks are in place for policy changes. We will also be investigating and prioritizing the following changes over the next few months: 1. Introducing a new environment will further stage changes within our deployment process and create another smoke test gate within our release lifecycle. 2. Simplifying coordination of service and policy deployment into a single deployable https://status.basistheory.com/ Wed, 26 Mar 2025 14:58:38 +0000 https://status.basistheory.com/#0521c090b731b1995a6801a4fc97211d0ab5e5897a114647f94397588af3426b 🇪🇺 API recovered https://status.basistheory.com/incident/534944 Wed, 26 Mar 2025 14:33:00 -0000 https://status.basistheory.com/incident/534944#4aaff324b294babf0bea6fed7eb20740eb9e070cb7e1796fc510baa10f25e490 This has been resolved. We will follow up with an RCA on this issue within the next 24 hours. https://status.basistheory.com/incident/534944 Wed, 26 Mar 2025 14:33:00 -0000 https://status.basistheory.com/incident/534944#4aaff324b294babf0bea6fed7eb20740eb9e070cb7e1796fc510baa10f25e490 This has been resolved. We will follow up with an RCA on this issue within the next 24 hours. https://status.basistheory.com/ Wed, 26 Mar 2025 14:28:43 +0000 https://status.basistheory.com/#0521c090b731b1995a6801a4fc97211d0ab5e5897a114647f94397588af3426b 🇪🇺 API went down https://status.basistheory.com/incident/534944 Wed, 26 Mar 2025 14:27:00 -0000 https://status.basistheory.com/incident/534944#e42f044f3c3c7f30b0f469f7f86a571bf863126829165af23556d5ee283da0f0 API is currently having issues serving requests. https://status.basistheory.com/incident/534944 Wed, 26 Mar 2025 14:27:00 -0000 https://status.basistheory.com/incident/534944#e42f044f3c3c7f30b0f469f7f86a571bf863126829165af23556d5ee283da0f0 API is currently having issues serving requests. https://status.basistheory.com/incident/516835 Fri, 21 Feb 2025 04:00:00 -0000 https://status.basistheory.com/incident/516835#934e26e80ef97295228ce929533133ad9a43e0dfbf7f1e511b3dc1604b856132 # Problem Description, Impact, and Resolution At 8:30 p.m. CST on February 20, 2025, we observed increased errors in loading Basis Theory Elements in our `js.basistheory.com` service. This prevented customers from loading Elements when using the CDN-hosted version. The issue was caused by an incorrect base URL configuration when building the static version of `basis-theory-js` for the `js.basistheory.com` CDN version, which was deployed at 4:00 p.m. CST and slowly rolled out to customers over the next 3 hours as the cache was invalidated. This resulted in Elements not loading correctly for customers relying on the CDN version. By 9:00 p.m. CST, we determined that the problem affected all customers using the CDN version. We deployed a fix at 9:57 p.m. CST, and by 10:00 p.m. CST, errors immediately decreased as our CDN cache and browser caches began to retrieve the new version. # Mitigation Steps and Future Preventative Measures To prevent this issue, we have already implemented versioning for our packaged Elements ([web-elements](https://developers.basistheory.com/docs/sdks/web/web-elements/) and [react-elements](https://developers.basistheory.com/docs/sdks/web/react/)) and all versions (down to patch) of our new [web-elements](https://developers.basistheory.com/docs/sdks/web/web-elements/) package in the CDN - enabling customers to pin to a specific version of Elements from our CDN. Additionally, we will immediately add monitoring for the following: - Enhance our automated smoke tests for [`js.basistheory.com`](http://js.basistheory.com) elements in our pre-deploy steps. - Implement enhanced monitoring for failures to load files from `js.basistheory.com`. - Increase visibility on pinning specific patch versions of `web-elements` in [developer documentation](https://developers.basistheory.com) and add a check for version pinning to our [production checklist](https://developers.basistheory.com/docs/guides/production-checklist). https://status.basistheory.com/incident/516835 Fri, 21 Feb 2025 02:30:00 -0000 https://status.basistheory.com/incident/516835#c0744c32bedb3470ad15b87d775886ae1bd059ba4ee4cacab0acc5c5a29151bd At 8:30 p.m. CST on February 20, 2025, we observed increased errors in loading Basis Theory Elements in our [js.basistheory.com](http://js.basistheory.com/) service. This prevented customers from loading Elements when using the CDN-hosted version. NOTE: This does not affect customers using our NPM modules to load elements. https://status.basistheory.com/incident/417729 Thu, 22 Aug 2024 18:40:00 -0000 https://status.basistheory.com/incident/417729#e2fdcb9706d9143afa698c34efbdb509370b9e64237e0505944fd3ecef421ac7 Latencies have returned to normal https://status.basistheory.com/incident/417729 Thu, 22 Aug 2024 18:28:00 -0000 https://status.basistheory.com/incident/417729#ebc67eb9a8e4afa43d057ac49bb51934b248069c2c45dcabd9f19aaa9695e5b8 Some customers are experiencing high latencies while creating tokens. https://status.basistheory.com/incident/403217 Fri, 26 Jul 2024 02:30:00 +0000 https://status.basistheory.com/incident/403217#f3ce7ffeab926633db7a18040475a08730ffab1491c0a331dc2870b0d222ea95 Maintenance completed https://status.basistheory.com/incident/403217 Fri, 26 Jul 2024 02:00:00 -0000 https://status.basistheory.com/incident/403217#c1f1ffcb61ad2578567b63ba596d5dcd16801d3bb74f7532d327dfc21bb65ab0 Routine maintenance is scheduled for 10 PM EDT on July 25th, 2024. During this time, the Portal may be intermittently inaccessible, and users may experience logouts. The maintenance is expected to last 10-15 minutes. https://status.basistheory.com/incident/294848 Fri, 01 Dec 2023 21:39:00 -0000 https://status.basistheory.com/incident/294848#8a6ba053fdfdbd361f4a93dcf736555b84cc67451494c1991e0d0569575ed655 ### Problem Description, Impact, and Resolution At 18:08 UTC on November 29th customers in our US region started experiencing higher than normal latencies, at points reaching as high as 5 seconds, for all requests and some requests were failing with `504` or `499` status codes. During this incident, 7.1% of all requests failed. The immediate cause of the incident was a drastic increase in CPU usage on our API’s primary database in the US region because there was an issue with data indexing leading to some queries requiring increased CPU and Memory. At 18:33 UTC we deployed a maintenance solution that optimized the indexes and we saw an immediate decrease in CPU usage and latencies though it took until 19:29 UTC for the fix to be fully rolled out across the US region and for the incident to be fully resolved with latencies back to normal ranges. The root cause for the incident was not immediately apparent and we prioritized a process to regularly optimize indexes as system load and volume increase and we put in place additional monitoring and alerts triggers for CPU level thresholds in our databases. The following day, on November 30th at 18:10 UTC, a similar increase in CPU usage occurred although the optimized indexes and monitoring in place enabled us to take mitigating actions to ensure that there was no impact on customers. Following the second occurrence, we determined the root cause for this behavior in the database was an upgrade in our client library on November 29th resulting in queries that our database could not optimize for larger volume customers. We immediately rolled back the upgrade and have not seen a re-occurrence of the issue. ### Mitigation Steps and Future Preventative Measures **To ensure this issue does not occur again we have:** - Increased the volume of data our automated tests use to ensure we catch any query degradations - Regularly scheduled index maintenance **To ensure similar issues to this one do not arise in the future, we will be:** - Updating our load and acceptance testing plans with significantly more aggressive loads and data volumes. - Increasing visibility and monitoring of low-level metrics that may indicate similar problems. - Increased monitoring and alerting around degraded latencies experienced by our customers. https://status.basistheory.com/incident/294848 Wed, 29 Nov 2023 19:22:00 -0000 https://status.basistheory.com/incident/294848#4e30ae2e429a480c63a8ee52357b83a6aad7d69a640024e8853f2ad159f07f09 Systems are fully operational. https://status.basistheory.com/incident/294848 Wed, 29 Nov 2023 18:08:00 -0000 https://status.basistheory.com/incident/294848#596e73b7de6687e58504fc71d37f166fa624727918df5672327a98d2a315d0da We're experiencing instability and are investigating. https://status.basistheory.com/incident/289959 Fri, 17 Nov 2023 20:00:00 -0000 https://status.basistheory.com/incident/289959#2bf8068d16d477de34044adf38bb368568db14b1f1b7eb1377aa1400b114c6ff # Problem Description, Impact, and Resolution At 19:18 UTC on Nov 17, 2023 we observed networking issues on the Reactors service in the US region, which resulted in intermittent 404 errors on token retrieves. The issue was caused by an internal and private DNS route update that temporarily broke the traffic route for the `bt` instantiated Basis Theory SDK in Reactors. We immediately updated the DNS record to route all traffic to the correct destination and the issue was fully resolved at 20:00 UTC. # Mitigation Steps and Future Preventative Measures To mitigate the recurrence of this issue, we have decommissioned the specific route configuration within our reactor services. This action will prevent any potential disruptions in future DNS changes when leveraging the `bt` property within Reactors. https://status.basistheory.com/incident/285044 Mon, 13 Nov 2023 04:00:00 +0000 https://status.basistheory.com/incident/285044#4a8b51890d8942f6fa3dbec361430addd7f1dc7575177baafdac50a3adf70b3f Maintenance completed https://status.basistheory.com/incident/285044 Mon, 13 Nov 2023 04:00:00 +0000 https://status.basistheory.com/incident/285044#4a8b51890d8942f6fa3dbec361430addd7f1dc7575177baafdac50a3adf70b3f Maintenance completed https://status.basistheory.com/incident/285044 Mon, 13 Nov 2023 01:00:00 -0000 https://status.basistheory.com/incident/285044#d5dd56f0216021dd117cff7fb7b1b9c1fbd5c663d6a2f0b51023e6165e47d8f2 Basis Theory will be performing a scheduled system upgrade on Monday, November 13, 2023, from 01:00 UTC to 04:00 UTC. (Sunday, November 12th, 2023 from 8:00 PM ET to 11:00 PM ET) We do not anticipate any outage as a part of this upgrade, although a brief increase in latency and errors may occur for a short number of seconds. https://status.basistheory.com/incident/285044 Mon, 13 Nov 2023 01:00:00 -0000 https://status.basistheory.com/incident/285044#d5dd56f0216021dd117cff7fb7b1b9c1fbd5c663d6a2f0b51023e6165e47d8f2 Basis Theory will be performing a scheduled system upgrade on Monday, November 13, 2023, from 01:00 UTC to 04:00 UTC. (Sunday, November 12th, 2023 from 8:00 PM ET to 11:00 PM ET) We do not anticipate any outage as a part of this upgrade, although a brief increase in latency and errors may occur for a short number of seconds. https://status.basistheory.com/incident/214366 Fri, 02 Jun 2023 03:00:00 +0000 https://status.basistheory.com/incident/214366#5c09f790b6b79507c13e530b16680b1598ceb74d3b54f19ecda91ac0ea9de133 Maintenance completed https://status.basistheory.com/incident/214366 Fri, 02 Jun 2023 02:00:00 -0000 https://status.basistheory.com/incident/214366#635aaf622f9ce047507613a7481930c26149077572fa7add7577f36ea31b15b1 Basis Theory will be performing a scheduled system upgrade on Friday, June 2, 2023, from 0200 UTC to 0300 UTC. (Thursday, June 1st, 2023 from 2200 ET to 2300 ET) We do not anticipate any outage or service interruption as a part of this upgrade. https://status.basistheory.com/incident/188429 Thu, 23 Mar 2023 18:29:00 -0000 https://status.basistheory.com/incident/188429#a8739f50ccab6ab9ab36d269d9e69ede128d445a4bc8eae52745a7dcdc375b51 Between 14:50 UTC and 15:19 UTC some customers experienced intermittent errors resulting in Web Elements being slow to load for some of their users. This issue was resolved but we will be investigating further and will release an RCA. https://status.basistheory.com/incident/129595 Sat, 29 Oct 2022 15:00:00 +0000 https://status.basistheory.com/incident/129595#b3581bf999eef79c6b6fa14126052afaf82f31bb65a2418afaa7a9fdfaf2aa95 Maintenance completed https://status.basistheory.com/incident/129595 Sat, 29 Oct 2022 13:00:00 -0000 https://status.basistheory.com/incident/129595#b6d97aed5c900fb4a962ebef5439a35a5c40927e96a3479f61e017e3e06129cd Basis Theory will be performing a scheduled system upgrade on Saturday, October 29, 2022 from 1300 UTC to 1600 UTC. We do not anticipate any outage or service interruption as a part of this upgrade. https://status.basistheory.com/incident/128191 Thu, 20 Oct 2022 21:12:00 -0000 https://status.basistheory.com/incident/128191#25da18c8edf93a140370ddfb6ab0bba63753c0dc7c4011a63522c5daa9d7f7e1 # Problem Description, Impact, and Resolution At 20:05 UTC on October 20, 2022 we observed 500 errors coming from creating Tokens in all regions, which resulted in intermittent issues for some customers to create and detokenize tokens. The issue was caused by a manual configuration change during a deployment for new production infrastructure. At 20:30 UTC we identified the issue was a manual configuration change and the issue was fully resolved at 20:35 UTC. # Mitigation Steps and Future Preventative Measures We have updated our checklist for manual changes to production systems that would prevent this type of configuration error and ensure that a similar issue does not occur in the future. https://status.basistheory.com/incident/128191 Thu, 20 Oct 2022 20:34:00 -0000 https://status.basistheory.com/incident/128191#88b60501eeeecce69e68a2f45e3454020d28457773e0e54a78c272b65b369765 We saw intermittent connection issues that would have impacted creating Tokens. The issue has now been resolved https://status.basistheory.com/incident/116207 Wed, 07 Sep 2022 19:24:00 -0000 https://status.basistheory.com/incident/116207#fa9cbbec872d6527e552bde71373730efbbf0a061ab4b8c5cae373318d480d35 Problem Description, Impact, and Resolution At 16:25 UTC on September 7, 2022 we observed intermittent issues in the API and Elements services as well as in the Portal in parts of the US, which resulted in some requests from customer applications being unable to access the service in those regions. The issue was caused by an incident with a provider for our CDN. We pushed a fix to route around the provider for the API and Elements at 17:22pm UTC and saw that the issue was fully resolved for the API though there continued to be intermittent issues with Elements. We made an additional change to the Elements service to route around the issue and it was fully resolved at 18:35 UTC. Additional changes to the Portal were made to fully resolve the incident at 19:11 UTC. Mitigation Steps and Future Preventative Measures To ensure this issue does not occur again we have updated our runbook for routing around this service to ensure it can be quickly resolved for the Elements service and the Admin Portal. We will also be reviewing our dependence on this service. https://status.basistheory.com/incident/116207 Wed, 07 Sep 2022 19:24:00 -0000 https://status.basistheory.com/incident/116207#fa9cbbec872d6527e552bde71373730efbbf0a061ab4b8c5cae373318d480d35 Problem Description, Impact, and Resolution At 16:25 UTC on September 7, 2022 we observed intermittent issues in the API and Elements services as well as in the Portal in parts of the US, which resulted in some requests from customer applications being unable to access the service in those regions. The issue was caused by an incident with a provider for our CDN. We pushed a fix to route around the provider for the API and Elements at 17:22pm UTC and saw that the issue was fully resolved for the API though there continued to be intermittent issues with Elements. We made an additional change to the Elements service to route around the issue and it was fully resolved at 18:35 UTC. Additional changes to the Portal were made to fully resolve the incident at 19:11 UTC. Mitigation Steps and Future Preventative Measures To ensure this issue does not occur again we have updated our runbook for routing around this service to ensure it can be quickly resolved for the Elements service and the Admin Portal. We will also be reviewing our dependence on this service. https://status.basistheory.com/incident/116207 Wed, 07 Sep 2022 19:24:00 -0000 https://status.basistheory.com/incident/116207#fa9cbbec872d6527e552bde71373730efbbf0a061ab4b8c5cae373318d480d35 Problem Description, Impact, and Resolution At 16:25 UTC on September 7, 2022 we observed intermittent issues in the API and Elements services as well as in the Portal in parts of the US, which resulted in some requests from customer applications being unable to access the service in those regions. The issue was caused by an incident with a provider for our CDN. We pushed a fix to route around the provider for the API and Elements at 17:22pm UTC and saw that the issue was fully resolved for the API though there continued to be intermittent issues with Elements. We made an additional change to the Elements service to route around the issue and it was fully resolved at 18:35 UTC. Additional changes to the Portal were made to fully resolve the incident at 19:11 UTC. Mitigation Steps and Future Preventative Measures To ensure this issue does not occur again we have updated our runbook for routing around this service to ensure it can be quickly resolved for the Elements service and the Admin Portal. We will also be reviewing our dependence on this service. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 19:09:00 -0000 https://status.basistheory.com/incident/116186#597735dcb9626c183b640ace51155aba558b35c897cf28089fab247f84d676a7 Our cloud service provider was experiencing intermittent issues at 4:25 PM UTC. We have restored all services at 7:00 PM UTC. We will follow up with a post-mortem in the next 24 hours on status.basistheory.com. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 19:09:00 -0000 https://status.basistheory.com/incident/116186#597735dcb9626c183b640ace51155aba558b35c897cf28089fab247f84d676a7 Our cloud service provider was experiencing intermittent issues at 4:25 PM UTC. We have restored all services at 7:00 PM UTC. We will follow up with a post-mortem in the next 24 hours on status.basistheory.com. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 19:09:00 -0000 https://status.basistheory.com/incident/116186#597735dcb9626c183b640ace51155aba558b35c897cf28089fab247f84d676a7 Our cloud service provider was experiencing intermittent issues at 4:25 PM UTC. We have restored all services at 7:00 PM UTC. We will follow up with a post-mortem in the next 24 hours on status.basistheory.com. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 18:36:00 -0000 https://status.basistheory.com/incident/116186#8c482a64a3dfb3ebac85cd4d70044f494818d1b5d01f34a1b2a43c2d29f5190b Elements connectivity issues have been resolved. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 18:36:00 -0000 https://status.basistheory.com/incident/116186#8c482a64a3dfb3ebac85cd4d70044f494818d1b5d01f34a1b2a43c2d29f5190b Elements connectivity issues have been resolved. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 18:36:00 -0000 https://status.basistheory.com/incident/116186#8c482a64a3dfb3ebac85cd4d70044f494818d1b5d01f34a1b2a43c2d29f5190b Elements connectivity issues have been resolved. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 17:49:00 -0000 https://status.basistheory.com/incident/116186#62b49bff3176a8c31885d661dd12b04f6ac282a547c44db4f99d16a7acf4f03f Fix with Elements connectivity only partially resolved the underlying connectivity issues and are still seeing intermittent regional issues. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 17:49:00 -0000 https://status.basistheory.com/incident/116186#62b49bff3176a8c31885d661dd12b04f6ac282a547c44db4f99d16a7acf4f03f Fix with Elements connectivity only partially resolved the underlying connectivity issues and are still seeing intermittent regional issues. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 17:49:00 -0000 https://status.basistheory.com/incident/116186#62b49bff3176a8c31885d661dd12b04f6ac282a547c44db4f99d16a7acf4f03f Fix with Elements connectivity only partially resolved the underlying connectivity issues and are still seeing intermittent regional issues. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 16:53:00 -0000 https://status.basistheory.com/incident/116186#471c3aacb1f5c622432df2cdfbe8299eb5199e631ff120b9dc60fe5588b777db Resolved issues with API and Elements, still seeing connectivity issues with the Web Portal and are working with a downstream service provider. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 16:53:00 -0000 https://status.basistheory.com/incident/116186#471c3aacb1f5c622432df2cdfbe8299eb5199e631ff120b9dc60fe5588b777db Resolved issues with API and Elements, still seeing connectivity issues with the Web Portal and are working with a downstream service provider. https://status.basistheory.com/incident/116186 Wed, 07 Sep 2022 16:53:00 -0000 https://status.basistheory.com/incident/116186#471c3aacb1f5c622432df2cdfbe8299eb5199e631ff120b9dc60fe5588b777db Resolved issues with API and Elements, still seeing connectivity issues with the Web Portal and are working with a downstream service provider. https://status.basistheory.com/incident/93727 Thu, 09 Jun 2022 19:03:00 -0000 https://status.basistheory.com/incident/93727#494336e7f9b2cb09bfb531e34a4a9a9597682f490ef8dec3177d87048d0297aa Problem Description, Impact, and Resolution At 6:10 PM UTC on June 8, 2022 we were alerted that requests to all services began failing, which resulted in requests not resolving to the Basis Theory hostnames resulting in a DNS Resolution error for all connections to Basis Theory services. A global DNS outage at our service provider caused the incident. While the incident was ongoing, we were unable to route traffic around the affected service and the provider resolved the incident before we were able to implement the mitigation. The service was restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC. Mitigation Steps and Future Preventative Measures To ensure this issue does not occur again we have created a process and run book to route around this service and limit the impact on customers. By end of Q2 2022, we will have a long-term plan to eliminate service impacts to be completed by end of Q3 2022. https://status.basistheory.com/incident/93727 Thu, 09 Jun 2022 19:03:00 -0000 https://status.basistheory.com/incident/93727#494336e7f9b2cb09bfb531e34a4a9a9597682f490ef8dec3177d87048d0297aa Problem Description, Impact, and Resolution At 6:10 PM UTC on June 8, 2022 we were alerted that requests to all services began failing, which resulted in requests not resolving to the Basis Theory hostnames resulting in a DNS Resolution error for all connections to Basis Theory services. A global DNS outage at our service provider caused the incident. While the incident was ongoing, we were unable to route traffic around the affected service and the provider resolved the incident before we were able to implement the mitigation. The service was restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC. Mitigation Steps and Future Preventative Measures To ensure this issue does not occur again we have created a process and run book to route around this service and limit the impact on customers. By end of Q2 2022, we will have a long-term plan to eliminate service impacts to be completed by end of Q3 2022. https://status.basistheory.com/incident/93727 Thu, 09 Jun 2022 19:03:00 -0000 https://status.basistheory.com/incident/93727#494336e7f9b2cb09bfb531e34a4a9a9597682f490ef8dec3177d87048d0297aa Problem Description, Impact, and Resolution At 6:10 PM UTC on June 8, 2022 we were alerted that requests to all services began failing, which resulted in requests not resolving to the Basis Theory hostnames resulting in a DNS Resolution error for all connections to Basis Theory services. A global DNS outage at our service provider caused the incident. While the incident was ongoing, we were unable to route traffic around the affected service and the provider resolved the incident before we were able to implement the mitigation. The service was restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC. Mitigation Steps and Future Preventative Measures To ensure this issue does not occur again we have created a process and run book to route around this service and limit the impact on customers. By end of Q2 2022, we will have a long-term plan to eliminate service impacts to be completed by end of Q3 2022. https://status.basistheory.com/incident/93465 Wed, 08 Jun 2022 19:30:00 -0000 https://status.basistheory.com/incident/93465#48ebf799a6a232f103455bb4f324d47dbfe7e48446306bb5ca22ef8483b515b9 Our cloud service provider experienced a global DNS outage at 6:10 PM UTC. Services were restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC. • Basis Theory API • Basis Theory Portal • Basis Theory Elements We will follow up with a post-mortem in the next 24 hours on status.basistheory.com. https://status.basistheory.com/incident/93465 Wed, 08 Jun 2022 19:30:00 -0000 https://status.basistheory.com/incident/93465#48ebf799a6a232f103455bb4f324d47dbfe7e48446306bb5ca22ef8483b515b9 Our cloud service provider experienced a global DNS outage at 6:10 PM UTC. Services were restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC. • Basis Theory API • Basis Theory Portal • Basis Theory Elements We will follow up with a post-mortem in the next 24 hours on status.basistheory.com. https://status.basistheory.com/incident/93465 Wed, 08 Jun 2022 19:30:00 -0000 https://status.basistheory.com/incident/93465#48ebf799a6a232f103455bb4f324d47dbfe7e48446306bb5ca22ef8483b515b9 Our cloud service provider experienced a global DNS outage at 6:10 PM UTC. Services were restored in North America at 6:56 PM UTC and globally at 7:01 PM UTC. • Basis Theory API • Basis Theory Portal • Basis Theory Elements We will follow up with a post-mortem in the next 24 hours on status.basistheory.com. https://status.basistheory.com/incident/93452 Wed, 08 Jun 2022 18:40:00 -0000 https://status.basistheory.com/incident/93452#81a45a54d583f8aaa17a4f1da62c910258f4b0d67cfca04df0ba64c910a4467a At 6:39 PM UTC, Basis Theory’s cloud service provider has experienced a global outage, rendering the following services unresponsive and unavailable to customers. • Basis Theory API • Basis Theory Portal • Basis Theory Elements Basis Theory will post updates on status.basistheory.com every 15-minutes. https://status.basistheory.com/incident/93452 Wed, 08 Jun 2022 18:40:00 -0000 https://status.basistheory.com/incident/93452#81a45a54d583f8aaa17a4f1da62c910258f4b0d67cfca04df0ba64c910a4467a At 6:39 PM UTC, Basis Theory’s cloud service provider has experienced a global outage, rendering the following services unresponsive and unavailable to customers. • Basis Theory API • Basis Theory Portal • Basis Theory Elements Basis Theory will post updates on status.basistheory.com every 15-minutes. https://status.basistheory.com/incident/93452 Wed, 08 Jun 2022 18:40:00 -0000 https://status.basistheory.com/incident/93452#81a45a54d583f8aaa17a4f1da62c910258f4b0d67cfca04df0ba64c910a4467a At 6:39 PM UTC, Basis Theory’s cloud service provider has experienced a global outage, rendering the following services unresponsive and unavailable to customers. • Basis Theory API • Basis Theory Portal • Basis Theory Elements Basis Theory will post updates on status.basistheory.com every 15-minutes.