Root Cause Analysis

Incident: Socure APIs Outage

Date Range: 15-Oct-2026

Impact: Interruption in service for Socure APIs

Status: Resolved

1. Summary

On 15-Oct-2026, between 18:12 and 19:00 Eastern time Socure experienced an interruption to its RiskOS and ID+ APIs, causing errors and timeouts to users. The first phase ran from 18:12 to 18:20, and was a major interruption for virtually every service. The second phase of the outage ran from 18:20 to 19:00, and was a partial outage for services that rely on KYC data, where customers saw elevated error rates and latencies for some services, but traffic was still being served.

2. Timeline

Date / Time (ET)  |  Event

18:12 | A routine cleanup change is merged, moving a source repo from one code repository to another. A bug in our review tool hides the scope of the change.

18:13 | ArgoCD restarts most services in a major Kubernetes cluster all at once.

18:20 | All service pods are running. Partial API service restored, although KYC services are still seeing an elevated error rate.

18:21 |  The flood of startup traffic overwhelms our KYC Search service. Error rates spike.

18:42 |  SRE diagnoses the cascading failure impacting KYC Search and blocks traffic to it so it can come up.

18:56 |  The KYC Search service is fully up, and SRE returns traffic to it. Downstream services stabilize. 

19:00 | All traffic running as normal, incident declared stable.

3. Root Cause

• Primary Root Cause:

As a part of an ongoing migration, an on-call SRE made a change to ArgoCD, the system we use to manage the kubernetes clusters Socure runs its services on. The change was scoped to a single cluster, was reviewed carefully before pushing, caused no issues in other environments it was tested in, and should have had no visible impact. Unfortunately, the change updated a legacy setting that did not appear in the change review. That legacy setting governed what services ArgoCD managed, and removed many live services from ArgoCD. In consequence, ArgoCD removed those services from production briefly, before the setting was corrected.

• Contributing Factors:

The KYC Search service is a backend piece of Socure infrastructure that provides services to several of our internal products. When that service restarted from scratch at the same time as all of the clients of the service did, a small number of pods in the service crashed under load. This increased the load on the remaining pods, some of which crashed in turn. This cascading failure prevented the service from coming back on its own the way the rest of Socure’s infrastructure did. Diagnosing this separate failure mode took additional time, and remediating it fixed the remaining errors after the initial incident.

4. Resolution

The ArgoCD change was reverted immediately, causing services to come back to their prior state. The KYC Search service required manual effort to block traffic from reaching it until a majority of pods in the service could come up. After the service was healthy, we could restore traffic to it and resolve the overall incident.

5. Corrective and Preventive Actions

6. Lessons Learned

• Socure’s tooling for viewing the changes it makes to production did not work correctly. We are reviewing these tools to ensure they operate correctly under all use cases and fix any cases like this where the full scope of a change is not clear.
• Augment the existing change review process with agentic support. While in this case, the specific change was not visible in the plan, having an extra reviewer that is supported by an extensive training model will help ensure issues are caught before submission.
• Even changes that appear to have no impact, when they are made to the governance system for our production jobs, can be risky, and should be shifted further out of US business hours.
• Our KYC Search service is vulnerable to a “thundering-herd” problem when it receives full traffic flow when it is not fully online. We need to strengthen our circuit breakers to prevent this issue recurring.
• We run too many of our core API services in a single kubernetes cluster. Even though that cluster is distributed across multiple availability zones, the shared control plane creates too high a blast radius for changes to that cluster.

7. Next Steps & Ongoing Commitment

Socure understands our customers rely on us as a key provider in critical business flows, and that this service interruption is not acceptable. We apologize for the impact this issue caused, and commit to ensuring it does not recur. To that end, we will deliver details on our plans to re-architect our application layout by November, as well as improve the resilience of the KYC Search service in Q1 of next year.