Root Cause Analysis

Incident: RiskOS Evaluation API Elevated 502 Errors
Date Range: June 22, 2026, 10:23 PM EDT – 10:59 PM EDT
Impact: Customers calling the Socure RiskOS Evaluation APIs in production experienced an elevated rate of HTTP 502 (Bad Gateway) responses for approximately 36 minutes.
Status: Resolved

1. Summary

On June 22, 2026, between approximately 10:23 PM and 10:59 PM EDT, Socure’s RiskOS Evaluation APIs in the production environment experienced an elevated rate of HTTP 502 errors.

The disruption was caused by a misconfiguration in the aggregations engine — a component that enriches Evaluation API processing by computing velocity based signals across entity data. The misconfiguration was introduced during a configuration update as part of Local Graph GA readiness, and was not caught by existing validation controls. When the service attempted to use this configuration during evaluation processing an unexpected runtime exception occurred that triggered automated service restarts. As the affected services cycled, backend capacity became intermittently unavailable, causing requests during those periods to receive HTTP 502 (Bad Gateway) responses.

Socure's automated monitoring detected the elevated error rate at 10:24 PM EDT and the Engineering team began investigation immediately. The team mitigated the issue by disabling aggregations processing to stabilize traffic, remediated the invalid configuration, and validated recovery before re-enabling aggregations.

Evaluation APIs returned to normal operation by 10:59 PM EDT. Aggregations were re-enabled after validation at approximately 11:13 PM EDT.

Requests that received HTTP 502 responses during the incident window were not processed and no corresponding evaluation was created. Customers who experienced HTTP 502 responses between 10:23 PM and 10:59 PM EDT on June 22, 2026 should verify any intended evaluations and resubmit them if necessary.

2. Timeline

June 22, 2026 / Time Event 10:22 PM EDT A configuration change in Entity Aggregations introduced an invalid value. 10:23 PM EDT Elevated HTTP 502 error rates begin on RiskOS Evaluation APIs in the production environment. 10:24 PM EDT Automated monitoring detected the elevated error rate and the on-call Engineering team was notified. Investigation began immediately. 10:25 PM to 10:40 PM EDT Engineering observed multiple servers failing intermittently. The team reviewed logs and traced the failures across affected servers to isolate the root cause. ~10:40 PM EDT Engineering identified that the issue was related to an aggregations configuration affecting Evaluation API processing. ~10:45 PM EDT Engineering began targeted mitigation, initially isolating specific aggregations-dependent components. Instability persisted, and the team proceeded to disable aggregations processing entirely to stabilize traffic. 10:55 PM EDT Aggregations processing was fully disabled. Evaluation API traffic stabilized. 10:59 PM EDT Evaluation APIs returned to normal operation. Elevated 502 errors were resolved.  10:59 PM to 11:13 PM EDT Engineering identified and remediated the specific invalid aggregation configuration. Recovery was validated against the corrected configuration. 11:13 PM EDT Aggregations re-enabled following successful validation of the corrected configuration.

3. Root Cause

Primary Root Cause:

An invalid configuration was applied to the aggregations engine. When the service attempted to use this configuration during Evaluation processing, it triggered an unhandled failure path inside the processing pipeline. Rather than gracefully rejecting the invalid configuration, the failure path caused service-level crashes and the affected servers entered a restart cycle.

While the affected services were cycling, the RiskOS Evaluation APIs' available capacity to serve traffic dropped, and a portion of incoming requests reached the gateway without a healthy backend service available to respond. The gateway returned HTTP 502 (Bad Gateway) responses for those requests.

Contributing Factors:

• Insufficient configuration validation: The server did not gracefully reject the invalid Aggregation configuration at the load time. Instead of failing safely, the invalid spec propagated into a runtime failure path that crashed the affected pod, producing the customer-visible 502 responses.
• Shared processing impact: Because the issue affected shared Evaluation API processing, a configuration-specific issue resulted in elevated API failures for customers during the incident window.

Non-contributing Factors:

• Infrastructure health: No underlying cluster, network, storage, or platform infrastructure issue contributed to the incident. Compute, network, and storage layers supporting RiskOS remained healthy throughout the incident window. Once the offending configuration was mitigated, the backend service stabilized with Evaluation processing restored back to normal.

4. Resolution

Socure restored service by disabling aggregations processing entirely, which stabilized Evaluation API traffic and returned normal operation by 10:59 PM EDT. The team then identified and removed the specific invalid aggregation configuration, validated the correction, and re-enabled aggregations at 11:13 PM EDT.

Runtime safeguards have been implemented to handle invalid aggregation configurations gracefully without affecting service availability. Pre-activation validation checks currently in development will prevent invalid configurations from reaching production.

5. Corrective and Preventive Actions

Action Description ETA / Status Runtime safeguards for Aggregations Configurations Updated Evaluation API processing to handle incomplete or missing aggregation configuration values gracefully across all similar configuration processing paths, preventing service crashes. The specific condition that triggered this incident is now handled safely without impacting service stability. Completed Aggregations configuration validation Added stricter validation so invalid, incomplete, or unsupported configurations - are rejected before they can become active in production. In Progress Pre-activation configuration checks Added checks before aggregation configurations are used in Evaluation API processing to identify invalid or conflicting states earlier in the pipeline, before they can impact live traffic. In Progress Blast radius control  Strengthen isolation so an issue in a specific workflow path does not affect broader shared Evaluation API processing. Q3, 2026

6. Lessons Learned

This incident highlighted a specific gap between Aggregations configuration validation and Evaluation API runtime handling. Follow-up work is focused on preventing invalid configurations from becoming active, improving runtime safeguards, and strengthening isolation for workflow-specific issues.

7. Next Steps & Ongoing Commitment

Socure recognizes that customers rely on RiskOS Evaluation APIs for reliable, real-time decisioning. We take ownership of the disruption caused by this incident and are implementing changes to reduce the likelihood and impact of similar issues.

The corrective actions underway focus on three areas: strengthening aggregations configuration validation, improving Evaluation API runtime safeguards, and limiting the blast radius of any future configuration-related issues.  We remain committed to improving the resilience and maintaining a reliable experience for our customers.