Incident: RiskOS Delayed Webhook Deliveries
Date Range: June 25, 2026 2:06 PM ET to June 26, 2026 6:03 AM ET
Impact: Customers subscribed to webhook events experienced delayed webhook deliveries during the incident window. Approximately half of webhook events were delayed; no webhook events were lost. The webhook processing pipeline auto recovered without manual intervention, and no customer action was required.
Status: Resolved
1.Summary
Between approximately 2:06 PM ET on June 25, 2026 and 6:03 AM ET on June 26, 2026, customers subscribed to RiskOS webhook events experienced delayed webhook deliveries due to an unexpected degradation in the webhook processing pipeline. During the impact window, approximately half of webhook events were delayed. No webhook events were lost.
The processing backlog did not surface within the webhook workers until 5:06 AM ET on June26. Once it became available, the queued webhook events were processed, and webhook delivery latency returned to normal at approximately 6:03 AM ET. The pipeline auto-recovered without any manual intervention.
Subsequent log and metric analysis identified an issue in the upstream message consumption layer as the primary contributor: following an infrastructure scale-up, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. As a result, webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition self-resolved when consumer group membership stabilized, at which point the queued events were processed quickly and deliveries returned to normal.
Standard Kafka topic-lag monitoring did not surface the condition in a timely manner because aggregate topic lag remained within thresholds while the affected partitions accumulated work in the background. Socure engineering has identified a set of corrective and preventive actions — described below — to harden the webhook processing pipeline against partition rebalance failures and to improve observability of partition-level consumption health. As an immediate preventive action, an additional alert that detects this class of failure has already been deployed.
2. Timeline
| Time (ET) | Event |
|---|---|
| Thu, Jun 25 — 2:06 PM | Webhook delivery for a subset of events starts getting impacted. This was following an infrastructure scale-up performed, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. Webhook events on those partitions begin to accumulate without being delivered. Standard Kafka topic-lag monitoring does not surface because aggregate topic lag remains within thresholds. |
| Thu, Jun 25 2:06 PM – Fri, Jun 26 – 5:06 AM | Webhook delivery latency remains elevated for approximately half of webhook events. Events on unaffected partitions continue to be delivered normally. No webhook events are lost. |
| Fri, Jun 26 — 5:06 AM | The processing backlog surfaces within the webhook workers. Consumer group membership stabilizes, the affected partitions begin to be consumed, and the webhook workers report a steep increase in lag as they begin processing the queued events. |
| Fri, Jun 26 — 5:06 AM –6:03 AM | Queued webhook events are processed rapidly. Webhook delivery latency returns to normal at approximately 6:03 AM ET. No manual mitigation is required. |
| Fri, Jun 26 — 10:00 AM | On-call engineering reviews telemetry from the overnight processing spike and correlates it with customer reports of delayed webhook deliveries. |
| Fri, Jun 26 — 11:57 AM | Incident formally declared in Socure’s incident management system as a retrospective incident. Incident response team assembled and investigation bridge convened. Telemetry for the webhook processing pipeline reviewed in parallel. |
| Post Incident | Detailed log and metric analysis of the webhook processing pipeline. Primary contributor identified as a Confluent Kafka |
3. Root Cause
Primary Root Cause:
Following an infrastructure scale-up performed Thu, Jun 25 02:06 PM, the Kafka consumer group that feeds Socure’s webhook processing pipeline did not rebalance correctly. As a result, a subset of Kafka partitions was effectively unassigned for consumption, and webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition was transient: when consumer group membership stabilized at approximately 5:06 AM ET on June 26, the affected partitions resumed consumption, the queued events were processed rapidly, and webhook delivery latency returned to normal at approximately 6:03 AM ET. No webhook events were lost.
Contributing Factors:
Non-contributing Factors:
4. Resolution
Service restoration: The condition was transient and self-resolved when the affected Confluent Kafka consumer group’s partition assignment stabilized at approximately 5:06 AM ET on June 26, 2026. At that point, the previously unassigned partitions resumed consumption, and the webhook workers processed the queued events rapidly. Webhook delivery latency returned to normal at approximately 6:03 AM ET. No manual mitigation was required. Socure engineering verified recovery through telemetry and continued monitoring through and beyond the incident window.
Validation and verification: A post-incident review was conducted to confirm the scope and duration of impact. Detailed log and metric analysis identified the Confluent Kafka consumer group rebalance failure as the primary contributor, occurring in the context of an infrastructure scale-up performed the prior day. As an immediate preventive action, Socure engineering deployed an additional alert that monitors divergence between message production rate and message polling (consumption) rate over a time period — designed specifically to detect scenarios where partitions are not being consumed even when topic-aggregate lag does not surface the delay. Additionally, we have now added a safety net for watching partition updates for ensuring the partition assignments.
5. Corrective and Preventive Actions
| Action | Description | ETA / Status |
|---|---|---|
| Partition-level consumption monitoring | Deploy an additional alert that monitors divergence between message production rate and message polling (consumption) rate over a time period, complementing existing Kafka topic-lag monitoring to detect partition-level consumption gaps that aggregate lag alone does not surface. | Completed |
| Consumer group handling partitional updates | We have implemented application-level event listeners to dynamically monitor Kafka partition state changes. In the event of a recurring Kafka broker or partition failure, the application will now automatically detect the disruption and execute a self-healing recovery sequence. | June 26th, 2026 |
| Scale-up runbook hardening | Update the infrastructure scale-up runbook for the webhook processing pipeline to include explicit verification of Kafka consumer group rebalance completion and partition assignment health before the scale-up is declared complete. | mid-Q3, 2026 |
6. Lessons Learned
This incident exposed two specific gaps in the webhook processing pipeline. First, infrastructure scale-up operations can leave the Confluent Kafka consumer group in an incomplete rebalance state, with a subset of partitions effectively unassigned, without that condition surfacing to operators in real time. Second, Kafka topic-aggregate lag is not a sufficient indicator of partition-level consumption health when only a subset of partitions is affected, because lag on actively consumed partitions can mask accumulating lag on unassigned ones. Follow-up work is focused on detecting and preventing partition-level consumption gaps, hardening the scale-up runbook for the webhook processing pipeline, and strengthening observability so that rebalance anomalies are surfaced as they occur rather than after they have already produced customer-visible impact.
7. Next Steps & Ongoing Commitment
Socure takes accountability for this incident and the impact it had on the affected customers. We remain committed to improving the resilience and maintaining a reliable experience for our customers.
The corrective actions underway focus on three areas: strengthening partition-level consumption monitoring for the webhook processing pipeline so that rebalance-related consumption gaps are detected as they occur, identifying and remediating the underlying cause of the Confluent Kafka consumer group rebalance failure, and hardening the infrastructure scale-up runbook to verify consumer group health and partition assignment before any scale-upis declared complete. We remain committed to improving the resilience and maintaining a reliable experience for our customers.