From a SageMaker Random Cut Forest proof of concept to a validated multivariate approach — issue, approaches, and way forward.
The Issue
The original architecture assessment reviewed a proof-of-concept anomaly detection layer built on SageMaker Random Cut Forest (RCF), sitting on top of deterministic dbt QA checks for EMIR/MiFID II regulatory reporting. The PoC demonstrated 93% detection at a 20% multi-category shift, but the review surfaced problems across four areas that made the design unsuitable for production as proposed:
Infrastructure. Synchronous Redshift queries inside a Lambda scoring path — Redshift is built for analytical queries, not sub-second lookups, and its concurrency limits meant Lambda invocations would queue and stall under load. Unpooled RDS Postgres connections risked exhausting max_connections under bursty traffic. SageMaker Serverless Endpoints added their own constraints: a 200-concurrent-invocation cap per region and cold starts of several seconds to tens of seconds — a real risk during peak compliance-testing windows.
Statistics. To get RCF to produce usable output at all, 117 raw incident codes had to be aggregated into 23 categories — RCF’s default tree sampling degenerated on the ~160 available training samples otherwise. That aggregation diluted exactly the kind of signal the system needed to catch: a severe, isolated spike in one low-volume code gets averaged into invisibility inside its category. The detector also had a hard sensitivity floor — below roughly a 20% coordinated shift across categories, its output was statistically indistinguishable from noise, so it could catch large coordinated anomalies but not early-stage drift. Static, monthly-refit normalization meant the model went stale against non-stationary trading data well before its next scheduled retrain.
Privacy and multi-tenancy. To reach viable sample sizes quickly, the design pooled raw data across tenants — a real risk under EMIR/MiFID II/DORA/GDPR, and a leak of operational signal between competing institutions. Pooling also cut both ways statistically: it produced false positives for tenants with unusually complex operations and missed genuine anomalies for simpler ones, since both got scored against a blended, unrepresentative baseline.
Operations. There was no schema registry, so a silent change to the underlying data — a new incident code inserted, a column reindexed — would corrupt feature vectors without ever raising an error. The alert-suppression logic, tied to a dbt changelog, was race-prone and mismatched in granularity to what it was trying to suppress.
Approaches
Algorithm: away from RCF, toward multivariate statistical process control
The core mismatch was that RCF is built for high-sample, static, any-shape anomaly detection, while the actual requirement was to catch a small, coordinated shift across a vector of correlated variables, continuously, from limited samples per tenant. That’s a much older and more directly applicable field: multivariate statistical process control.
MEWMA (Multivariate Exponentially Weighted Moving Average) and its cousin MCUSUM are purpose-built for exactly this: a small, sustained shift across correlated variables, detected faster and more sensitively than a static-baseline tree ensemble. Critically, MEWMA doesn’t need the 117-into-23 aggregation that diluted RCF’s signal — it operates on the full covariance structure directly, so an isolated single-code spike and a diffuse multi-code drift can both be caught by the same family of statistic, at different levels of aggregation.
For tenants with too few historical runs to support a full-dimensional covariance estimate, Ledoit-Wolf shrinkage covariance (rather than a raw sample covariance, and rather than classical Minimum Covariance Determinant, which needs more samples than features to be well-posed) degrades gracefully as sample size shrinks.
For the multi-tenant privacy problem, the fix that emerged was covariance-only cross-tenant sharing: blend a tenant’s own covariance estimate with one built only from other tenants’ summary statistics — never raw records — while keeping each tenant’s mean strictly local (since different tenants run at genuinely different absolute rates, and blending means would blur the signal). This is architecturally similar to federated learning’s “share parameters, not data” principle, without the full complexity of an iterative federated training loop.
Architecture: async, schema-validated, and horizontally reusable across engines
The infrastructure fix mirrored the original assessment’s own recommendation: decouple ingestion from scoring via a durable, replayable event log, add a schema registry at the ingestion boundary, and move to an online feature store with a rolling baseline rather than a static monthly refit.
This was built and proven twice, on two different substrates, to confirm the pattern wasn’t tied to one vendor’s primitives:
- Redis Streams + asyncio workers — consumer groups, ack/pending tracking, and a deliberately slowed-down scoring worker to prove ingestion never blocks on scoring (backlog absorbed to 800+ pending messages, then drained without the producer ever waiting).
- Spark Structured Streaming — the same detection code, unchanged, running inside
applyInPandasWithStatefor per-tenant persistent detector state, withforeachBatchwriting to a durable audit store.
Both surfaced a real, reusable lesson: Spark’s foreachBatch sink is at-least-once, not exactly-once — a retried micro-batch will call the sink function again with the same rows. The fix is a natural-key upsert (UNIQUE(tenant_id, run_id, model_version) + INSERT OR IGNORE), not a Spark configuration flag. Any foreachBatch sink needs this or it will silently double-write.
The hard lesson: calibration is not a detail
Scaling the demonstration from a toy 24-code system to the real 117-code, 23-category scale surfaced a genuine, not-cosmetic problem: a single flat MEWMA statistic over all 117 dimensions loses power to detect a shift confined to one category, because that shift becomes a small fraction of total variance and gets swamped by noise from the 112 unaffected codes. Detection latency degraded from single-digit runs to over a hundred, sometimes missing the shift entirely.
The architecturally correct fix — one MEWMA chart per category plus a raw-level chart for isolated spikes — introduced a second, equally real problem: running 24 simultaneous statistical tests multiplies the false-alarm rate unless corrected for multiple comparisons. An uncorrected “flag if any chart trips” pushed false alarms to 70–87% on clean data.
Both problems were compounded by a third, self-inflicted issue: the synthetic data used to validate all of this had a persistent, autocorrelated drift baked into its generator. That autocorrelation broke the standard assumption (independent, identically distributed observations) behind both theoretical and empirically-bootstrapped control limits, and no amount of recalibration tuning fully fixed it — because the problem wasn’t the calibration method, it was that the test data didn’t match the method’s assumptions.
Rebuilding the synthetic data to be genuinely close to i.i.d. — a fresh random draw each run rather than a carried-over drift — combined with theoretical (chi-square) control limits and a standard Bonferroni correction across the 24 simultaneous charts, resolved this cleanly. Validated across 20 random seeds rather than a single run: false-alarm rate stable at ~1–2% (target ~1%), coordinated shifts detected within single-digit runs on average, and — the part that had been silently wrong in earlier attempts — correct category attribution 20 out of 20 times, not a plausible-looking but unreliable guess.
Way Forward
- Validate against real historical QA data, not synthetic data with an assumed correlation structure. The i.i.d. assumption behind the current calibration is a reasonable starting point, not a proven property of the real system — the only way to know whether it holds is to backtest against actual run history spanning enough time to see genuine seasonality and drift.
- Establish a recalibration cadence. Real QA metrics will drift in ways the current static reference sample doesn’t capture. A periodic (e.g. monthly) recalibration against a trailing clean window is the standard operational answer — but “clean” needs a definition (excluding known-anomalous periods) that should be agreed with the compliance stakeholders, not decided unilaterally by the pipeline.
- Decide on the multiple-testing correction’s rigor level. Bonferroni is simple and was sufficient once the underlying data assumption held, but it’s conservative; if false-alarm budget is tight, a Benjamini-Hochberg FDR-controlling procedure is worth evaluating once real data is available to compare against.
- Choose the production streaming substrate. Both the Redis Streams and Spark Structured Streaming versions proved the pattern; the real choice is an infrastructure and operations one (managed Kafka/MSK or Kinesis vs. a Spark-based platform the team already runs), not an algorithmic one.
- Formalize the schema registry as a first-class, versioned artifact (AWS Glue Schema Registry or equivalent) rather than an in-repo Python module, so schema evolution is governed the same way for every producer, not just this pipeline.
SageMaker Algorithms, scikit-learn, and PySpark: Where Each Fits
One of the more useful findings in this process was negative: none of the three major ML ecosystems available to the team had a built-in algorithm that actually matched the problem.
SageMaker built-in algorithms
SageMaker’s built-in algorithms are strong for the categories they cover — Random Cut Forest for general-purpose anomaly detection, XGBoost and Linear Learner for supervised tabular problems, DeepAR for time-series forecasting — but none of them are multivariate statistical process control charts. RCF’s use here wasn’t a wrong instinct (unsupervised, multivariate, no labeled anomalies is exactly RCF’s niche); the problem was forcing RCF’s tree-based, high-sample assumptions onto a problem that needed a small-sample, correlation-structure-aware method instead.
Worth noting for deployment planning: SageMaker Serverless Inference doesn’t support GPU compute at all, so only algorithms with CPU-capable inference containers are viable there — which includes RCF, Linear Learner, XGBoost, K-Means, PCA, k-NN, Factorization Machines, IP Insights, LDA, NTM, BlazingText, Object2Vec, and DeepAR, but excludes Image Classification, Object Detection, Semantic Segmentation, and Sequence-to-Sequence. That constraint was never actually the bottleneck in the original RCF design, though — the 200-concurrency cap and multi-second cold starts were.
scikit-learn
scikit-learn doesn’t have MEWMA or MCUSUM either — these are classical SPC methods, not mainstream ML library territory. What it did provide, and what the final working solution actually depends on, is sklearn.covariance.LedoitWolf — shrinkage covariance estimation that degrades gracefully as the ratio of samples to features gets unfavorable, which is precisely the low-sample-tenant problem this project needed solved. MinCovDet (classical Minimum Covariance Determinant) was tried first and rejected: it requires more samples than features to be well-posed, which fails exactly in the low-frequency-tenant case it needed to handle.
PySpark MLlib
PySpark’s pyspark.ml covers classification (logistic regression, tree ensembles, SVM, naive Bayes, factorization machines), regression (linear, generalized linear, survival, isotonic), clustering (K-Means, bisecting K-Means, Gaussian mixture, LDA, power iteration clustering), recommendation (ALS), and frequent pattern mining (FP-Growth, PrefixSpan) — a general-purpose distributed ML toolkit. It has no anomaly detection module, and mapping its algorithms onto SageMaker’s built-ins is lopsided at best: logistic regression and linear learner line up reasonably, decision-tree-family algorithms map loosely onto XGBoost, and a meaningful fraction of each ecosystem (RCF, IP Insights, DeepAR, and BlazingText on the SageMaker side; isotonic regression, GMM, and power iteration clustering on the Spark side) has no real counterpart in the other.
PySpark’s actual role in this project ended up being architectural rather than algorithmic: Structured Streaming’s applyInPandasWithState provided the right native mechanism for maintaining one stateful detector object per tenant across a stream of incoming events — the same job Redis Streams did by hand in the asyncio version, but using Spark’s own checkpointing and state management instead of hand-rolled consumer-group bookkeeping.
The pattern across all three ecosystems is the same: the infrastructure and general-purpose ML tooling are mature and reusable, but the actual detection algorithm — MEWMA/MCUSUM with covariance shrinkage and cross-tenant parameter sharing — had to be implemented from first principles. No built-in library call replaces that; it’s classical statistics, correctly applied, sitting on top of whichever platform’s plumbing.