Tuning a fraud threshold without drowning your analysts
The setting with perfect precision looked like the safe choice. It quietly missed two of the seven accounts that needed review.
The setup
A payments platform sees a possible fraud spike. The lab models it with 60 synthetic accounts, 3,771 transactions and 64 devices. SQL features feed a risk score per account: distinct inbound senders in 24 hours, rapid outbound movement, new foreign devices and shared-device links. Seven accounts are seeded as review-worthy. The review team can work about eight alerts a day.
The question a fraud lead actually faces: where do we set the alert threshold?
What each threshold buys you
| Threshold | Alerts | Precision | Recall | Capacity |
|---|---|---|---|---|
| 20 | 10 | 70.0% | 100% | Over |
| 30–40 | 8 | 87.5% | 100% | Within |
| 50+ | 5 | 100% | 71.4% | Within |
At 50 and above, every alert is a true positive. On a dashboard that reads as a clean, efficient control. But two of the seven review-worthy accounts never reach an analyst, and nobody sees the ones you didn't alert on.
Precision tells you how much analyst time you waste. Recall tells you how much risk you let through. A fraud control has to be judged on both, and on whether the team can actually work the queue.
Why I'd pick 40
At 40 the queue is eight alerts, exactly the team's daily capacity. It catches all seven review-worthy accounts at the cost of one benign lookalike. Dropping to 20 adds two more false positives and pushes the team over capacity. Once a queue overflows, real cases wait behind noise.
The lookalike is also useful. It shares surface features with the fraud cluster, so reviewing it teaches the team what a false positive looks like here. That's input for the next round of feature work.
What I'd check before shipping it
- Stability: does 40 still fit capacity in a heavier week, or does it need a volume-based guardrail?
- Single signals: a shared device is a lead, not proof. Several legitimate users can sit behind one household device or a business network.
- Label maturity: recall is only as good as your labels. Some fraud surfaces weeks later as chargebacks.
- Monitoring: track alert rate and data quality weekly, so a broken feed doesn't look like a quiet week.
Try it yourself. Move the threshold slider and watch precision, recall and the queue change.