DETECTSecurity & safety
Scam Signal Lens
Danny Livshits@dannylivshits
0:12A proof-of-concept tool for classifying suspicious emails as scams, utilizing zero-shot classification with a configurable policy and provider API key for private data processing.
TakeWhat it does - Runs zero-shot style scam-signal / concern classification with a fixed policy.
I was exploring Jev by @typesafeai AI for safety use cases and it is an impressive and useful model, however don't expect it to magically auto classify any data without proper parameters and thresholds (see finding below). I open-sourced a scam-email classification where you can use your own API key to try it.
What I built - a proof of concept for suspicious email classification aimed at scam emails: Scam Signal Lens. Open-source evaluation tooling so teams can run and inspect concern-policy analyses instead of staring at a black-box score.
What it does - Runs zero-shot style scam-signal / concern classification with a fixed policy. With a provider API key (TYPESAFE_API_KEY) it can run live private captures, write private output outside the public repo, and later promote reviewed records into the static demo. The browser/GitHub Pages build deliberately has no key field and no live inference path.
What it does not do - It does not authenticate senders, inspect destinations, determine whether a message is safe, replace a production detector, or guarantee scam blocking.
What data I used
(a) AI Email 200 - synthetic English data created by a large language model (LLM): 200 rows, with 100 dataset-author benign labels and 100 phishing labels.
(b) SpaPhish v5 public projection - a Spanish phishing/benign email dataset: 499 published rows (250 upstream benign / 249 upstream phishing after one documented post-capture exclusion), projected from a 500-row source capture.
(c) SpaPhish v5 - with Spanish questions as input for Jev (vs English in option b) and lower threshold
How well it did
- AI Email 200: precision 100.00%, recall 92.00%
- SpaPhish v5 public projection: precision 93.48%, recall 17.27%
- SpaPhish v5 Spanish questions + YES=0.70 : precision 90.62%, recall 23.29%
Interesting finding: SpaPhish Alert-recall was very low (~17%) while precision stayed high, we only count Alert, and most Spanish phishing stopped at Verify first under English questions and YES≥0.8. I validated the fix hypothesis with a third run (Spanish questions + YES=0.70): Alert phishing rose 43→58 and recall ~17%→~23%, with a small precision dip (~93%→~91%). Most phishing still stayed in Verify first (~165), so the English recall gap (~92%) remains. Takeaway: localizing questions and retuning thresholds helps a bit; it doesn’t close the full gap. The Spanish recall remained lower as the classifier needed to clear 5 rules (login credentials ask, fee/refund trap, ask for payment, ask for remote access, asks for sensitive data), the Spanish data mostly scored on (authority, urgency) which were softer signals, so different data patterns can benefit from different screening rules in zero shot classification.
What the live demo shows
The public demo replays those checked-in recordings only: Replay classifications case by case, Data samples, Results (precision/recall graphs), and Methodology.
Link to demo and source code in the first comment below,
Related builds
Vectorless RAG with Jev
This project introduces a novel approach to Retrieval Augmented Generation (RAG) that replaces traditional embedding methods with a structured, table-of-contents-based navigation system, significantly improving speed and reducing costs by offloading non-generative tasks to Jev.
