How well does it detect?
Frozen external confirmation on PhreshPhish, followed by the earlier revised exploratory PhiUSIIL benchmark. Each result has its own scope and limitations.
Verify inference on this device
Run 600 retained Python reference vectors against the same local model. This checks score arithmetic and model-threshold decisions; it does not measure phishing accuracy. Only fixed model/reference files are downloaded.
A separate dataset. The same model.
PhreshPhish test records collected September–December 2025. The model, features and thresholds were committed before evaluation. No retraining or threshold tuning used this cohort.
The external false-positive rate rose above the 2% validation budget. A low score cannot certify safety. The model threshold and the application’s review/high policy have different tradeoffs.
| Frozen model / archived v3 policy | Precision | Recall | F1 | FPR |
|---|---|---|---|---|
| Model threshold | 91.41% | 65.43% | 76.27% | 4.84% |
| Review or high | 70.21% | 83.77% | 76.39% | 27.98% |
| High verdict only | 100.00% | 3.71% | 7.16% | 0.00% |
Policy update: these application-policy rows describe archived evidence-v1, not the current evidence-v2 release. Current URL-only analysis never issues a high verdict; observed submitted HTTP password exposure can produce a separate data-exposure warning. The model and its operating-threshold measurements are unchanged.
“Review” asks the user to investigate; it is not a phishing accusation. High-only verdicts are rare and miss most phishing. Observing zero high-only false positives here does not establish a zero real-world error rate.
External matrix, coverage & uncertainty
PR-AUC: 0.8805 · Model matrix [[TN, FP], [FN, TP]]: [[70318,3576],[20107,38053]]
Recall 95% domain-cluster interval: 63.53% – 67.69%
From 168,060 source rows: 35,214 excluded for domain overlap with any original-source row, 776 outside model scope, 4 invalid and 12 duplicate URLs. Retained: 58,160 phishing and 73,894 benign.
Source labels are inherited, not newly adjudicated by this project. Language metadata describes saved pages; no sufficient Arabic cohort is established. HTML/brand additions remain unmeasured.
PhreshPhish · CC BY 4.0 · Download results · Frozen protocol
Earlier exploratory benchmark
Baseline comparison
Positive class = phishing. Each model uses its own validation-selected threshold; comparisons are not at identical false-positive rates.
| Model | Precision | Recall | F1 | FPR | PR-AUC |
|---|---|---|---|---|---|
| Original rules | 95.69% | 54.34% | 69.31% | 1.97% | 0.7234 |
| Shape Logistic Regression | 91.67% | 28.18% | 43.11% | 2.06% | 0.7249 |
| Shape Random Forest | 92.39% | 32.63% | 48.23% | 2.16% | 0.7582 |
| Character Logistic ★ | 95.38% | 50.93% | 66.40% | 1.98% | 0.9080 |
The model misses many phishing cases at this threshold. Source labels describe URLs, including redirect links on legitimate hosts, so domain associations can falsely flag familiar domains. The model alone never produces the interface’s high-risk verdict. It is not a standalone security control.
Model selection: maximize validation recall with a validation FPR budget of 2%; the held-out FPR may differ. Scores are uncalibrated.
Confusion matrix
| Actual / Predicted | Legitimate | Phishing |
|---|---|---|
| Legitimate | 20,143 | 407 |
| Phishing | 8,101 | 8,407 |
95% domain-cluster bootstrap interval for phishing recall: 40.63% – 62.23%
Experiment record
| Partition | URLs | Domains | Phishing |
|---|---|---|---|
| train | 163,467 | 137,918 | 69,073 |
| validation | 33,618 | 29,554 | 13,713 |
| test | 37,058 | 29,554 | 16,508 |
Zero overlapping registrable domains. Canonical duplicates and conflicting labels are removed before splitting. Group split seed: 20261020
Version: linklens-domain-char-3.0.0 · 14 domain-shape features · 286.7 KiB
Artifact identity & selection details
SHA-256: 241259632fdf6f487613835acebbbe3578cc5614ab1f7ac1389492b0de61d077
Maximize validation recall at FPR <= 2%; follow-up character candidate selected on validation only. Corpus was previously explored; this is not an independent final confirmation.
300 domain-cluster bootstrap replicates, seed 20261020
Read these limits with the numbers
- The PhiUSIIL results below remain exploratory. The separate frozen PhreshPhish confirmation is reported above; neither proves universal deployment safety.
- Domain-group random split; no trusted collection timestamps for a temporal test.
- IDN / internationalized domains abstain; no sufficient labeled Arabic subset validates model accuracy.
- HTML and brand checks are contextual rules, not a separately validated hybrid model.
- The UI may elevate a URL-only verdict to review based on rule findings. These metrics evaluate classifier thresholds, not the interface’s corroboration policy. Model-only anomalies are shown as review; high verdicts in the current release require uploaded evidence of submitted HTTP password exposure.