The problem, precisely
A client wants to know whether their data — corporate domains, employee emails, API keys, customer card numbers — has leaked onto Tor hidden services or is being sold on a criminal marketplace. Commercial platforms answer that question as a service, which means your watchlist — a list of exactly what you are most afraid of losing — lives on someone else's infrastructure. dwcollector answers it as infrastructure the client runs and controls.
Evidence-only by design
It stores the fact of a match plus roughly sixty characters of surrounding context — never the leaked dataset itself. Secrets and PII are masked at the moment of capture, not at display time. If the monitoring system is ever compromised, it holds no leaked data to lose. That constraint shaped every other decision in the tool.
The clone problem
The dark web is thick with phishing mirrors — pixel-perfect copies of real marketplaces built to steal deposits. A naive crawler that finds a client's email on some onion that merely looks like a known forum produces an alert an analyst cannot act on, because nobody knows whether that site is real or a scammer's replica. Solving this is the architectural spine of the whole tool.
Fingerprinting and clustering
Every live page gets a 64-bit simhash over word tokens. Pages within a Hamming distance of 3 are unioned into clusters using union-find, so single-linkage transitivity holds: if A matches B and B matches C, all three cluster even when A and C are not directly similar. Pages with an empty fingerprint are deliberately never unioned, which prevents every untokenizable page from collapsing into one bogus cluster.
Trust flows only from anchors
Each cluster elects a canonical member by strict precedence: an operator allowlist wins; failing that, a page whose scraped PGP fingerprint matches a known-good key; failing that, the earliest seen; last, the most inbound links. An allowlisted canonical becomes verified. A PGP-matched canonical becomes probable. Anything elected by the fallback routes becomes unknown — explicitly not trusted. Every non-canonical member becomes a suspected clone. The safety property this buys: a phishing clone whose genuine original was never crawled sits alone, becomes its own canonical by first-seen, and lands at unknown. It cannot be mistaken for legitimate merely by standing alone.
The safety cutout
Every cycle, each trusted anchor is re-fetched and — for PGP-anchored sites — its advertised fingerprint re-checked. On mismatch the code bails before that URL is ever added to the covered set. That ordering does two jobs at once: the client's watchlist is never transmitted to a hijacked or impersonated site, and existing findings stay open rather than falsely resolving.
Matching and validation
Five term modes: literal, regex, defang-aware fuzzy (bracket notation, dot and at spellings, zero-width characters, Cyrillic homoglyph folding), encoded (base64, hex, percent), and credential-pair — which records the identifier and masks the secret. PII detectors validate rather than pattern-match: Luhn for cards, mod-97 for IBANs, area/group/serial rules for SSNs. Each produces one finding per site per detector with an occurrence count, so a dump of ten thousand cards does not generate ten thousand alerts.
The analyst layer
Findings carry a machine-owned status and an analyst-owned disposition, and no reconciler ever writes the latter. Detection is the machine's job; judgment is the analyst's. Dispositions are sticky across the open and resolved lifecycle, so a finding dismissed once stays dismissed when it reopens months later. Above that sit triage queues, incident cases with append-only timelines, and per-client reports.
Engineering notes
All egress goes through a containerised Tor SOCKS5 proxy — there is no clearnet path anywhere in the code, including seed harvesting. The crawler recovers from being killed mid-flight by resetting stranded rows on startup, which is what lets it survive OOM and reboots in a 24/7 loop. Alert emission is wrapped per-sink so no webhook or database failure can propagate into the supervisor loop.