Nineteen Antivirus Labels, None of Them Right
Checking malware family tags against the actual binaries, across 1.2 million samples
The vx-underground InTheWild collection sitting on our storage box runs to 1,193,747 PE files. Nobody is reading that by hand. The only way to say anything true about an archive that size is to build a pipeline: cluster it, sample it, attribute the samples, and trust the clusters to speak for their members.
Before any of that, credit where it belongs. vx-underground is the best collection of Windows malware available to researchers, by a distance, and it exists because one person decided the field needed an open archive and then did the unglamorous work of building and maintaining it for years. Nothing in this post is possible without it. If you do this work and you haven’t sent him a thank-you, consider it.
We built that pipeline this month. It works. What it told us was not what we expected.
Clustering first, reading second
Every file in the archive gets reduced to an import-derived fingerprint before anything else happens — imphash, which DLLs it touches, which API sets it calls. UMAP folds that fingerprint space down to two dimensions, HDBSCAN finds the dense regions in it. Run against the full archive with a minimum cluster size of 100, that produced 2,668 clusters and 154,673 outliers that didn’t belong anywhere.
A cluster is a bet: everything inside it was probably built by the same tool, the same actor, or the same crypter. Some clusters are pure — every member shares one imphash, meaning one builder stamped all of them. We picked those first, 1,052 of them, one representative sample per cluster, covering 462,585 files without opening more than a thousand.
From there the pipeline is mechanical. Pull the representative binary off storage. Run it through pefile and strings. Pull URLs, IPs, registry paths, mutex names. Compare the computed imphash against what the cluster promised — if they don’t match, either the clustering broke or someone tampered with the archive entry. Write out a one-page static analysis summary per sample. That gave us 967 clusters with a real static profile behind them, ready to be identified.
Handing it to something that argues back
Identification is the part that doesn’t parallelize well by hand, so we handed it to Kimi K3 — full tool access, told to read the static analysis, cross-reference PDB paths and version strings and C2 indicators against whatever local threat intel we already had, and come back with a family name, a confidence score, and the evidence for both.
It is, credit where due, good at this. Watching it work a cluster tagged “mespinoza” is a small pleasure: it doesn’t just eyeball the strings, it finds the embedded PDB path — something like P:\Target\x64\ship\lync\ — recognizes a Microsoft internal build tree when it sees one, and returns benign with 0.97 confidence and the path as its evidence. That is not a guess. That is a receipt.
It also, less gloriously, ran into a billing-cycle rate limit partway through the first batch and returned empty responses for a while without raising an error, which is its own small lesson about API dispatchers: check the raw text, don’t trust that a clean exit code means you got an answer.
What nineteen attributions actually found
Here is the batch that finished first, family tag on the left, what Kimi actually found on the right:
| AV family tag | What it actually is | Confidence |
|---|---|---|
| mespinoza | benign Microsoft Lync / Skype for Business | 0.97 |
| amadey | Xiang Corporation crypter, not amadey | 0.85 |
| coinminer | Snojan, a GCC/MinGW UPX-packed backdoor | 0.80 |
| wannacry | StormAttack DDoS bot plus Mandaph, not wannacry | 0.78 |
| cobalt-strike | Xiang Corp crypter again, not Cobalt Strike | 0.80 |
| rhadamanthys | benign, a crypter wrapper | 0.82 |
| glassworm | benign Microsoft / Adobe plus old Chinese adware | 0.90 |
| vidar | benign Adobe Acrobat installer | 0.90 |
Zero of nineteen came back as the malware family the tag claimed. Not “mostly right.” Not “close enough.” Zero. Some of the clusters were genuinely malicious under a different name — Snojan is real crimeware, just not a coinminer. Others were entirely benign software that got swept into a malware corpus because a crypter wrapped around a legitimate installer shares import-table shape with something someone once flagged. The single biggest offender, Xiang Corporation’s crypter, shows up masquerading under at least four different family tags across our sample — amadey, cobalt-strike, rhadamanthys, and smoke-loader all turned out to be the same wrapper in different clusters.
The tag “amadey” alone accounts for 167,174 samples in the archive. If that entire cluster is actually crypter wrapper rather than banking malware, that’s not a rounding error in a research corpus. That’s the kind of mislabeling that would quietly poison any detection signature built on the assumption that the tag means what it says.
To be clear about where the fault lies: not with vx-underground. Those tags are not its own judgements. They’re the labels the samples arrived wearing, inherited from the scanners and feeds that first flagged the files, and an archive that silently rewrote them would be worse than one that preserves them. Keeping the original label is the right call even when the label is garbage — you can always re-derive a better one from the binary, and you can never recover provenance that somebody helpfully cleaned up. The interesting result here isn’t that an archive is wrong. It’s how far the entire industry’s habit of trusting a scanner’s family name drifts from what the files actually are.
Turning the same test on ourselves
It would be easy to end there and let this read as a story about how sloppy everyone else’s tagging is. It isn’t only that. Our own pipeline got fooled too, and the honest version of this writeup has to include it.
During earlier work on a separate driver sample we were calling cand-A, the import-table-overlap heuristics that make clustering possible in the first place pointed at something that looked, on paper, like a textbook bring-your-own-vulnerable-driver setup: a binary statically importing internal ordinals from cspcm4.sys, the real CrowdStrike Falcon kernel component. Ordinal-level imports into a security vendor’s own driver is exactly the shape a BYOVD evasion tool would have. We wrote it up that way.
It was wrong. cand-A turned out to be the CrowdStrike driver itself, not a separate binary abusing it — a false positive generated by the same kind of surface-level structural signal that misfiles Microsoft’s Lync installer into a malware corpus. We’ve corrected the record on our side. We’re including the mistake here because it is the same failure mode as every row in the table above, just running on our own tooling instead of a vendor’s. Structural similarity is a lead, never a verdict. It took actually reading the sample, not just fingerprinting it, to find out we were looking at something legitimate both times.
What this is actually good for
None of this means AV tags are useless — they’re often the only signal available at scale, and even a noisy label narrows a haystack. What it means is that a tag should never be the last step in an investigation, only the first. The pipeline we built exists to make that first step cheap across a million files and the second step, the actual reading, targeted enough that a human or a tool with real reasoning can do it properly instead of skimming.
The corpus keeps growing as we go — every confirmed attribution gets written back as a family signature, so the next sampling round already knows what Xiang Corp’s crypter looks like and doesn’t waste a Kimi call rediscovering it. Roughly 4,269 multi-imphash clusters are still waiting behind this first pass, and the plan is to burn through them the same way: cluster, sample, read, correct, repeat.
The next two writeups here go the other direction, from the archive down into a single sample each. One is a Turla-attributed backdoor with a C2 URL that matches independently published research byte for byte. The other is a kernel driver we’re still taking apart. Both started as one entry in a cluster just like the ones above.