Ninja Check
Head-to-head benchmark of Ninja Scan and Web3 Antivirus on real Ethereum smart contracts
About
Ninja is a pre-transaction security API for wallets, dapps and AI agents, built by Tokyo KSC. Before a user signs, it answers two questions: is this contract safe to interact with (Ninja Scan, bytecode-level address risk scoring), and what will this transaction actually do (transaction analysis with decoded calls and risk alerts). It is exposed as REST and as an MCP endpoint so LLM agents can call it directly, and runs on 11 EVM chains today. This project is a measurement report, not a demo. We ran Ninja and Web3 Antivirus (W3A) against the same labeled dataset: 40 Ethereum mainnet transactions from 22 real incidents (Bybit, WazirX, Ronin, Euler, Wintermute, Poloniex and others), 59 benign transactions, and the 28 contracts those transactions touched. Every per-transaction verdict, the label rules, and the raw API responses are in the repo so the numbers can be checked. The headline result is not flattering to us and we publish it anyway. On historical attack transactions W3A flagged 79% and Ninja 35%, a statistically significant gap. But a control experiment shows W3A's verdicts are driven by post-incident labels on attacker addresses: the same 1 wei call to a labeled attacker address scores High 90, and to a random fresh address scores Low 10. Ninja's verdicts come from decoding what the transaction does and from the counterparty's bytecode and provenance, so they would have been identical on day zero. On the two transactions in the set that a wallet could actually have blocked, the victim-signed Safe delegatecalls in the Bybit and WazirX takeovers, Ninja returned DANGER on both while W3A returned Low 0 on WazirX. Ninja also scored all four attacker-deployed contracts HIGH or CRITICAL before any label existed, where W3A topped out at Medium. We also report where we lose: W3A's address intelligence caught 20 transactions Ninja missed, including every drain to a plain EOA, and Ninja's bytecode classifier flags 9 of 22 legitimate protocol contracts (Uniswap UniversalRouter, Safe proxies, 1inch) as HIGH. The report closes with the fixes on the table and the next experiment: synthetic transactions to unlabeled addresses that isolate mechanism from label effect.
How it's made
Ninja's transaction analysis decomposes calldata deterministically into operations (delegatecall, approve, permit, setApprovalForAll, ownership transfer, proxy upgrade) and combines that with counterparty provenance: is the target verified, how new is it, is it sanctioned. It follows proxies and EIP-7702 delegations to the code that actually executes, which is how it caught the Safe-to-unverified-implementation pattern in Bybit and WazirX. Ninja Scan sits underneath as a bytecode-level ML classifier derived from Forta's model lineage, trained on 15,443 benign and 174 malicious contracts, scoring what a contract does rather than who deployed it. Both are served over REST and an MCP endpoint. The benchmark itself is a set of shell and TypeScript scripts. The attack set comes from our internal evaluation harness of 100 labeled verdicts across 24 incidents, filtered to the 40 Ethereum mainnet transactions Ninja covers. Ninja results are stored tx_analysis responses from June 2026. W3A was called on 2026-09-27 through Scan Transaction (chainId 1, transaction hash, mode full) and Scan Contract on each transaction's destination, with every raw JSON response saved. Positives are defined per tool: Ninja signal not SAFE, W3A Scan Transaction riskGroup High, W3A Scan Contract Medium or above. Aggregation computes rates with 95% confidence intervals, McNemar's test on the paired verdicts, incident-level coverage, and a signer split (victim-signed vs attacker-sent), since only victim-signed transactions are ones a pre-signing tool could block. The hacky part worth mentioning is the label-dependency control. Instead of arguing about whether W3A was reading labels, we resubmitted the Poloniex attack transaction to W3A as unsigned parameters with the same sender and a 1 wei value, once to the real attacker address and once to a random fresh address. High 90 versus Low 10 settled it. We also ran a false-positive smoke on 29 CEX hot-wallet transfers and 30 recent-block transactions, and scored 174 destination contracts from 1,000 recent transactions to measure our own classifier's false-positive exposure, which is how we found the 45% HIGH-or-above rate on unlabeled real traffic.