Fix four unicode-spoofing false-positive classes - #16
Merged
Conversation
TOKEN_RE deliberately admits U+2019 so contractions tokenize whole, but
that single character was enough to fail the ASCII gate on both
confusable_word paths. The UTS #39 skeleton then folds it back to "'",
so "I’ll" resolved to "l'll" — plain ASCII with letters — and reported
as a disguised word. Every contraction typed with a smart quote
("don’t", "it’s") was a false positive, which is any text that has been
through a phone keyboard or word processor.
Judge a token's ASCII-ness against its straight-apostrophe form. The
exemption covers punctuation only, so letters still decide: "Ivа’s"
with a Cyrillic а is still caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
U+FFFD was classified as `illegal`, so a message whose name field had been
mangled upstream reported as a spoofing attack. On a 115-row sample of real
SMS traffic that was 70 rows — 61% of every flag — all of them one broken
merge field ("Jos Luis", "Jess") in a plumbing company's appointment
reminders.
U+FFFD cannot evidence intent. A decoder emits it when handed bytes it cannot
read, and those bytes are already destroyed by the time it exists, so no
payload survives inside it. That is what makes it safe to exempt from
`spoofed` without opening a bypass — unlike mojibake fingerprinting
("looks like broken UTF-8"), which an attacker could imitate on purpose to
suppress detection, and which is deliberately NOT implemented here.
The signal is reported, not hidden: `signals.encoding_damage` and the word
findings still fire, so data-quality alerting keeps working. It just does not
assert an attack. The normalizer also leaves U+FFFD in place — stripping it
would repair a corrupted message into one that reads as intact.
C1 controls, NUL and non-characters stay `illegal`: those are real bytes that
survived, so they remain attacker-controllable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a 48-row sample of real SMS traffic, 17 were one company's template with two zero-width spaces sitting between a colon and a line break, left behind by a rich-text editor. A zero-width character in that position cannot split a word — the break is already there — so it changes nothing about how the text renders or tokenizes, and reporting it read as accusing a sender of attacking their own recipients. The rule requires whitespace or a string boundary on BOTH sides, and that is the whole of its safety. My first attempt accepted a break on EITHER side and the existing suite caught it immediately: `admin<ZWSP>` and `a = <ZWSP>= b` stopped flagging. Those are not inert — a trailing zero-width renders as the bare word while comparing unequal to it, which passes an exact match, and the same goes for `<ZWSP>Valencia` glued to the front of a token. Length is the second limit: every position in a run carries at least a bit, so past ZERO_WIDTH_INERT_RUN (4) a run is a payload channel wherever it sits. Scoped to characters that are zero-width AND non-reordering. Bidi controls reorder the line, tag characters carry a payload, and blank glyphs occupy space, so none become inert next to a space — Trojan Source and ASCII smuggling are untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"Ayalaæs Lashes" reported as a disguised word, and normalized to "Ayalaaes". The message was a corrupted "Ayala's" — a codepage swap had put æ where the apostrophe belonged — but nothing about it was a disguise, and our correction made it worse. The cause is that a Latin word was being skeleton-tested against Latin. There is no cross-script evidence to find there, so all the test measured was whether the fold happened to land in ASCII. It does for ordinary European letters, and it does so arbitrarily: æ is a ligature so UTS #39 dissolves it to "ae", while ø keeps its stroke as a combining mark and never gets there. Ægir and Þór were reported; Ålborg and Straße were not. Coin flip, decided by table shape. Identifier_Status is the distinction Unicode defines for exactly this question. æ ø å ß þ œ ı and the Hawaiian ʻokina are Allowed — letters of living alphabets. IPA alpha and the fi ligature are Restricted. Requiring a Restricted character keeps intra-Latin homoglyphs like "pɑypal", which mixed_script and the cross-script path are both blind to, while clearing every Nordic, Turkish, German, French and Hawaiian word we were indicting. Scoped to Latin words deliberately. Cyrillic а and о are Allowed as well — they are ordinary Russian — so a universal gate would have let раураl through. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four false-positive classes surfaced by scanning ~780M rows of real SMS traffic for one org. Each is a distinct root cause with its own commit, changeset, and tests. On a 115-row sample of flagged messages, zero were actual spoofing attempts — every flag was ordinary text, broken text, or benign editor cruft.
The fixes
1. Curly apostrophes (
e83da95) —I’ll,don’t,it’sall flagged.TOKEN_REkeeps U+2019 inside words so contractions tokenize whole, but that one character failed the ASCII gate and its skeleton folds back to', soI’llresolved tol'll— a "disguised" ASCII word. Any text through a phone keyboard hit this. Now judged against the straight-apostrophe form;Ivа’swith a Cyrillicаis still caught.2. Decode damage (
4ae1dc1) — U+FFFD counted asillegal, so a mangled merge field (Hi Jos�� Luis) reported as an attack — 61% of all flags. U+FFFD is a decoder's output, never an author's input; the bytes are already destroyed, so no payload survives. Newencoding_damagesignal is reported insignals/wordsbut does not setspoofed, and the normalizer leaves it in place rather than silently "repairing" it.3. Whitespace-isolated zero-width runs (
d588538) — a ZWSP evades filters by splitting a word (fr<ZWSP>ee); with whitespace on both sides it splits nothing. Rich-text editors leave these in templates constantly. Now inert only when isolated on both sides and short —admin<ZWSP>and<ZWSP>Valenciastill flag (they render as the bare word but compare unequal), and runs pastZERO_WIDTH_INERT_RUN(4) always flag.4. Latin words judged on Identifier_Status (
537a8dc) — a Latin word can't impersonate Latin, so the skeleton test only measured whether the fold reached ASCII — arbitrarily.æis a ligature →ae(flagged);økeeps its stroke as a combining mark → never ASCII (clean).Ægirindicted,Ålborgcleared. Now a Latin word needs a UTS #39 Restricted character first:æ ø å ß þ œ ı ʻare Allowed and pass, whilepɑypal(IPA alpha) andfirst(ligature) are Restricted and still caught. AddsIdentifierStatus.txtas a second generated table.Impact on the sample
115 flags → 31, all remaining ones correctly-classified broken or genuinely-suspicious text (mojibake, C1 controls, one-sided zero-width, styled-font names).
Notes
minorchangesets.SpoofSignalgainsencoding_damageand three constants are exported (SPOOFING_SIGNALS,ZERO_WIDTH_INERT_RUN) — a stricttoEqualon the signals object or an exhaustiveswitchwill need a line.tscand eslint clean.𝖎𝖓𝖆𝐣) flags via the styled path — same family as fullwidth-romaji-in-CJK. Left for a separate decision.🤖 Generated with Claude Code