How Detection Works

jscpd is a token-based detector. The important word is token: a clone is a repeated sequence of language tokens, not a repeated run of characters. The tokens come from each language's own syntax, so what counts as a comment, a string, a keyword or an identifier is decided per language before any matching happens. This page walks through the pipeline.

1. Format detection

The file extension picks one of the 224 formats. Extensionless files such as Makefile or Dockerfile are mapped with --formats-names, and unknown extensions with --formats-exts. The format selects the tokenizer and, for cross-format groups, the pool of files a clone may span.

2. Tokenization

Every format is read with its own lexical rules:

• Comments. // and /* */ for the C family, # for Python, Ruby, shells, YAML and friends, -- for SQL, Haskell and Ada, --[[ ]] for Lua, ; for Lisp dialects and INI, ' for Visual Basic, and none for Markdown. A # inside a JavaScript string is not a comment, and a /* inside Markdown prose does not open one that never closes.
• Literals. String and numeric literals are recognized as single tokens, so a comment marker or a brace inside a string cannot break the stream.
• Classification. Each token is an identifier, a literal, punctuation, a comment or whitespace. That classification is what the normalization options act on later.

JavaScript and TypeScript

JavaScript, TypeScript, JSX and TSX are not handled by the generic rules. They go through the oxc parser, the one behind oxlint. Template literals, regular expressions, JSX elements and decorators come out as the tokens the grammar defines. A recoverable parse error, a redeclaration for instance, leaves the token stream intact, so such files still match files that parse cleanly. Only a file the parser gives up on entirely falls back to a word-split tokenizer.

With --cross-formats the parser also erases TypeScript-only syntax from the detection stream: type annotations, interfaces, type aliases, generics and access modifiers are removed via the syntax tree, so a .ts file matches its plain-JavaScript equivalent. Constructs with runtime meaning stay, among them enum, non-declare namespace and parameter properties, because removing them would change what the code does.

Embedded languages

Some files are several languages at once, and jscpd splits them before tokenizing:

Clones inside a block are reported with the block's own line range in the original file, and a <script lang="ts"> block matches .ts files. See the cross-format benchmark for what this finds in practice.

3. What counts

Not every token takes part in matching. --mode mild, the default, drops whitespace tokens; --mode weak also drops comments, so duplicated comment blocks no longer count; --mode strict keeps every token. jscpd:ignore-start / jscpd:ignore-end comments exclude a region, and --ignore-pattern regular expressions exclude whatever they match, a license header for example. Skipped tokens leave the stream without shifting the positions reported for the remaining ones.

4. Normalization

The classification from step 2 is what makes Type-2 clones detectable:

• --ignore-identifiers hashes every identifier as the same placeholder, so total(items) and sum(rows) produce the same tokens. Keywords keep their value: the oxc token kinds tell them apart in JavaScript and TypeScript, and a shared keyword table does it for other languages, so return never matches retry.
• --ignore-literals does the same for string and numeric literals.
• --ignore-annotations drops @Name and @Name(...) sequences, but only in languages where @ introduces an annotation or decorator: Java, Kotlin, Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript. In Ruby, Perl, T-SQL, Razor and CSS, @ means a variable or a directive and is left alone.

Clones found this way are reported as renamed, with the exact-match fingerprint kept separately so the default report is unchanged.

5. Matching

A rolling Rabin-Karp hash slides over the token stream and finds every repeated window of at least --min-tokens tokens spanning at least --min-lines lines. Files of one format share a pool, and --cross-formats groups merge pools, which is how a Vue script block can match a TypeScript file. Detection runs in parallel across pools.

6. Near-miss passes

Two opt-in passes reach into Type-3 territory:

• --max-gap-lines N merges clone pieces of the same file pair that are separated by at most N unmatched lines into one similar clone with a similarity score, which catches a copy where a few lines were inserted or changed.
• --similarity RATIO extracts every JavaScript and TypeScript function from the syntax tree and compares the sequence of node types inside it. Names, literal values and small edits do not change the sequence much, so two functions with the same shape match even when the token stream does not.

What it does not do

jscpd does not analyze what the code means. Two functions that compute the same result with different statements, Type-4 clones, are out of scope, as they are for every token-based detector. What the language-aware tokenization buys is precision within that scope: fewer false positives from comments and strings, correct handling of embedded and typed code, and normalization that knows a keyword from a name.