fix(tokenizer): recognize ';' line comments for lisp/clojure/scheme/racket

[laurynas-biveinis]

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Bug fix.

• What is the current behavior? (You can also link to an open issue here)

Fixes #849. lisp, clojure, scheme, and racket are absent from comment_style() in rust/crates/cpd-tokenizer/src/generic.rs, so they fall through to _ => CommentStyle::CStyle. Their ; line comments are never recognized and are tokenized as code, so --mode weak / --skip-comments cannot drop them — e.g. the identical GNU GPL header that every Emacs Lisp file carries gets reported as duplicate code in every file pair.

• What is the new behavior (if this is a feature change)?

Route lisp/clojure/scheme/racket to the existing CommentStyle::Semicolon (their line comment is ;). --mode weak / --skip-comments now drops their comments.

Verified by building cpd and running it on two .el files sharing an identical license header but different code: Found 1 clones → Found 0 clones (exit 0). Added a cpd-tokenizer regression test lisp_family_semicolon_comment_marked_as_comment; cargo test -p cpd-tokenizer → 109 passed, 0 failed.

• Other information:

Block comments (#| |#, datum #;) in Common Lisp/Scheme/Racket are a separate, secondary concern not handled by the Semicolon style and are out of scope here. Disclosure: drafted with AI assistance and reviewed before submitting.

---

This change is 

Summary by CodeRabbit

• New Features
  • Expanded comment recognition for additional Lisp-family formats, including Lisp, Clojure, Scheme, and Racket.
• Bug Fixes
  • ;; comments are now tokenized correctly in these languages, improving syntax highlighting and parsing consistency.

[coderabbitai]

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (1)

• rust/Cargo.lock is excluded by !**/*.lock

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7b7b8ce6-f46b-46d3-af57-0b7a31044e40

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

The comment_style function in the generic tokenizer adds lisp, clojure, scheme, and racket to the CommentStyle::Semicolon match arm. A new unit test verifies that ;;-style comments in these formats are tokenized as TokenKind::Comment.

Changes

Lisp-family semicolon comment routing

Layer / File(s)	Summary
Semicolon comment routing and test rust/crates/cpd-tokenizer/src/generic.rs	Adds lisp, clojure, scheme, and racket to the CommentStyle::Semicolon match arm, and adds lisp_family_semicolon_comment_marked_as_comment test asserting ;; produces TokenKind::Comment for all four formats.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐇 A semicolon, long ignored,
For Lisp and Clojure, Scheme adored,
Now comments bloom where code once sat,
No clones from headers — fancy that!
The rabbit hops, the diff is small. 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	It clearly summarizes the main fix: recognizing semicolon line comments for the lisp-family formats.
Linked Issues check	✅ Passed	The change maps lisp, clojure, scheme, and racket to semicolon comments, directly fixing #849’s stated root cause.
Out of Scope Changes check	✅ Passed	The diff is limited to the requested tokenizer mapping and a regression test, with no unrelated changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}