feat: add TLSH similarity clustering (offsetscan cluster) — 0.3.0 - #21
Merged
Conversation
Contributor
Reviewer's GuideImplements a new Flow diagram for offsetscan cluster TLSH similarity clustering pipelineflowchart TD
A[CLI Commands::Cluster] --> B[main expand_paths]
B --> C[cluster_run]
C --> D[Read files in parallel std_fs_read]
D --> E[cluster_reads]
E --> F[Compute SHA256 sha256_hex]
F --> G[Build TLSH digest TlshDefaultBuilder]
G --> H{TLSH digest present?}
H -- Yes --> I[Store ClusterRecord with tlsh]
H -- No --> J[Store ClusterRecord with Tlsh null and warning]
I --> K[Pairwise TLSH diff and edge discovery]
J --> K
K --> L[UnionFind union connected indices]
L --> M[Group components with size >= 2]
M --> N[Deterministically assign ClusterId]
N --> O[Emit JSON or NDJSON in main]
subgraph cluster_rs
C
E
F
G
K
L
M
N
end
subgraph union_find
L
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- Consider validating/clamping the
thresholdargument inCommands::Cluster/cluster::run(e.g., rejecting negative values or values above the maximum meaningful TLSH distance) so obviously invalid inputs don’t produce surprising clustering behavior. - The
run/cluster_readspath currently reads each file fully into memory before hashing; if you anticipate very large files or corpora, it may be worth switching SHA-256 and TLSH to streaming overReadto bound peak memory per file.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider validating/clamping the `threshold` argument in `Commands::Cluster`/`cluster::run` (e.g., rejecting negative values or values above the maximum meaningful TLSH distance) so obviously invalid inputs don’t produce surprising clustering behavior.
- The `run`/`cluster_reads` path currently reads each file fully into memory before hashing; if you anticipate very large files or corpora, it may be worth switching SHA-256 and TLSH to streaming over `Read` to bound peak memory per file.
## Individual Comments
### Comment 1
<location path="README.md" line_range="197" />
<code_context>
+]
+```
+
+`ClusterId` is shared across near-duplicates and `null` for a singleton. Files too small or too
+uniform for TLSH (and unreadable files) come back with `Tlsh: null` and a `Warnings` entry
+rather than failing the run. Digests carry the `T1` version prefix, so they compare directly
</code_context>
<issue_to_address>
**nitpick (typo):** Consider adding "is" before `null` for smoother grammar.
This minor tweak improves the sentence flow while keeping the meaning unchanged.
```suggestion
`ClusterId` is shared across near-duplicates and is `null` for a singleton. Files too small or too
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| ] | ||
| ``` | ||
|
|
||
| `ClusterId` is shared across near-duplicates and `null` for a singleton. Files too small or too |
Contributor
There was a problem hiding this comment.
nitpick (typo): Consider adding "is" before null for smoother grammar.
This minor tweak improves the sentence flow while keeping the meaning unchanged.
Suggested change
| `ClusterId` is shared across near-duplicates and `null` for a singleton. Files too small or too | |
| `ClusterId` is shared across near-duplicates and is `null` for a singleton. Files too small or too |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Add a new TLSH-based corpus clustering subcommand and wire it into the CLI, backed by a pure-Rust TLSH implementation.
New Features:
offsetscan clustersubcommand for TLSH-based similarity hashing and single-linkage clustering across a file corpus, producing per-file records with TLSH digests and deterministic cluster IDs.Enhancements:
Build:
tlsh2dependency for TLSH hashing, updating the package description to mention similarity clustering.Documentation:
offsetscan clustercommand, its threshold semantics, output shape, and operational characteristics in the README, and record the 0.3.0 release and plannedoffsetscan richfeature in the changelog.Tests: