diff --git a/docs/indexing/fts-index.mdx b/docs/indexing/fts-index.mdx index 400e07f..a3b33f4 100644 --- a/docs/indexing/fts-index.mdx +++ b/docs/indexing/fts-index.mdx @@ -92,7 +92,7 @@ await async_table.create_index("payload.text", config=FTS(with_position=True)) | `stem` | bool | `True` | Apply stemming (`running` → `run`) | | `remove_stop_words` | bool | `True` | Drop common stop words | | `ascii_folding` | bool | `True` | Normalize accented characters | -| `custom_stop_words` | list[str] | `None` | Extra stop words to drop in addition to the language defaults. Requires `remove_stop_words=True`. | +| `custom_stop_words` | list[str] | `None` | Explicit stop-word list that replaces the built-in language list. Requires `remove_stop_words=True`. `None` uses the built-in list; an empty list disables stop-word removal even though it is enabled. | | `ngram_min_length` | int | `3` | Minimum n-gram length. Applies only when `base_tokenizer="ngram"`. | | `ngram_max_length` | int | `3` | Maximum n-gram length. Applies only when `base_tokenizer="ngram"`. | | `prefix_only` | bool | `False` | Index only prefix n-grams rather than all substrings. Applies only when `base_tokenizer="ngram"`. | @@ -145,6 +145,100 @@ Enable phrase queries by setting: | `with_position` | `True` | Track token positions for phrase matching | | `remove_stop_words` | `False` | Preserve stop words for exact phrase matching | +## Inspecting query tokenization + +When a query returns unexpected FTS results, it helps to see the exact tokens the index will match. LanceDB exposes two ways to run the tokenizer directly: + +- `Table.tokenize` uses the tokenizer configured on an existing FTS index. This is the fastest way to reproduce what happens inside `search(...)` for a given column or named index. +- The top-level `tokenize` helper runs the tokenizer without touching a table, so you can preview the behavior of a `base_tokenizer`, language, or `ngram` configuration before you build an index. + +Both entry points return a list of tokens with the token text (after filters such as lowercasing, stemming, and stop-word removal) and the token position used by phrase matching. + + +Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in the client from index metadata. For remote tables, the same tokenizer model files must also exist locally, in Lance's language model home. + + +### Tokenize against an existing FTS index + +Specify exactly one of `column` or the index name. Passing a column requires that column to have exactly one FTS index; passing an index name uses that index regardless of column. + + +```python Python icon="python" +# Sync table +tokens = table.tokenize("Running in cafés", column="text") +for tok in tokens: + print(tok.text, tok.position) +# run 0 +# cafe 2 + +# Or select a specific index by name +tokens = table.tokenize("Running in cafés", index_name="text_idx") +``` + +```python Async Python icon="python" +# Async table +tokens = await async_table.tokenize("Running in cafés", column="text") +``` + +```typescript TypeScript icon="square-js" +// Pass either { column } or { indexName } +const tokens = await tbl.tokenize("Running in cafés", { column: "text" }); +for (const t of tokens) { + console.log(t.text, t.position); +} +``` + +```rust Rust icon="rust" +// By index name +let tokens = table.tokenize("Running in cafés", "text_idx").await?; + +// Or resolve the FTS index by column +let tokens = table.tokenize_with_column("Running in cafés", "text").await?; +``` + + +### Tokenize without an index + +Use this to explore tokenizer settings before you commit to a specific `FTS` configuration. The options mirror the FTS index parameters described above. + + +```python Python icon="python" +import lancedb + +tokens = lancedb.tokenize( + "Running in cafés", + base_tokenizer="simple", + language="English", + lower_case=True, + stem=True, + ascii_folding=True, + # Optional: replace the built-in stop-word list for `language`. + custom_stop_words=["in"], +) +``` + +```typescript TypeScript icon="square-js" +import { tokenize } from "@lancedb/lancedb"; + +const tokens = await tokenize("Running in cafés", { + baseTokenizer: "simple", + language: "English", + lowercase: true, + stem: true, + asciiFolding: true, + // Optional: replace the built-in stop-word list for `language`. + customStopWords: ["in"], +}); +``` + +```rust Rust icon="rust" +use lancedb::{index::scalar::FtsIndexBuilder, tokenize}; + +let params = FtsIndexBuilder::default().base_tokenizer("simple".to_string()); +let tokens = tokenize("Running in cafés", ¶ms)?; +``` + + ## Indexing nested string fields You can build an FTS index on a string field inside a struct by passing its full dotted path, like `nested.text`. The same path is used when you query the index through `fts_columns`, and the indexed column is reported back as the full path from `list_indices()`.