feat: document the new streaming dataset for data loading - #294
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
Thanks for the PR @westonpace! As I understand it, this is in main, but not yet in the official release, right? If so, we should wait till we release it before merging. |
It's not even in main yet 😅 so yes, we should hold off for now. |
prrao87
added
the
needs_new_release
prrao87
added
the
wip
prrao87
removed
needs_new_release
prrao87
left a comment
prrao87
left a comment
There was a problem hiding this comment.
Updated to the latest state of the code and improved conciseness/clarity. LGTM!
| stored in memory but they can be persisted to storage as well. This is useful when you want to share a permutation | ||
| table across processes or nodes. | ||
| `StreamingDataset` relies on the permutation API and requires direct access to an embedded LanceDB table. The table | ||
| data can live on local or cloud object storage, but a remote table accessed through the LanceDB Cloud API is not |
There was a problem hiding this comment.
Is it still true that the remote table API isn't supported?
There was a problem hiding this comment.
No, the remote table API should be supported.
There was a problem hiding this comment.
I'll clean that up.
The Note incorrectly stated remote tables accessed through the LanceDB Cloud API were unsupported. The remote table API is supported, so the note now covers both embedded and remote tables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Merging this version as this will all be reflected in the soon-to-be-released next version of LanceDB |
2 participants
No description provided.