Skip to content

Add blog: Streaming Real-Time Features to Lakebase with the Native Sink - #101

Draft
muralitalluri-db wants to merge 5 commits into
databricks-solutions:mainfrom
muralitalluri-db:rtm-native-lakebase-sink
Draft

muralitalluri-db wants to merge 5 commits into
databricks-solutions:mainfrom
muralitalluri-db:rtm-native-lakebase-sink

Conversation

@muralitalluri-db

Copy link
Copy Markdown
Contributor

Adds a self-serve example under 2026-09-rtm-native-lakebase-sink/ that uses Spark RTM to compute features and ingest them to Databricks Lakebase via the native Lakebase sink — for a real-time personalization use case.

What it does:

  • Reads video engagement events (watches/likes) from Kafka
  • Computes each user's last-6 watched / last-3 liked with transformWithState (single ValueState)
  • Writes one feature row per user to a UC-registered Lakebase table via .toTable() (native sink) — no ForeachWriter, no offline→online sync

Contents:

  • ingest-source-data/ — data generator, Kafka topic admin, Delta→Kafka replay
  • sss-rtm-to-lakebase/ — Lakebase table creation + the RTM feature pipeline
  • README.md — end-to-end self-serve instructions (workspace-specific values are placeholders)

Requires DBR 18 LTS.

@dichlorodiphen dichlorodiphen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (david-young_data)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants