Refactor(spark-3.5): Make OpenHouse SQL extensions standalone - #660
Draft
mkuchenbecker wants to merge 2 commits into
Draft
Refactor(spark-3.5): Make OpenHouse SQL extensions standalone #660mkuchenbecker wants to merge 2 commits into
mkuchenbecker wants to merge 2 commits into
Conversation
…park-3.1) spark-3.5 previously borrowed the ANTLR grammar (via spark-3.1's generated-src srcDir) and several classes (OpenHouseCatalog, IcebergCatalogMapper, OpenhouseSparkSessionExtensions, OpenhouseSqlExtensionsAstBuilder, OpenhouseDataSourceV2Strategy, GrantableResourceTypes, Principal) from the spark-3.1 shadow jar. This coupling meant the SQL grammar could not evolve for spark-3.5 without also changing spark-3.1. Give spark-3.5 its own copy of the grammar and these classes, add the ANTLR generation task (antlr4 4.7.1, matching spark-3.1), and drop the spark-3.1 module dependency. No behavior change; runtime + itest compile against the standalone module. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Jul 24, 2026
Draft
mkuchenbecker
changed the title
mkuchenbecker
commented
Jul 24, 2026
Comment on lines
-25
to
+32
| srcDirs += "${project(':integrations:spark:spark-3.1:openhouse-spark-runtime_2.12').buildDir}/generated-src/antlr/main" | ||
| srcDirs antlrMainGeneratedSrcDir |
Contributor
Author
There was a problem hiding this comment.
The difference is directly compiling rather than using the 3.1 generation.
Contributor
Author
There was a problem hiding this comment.
Yes — that's intentional. The Spark 3.5 module now owns and runs ANTLR over its own src/main/antlr/.../OpenhouseSqlExtensions.g4, with sourceSets.main.java.srcDirs pointing at its own build/generated-src/antlr/main and compileJava.dependsOn runAntlr. Reusing the Spark 3.1 generated sources would keep Spark 3.5 coupled to the 3.1 module's build output/task ordering and would prevent the 3.5 grammar from diverging for 3.5-specific SQL extensions. This keeps the runtime self-contained while preserving the same generated parser package/shape.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gives the spark-3.5 module its own copy of the OpenHouse SQL-extension machinery so the grammar can evolve for spark-3.5.
Previously spark-3.5 took a dependency on the 3.1 code to avoid duplication. That coupling meant the SQL grammar could not change for spark-3.5 without also changing spark-3.1.
This change will mean that we will explicitly release grammer on each version.
Changes
srcDirsat the module's own generated dir.
Testing Done
the standalone module.
Additional Information
No breaking changes; no client-facing changes.