feat: add basic openlineage support#718
Draft
sweb wants to merge 1 commit into
Draft
Conversation
sweb
commented
Jul 13, 2026
| /// IMPORTANT: the version MUST stay in sync with the `openlineage-spark-version` build-argument in | ||
| /// `docker-images/spark-k8s/boil-config.toml` (the image is what places the jar at this path). | ||
| pub const OPENLINEAGE_JAR_LOCAL_URI: &str = | ||
| "local:///stackable/spark/openlineage/openlineage-spark_2.13-1.51.0.jar"; |
Member
Author
There was a problem hiding this comment.
this needs a way that is version agnostic / some proper way to keep this in sync other than a comment.
Member
There was a problem hiding this comment.
The way we solved similar problems in the past was to use symlinks
| /// `ADDRESS` contract of the existing `vectorAggregatorConfigMapName` discovery ConfigMap. | ||
| pub const OPENLINEAGE_CONFIG_MAP_ADDRESS_KEY: &str = "ADDRESS"; | ||
|
|
||
| /// Java module-system flag OpenLineage requires on Spark 4.x: without it the driver throws a |
Member
Author
There was a problem hiding this comment.
Check how this changes for Spark 3.5.8
|
|
||
| submit_conf.insert( | ||
| "spark.openlineage.transport.type".to_string(), | ||
| "http".to_string(), |
Member
Author
There was a problem hiding this comment.
right now limited to http - first version probably needs https support
| // OpenLineage on Spark 4.x needs `java.base/java.security` opened to the unnamed module, | ||
| // otherwise the driver throws a non-fatal `InaccessibleObjectException` and silently degrades | ||
| // extension-interface lineage (MVP §7). Added to both driver and executor. | ||
| if spark_application |
Member
Author
There was a problem hiding this comment.
see other comment, this needs proper Spark 3 treatment.
| }; | ||
|
|
||
| let name = config_map_name.as_ref(); | ||
| let namespace = spark_application.namespace().unwrap_or_default(); |
Member
Author
There was a problem hiding this comment.
this needs to become an error instead of default
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Allows to configure openlineage for Spark
Definition of Done Checklist
Author
Reviewer
Acceptance
type/deprecationlabel & add to the deprecation scheduletype/experimentallabel & add to the experimental features tracker