feat: add basic openlineage support

CHANGELOG.md

@@ -8,6 +8,7 @@ All notable changes to this project will be documented in this file.
 
 - BREAKING: Add required CLI argument and env var to set the image repository used to construct final product image names: `IMAGE_REPOSITORY` (`--image-repository`), eg. `oci.example.org/my/namespace` ([#684]).
 - Support for Spark `4.1.2` ([#708])
+- Add `spec.openLineage` to enable OpenLineage lineage emission: injects the OpenLineage Spark listener, HTTP transport (endpoint resolved from a discovery ConfigMap) and a stable job name. The listener jar is referenced through a stable, version- and Scala-independent symlink maintained by the Spark image, so the operator tracks neither. The `--add-opens java.base/java.security` JVM flag is emitted only on Spark 4.x (it is unnecessary on the Spark 3.5.x images and must not be set there) ([#XXXX]).
 
 ### Fixed
 
@@ -30,7 +31,7 @@ All notable changes to this project will be documented in this file.
 - Internal operator refactoring: introduce dereference() and validate() steps in the reconciler for spark application, spark connect and spark history server([#687]).
 - test: Bump vector-aggregator to 0.55.0, replace /graphql call with gRPC call ([#689]).
 - Fix the `SparkApplication` CRD description, which incorrectly described it as a "Spark cluster stacklet" rather than a Spark application ([#705]).
-- BREAKING: make application templates namespaced instead of cluster wide objects ([#694]).
+- BREAKING: make application templates namespaced instead of cluster wide objects ([#694]), reverted in [#719] and readded in [#720].
 
 [#674]: https://github.com/stackabletech/spark-k8s-operator/pull/674
 [#679]: https://github.com/stackabletech/spark-k8s-operator/pull/679
@@ -44,6 +45,8 @@ All notable changes to this project will be documented in this file.
 [#705]: https://github.com/stackabletech/spark-k8s-operator/pull/705
 [#708]: https://github.com/stackabletech/spark-k8s-operator/pull/708
 [#716]: https://github.com/stackabletech/spark-k8s-operator/pull/716
+[#719]: https://github.com/stackabletech/spark-k8s-operator/pull/719
+[#720]: https://github.com/stackabletech/spark-k8s-operator/pull/720
 
 ## [26.3.0] - 2026-03-16
 

docs/modules/spark-k8s/pages/usage-guide/openlineage.adoc

@@ -0,0 +1,73 @@
+= OpenLineage
+
+https://openlineage.io/[OpenLineage] is an open standard for data lineage collection.
+The Spark operator can automatically emit lineage events for a `SparkApplication` to an OpenLineage-compatible backend such as https://marquezproject.github.io/marquez/[Marquez], without any manual `sparkConf` wiring.
+
+When enabled, the operator injects everything required to make the https://openlineage.io/docs/integrations/spark/[OpenLineage Spark listener] work:
+
+* the OpenLineage Spark listener (appended to `spark.extraListeners`, so any listener you configure yourself is preserved),
+* the OpenLineage jar baked into the Spark image, referenced via `spark.jars` so it shares the same classloader as connectors pulled in through xref:usage-guide/job-dependencies.adoc[`deps.packages`] (for example Apache Iceberg). The operator references a stable symlink maintained by the image, which points at the correct build for that image's Spark version (Scala 2.13 for Spark 4.x, Scala 2.12 for Spark 3.5.x), so neither the jar version nor its Scala suffix has to be tracked in the operator,
+* an HTTP transport pointing at the backend endpoint,
+* a stable lineage namespace and job name (see <<job-name>>), and
+* on Spark 4.x only, the `--add-opens java.base/java.security=ALL-UNNAMED` JVM flag the listener requires on the driver and executors. This flag is not added on the Spark 3.5.x images, where it is unnecessary (and, on their JVM, potentially harmful).
+
+== Enabling OpenLineage
+
+The backend endpoint is not configured on the `SparkApplication` directly.
+Instead, mirroring the xref:usage-guide/logging.adoc[vector aggregator discovery], the operator reads it from a discovery ConfigMap referenced by `configMapName`:
+
+[source,yaml]
+----
+apiVersion: spark.stackable.tech/v1alpha1
+kind: SparkApplication
+metadata:
+  name: orders-by-nation
+spec:
+  openLineage:
+    enabled: true # <1>
+    configMapName: marquez-discovery # <2>
+    # namespace: orders-lineage # <3>
+    # appName: orders-by-nation # <4>
+  ...
+----
+<1> Enable OpenLineage event emission. Defaults to `false`, in which case the operator emits nothing OpenLineage-related and behaviour is unchanged.
+<2> Name of a discovery ConfigMap holding the backend endpoint (see below).
+    Must be in the same namespace as the `SparkApplication`.
+<3> Optional. The OpenLineage namespace events are reported under.
+    Defaults to the application's Kubernetes namespace.
+<4> Optional. The stable OpenLineage job name (see <<job-name>>).
+
+The discovery ConfigMap holds the backend base URL under the `ADDRESS` key:
+
+.Example discovery ConfigMap
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: marquez-discovery
+data:
+  ADDRESS: http://marquez:5000 # <1>
+----
+<1> Base URL of the OpenLineage backend, for example the Marquez API service.
+
+Alternatively, you can set the endpoint directly through `sparkConf` (`spark.openlineage.transport.url`).
+If `enabled: true` and no endpoint can be resolved from either the discovery ConfigMap or `sparkConf`, reconciliation fails with a clear error rather than emitting a config that silently drops events.
+
+[#job-name]
+== Job name
+
+The OpenLineage job name is resolved in the following order:
+
+. `spec.openLineage.appName`, if set.
+. `spark.app.name` from `sparkConf`, if set.
+. `metadata.name` as a last resort.
+
+The last case additionally emits a Kubernetes warning event.
+Falling back to `metadata.name` is discouraged: if that name carries a timestamp or other run-specific suffix (as generated names often do), every run becomes a separate job in the backend and the run history fragments.
+Set `spec.openLineage.appName` (or `spark.app.name`) to a stable value to keep all runs of the same logical job together.
+
+== Overriding injected configuration
+
+Every injected value is a default that goes into the submit configuration *before* your xref:usage-guide/overrides.adoc[`sparkConf`] is merged, so any key you set yourself wins.
+`spark.extraListeners` and `spark.jars` are the exception: the operator *appends* to any value you provide (comma-merged) rather than overwriting it, so your own listeners and jars are kept alongside OpenLineage's.

docs/modules/spark-k8s/partials/nav.adoc

@@ -9,6 +9,7 @@
 ** xref:spark-k8s:usage-guide/app_templates.adoc[]
 ** xref:spark-k8s:usage-guide/security.adoc[]
 ** xref:spark-k8s:usage-guide/logging.adoc[]
+** xref:spark-k8s:usage-guide/openlineage.adoc[]
 ** xref:spark-k8s:usage-guide/history-server.adoc[]
 ** xref:spark-k8s:usage-guide/spark-connect.adoc[]
 ** xref:spark-k8s:usage-guide/examples.adoc[]

docs/modules/spark-k8s/partials/supported-versions.adoc

@@ -5,7 +5,7 @@
 
 - 4.1.2 (Hadoop 3.4.3, Scala 2.13, Python 3.12, Java 21) (LTS)
 - 4.1.1 (Hadoop 3.4.3, Scala 2.13, Python 3.12, Java 21) (Deprecated)
-- 3.5.8 (Hadoop 3.4.2, Scala 2.12, Python 3.11, Java 17) (Deprecated)
+- 3.5.8 (HBase 2.6.6, Hadoop 3.4.3, Scala 2.12, Python 3.11, Java 17) (Deprecated)
 
 Apache Spark 4.1.1 & 4.1.2 have the following known issues (as of July 2026):
 

extra/crds.yaml

@@ -1957,6 +1957,42 @@ spec:
                 - cluster
                 - client
                 type: string
+              openLineage:
+                description: |-
+                  Emit [OpenLineage](https://openlineage.io/) lineage events for this application.
+                  The OpenLineage Spark listener runs on the driver and describes the whole application,
+                  so this is application-scoped config (not per driver/executor role).
+                  See the [OpenLineage usage guide](https://docs.stackable.tech/home/nightly/spark-k8s/usage-guide/openlineage).
+                nullable: true
+                properties:
+                  appName:
+                    description: |-
+                      A stable OpenLineage job/application name. Setting this prevents fragmented run history
+                      (and the intermittent `unknown` job-name bug). If unset, the operator resolves it from
+                      `spark.app.name`, falling back to `metadata.name` (with a warning event).
+                    nullable: true
+                    type: string
+                  configMapName:
+                    description: |-
+                      Name of the OpenLineage backend [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery).
+                      It must contain the key `ADDRESS` with the base URL of the OpenLineage backend
+                      (e.g. `http://marquez:5000`). Mirrors the `vectorAggregatorConfigMapName` field on this CRD.
+                    maxLength: 253
+                    minLength: 1
+                    nullable: true
+                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                    type: string
+                  enabled:
+                    default: false
+                    description: Enable OpenLineage event emission. Defaults to `false` (nothing is injected).
+                    type: boolean
+                  namespace:
+                    description: |-
+                      The OpenLineage namespace lineage is reported under.
+                      Defaults to the application's Kubernetes namespace (`metadata.namespace`).
+                    nullable: true
+                    type: string
+                type: object
               s3connection:
                 description: |-
                   Configure an S3 connection that the SparkApplication has access to.
@@ -6563,6 +6599,42 @@ spec:
                 - cluster
                 - client
                 type: string
+              openLineage:
+                description: |-
+                  Emit [OpenLineage](https://openlineage.io/) lineage events for this application.
+                  The OpenLineage Spark listener runs on the driver and describes the whole application,
+                  so this is application-scoped config (not per driver/executor role).
+                  See the [OpenLineage usage guide](https://docs.stackable.tech/home/nightly/spark-k8s/usage-guide/openlineage).
+                nullable: true
+                properties:
+                  appName:
+                    description: |-
+                      A stable OpenLineage job/application name. Setting this prevents fragmented run history
+                      (and the intermittent `unknown` job-name bug). If unset, the operator resolves it from
+                      `spark.app.name`, falling back to `metadata.name` (with a warning event).
+                    nullable: true
+                    type: string
+                  configMapName:
+                    description: |-
+                      Name of the OpenLineage backend [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery).
+                      It must contain the key `ADDRESS` with the base URL of the OpenLineage backend
+                      (e.g. `http://marquez:5000`). Mirrors the `vectorAggregatorConfigMapName` field on this CRD.
+                    maxLength: 253
+                    minLength: 1
+                    nullable: true
+                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                    type: string
+                  enabled:
+                    default: false
+                    description: Enable OpenLineage event emission. Defaults to `false` (nothing is injected).
+                    type: boolean
+                  namespace:
+                    description: |-
+                      The OpenLineage namespace lineage is reported under.
+                      Defaults to the application's Kubernetes namespace (`metadata.namespace`).
+                    nullable: true
+                    type: string
+                type: object
               s3connection:
                 description: |-
                   Configure an S3 connection that the SparkApplication has access to.