Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ All notable changes to this project will be documented in this file.

- Fix broken link to the NiFi authorization usage guide in the `spec.clusterConfig.authorization` CRD doc (`usage-guide` -> `usage_guide`) ([#924]).

### Removed

- BREAKING: Remove support for NiFi 1.x.
This removes the Prometheus reporting-task Job (and its `spec.clusterConfig.createReportingTaskJob` field), the pre-2.x non-rolling upgrade handling, the dedicated metrics port, and the sensitive-properties algorithms that were only supported on NiFi 1.x.
`status.deployed_version` is retained even though it no longer drives the (now removed) non-rolling upgrade state machine, as we don't want a breaking change to the status just for this ([#954]).

[#903]: https://github.com/stackabletech/nifi-operator/pull/903
[#916]: https://github.com/stackabletech/nifi-operator/pull/916
[#921]: https://github.com/stackabletech/nifi-operator/pull/921
Expand All @@ -41,6 +47,7 @@ All notable changes to this project will be documented in this file.
[#940]: https://github.com/stackabletech/nifi-operator/pull/940
[#945]: https://github.com/stackabletech/nifi-operator/pull/945
[#953]: https://github.com/stackabletech/nifi-operator/pull/953
[#954]: https://github.com/stackabletech/nifi-operator/pull/954
[#959]: https://github.com/stackabletech/nifi-operator/pull/959

## [26.3.0] - 2026-03-16
Expand Down
11 changes: 0 additions & 11 deletions deploy/helm/nifi-operator/templates/clusterrole-operator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,17 +81,6 @@ rules:
- list
- patch
- watch
# Optional reporting-task Job (NiFi 1.x only). Applied via SSA and tracked for orphan cleanup.
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
# PodDisruptionBudget created per role. Applied via SSA and tracked for orphan cleanup.
- apiGroups:
- policy
Expand Down
3 changes: 2 additions & 1 deletion docs/modules/nifi/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ Every role group is accessible through it's own Service, and there is a Service

== Dependencies

Apache NiFi 1.x depends on Apache ZooKeeper which you can run in Kubernetes with the xref:zookeeper:index.adoc[].
NiFi builds its cluster quorum using Kubernetes by default and needs no additional dependencies.
It can optionally use Apache ZooKeeper instead, which you can run in Kubernetes with the xref:zookeeper:index.adoc[].

== [[demos]]Demos

Expand Down
2 changes: 0 additions & 2 deletions docs/modules/nifi/pages/usage_guide/clustering.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ CAUTION: The cluster backend of an existing cluster should never be changed. Oth
[#backend-kubernetes]
== Kubernetes

NOTE: The Kubernetes provider is only supported by Apache NiFi 2.0 or newer. When using NiFi 1.x, use the xref:#backend-zookeeper[] backend instead.

The Kubernetes backend is used by default (unless the xref:#backend-zookeeper[] backend is configured), and stores all state in Kubernetes objects, in the same namespace as the `NifiCluster` object.

It takes no configuration.
Expand Down
12 changes: 6 additions & 6 deletions docs/modules/nifi/pages/usage_guide/custom-components.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ There are currently two types of custom components:

1. Custom NiFi Archives (NARs) +
The {nifi-docs-developers-guide}[NiFi Developer’s Guide] provides further information on how to develop custom NARs.
2. Starting with NiFi 2.0, also custom Python extensions can be used. +
2. Additionally, custom Python extensions can be used. +
In the {nifi-docs-python-developers-guide}[NiFi Python Developer’s Guide], the development of custom components using Python is described.

The Stackable image contains the required tooling for both types.
Expand Down Expand Up @@ -165,7 +165,7 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 2.2.0
productVersion: 2.9.0
clusterConfig:
authentication:
- authenticationClass: simple-nifi-admin-user
Expand Down Expand Up @@ -202,7 +202,7 @@ The basic Dockerfile below shows how to achieve this:

[source,Dockerfile]
----
FROM oci.stackable.tech/sdp/nifi:2.2.0-stackable25.3.0
FROM oci.stackable.tech/sdp/nifi:2.9.0-stackable26.7.0
COPY /path/to/your/nar.file /stackable/nifi/lib/
COPY /path/to/your/Python.file /stackable/nifi/python/extensions/
----
Expand All @@ -213,8 +213,8 @@ You then need to make this image available to your Kubernetes cluster and specif
----
spec:
image:
productVersion: 2.2.0
custom: oci-registry.company.org/nifi:2.2.0-customprocessors
productVersion: 2.9.0
custom: oci-registry.company.org/nifi:2.9.0-customprocessors
----

Also read the xref:guides:custom-images.adoc[Using customized product images] guide for additional information.
Expand Down Expand Up @@ -285,7 +285,7 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 2.2.0
productVersion: 2.9.0
clusterConfig:
authentication:
- authenticationClass: simple-nifi-admin-user
Expand Down
5 changes: 2 additions & 3 deletions docs/modules/nifi/pages/usage_guide/flow-versioning.adoc
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
= Flow Versioning

IMPORTANT: The way described in this document to version flows is only available in NiFi 2.0.0 and later versions. Previous versions of NiFi relied exclusively on the NiFi Registry service for flow versioning.

Starting with Apache NiFi 2.0.0, you can connect directly to Git repositories using either GitHub or GitLab as your flow registry without requiring a separate NiFi Registry service. This document explains how to configure and use these registry clients to version your NiFi flows.
You can connect directly to Git repositories using either GitHub or GitLab as your flow registry without requiring a separate NiFi Registry service.
This document explains how to configure and use these registry clients to version your NiFi flows.

== Setting up the Registry Client

Expand Down
40 changes: 4 additions & 36 deletions docs/modules/nifi/pages/usage_guide/monitoring.adoc
Original file line number Diff line number Diff line change
@@ -1,42 +1,10 @@
= Monitoring
:description: The Stackable Operator for Apache NiFi automatically configures NiFi to export Prometheus metrics.
:k8s-job: https://kubernetes.io/docs/concepts/workloads/controllers/job/
:k8s-network-policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
:prometheus-operator: https://prometheus-operator.dev/

In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`,window=_blank].
== Configure metrics

The NiFi `2.0.0` release changed the way of exposing Prometheus metrics significantly.
The following steps explain on how to expose Metrics in NiFi versions `1.x.x` and `2.x.x`.

== Configure metrics in NiFi `1.x.x`

For NiFi versions `1.x.x`, the operator automatically configures NiFi to export Prometheus metrics.
This is done by creating a {k8s-job}[Job,window=_blank] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task,window=_blank].

IMPORTANT: Network access from the Job to NiFi is required.
If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies,window=_blank], make sure to allow access from the Job to NiFi.

See xref:operators:monitoring.adoc[] for more details.

== Disabling create-reporting-task Job

It can be helpful to disable the Job, e.g. when you configOverride an authentication mechanism, which the Job currently cannot use to authenticate against NiFi.

To achieve this use the following configuration:

[source,yaml]
----
spec:
clusterConfig:
createReportingTaskJob:
enabled: false
----

== Configure metrics in NiFi `2.x.x`

The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507,window=_blank].
Metrics are now always exposed and can be scraped using the NiFi `metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`.
Metrics are always exposed and can be scraped using the NiFi `metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`.

For a deployed NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under:

Expand All @@ -49,9 +17,9 @@ To scrape metrics from a particular Pod, the FQDN of the Pod and the `headless`

IMPORTANT: If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized.

=== Authentication with NiFi `2.x.x`
=== Authentication with NiFi

To authenticate against the NiFi `2.x.x` API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between
To authenticate against the NiFi API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between
Kubernetes Pods, check out the xref:home:secret-operator:index.adoc[Secret Operator documentation].

The following example illustrates the configuration of a Prometheus scraper for NiFi, using the aforementioned method of configuring mTLS
Expand Down
19 changes: 0 additions & 19 deletions docs/modules/nifi/pages/usage_guide/overrides.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -94,25 +94,6 @@ Pod overrides allow you to configure any attributes that can be configured on a

Read the xref:concepts:overrides.adoc#pod-overrides[Pod overrides concepts page] to learn more.

=== Pod overrides on create-reporting-task Job

In addition to podOverrides on the created StatefulSet we also support podOverrides on the created Kubernetes Job, which enables the export of Prometheus metrics within NiFi.

[source,yaml]
----
spec:
clusterConfig:
createReportingTaskJob:
# enabled: false # You can also turn off the Job entirely
podOverrides: # podOverrides as usual
spec:
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
----

== JVM argument overrides

Stackable operators automatically determine the set of needed JVM arguments, such as memory settings or trust- and keystores.
Expand Down
14 changes: 1 addition & 13 deletions docs/modules/nifi/pages/usage_guide/updating.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,4 @@ spec:

<1> Change the NiFi version here

[WARNING]
====
NiFi clusters cannot be upgraded or downgraded in a rolling fashion due to a limitation in NiFi prior to version 2.

When upgrading between NiFi 1 versions or from NiFi 1 to NiFi 2, any change to the NiFi version in the CRD triggers a full cluster restart with brief downtime.
However, the Stackable image version can be updated in a rolling manner, provided the NiFi version remains unchanged.

Since NiFi version 2, rolling upgrades are supported.
====

== NiFi 2.0.0

Before you can upgrade to `2.0.0` you https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance[need to update] to at least version 1.27.x!
Rolling upgrades are supported, so changing the NiFi version in the CRD updates the cluster without downtime.
45 changes: 1 addition & 44 deletions extra/crds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -139,33 +139,6 @@ spec:
- accessPolicyProvider
type: object
type: object
createReportingTaskJob:
default:
enabled: true
podOverrides: {}
description: |-
This section creates a `create-reporting-task` Kubernetes Job, which enables the export of
Prometheus metrics within NiFi.
properties:
enabled:
default: true
description: |-
Whether the Kubernetes Job should be created, defaults to true. It can be helpful to disable
the Job, e.g. when you configOverride an authentication mechanism, which the Job currently
can't use to authenticate against NiFi.
type: boolean
podOverrides:
default: {}
description: |-
Here you can define a
[PodTemplateSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#podtemplatespec-v1-core)
to override any property that can be set on the Pod of the create-reporting-task Kubernetes Job.
Read the
[Pod overrides documentation](https://docs.stackable.tech/home/nightly/concepts/overrides#pod-overrides)
for more information.
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
customComponentsGitSync:
default: []
description: |-
Expand Down Expand Up @@ -347,26 +320,11 @@ spec:
`nifiPbkdf2AesGcm256` (the default value),
`nifiArgon2AesGcm256`,

The following algorithms are deprecated and will be removed in future versions:

`nifiArgon2AesGcm128`,
`nifiBcryptAesGcm128`,
`nifiBcryptAesGcm256`,
`nifiPbkdf2AesGcm128`,
`nifiScryptAesGcm128`,
`nifiScryptAesGcm256`.

Learn more about the specifics of the algorithm parameters in the
[NiFi documentation](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms).
enum:
- nifiPbkdf2AesGcm256
- nifiArgon2AesGcm256
- nifiBcryptAesGcm128
- nifiBcryptAesGcm256
- nifiPbkdf2AesGcm128
- nifiArgon2AesGcm128
- nifiScryptAesGcm128
- nifiScryptAesGcm256
- null
nullable: true
type: string
Expand Down Expand Up @@ -422,8 +380,7 @@ spec:
When using the [Stackable operator for Apache ZooKeeper](https://docs.stackable.tech/home/nightly/zookeeper/)
to deploy a ZooKeeper cluster, this will simply be the name of your ZookeeperCluster resource.

The Kubernetes provider will be used if this field is unset. Kubernetes is only supported for NiFi 2.x and newer,
NiFi 1.x requires ZooKeeper.
The Kubernetes provider will be used if this field is unset.
maxLength: 253
minLength: 1
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
Expand Down
2 changes: 0 additions & 2 deletions rust/operator-binary/src/controller/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,6 @@ pub const PROTOCOL_PORT_NAME: &str = "protocol";
pub const PROTOCOL_PORT: Port = Port(9088);
pub const BALANCE_PORT_NAME: &str = "balance";
pub const BALANCE_PORT: Port = Port(6243);
pub const METRICS_PORT_NAME: &str = "metrics";
pub const METRICS_PORT: Port = Port(8081);
Comment thread
NickLarsenNZ marked this conversation as resolved.

// Filesystem paths shared by multiple builders. Single-consumer paths live in their builder.
pub const NIFI_CONFIG_DIRECTORY: &str = "/stackable/nifi/conf";
Expand Down
16 changes: 3 additions & 13 deletions rust/operator-binary/src/controller/build/properties.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,8 @@ pub(crate) mod test_support {

use crate::{
controller::{
NifiRoleGroupConfig, ValidatedCluster, ValidatedClusterConfig, ValidatedReportingTask,
ValidatedRoleConfig, ValidatedSensitiveProperties, validate::build_role_group_configs,
NifiRoleGroupConfig, ValidatedCluster, ValidatedClusterConfig, ValidatedRoleConfig,
ValidatedSensitiveProperties, validate::build_role_group_configs,
},
crd::{NifiRole, v1alpha1},
security::{
Expand All @@ -105,8 +105,7 @@ pub(crate) mod test_support {
};

/// A minimal NiFi cluster YAML. Mirrors the fixture used by bootstrap_conf tests,
/// stripped down to the mandatory fields only (NiFi 2.x, Kubernetes clustering backend,
/// SingleUser auth).
/// stripped down to the mandatory fields only (Kubernetes clustering backend, SingleUser auth).
pub const MINIMAL_NIFI_YAML: &str = r#"
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
Expand Down Expand Up @@ -196,15 +195,6 @@ pub(crate) mod test_support {
},
server_tls_secret_class: nifi.server_tls_secret_class().clone(),
extra_volumes: nifi.spec.cluster_config.extra_volumes.clone(),
reporting_task: ValidatedReportingTask {
enabled: nifi.spec.cluster_config.create_reporting_task_job.enabled,
pod_overrides: nifi
.spec
.cluster_config
.create_reporting_task_job
.pod_overrides
.clone(),
},
host_header_check: nifi.spec.cluster_config.host_header_check.clone(),
},
)
Expand Down
Loading
Loading