Add CodSpeed performance benchmarks and CI workflow

.github/workflows/codspeed.yml

@@ -0,0 +1,59 @@
+name: CodSpeed
+
+on:
+  push:
+    branches:
+      - main
+  # Only fire on the events the job's `if:` below actually needs: `labeled`
+  # (adding the `runcodespeed` label) and `synchronize` (further pushes,
+  # which re-benchmark for as long as the label stays attached). Doc-only
+  # changes are skipped entirely, since they can't affect performance.
+  pull_request:
+    types: [labeled, synchronize]
+    paths-ignore:
+      - '**.md'
+      - '**.rst'
+      - 'docs/**'
+      - 'paper/**'
+  # `workflow_dispatch` allows CodSpeed to trigger backtest
+  # performance analysis in order to generate initial data.
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  # Cancel a stale in-progress run when a PR gets pushed to again, but let
+  # every push to main finish, since each one records a CodSpeed baseline.
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+permissions:
+  contents: read
+  id-token: write # for OpenID Connect authentication with CodSpeed
+
+jobs:
+  benchmarks:
+    name: Run benchmarks
+    runs-on: ubuntu-latest
+    # Always run for push (main) and workflow_dispatch. For pull_request
+    # events, only run while the PR carries the `runcodespeed` label.
+    if: >
+      github.event_name != 'pull_request' ||
+      contains(github.event.pull_request.labels.*.name, 'runcodespeed')
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install pytest pytest-codspeed
+
+      - name: Run benchmarks
+        uses: CodSpeedHQ/action@v5
+        with:
+          mode: simulation
+          run: pytest benchmarks/ --codspeed

README.md

@@ -11,7 +11,7 @@
 | **Meta** | [![GitHub contributors](https://img.shields.io/github/contributors/feature-engine/feature_engine?logo=GitHub)](https://github.com/feature-engine/feature_engine/graphs/contributors) [![first-timers-only](https://img.shields.io/badge/first--timers--only-friendly-blue.svg?style=flat)](https://www.firsttimersonly.com/) |
 | **Documentation** | [![Read the Docs](https://img.shields.io/readthedocs/feature_engine?logo=readthedocs)](https://feature-engine.readthedocs.io/en/latest/index.html) |
 | **Citation** | [![DOI](https://zenodo.org/badge/163630824.svg)](https://zenodo.org/badge/latestdoi/163630824) [![JOSS](https://joss.theoj.org/papers/10.21105/joss.03642/status.svg)](https://doi.org/10.21105/joss.03642) |
-| **Testing** | [![CircleCI](https://img.shields.io/circleci/build/github/feature-engine/feature_engine/main?logo=CircleCI)](https://app.circleci.com/pipelines/github/feature-engine/feature_engine) [![Codecov](https://img.shields.io/codecov/c/github/feature-engine/feature_engine?logo=CodeCov&token=ZBKKSN6ERL)](https://codecov.io/github/feature-engine/feature_engine) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) |
+| **Testing** | [![CircleCI](https://img.shields.io/circleci/build/github/feature-engine/feature_engine/main?logo=CircleCI)](https://app.circleci.com/pipelines/github/feature-engine/feature_engine) [![Codecov](https://img.shields.io/codecov/c/github/feature-engine/feature_engine?logo=CodeCov&token=ZBKKSN6ERL)](https://codecov.io/github/feature-engine/feature_engine) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![CodSpeed](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://app.codspeed.io/feature-engine/feature_engine?utm_source=badge) |
 <div align="center">
 
 

benchmarks/README.md

@@ -0,0 +1,60 @@
+# Benchmarks
+
+This folder contains the performance benchmarks of Feature-engine. They are
+written with [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed)
+and run on every push and pull request by the `CodSpeed` GitHub Actions
+workflow, which reports the results to
+[CodSpeed](https://app.codspeed.io/feature-engine/feature_engine).
+
+## What is covered
+
+One module per transformer family, benchmarking `fit` and `transform`
+separately, since they have very different performance profiles:
+
+| File                       | Covers                                                        |
+| -------------------------- | ------------------------------------------------------------- |
+| `test_imputation.py`       | Missing data imputers                                         |
+| `test_encoding.py`         | Categorical encoders                                          |
+| `test_discretisation.py`   | Discretisers                                                  |
+| `test_outliers.py`         | Outlier cappers and trimmers                                  |
+| `test_transformation.py`   | Mathematical transformers and scalers                         |
+| `test_creation.py`         | Feature creation transformers                                 |
+| `test_datetime.py`         | Datetime feature extraction                                   |
+| `test_timeseries.py`       | Lag, window and expanding window features                     |
+| `test_selection.py`        | Feature selectors                                             |
+| `test_variable_handling.py`| Variable handling helpers, called by every transformer's `fit` |
+| `test_pipeline.py`         | End to end pipelines and the preprocessing transformers       |
+
+The data is synthetic and built in `conftest.py` fixtures, so data generation is
+never part of what is measured. Dataframes are session scoped and shared by all
+benchmarks.
+
+## Running them locally
+
+```bash
+pip install -e .
+pip install pytest pytest-codspeed
+
+# quick check that the benchmarks run, with walltime measurements
+pytest benchmarks/ --codspeed
+
+# same measurements as CI, requires the CodSpeed CLI
+codspeed run --mode simulation -- pytest benchmarks/ --codspeed
+```
+
+Running a single file or benchmark works as with any other pytest test:
+
+```bash
+pytest benchmarks/test_encoding.py --codspeed
+pytest benchmarks/test_encoding.py::test_woe_encoder_fit --codspeed
+```
+
+## Adding a benchmark
+
+- Reuse the dataframe fixtures from `conftest.py`. Use `df_big` for the
+  vectorised transformers, `df_small` for the ones that train models
+  (decision trees, cross-validation) and `df_tiny` for the row-wise ones.
+- Do the `fit` outside of the measured section when benchmarking `transform`.
+- Keep a single benchmark in the millisecond range: the whole suite runs under
+  CPU simulation in CI, which is roughly two orders of magnitude slower than a
+  plain run.

benchmarks/conftest.py

@@ -0,0 +1,133 @@
+"""Shared data fixtures for the benchmark suite.
+
+The dataframes built here are synthetic but representative of the kind of data
+Feature-engine transformers are used on: a mix of numerical, categorical and
+datetime variables, with missing values.
+
+Data generation happens in fixtures so that it is never included in the
+measured section of a benchmark.
+"""
+
+import numpy as np
+import pandas as pd
+import pytest
+
+# Number of rows used for the transformers whose fit/transform is cheap.
+BIG_N = 10_000
+
+# Number of rows used for the transformers that train models under the hood
+# (decision trees, cross-validation, ...) so benchmarks stay in the millisecond
+# to low second range.
+SMALL_N = 1_000
+
+# Number of rows used for the row-wise transformers, which are an order of
+# magnitude slower per row than the vectorised ones.
+TINY_N = 500
+
+N_NUMERICAL = 8
+N_CATEGORICAL = 4
+
+
+def _make_dataframe(n_rows: int, seed: int = 0, with_na: bool = False):
+    rng = np.random.default_rng(seed)
+
+    data = {
+        f"num_{i}": rng.normal(loc=i, scale=i + 1, size=n_rows)
+        for i in range(N_NUMERICAL)
+    }
+
+    # A couple of strictly positive variables, needed by log/box-cox style
+    # transformers.
+    data["pos_0"] = rng.gamma(shape=2.0, scale=3.0, size=n_rows) + 0.1
+    data["pos_1"] = rng.gamma(shape=5.0, scale=1.0, size=n_rows) + 0.1
+
+    # A variable bounded between 0 and 1, needed by the arcsin transformer.
+    data["frac_0"] = rng.uniform(0.0, 1.0, size=n_rows)
+
+    # Categorical variables with a decreasing cardinality, including rare
+    # categories to exercise the rare label encoder.
+    for i in range(N_CATEGORICAL):
+        n_categories = 5 * (i + 1)
+        weights = np.linspace(1.0, 0.02, num=n_categories)
+        weights = weights / weights.sum()
+        data[f"cat_{i}"] = rng.choice(
+            [f"cat_{i}_value_{j}" for j in range(n_categories)],
+            size=n_rows,
+            p=weights,
+        )
+
+    data["date_0"] = pd.date_range("2015-01-01", periods=n_rows, freq="h")
+    data["date_1"] = pd.date_range("2018-06-15", periods=n_rows, freq="7min")
+
+    df = pd.DataFrame(data)
+
+    if with_na:
+        for column in ["num_0", "num_1", "pos_0", "cat_0", "cat_1"]:
+            mask = rng.random(n_rows) < 0.15
+            df.loc[mask, column] = np.nan
+
+    return df
+
+
+def numerical_vars():
+    return [f"num_{i}" for i in range(N_NUMERICAL)]
+
+
+def categorical_vars():
+    return [f"cat_{i}" for i in range(N_CATEGORICAL)]
+
+
+@pytest.fixture(scope="session")
+def df_big():
+    """Complete dataframe, no missing data."""
+    return _make_dataframe(BIG_N, seed=0)
+
+
+@pytest.fixture(scope="session")
+def df_big_na():
+    """Complete dataframe with missing data in numerical and categorical vars."""
+    return _make_dataframe(BIG_N, seed=1, with_na=True)
+
+
+@pytest.fixture(scope="session")
+def df_small():
+    """Smaller dataframe, for the estimator based transformers."""
+    return _make_dataframe(SMALL_N, seed=2)
+
+
+@pytest.fixture(scope="session")
+def df_tiny():
+    """Smallest dataframe, for the row-wise transformers."""
+    return _make_dataframe(TINY_N, seed=7)
+
+
+@pytest.fixture(scope="session")
+def y_binary():
+    """Binary target aligned with ``df_small``."""
+    rng = np.random.default_rng(3)
+    return pd.Series(rng.integers(0, 2, size=SMALL_N), name="target")
+
+
+@pytest.fixture(scope="session")
+def y_binary_big():
+    """Binary target aligned with ``df_big``."""
+    rng = np.random.default_rng(4)
+    return pd.Series(rng.integers(0, 2, size=BIG_N), name="target")
+
+
+@pytest.fixture(scope="session")
+def y_continuous():
+    """Continuous target aligned with ``df_small``."""
+    rng = np.random.default_rng(5)
+    return pd.Series(rng.normal(size=SMALL_N), name="target")
+
+
+@pytest.fixture(scope="session")
+def df_timeseries():
+    """Time indexed dataframe with numerical variables only."""
+    rng = np.random.default_rng(6)
+    index = pd.date_range("2020-01-01", periods=BIG_N, freq="15min")
+    return pd.DataFrame(
+        {f"num_{i}": rng.normal(size=BIG_N).cumsum() for i in range(4)},
+        index=index,
+    )

benchmarks/test_creation.py

@@ -0,0 +1,64 @@
+"""Benchmarks for the feature creation transformers."""
+
+import pytest
+
+from feature_engine.creation import (
+    CyclicalFeatures,
+    DecisionTreeFeatures,
+    MathFeatures,
+    RelativeFeatures,
+)
+
+from .conftest import numerical_vars
+
+NUM_VARS = numerical_vars()
+
+
+@pytest.mark.parametrize(
+    "func", [["sum", "mean"], ["sum", "mean", "std", "min", "max"]]
+)
+def test_math_features_transform(benchmark, df_tiny, func):
+    # MathFeatures aggregates row-wise, which is orders of magnitude slower per
+    # row than the vectorised transformers, hence the smallest dataframe.
+    creator = MathFeatures(variables=NUM_VARS, func=func)
+    creator.fit(df_tiny)
+    benchmark(creator.transform, df_tiny)
+
+
+def test_relative_features_transform(benchmark, df_big):
+    creator = RelativeFeatures(
+        variables=NUM_VARS[:4],
+        reference=["num_4"],
+        func=["sub", "div"],
+    )
+    creator.fit(df_big)
+    benchmark(creator.transform, df_big)
+
+
+def test_cyclical_features_transform(benchmark, df_big):
+    creator = CyclicalFeatures(variables=NUM_VARS)
+    creator.fit(df_big)
+    benchmark(creator.transform, df_big)
+
+
+def test_decision_tree_features_fit(benchmark, df_small, y_continuous):
+    creator = DecisionTreeFeatures(
+        variables=NUM_VARS[:3],
+        features_to_combine=2,
+        regression=True,
+        cv=2,
+        random_state=0,
+    )
+    benchmark(creator.fit, df_small, y_continuous)
+
+
+def test_decision_tree_features_transform(benchmark, df_small, y_continuous):
+    creator = DecisionTreeFeatures(
+        variables=NUM_VARS[:3],
+        features_to_combine=2,
+        regression=True,
+        cv=2,
+        random_state=0,
+    )
+    creator.fit(df_small, y_continuous)
+    benchmark(creator.transform, df_small)

benchmarks/test_datetime.py

@@ -0,0 +1,51 @@
+"""Benchmarks for the datetime feature extraction transformers."""
+
+import pytest
+
+from feature_engine.datetime import (
+    DatetimeFeatures,
+    DatetimeOrdinal,
+    DatetimeSubtraction,
+)
+
+DATE_VARS = ["date_0", "date_1"]
+
+
+@pytest.mark.parametrize(
+    "features_to_extract",
+    [
+        ["year", "month", "day_of_month"],
+        None,
+        "all",
+    ],
+    ids=["basic", "default", "all"],
+)
+def test_datetime_features_transform(benchmark, df_big, features_to_extract):
+    transformer = DatetimeFeatures(
+        variables=DATE_VARS, features_to_extract=features_to_extract
+    )
+    transformer.fit(df_big)
+    benchmark(transformer.transform, df_big)
+
+
+def test_datetime_features_from_string_transform(benchmark, df_big):
+    # Dates stored as strings: parsing dominates the runtime.
+    df = df_big.copy()
+    df["date_0"] = df["date_0"].astype(str)
+    transformer = DatetimeFeatures(
+        variables=["date_0"], features_to_extract=["year", "month", "day_of_month"]
+    )
+    transformer.fit(df)
+    benchmark(transformer.transform, df)
+
+
+def test_datetime_subtraction_transform(benchmark, df_big):
+    transformer = DatetimeSubtraction(variables=["date_0"], reference=["date_1"])
+    transformer.fit(df_big)
+    benchmark(transformer.transform, df_big)
+
+
+def test_datetime_ordinal_transform(benchmark, df_big):
+    transformer = DatetimeOrdinal(variables=DATE_VARS)
+    transformer.fit(df_big)
+    benchmark(transformer.transform, df_big)