diff --git a/mkdocs/docs/configuration.md b/mkdocs/docs/configuration.md index 44b5e395a9..d5ab6e19bc 100644 --- a/mkdocs/docs/configuration.md +++ b/mkdocs/docs/configuration.md @@ -425,7 +425,9 @@ Legacy OAuth2 Properties will be removed in PyIceberg 1.0 in place of pluggable ##### Pluggable Authentication via AuthManager -The RESTCatalog supports pluggable authentication via the `auth` configuration block. This allows you to specify which how the access token will be fetched and managed for use with the HTTP requests to the RESTCatalog server. The authentication method is selected by setting the `auth.type` property, and additional configuration can be provided as needed for each method. +The RESTCatalog supports pluggable authentication via the `auth` configuration block. This selects how an access token or +authorization header is fetched and managed for requests to the REST Catalog server. The authentication method is selected by +setting `auth.type`, and additional configuration can be provided as needed for each method. ###### Supported Authentication Types @@ -438,7 +440,17 @@ The RESTCatalog supports pluggable authentication via the `auth` configuration b ###### Configuration Properties -The `auth` block is structured as follows: +Authentication can be configured through: + +- Python arguments passed to `load_catalog`. +- Nested or flat properties in `.pyiceberg.yaml`. +- `PYICEBERG_CATALOG____AUTH__...` environment variables. +- Environment variables and credential chains defined by the authentication provider. + +PyIceberg guarantees that `auth.type` selects the authentication manager. Manager-specific properties are a best-effort +passthrough to the manager constructor. PyIceberg does not normalize their names or coerce their values. + +The nested YAML form is: ```yaml catalog: @@ -452,67 +464,208 @@ catalog: impl: # Only for custom auth ``` +The equivalent flat YAML form is: + +```yaml +catalog: + default: + type: rest + uri: http://rest-catalog/ws/ + auth.type: entra +``` + +Python accepts the same flat properties and is the most reliable way to pass exact constructor names and typed values. + +Environment variables use two underscores to separate the catalog name from the property path and to represent dots inside +that path: + +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=entra +``` + +###### YAML and Environment Variable Limitations + +- Environment variables convert `_` in property names to `-`; literal underscores cannot be preserved. For example, + `AUTH__OAUTH2__CLIENT_ID` becomes `auth.oauth2.client-id`, not `auth.oauth2.client_id`. +- Environment variable values remain strings and are not decoded, so lists, mappings, integers, and booleans cannot be passed + with their expected types. YAML preserves lists and mappings, but its scalar values are also strings. +- When supported by the auth manager, prefer provider-native environment variables such as `GOOGLE_APPLICATION_CREDENTIALS` + or `AZURE_CLIENT_ID`. + +Flat `auth.*` properties override equivalent nested properties. Python arguments override environment variables, and +environment variables override YAML. + ###### Property Reference -| Property | Required | Description | -|------------------|----------|-------------------------------------------------------------------------------------------------| -| `auth.type` | Yes | The authentication type to use (`noop`, `basic`, `oauth2`, or `custom`). | -| `auth.impl` | Conditionally | The fully qualified class path for a custom AuthManager. Required if `auth.type` is `custom`. | -| `auth.basic` | If type is `basic` | Block containing `username` and `password` for HTTP Basic authentication. | -| `auth.oauth2` | If type is `oauth2` | Block containing OAuth2 configuration (see below). | -| `auth.custom` | If type is `custom` | Block containing configuration for the custom AuthManager. | -| `auth.google` | If type is `google` | Block containing `credentials_path` to a service account file (if using). Will default to using Application Default Credentials. | -| `auth.entra` | If type is `entra` | Block containing Entra ID configuration. Will default to using DefaultAzureCredential. | +| Property | Required | Description | +|----------|----------|-------------| +| `auth.type` | Yes | Authentication type: `noop`, `basic`, `oauth2`, `google`, `entra`, or `custom`. | +| `auth.impl` | For `custom` | Fully qualified class name of a custom `AuthManager`. | +| `auth.` | Type-specific | Nested YAML mapping passed to the selected manager. | +| `auth..*` | Type-specific | Flat properties passed to the selected manager after removing the prefix. | ###### Examples No Authentication: ```yaml -auth: - type: noop +catalog: + default: + type: rest + uri: https://rest-catalog.example.com + auth.type: noop +``` + +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=noop ``` +The `noop` manager accepts no options. + Basic Authentication: ```yaml -auth: - type: basic - basic: - username: myuser - password: mypass +catalog: + default: + type: rest + uri: https://rest-catalog.example.com + auth.type: basic + auth.basic.username: myuser + auth.basic.password: mypass ``` +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=basic +export PYICEBERG_CATALOG__DEFAULT__AUTH__BASIC__USERNAME=myuser +export PYICEBERG_CATALOG__DEFAULT__AUTH__BASIC__PASSWORD=mypass +``` + +`username` and `password` contain no underscores and both expect strings, so this manager can be fully configured using +PyIceberg-prefixed environment variables. Avoid storing passwords directly in `.pyiceberg.yaml`. + OAuth2 Authentication: +```yaml +catalog: + default: + type: rest + uri: https://rest-catalog.example.com + auth: + type: oauth2 + oauth2: + client_id: my-client-id + client_secret: my-client-secret + token_url: https://auth.example.com/oauth/token + scope: read +``` + +PyIceberg-prefixed environment variables can select the manager: + +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=oauth2 +``` + +OAuth2 cannot be fully configured through PyIceberg-prefixed environment variables because required names such as `client_id` +become unsupported names such as `client-id`; use YAML for required strings and Python for typed options. + +Google Authentication: + +```yaml +catalog: + default: + type: rest + uri: https://biglake.googleapis.com/iceberg/v1/restcatalog + auth.type: google +``` + +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=google +export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json +``` + +Install the manager with `pip install 'pyiceberg[gcp-auth]'`. Google Application Default Credentials handles +`GOOGLE_APPLICATION_CREDENTIALS`, workload identity, attached service accounts, and other native credential sources. + +The optional `credentials_path` string and `scopes` list can be passed through nested YAML: + ```yaml auth: - type: oauth2 - oauth2: - client_id: my-client-id - client_secret: my-client-secret - token_url: https://auth.example.com/oauth/token - scope: read - refresh_margin: 60 # (optional) seconds before expiry to refresh - expires_in: 3600 # (optional) fallback if server does not provide + type: google + google: + credentials_path: /path/to/service-account.json + scopes: + - https://www.googleapis.com/auth/cloud-platform ``` -Custom Authentication: +Google cannot use these options from PyIceberg-prefixed environment variables because `credentials_path` becomes +`credentials-path` and `scopes` remains a string. Google-native variables can configure credentials, but scopes require YAML or +Python. + +Microsoft Entra Authentication: + +```yaml +catalog: + default: + type: rest + uri: https://rest-catalog.example.com + auth.type: entra +``` + +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=entra +export AZURE_TENANT_ID= +export AZURE_CLIENT_ID= +export AZURE_CLIENT_SECRET= +``` + +Install the manager with `pip install 'pyiceberg[entra-auth]'`. The manager uses `DefaultAzureCredential`, which supports Azure +environment variables, managed identity, workload identity, Azure CLI login, and other native credential sources. The default +scope is `https://storage.azure.com/.default`. + +Scopes and string-valued `DefaultAzureCredential` keyword arguments can be passed through nested YAML: ```yaml auth: - type: custom - impl: mypackage.module.MyAuthManager - custom: - property1: value1 - property2: value2 + type: entra + entra: + scopes: + - https://storage.azure.com/.default + managed_identity_client_id: user-assigned-client-id ``` +Entra cannot use these options from PyIceberg-prefixed environment variables because `scopes` remains a string and +`managed_identity_client_id` becomes `managed-identity-client-id`; use Azure-native credentials, YAML, or Python instead. + +Custom Authentication: + +```yaml +catalog: + default: + type: rest + uri: https://rest-catalog.example.com + auth.type: custom + auth.impl: mypackage.module.MyAuthManager + auth.custom.property1: value1 + auth.custom.property2: value2 +``` + +```sh +export PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE=custom +export PYICEBERG_CATALOG__DEFAULT__AUTH__IMPL=mypackage.module.MyAuthManager +export PYICEBERG_CATALOG__DEFAULT__AUTH__CUSTOM__PROPERTY1=value1 +``` + +`auth.impl` must be the fully qualified class name of an `AuthManager`. Properties under `auth.custom.*` are passed directly to +its constructor after removing the prefix. Environment-variable compatibility depends on the names and value types accepted by +the custom manager; prefer its provider-native configuration when available. + ###### Notes - If `auth.type` is `custom`, you **must** specify `auth.impl` with the full class path to your custom AuthManager. - If `auth.type` is not `custom`, specifying `auth.impl` is not allowed. - The configuration block under each type (e.g., `basic`, `oauth2`, `custom`) is passed as keyword arguments to the corresponding AuthManager. +- PyIceberg does not deserialize a complete `auth` mapping from a single environment variable. Use individual `AUTH__...` + variables. diff --git a/pyiceberg/catalog/rest/__init__.py b/pyiceberg/catalog/rest/__init__.py index 97a71b429b..2f97cec0b7 100644 --- a/pyiceberg/catalog/rest/__init__.py +++ b/pyiceberg/catalog/rest/__init__.py @@ -31,7 +31,14 @@ from pyiceberg import __version__ from pyiceberg.catalog import BOTOCORE_SESSION, TOKEN, URI, WAREHOUSE_LOCATION, Catalog, PropertiesUpdateSummary -from pyiceberg.catalog.rest.auth import AUTH_MANAGER, AuthManager, AuthManagerAdapter, AuthManagerFactory, LegacyOAuth2AuthManager +from pyiceberg.catalog.rest.auth import ( + AUTH_MANAGER, + AuthManager, + AuthManagerAdapter, + AuthManagerFactory, + LegacyOAuth2AuthManager, + _resolve_auth_config, +) from pyiceberg.catalog.rest.response import _handle_non_200_response from pyiceberg.catalog.rest.scan_planning import ( FetchScanTasksRequest, @@ -260,8 +267,6 @@ class ScanPlanningMode(Enum): EMPTY_BODY_SHA256: str = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" OAUTH2_SERVER_URI = "oauth2-server-uri" SNAPSHOT_LOADING_MODE = "snapshot-loading-mode" -AUTH = "auth" -CUSTOM = "custom" SCAN_PLANNING_MODE = "scan-planning-mode" SCAN_PLANNING_MODE_DEFAULT = ScanPlanningMode.CLIENT.value # for backwards compatibility with older REST servers where it can be assumed that a particular @@ -441,20 +446,8 @@ def _create_session(self) -> Session: elif ssl_client_cert := ssl_client.get(CERT): session.cert = ssl_client_cert - if auth_config := self.properties.get(AUTH): - auth_type = auth_config.get("type") - if auth_type is None: - raise ValueError("auth.type must be defined") - auth_type_config = auth_config.get(auth_type, {}) - auth_impl = auth_config.get("impl") - - if auth_type == CUSTOM and not auth_impl: - raise ValueError("auth.impl must be specified when using custom auth.type") - - if auth_type != CUSTOM and auth_impl: - raise ValueError("auth.impl can only be specified when using custom auth.type") - - self._auth_manager = AuthManagerFactory.create(auth_impl or auth_type, auth_type_config) + if auth_config := _resolve_auth_config(self.properties): + self._auth_manager = AuthManagerFactory.create(auth_config.manager, auth_config.properties) session.auth = AuthManagerAdapter(self._auth_manager) else: self._auth_manager = self._create_legacy_oauth2_auth_manager(session) diff --git a/pyiceberg/catalog/rest/auth.py b/pyiceberg/catalog/rest/auth.py index 602074282c..df90452e07 100644 --- a/pyiceberg/catalog/rest/auth.py +++ b/pyiceberg/catalog/rest/auth.py @@ -21,6 +21,7 @@ import threading import time from abc import ABC, abstractmethod +from dataclasses import dataclass from functools import cached_property from typing import Any @@ -30,13 +31,55 @@ from pyiceberg.catalog.rest.response import TokenResponse, _handle_non_200_response from pyiceberg.exceptions import OAuthError +from pyiceberg.typedef import Properties AUTH_MANAGER = "auth.manager" +AUTH = "auth" +AUTH_TYPE = f"{AUTH}.type" +AUTH_IMPL = f"{AUTH}.impl" +CUSTOM = "custom" COLON = ":" logger = logging.getLogger(__name__) +@dataclass(frozen=True) +class _AuthConfig: + """Resolved declarative configuration for an AuthManager.""" + + manager: str + properties: Properties + + +def _resolve_auth_config(properties: Properties) -> _AuthConfig | None: + """Resolve nested and flat auth properties into the canonical property representation.""" + auth_config = properties.get(AUTH) + if auth_config is None: + auth_config = {} + elif not isinstance(auth_config, dict): + raise ValueError( + "Auth configuration must be a mapping; use auth.type in YAML or " + "PYICEBERG_CATALOG____AUTH__TYPE in the environment" + ) + + auth_type = properties.get(AUTH_TYPE, auth_config.get("type")) + if auth_type is None: + if auth_config or any(key.startswith(f"{AUTH}.") and key != AUTH_MANAGER for key in properties): + raise ValueError("auth.type must be defined") + return None + + auth_impl = properties.get(AUTH_IMPL, auth_config.get("impl")) + if auth_type == CUSTOM and not auth_impl: + raise ValueError("auth.impl must be specified when using custom auth.type") + if auth_type != CUSTOM and auth_impl: + raise ValueError("auth.impl can only be specified when using custom auth.type") + + auth_properties = dict(auth_config.get(auth_type, {})) + type_prefix = f"{AUTH}.{auth_type}." + auth_properties.update({key[len(type_prefix) :]: value for key, value in properties.items() if key.startswith(type_prefix)}) + return _AuthConfig(manager=auth_impl or auth_type, properties=auth_properties) + + class AuthManager(ABC): """ Abstract base class for Authentication Managers used to supply authorization headers to HTTP clients (e.g. requests.Session). diff --git a/tests/catalog/test_rest_auth.py b/tests/catalog/test_rest_auth.py index ae5d40f5aa..ea2380e3c1 100644 --- a/tests/catalog/test_rest_auth.py +++ b/tests/catalog/test_rest_auth.py @@ -16,18 +16,333 @@ # under the License. import base64 -from unittest.mock import MagicMock, patch +from pathlib import Path +from typing import cast +from unittest.mock import MagicMock, call, patch import pytest import requests from requests_mock import Mocker -from pyiceberg.catalog.rest.auth import AuthManagerAdapter, BasicAuthManager, EntraAuthManager, GoogleAuthManager, NoopAuthManager +from pyiceberg.catalog import load_catalog +from pyiceberg.catalog.rest import RestCatalog +from pyiceberg.catalog.rest.auth import ( + AuthManagerAdapter, + BasicAuthManager, + EntraAuthManager, + GoogleAuthManager, + NoopAuthManager, +) +from pyiceberg.typedef import Properties +from pyiceberg.utils.config import Config TEST_URI = "https://iceberg-test-catalog/" GOOGLE_CREDS_URI = "https://oauth2.googleapis.com/token" +def _assert_load_catalog_auth_config_from_yaml( + yaml_config: str, + tmp_path: Path, + requests_mock: Mocker, +) -> tuple[str, Properties]: + (tmp_path / ".pyiceberg.yaml").write_text(yaml_config, encoding="utf-8") + with patch.dict("os.environ", {"PYICEBERG_HOME": str(tmp_path)}, clear=True): + config = Config() + + requests_mock.get(f"{TEST_URI}v1/config", json={"defaults": {}, "overrides": {}}, status_code=200) + fake_auth_manager = MagicMock() + fake_auth_manager.auth_header.return_value = None + + with ( + patch("pyiceberg.catalog._ENV_CONFIG", config), + patch("pyiceberg.catalog.rest.AuthManagerFactory.create", return_value=fake_auth_manager) as create_auth_manager, + ): + catalog = load_catalog("default", type="rest", uri=TEST_URI) + + assert isinstance(catalog, RestCatalog) + assert create_auth_manager.call_args_list + configured_auth_manager_call = create_auth_manager.call_args_list[0] + assert all(auth_manager_call == configured_auth_manager_call for auth_manager_call in create_auth_manager.call_args_list) + configured_manager, configured_properties = configured_auth_manager_call.args + assert isinstance(configured_manager, str) + assert isinstance(configured_properties, dict) + return configured_manager, cast(Properties, configured_properties) + + +def _assert_load_catalog_auth_config_from_environment( + environment: dict[str, str], + requests_mock: Mocker, +) -> tuple[str, Properties]: + requests_mock.get(f"{TEST_URI}v1/config", json={"defaults": {}, "overrides": {}}, status_code=200) + fake_auth_manager = MagicMock() + fake_auth_manager.auth_header.return_value = None + + with patch.dict("os.environ", environment, clear=True), patch.object(Config, "_from_configuration_files", return_value=None): + config = Config() + with ( + patch("pyiceberg.catalog._ENV_CONFIG", config), + patch("pyiceberg.catalog.rest.AuthManagerFactory.create", return_value=fake_auth_manager) as create_auth_manager, + ): + catalog = load_catalog("default", type="rest", uri=TEST_URI) + + assert isinstance(catalog, RestCatalog) + assert create_auth_manager.call_args_list + configured_auth_manager_call = create_auth_manager.call_args_list[0] + assert all(auth_manager_call == configured_auth_manager_call for auth_manager_call in create_auth_manager.call_args_list) + configured_manager, configured_properties = configured_auth_manager_call.args + assert isinstance(configured_manager, str) + assert isinstance(configured_properties, dict) + return configured_manager, cast(Properties, configured_properties) + + +def test_load_catalog_with_yaml_and_environment_noop_auth(tmp_path: Path, requests_mock: Mocker) -> None: + yaml_config = """ +catalog: + default: + auth: + type: noop +""" + environment = {"PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE": "noop"} + + yaml_auth_config = _assert_load_catalog_auth_config_from_yaml(yaml_config, tmp_path, requests_mock) + environment_auth_config = _assert_load_catalog_auth_config_from_environment(environment, requests_mock) + + assert yaml_auth_config == environment_auth_config == ("noop", {}) + + +def test_load_catalog_with_yaml_and_environment_basic_auth(tmp_path: Path, requests_mock: Mocker) -> None: + yaml_config = """ +catalog: + default: + auth: + type: basic + basic: + username: user + password: password +""" + environment = { + "PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE": "basic", + "PYICEBERG_CATALOG__DEFAULT__AUTH__BASIC__USERNAME": "user", + "PYICEBERG_CATALOG__DEFAULT__AUTH__BASIC__PASSWORD": "password", + } + yaml_auth_config = _assert_load_catalog_auth_config_from_yaml(yaml_config, tmp_path, requests_mock) + environment_auth_config = _assert_load_catalog_auth_config_from_environment(environment, requests_mock) + + assert yaml_auth_config == environment_auth_config == ("basic", {"username": "user", "password": "password"}) + + +def test_load_catalog_with_yaml_and_environment_custom_auth(tmp_path: Path, requests_mock: Mocker) -> None: + yaml_config = """ +catalog: + default: + auth: + type: custom + impl: pyiceberg.catalog.rest.auth.BasicAuthManager + custom: + username: user + password: password +""" + environment = { + "PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE": "custom", + "PYICEBERG_CATALOG__DEFAULT__AUTH__IMPL": "pyiceberg.catalog.rest.auth.BasicAuthManager", + "PYICEBERG_CATALOG__DEFAULT__AUTH__CUSTOM__USERNAME": "user", + "PYICEBERG_CATALOG__DEFAULT__AUTH__CUSTOM__PASSWORD": "password", + } + yaml_auth_config = _assert_load_catalog_auth_config_from_yaml(yaml_config, tmp_path, requests_mock) + environment_auth_config = _assert_load_catalog_auth_config_from_environment(environment, requests_mock) + + assert ( + yaml_auth_config + == environment_auth_config + == ( + "pyiceberg.catalog.rest.auth.BasicAuthManager", + {"username": "user", "password": "password"}, + ) + ) + + +def test_load_catalog_with_yaml_and_environment_oauth2_auth(tmp_path: Path, requests_mock: Mocker) -> None: + yaml_config = """ +catalog: + default: + auth: + type: oauth2 + oauth2: + client_id: client + client_secret: secret + token_url: https://identity.example.com/token + scope: catalog + refresh_margin: 30 + expires_in: 3600 +""" + environment = { + "PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE": "oauth2", + "PYICEBERG_CATALOG__DEFAULT__AUTH__OAUTH2__CLIENT_ID": "client", + "PYICEBERG_CATALOG__DEFAULT__AUTH__OAUTH2__CLIENT_SECRET": "secret", + "PYICEBERG_CATALOG__DEFAULT__AUTH__OAUTH2__TOKEN_URL": "https://identity.example.com/token", + "PYICEBERG_CATALOG__DEFAULT__AUTH__OAUTH2__SCOPE": "catalog", + "PYICEBERG_CATALOG__DEFAULT__AUTH__OAUTH2__REFRESH_MARGIN": "30", + "PYICEBERG_CATALOG__DEFAULT__AUTH__OAUTH2__EXPIRES_IN": "3600", + } + + yaml_auth_config = _assert_load_catalog_auth_config_from_yaml(yaml_config, tmp_path, requests_mock) + environment_auth_config = _assert_load_catalog_auth_config_from_environment(environment, requests_mock) + + yaml_manager, yaml_properties = yaml_auth_config + environment_manager, environment_properties = environment_auth_config + assert yaml_manager == environment_manager == "oauth2" + + # OAuth2AuthManager requires client_id, but environment variables cannot preserve `_`, + # so options such as client_id cannot be configured through environment variables. + # Both parsed configurations are asserted explicitly to document this limitation. + assert yaml_properties == { + "client_id": "client", + "client_secret": "secret", + "token_url": "https://identity.example.com/token", + "scope": "catalog", + "refresh_margin": "30", + "expires_in": "3600", + } + assert environment_properties == { + "client-id": "client", + "client-secret": "secret", + "token-url": "https://identity.example.com/token", + "scope": "catalog", + "refresh-margin": "30", + "expires-in": "3600", + } + + +def test_load_catalog_with_yaml_and_environment_google_auth(tmp_path: Path, requests_mock: Mocker) -> None: + yaml_config = """ +catalog: + default: + auth: + type: google + google: + credentials_path: /path/to/credentials.json + scopes: + - scope-a + - scope-b +""" + environment = { + "PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE": "google", + "PYICEBERG_CATALOG__DEFAULT__AUTH__GOOGLE__CREDENTIALS_PATH": "/path/to/credentials.json", + "PYICEBERG_CATALOG__DEFAULT__AUTH__GOOGLE__SCOPES": "scope-a,scope-b", + "GOOGLE_APPLICATION_CREDENTIALS": "/path/to/credentials.json", + } + + yaml_auth_config = _assert_load_catalog_auth_config_from_yaml(yaml_config, tmp_path, requests_mock) + environment_auth_config = _assert_load_catalog_auth_config_from_environment(environment, requests_mock) + + yaml_manager, yaml_properties = yaml_auth_config + environment_manager, environment_properties = environment_auth_config + assert yaml_manager == environment_manager == "google" + + # GoogleAuthManager requires credentials_path and list scopes, but environment variables cannot preserve `_` or lists, + # so these options cannot be configured through PyIceberg-prefixed environment variables. + # Both parsed configurations are asserted explicitly to document this limitation. + assert yaml_properties == { + "credentials_path": "/path/to/credentials.json", + "scopes": ["scope-a", "scope-b"], + } + assert environment_properties == { + "credentials-path": "/path/to/credentials.json", + "scopes": "scope-a,scope-b", + } + # Credentials can instead be configured through Google Auth's native environment variables, but scopes cannot. + assert environment["GOOGLE_APPLICATION_CREDENTIALS"] == yaml_properties["credentials_path"] + + +def test_load_catalog_with_yaml_and_environment_entra_auth(tmp_path: Path, requests_mock: Mocker) -> None: + yaml_config = """ +catalog: + default: + auth: + type: entra + entra: + scopes: + - https://storage.azure.com/.default + managed_identity_client_id: client-id +""" + environment = { + "PYICEBERG_CATALOG__DEFAULT__AUTH__TYPE": "entra", + "PYICEBERG_CATALOG__DEFAULT__AUTH__ENTRA__SCOPES": "https://storage.azure.com/.default", + "PYICEBERG_CATALOG__DEFAULT__AUTH__ENTRA__MANAGED_IDENTITY_CLIENT_ID": "client-id", + "AZURE_TENANT_ID": "tenant-id", + "AZURE_CLIENT_ID": "client-id", + "AZURE_CLIENT_SECRET": "client-secret", + "AZURE_AUTHORITY_HOST": "https://login.microsoftonline.com", + } + + yaml_auth_config = _assert_load_catalog_auth_config_from_yaml(yaml_config, tmp_path, requests_mock) + environment_auth_config = _assert_load_catalog_auth_config_from_environment(environment, requests_mock) + + yaml_manager, yaml_properties = yaml_auth_config + environment_manager, environment_properties = environment_auth_config + assert yaml_manager == environment_manager == "entra" + + # EntraAuthManager requires list scopes and managed_identity_client_id, + # but environment variables cannot preserve lists or `_`, so these options cannot be configured through them. + # Both parsed configurations are asserted explicitly to document this limitation. + assert yaml_properties == { + "scopes": ["https://storage.azure.com/.default"], + "managed_identity_client_id": "client-id", + } + assert environment_properties == { + "scopes": "https://storage.azure.com/.default", + "managed-identity-client-id": "client-id", + } + # Credentials can instead be configured through Azure Identity's native environment variables. + assert environment["AZURE_CLIENT_ID"] == yaml_properties["managed_identity_client_id"] + + +def test_load_catalog_environment_auth_config_overrides_yaml(tmp_path: Path, requests_mock: Mocker) -> None: + (tmp_path / ".pyiceberg.yaml").write_text( + """ +catalog: + default: + auth: + type: basic + basic: + username: yaml-user + password: yaml-password +""", + encoding="utf-8", + ) + environment = { + "PYICEBERG_HOME": str(tmp_path), + "PYICEBERG_CATALOG__DEFAULT__AUTH__BASIC__PASSWORD": "environment-password", + } + with patch.dict("os.environ", environment, clear=True): + config = Config() + + requests_mock.get(f"{TEST_URI}v1/config", json={"defaults": {}, "overrides": {}}, status_code=200) + fake_auth_manager = MagicMock() + fake_auth_manager.auth_header.return_value = None + + with ( + patch("pyiceberg.catalog._ENV_CONFIG", config), + patch("pyiceberg.catalog.rest.AuthManagerFactory.create", return_value=fake_auth_manager) as create_auth_manager, + ): + catalog = load_catalog("default", type="rest", uri=TEST_URI) + + assert isinstance(catalog, RestCatalog) + assert create_auth_manager.call_args_list + assert all( + auth_manager_call == call("basic", {"username": "yaml-user", "password": "environment-password"}) + for auth_manager_call in create_auth_manager.call_args_list + ) + + +def test_load_catalog_rejects_bare_auth_string() -> None: + config = MagicMock(spec=Config) + config.get_catalog_config.return_value = {"auth": "entra"} + + with patch("pyiceberg.catalog._ENV_CONFIG", config), pytest.raises(ValueError, match="PYICEBERG_CATALOG____AUTH__TYPE"): + load_catalog("default", type="rest", uri=TEST_URI) + + @pytest.fixture def rest_mock(requests_mock: Mocker) -> Mocker: requests_mock.get(