Hua work common

README.md

@@ -2,12 +2,30 @@
 
 Python common library codes to be shared by other RDA python utility programs.
 
-## Installing and using in another RDA python repo
+## Environment setup
 
-`rda-python-common` is the foundation that every other `rda-python-*` repo
-builds on.  To consume it from a new or existing repo, follow these steps.
+Create a Python environment first; the install command in the next section
+runs inside whichever environment you activate here.
+
+### Option A — Python venv (DECS machines)
+
+```bash
+python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
+source $ENVHOME/bin/activate
+```
+
+### Option B — Conda (DAV/Casper)
 
-### 1. Install the package
+```bash
+conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
+conda activate $ENVHOME
+```
+
+## Installing rda-python-common
+
+Pick whichever install mode fits your workflow.  All four pull in the
+transitive dependencies (`psycopg`, `rda-python-globus`, `unidecode`,
+`hvac`) automatically.
 
 For local development, clone this repo alongside your project and install it
 in editable mode so that changes are picked up without re-installing:
@@ -18,6 +36,15 @@ cd rda-python-common
 pip install -e .
 ```
 
+To test a specific branch (e.g. an in-progress feature or fix branch), pass
+`-b/--branch` to `git clone`:
+
+```bash
+git clone -b <branch-name> https://github.com/NCAR/rda-python-common.git
+cd rda-python-common
+pip install -e .
+```
+
 For a regular (non-editable) install from a checkout:
 
 ```bash
@@ -30,10 +57,75 @@ For a production install on a system that uses the published distribution:
 pip install rda_python_common
 ```
 
-The package brings in its own transitive dependencies (`psycopg2-binary`,
-`rda-python-globus`, `unidecode`, `hvac`).
+### PostgreSQL driver: psycopg v3 (default) and psycopg2 (fallback)
+
+`rda-python-common` uses **psycopg v3** by default.  `pg_dbi.py`
+auto-detects which driver is installed at import time and prefers psycopg v3
+when both are present; no code changes are needed to switch drivers.
 
-### 2. Declare it as a dependency in your project
+The required dependency is the base `psycopg` package, which works whether
+psycopg was compiled from source or installed via a binary wheel.  If psycopg
+is not available on your system, install whichever driver works:
+
+```bash
+pip install psycopg || pip install psycopg2
+```
+
+To explicitly install the legacy psycopg2 driver:
+
+```bash
+pip install "rda_python_common[psycopg2]"          # build from source
+pip install "rda_python_common[psycopg2-binary]"   # pre-built wheel
+```
+
+## Configuration: COMMONUSER and ADMINUSER
+
+`PGLOG['COMMONUSER']` is the shared common user that setuid-wrapped programs
+execute as (default `gdexdata`), and `PGLOG['ADMINUSER']` is the admin
+specialist user that receives email notifications and is permitted to invoke
+`pgstart_<user>` (default `zji`).
+
+Both values are initialized via the `SETPGLOG(key, default)` helper, which
+reads the environment variable `PG<KEY>` and falls back to the supplied
+default when the variable is unset:
+
+```python
+# pg_log.py (class-based)
+self.SETPGLOG("COMMONUSER", "gdexdata")   # reads $PGCOMMONUSER
+self.SETPGLOG("ADMINUSER",  "zji")        # reads $PGADMINUSER
+
+# PgLOG.py (module-level) exposes the same helper as a function
+SETPGLOG("COMMONUSER", "gdexdata")
+SETPGLOG("ADMINUSER",  "zji")
+```
+
+To override the defaults per environment **once** so the values persist
+across `pip install --upgrade`, set the environment variables:
+
+```bash
+export PGCOMMONUSER=gdexdata    # overrides PGLOG['COMMONUSER']
+export PGADMINUSER=zji          # overrides PGLOG['ADMINUSER']
+```
+
+Place these `export` lines in `$ENVHOME/bin/activate` (venv), or set them as
+conda environment variables so they are applied whenever the environment is
+activated:
+
+```bash
+conda env config vars set PGCOMMONUSER=gdexdata PGADMINUSER=zji
+conda activate $ENVHOME   # reactivate to pick up the values
+```
+
+If the variables are unset, the built-in defaults (`gdexdata` / `zji`) are
+used, preserving existing behavior.
+
+## Using rda-python-common in another RDA python repo
+
+`rda-python-common` is the foundation that every other `rda-python-*` repo
+builds on.  Once it is installed in the active environment, consuming it from
+a new or existing repo takes three short steps.
+
+### 1. Declare it as a dependency in your project
 
 Add `rda_python_common` to the `dependencies` list of your project's
 `pyproject.toml` so that downstream installs pull it in automatically:
@@ -52,9 +144,10 @@ This is the same pattern used by `rda-python-dsarch`, `rda-python-dsupdt`,
 `rda-python-dsrqst`, `rda-python-dscheck`, `rda-python-metrics`, and
 `rda-python-miscs`.
 
-### 3. Import the modules you need
+### 2. Import the modules you need
 
-Two import styles are supported (see [Usage examples](#usage-examples) below):
+Two import styles are supported (see [Usage examples](#usage-examples) below
+for fuller patterns):
 
 ```python
 # Preferred for new code -- import the class from the lower-case module
@@ -66,26 +159,26 @@ from rda_python_common import PgLOG, PgDBI
 PgLOG.pglog("hello", PgLOG.LOGWRN)
 ```
 
-### 4. Verify the install
+### 3. Verify the install
 
 ```bash
 python -c "import rda_python_common; print(rda_python_common.__version__)"
 ```
 
-You should see the installed version (currently `2.1.11`).  If the import
+You should see the installed version (currently `3.0.0`).  If the import
 fails, double-check that the active Python environment is the one where you
 ran `pip install`.
 
 ## Modules
 
-All shared functionality lives under `src/rda_python_common/` and is organised as
-a single-inheritance class hierarchy.  Each module defines exactly one class;
-later classes extend earlier ones, so an application that instantiates the
-top-of-chain class (typically `PgOPT` or `PgCMD`) gets every helper through one
-object.
+All shared functionality lives under `src/rda_python_common/` and is organised
+as a (mostly) single-inheritance class hierarchy.  Each module defines exactly
+one class; later classes extend earlier ones, so an application that
+instantiates the top-of-chain class (typically `PgOPT` or `PgCMD`) gets every
+helper through one object.
 
-Inheritance tree (top-down; multi-inheritance shown as two arrows
-converging on the same child):
+The inheritance tree below is read top-down; the two multi-inheritance joins
+are shown as two arrows converging on the same child:
 
 ```
                           PgLOG
@@ -122,6 +215,8 @@ The tree is single inheritance everywhere except at two join points:
   operations (`PgDBI`) it needs to keep the shared `wfile` table and the
   per-dataset `wfile_<dsid>` partitions in sync.
 
+Each class lives in its own module.  Walking the tree from the root:
+
 - **`pg_log.py`** — `PgLOG`.  Root of the hierarchy.  Provides the central
   logging facility (bit-mask `logact` flags such as `MSGLOG`, `WARNLG`,
   `ERRLOG`, `EXITLG`), e-mail dispatch, system-command execution, process
@@ -145,7 +240,8 @@ The tree is single inheritance everywhere except at two join points:
   long-running batch jobs coordinate cleanly.
 
 - **`pg_dbi.py`** — `PgDBI(PgLOG)`.  PostgreSQL database interface built on
-  `psycopg2`.  Wraps connection management, batch `INSERT`/`SELECT`/
+  `psycopg` (v3 by default, with `psycopg2` as an opt-in fallback).  Wraps
+  connection management, batch `INSERT`/`SELECT`/
   `UPDATE`/`DELETE`, transaction control, and credential lookup from
   `.pgpass` or OpenBao.  All RDA tools talk to the `rdadb` database through
   this class.
@@ -179,9 +275,9 @@ The tree is single inheritance everywhere except at two join points:
 
 ## Usage examples
 
-Each class lives in its own submodule.  Import the class you need, then
-either instantiate it directly or subclass it to add application-specific
-state and methods.
+The patterns below show the typical ways the classes above are used in
+practice.  Import the class you need, then either instantiate it directly or
+subclass it to add application-specific state and methods.
 
 ### 1. Direct instantiation — use the helpers as-is
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rda_python_common"
-version = "2.1.11"
+version = "3.0.0"
 authors = [
   { name="Zaihua Ji", email="zji@ucar.edu" },
 ]
@@ -18,13 +18,20 @@ classifiers = [
     "Development Status :: 5 - Production/Stable",
 ]
 dependencies = [
-  "psycopg2-binary",
+  "psycopg",
   "psutil",
   "rda-python-globus",
   "unidecode",
   "hvac"
 ]
 
+[project.optional-dependencies]
+# Allow opting in to the legacy psycopg2 driver instead of psycopg (v3).
+# pg_dbi.py auto-detects which driver is installed and prefers psycopg (v3)
+# when both are available.
+psycopg2        = ["psycopg2"]           # psycopg2 built from source
+psycopg2-binary = ["psycopg2-binary"]    # psycopg2 pre-built C extension
+
 [project.urls]
 "Homepage" = "https://github.com/NCAR/rda-python-common"
 

requirements.txt

@@ -1,7 +1,7 @@
 iniconfig==2.1.0
 packaging==24.2
 pluggy==1.5.0
-psycopg2-binary==2.9.10
+psycopg[binary]
 psutil
 pytest==8.3.5
 rda-python-globus

src/rda_python_common/PgDBI.py

@@ -17,12 +17,53 @@
 import time
 import hvac
 from datetime import datetime
-import psycopg2 as PgSQL
-from psycopg2.extras import execute_values
-from psycopg2.extras import execute_batch
 from os import path as op
 from . import PgLOG
 
+# Prefer psycopg (v3); fall back to psycopg2 if v3 is not installed.
+try:
+   import psycopg as PgSQL
+   PG_DRIVER = 'psycopg3'
+
+   def execute_values(cursor, sql, argslist, page_size=100):
+      """Compatibility shim for psycopg2.extras.execute_values on psycopg3.
+
+      Rewrites ``VALUES %s`` placeholder to ``VALUES (%s, %s, ...)`` based on
+      the column count inferred from the first row, then dispatches to
+      psycopg3's ``executemany`` (which already batches efficiently).
+      """
+      if not argslist: return
+      ncol = len(argslist[0])
+      row_ph = '(' + ','.join(['%s'] * ncol) + ')'
+      new_sql = re.sub(r'(?i)\bVALUES\s+%s\b', 'VALUES ' + row_ph, sql, count=1)
+      cursor.executemany(new_sql, argslist)
+
+   def execute_batch(cursor, sql, argslist, page_size=100):
+      """Compatibility shim for psycopg2.extras.execute_batch on psycopg3."""
+      cursor.executemany(sql, argslist)
+
+   def get_pgcode(pgerr):
+      """Return SQLSTATE for a psycopg3 error (via err.diag.sqlstate)."""
+      diag = getattr(pgerr, 'diag', None)
+      return getattr(diag, 'sqlstate', None) if diag is not None else None
+
+   def get_pgerror(pgerr):
+      """Return primary error message for a psycopg3 error (via err.diag.message_primary)."""
+      diag = getattr(pgerr, 'diag', None)
+      return getattr(diag, 'message_primary', None) if diag is not None else None
+except ImportError:
+   import psycopg2 as PgSQL
+   from psycopg2.extras import execute_values, execute_batch
+   PG_DRIVER = 'psycopg2'
+
+   def get_pgcode(pgerr):
+      """Return SQLSTATE for a psycopg2 error (via err.pgcode)."""
+      return getattr(pgerr, 'pgcode', None)
+
+   def get_pgerror(pgerr):
+      """Return primary error message for a psycopg2 error (via err.pgerror)."""
+      return getattr(pgerr, 'pgerror', None)
+
 pgdb = None    # reference to a connected database object
 curtran = 0    # 0 - no transaction, 1 - in transaction
 NMISSES = []   # array of mising userno
@@ -439,8 +480,8 @@ def check_dberror(pgerr, pgcnt, sqlstr, ary, logact = PGDBI['ERRLOG']):
 
    ret = PgLOG.FAILURE
 
-   pgcode = pgerr.pgcode
-   pgerror = pgerr.pgerror
+   pgcode = get_pgcode(pgerr)
+   pgerror = get_pgerror(pgerr)
    dberror = "{} {}".format(pgcode, pgerror) if pgcode and pgerror else str(pgerr)
    if pgcnt < PgLOG.PGLOG['DBRETRY']:
       if not pgcode:
@@ -517,7 +558,7 @@ def pgconnect(reconnect = 0, pgcnt = 0, autocommit = True):
       reconnect = 0   # initial connection
 
    while True:
-      config = {'database' : PGDBI['DBNAME'],
+      config = {'dbname' : PGDBI['DBNAME'],
                     'user' : PGDBI['LNNAME']}
       if PGDBI['DBSHOST'] == PgLOG.PGLOG['HOSTNAME']:
          config['host'] = 'localhost'
@@ -526,7 +567,7 @@ def pgconnect(reconnect = 0, pgcnt = 0, autocommit = True):
          if not PGDBI['DBPORT']: PGDBI['DBPORT'] = get_dbport(PGDBI['DBNAME'])
       if PGDBI['DBPORT']: config['port'] = PGDBI['DBPORT']
       config['password'] = '***'
-      sqlstr = "psycopg2.connect(**{})".format(config)
+      sqlstr = "{}.connect(**{})".format(PG_DRIVER, config)
       config['password'] = get_pgpass_password()
       if PgLOG.PGLOG['DBGLEVEL']: PgLOG.pgdbg(1000, sqlstr)
       try:

src/rda_python_common/PgFile.py

@@ -788,7 +788,7 @@ def delete_backup_file(file, endpoint = None, logact = 0):
    return PgLOG.FAILURE
 
 #
-# reset local file/directory information to make them writable for PgLOG.PGLOG['GDEXUSER']
+# reset local file/directory information to make them writable for PgLOG.PGLOG['COMMONUSER']
 # file - file name (mandatory)
 # info - gathered file info with option 14, None means file not exists
 #