Hua work common

README.md

@@ -1,2 +1,211 @@
 # rda-python-common
+
 Python common library codes to be shared by other RDA python utility programs.
+
+## Modules
+
+All shared functionality lives under `src/rda_python_common/` and is organised as
+a single-inheritance class hierarchy.  Each module defines exactly one class;
+later classes extend earlier ones, so an application that instantiates the
+top-of-chain class (typically `PgOPT` or `PgCMD`) gets every helper through one
+object.
+
+Inheritance tree (top-down; multi-inheritance shown as two arrows
+converging on the same child):
+
+```
+                          PgLOG
+                       ┌────┴────┐
+                       ▼         ▼
+                    PgUtil     PgDBI
+                     │ │        │ │ │
+                     │ └────┐ ┌─┘ │ └─► PgPassword
+                     │      ▼ ▼   │
+                     │    PgSplit │       (multi-inherits
+                     │            │        PgUtil + PgDBI)
+                     │            ▼
+                     │          PgSIG
+                     │            │
+                     │ ┌──────────┘
+                     ▼ ▼
+                   PgFile                 (multi-inherits
+                     │                     PgUtil + PgSIG)
+                     ├─► PgOPT
+                     │
+                     └─► PgLock
+                          │
+                          └─► PgCMD
+```
+
+The tree is single inheritance everywhere except at two join points:
+
+- **`PgFile(PgUtil, PgSIG)`** — combines date/record utilities (`PgUtil`
+  via `PgLOG`) with daemon/signal/DB control (`PgSIG` → `PgDBI` → `PgLOG`),
+  so its descendants `PgOPT`, `PgLock`, and `PgCMD` inherit logging, DB,
+  util, signal, and file facilities through one MRO.
+- **`PgSplit(PgUtil, PgDBI)`** — combines record-manipulation helpers
+  (`PgUtil`) with the `pgadd`/`pgget`/`pgmget`/`pgupdt`/`pgdel` DB
+  operations (`PgDBI`) it needs to keep the shared `wfile` table and the
+  per-dataset `wfile_<dsid>` partitions in sync.
+
+- **`pg_log.py`** — `PgLOG`.  Root of the hierarchy.  Provides the central
+  logging facility (bit-mask `logact` flags such as `MSGLOG`, `WARNLG`,
+  `ERRLOG`, `EXITLG`), e-mail dispatch, system-command execution, process
+  metadata lookup, and the global `PGLOG` settings dictionary used by every
+  other module.
+
+- **`pg_util.py`** — `PgUtil(PgLOG)`.  Miscellaneous date/time, dataset-ID,
+  and column-oriented record-manipulation helpers.  Holds the `DATEFMTS`
+  regex table, `MONTHS`/`MNS`/`WDAYS`/`WDS` lookup lists, and the `MDAYS`
+  days-per-month array used for date arithmetic, formatting, parsing, and
+  record sort/search/classification across all RDA tools.
+
+- **`pg_file.py`** — `PgFile(PgUtil, PgSIG)`.  Unified file-operation layer
+  spanning local file systems, remote hosts (rsync/ssh/scp), AWS S3 / object
+  store, and Globus endpoints.  Used by `rdacp`, `dsarch`, `dsupdt`, and
+  related tools whenever data is moved, listed, or stat-ed.
+
+- **`pg_lock.py`** — `PgLock(PgFile)`.  RDADB record-locking primitives for
+  the `dscheck`, `dsrqst`, `dlupdt`, `dcupdt`, `ptrqst`, and `dataset`
+  tables.  Acquires, refreshes, and releases per-record locks so that
+  long-running batch jobs coordinate cleanly.
+
+- **`pg_dbi.py`** — `PgDBI(PgLOG)`.  PostgreSQL database interface built on
+  `psycopg2`.  Wraps connection management, batch `INSERT`/`SELECT`/
+  `UPDATE`/`DELETE`, transaction control, and credential lookup from
+  `.pgpass` or OpenBao.  All RDA tools talk to the `rdadb` database through
+  this class.
+
+- **`pg_sig.py`** — `PgSIG(PgDBI)`.  Daemon process control, POSIX signal
+  handling, child/background-process management, and PBS/Torque batch-job
+  status queries.  Provides the `PGSIG` runtime dictionary plus `VUSERS`,
+  `CPIDS`, `CBIDS`, and `SDUMP` tables that drive RDA daemon programs.
+
+- **`pg_cmd.py`** — `PgCMD(PgLock)`.  Manages `dscheck` batch and delayed-
+  mode command tracking.  Records, updates, and reaps the per-command rows
+  that let RDA utilities resume or be monitored across PBS batch jobs.
+
+- **`pg_split.py`** — `PgSplit(PgUtil, PgDBI)`.  Synchronises `wfile` records
+  between the shared `wfile` table and the per-dataset `wfile_<dsid>`
+  partition tables.  Provides compare/add/update/delete helpers used when
+  archiving or reconciling dataset file inventories.
+
+- **`pg_opt.py`** — `PgOPT(PgFile)`.  Command-line option parsing and
+  application configuration framework for RDA tools (`dsarch`, `dsupdt`,
+  `dsrqst`, ...).  Holds the master `OPTS` definition table, parsed
+  `params`, command-line vs. input-file option tracking (`CMDOPTS`/
+  `INOPTS`), output formatting, dataset/help/media/storage/backup type
+  maps, and the global `PGOPT` settings.
+
+- **`pgpassword.py`** — `PgPassword(PgDBI)`.  Standalone CLI entry point
+  (`pgpassword`) that resolves a PostgreSQL login password from OpenBao
+  (`get_baopassword`) or `~/.pgpass` (`get_pgpassword()`) given database/schema/
+  host/port/user selectors via `-d`, `-c`, `-h`, `-p`, `-u`, `-l`, `-k`.
+  Prints the resolved password to stdout so shell scripts can capture it.
+
+## Usage examples
+
+Each class lives in its own submodule.  Import the class you need, then
+either instantiate it directly or subclass it to add application-specific
+state and methods.
+
+### 1. Direct instantiation — use the helpers as-is
+
+```python
+# Logging only
+from rda_python_common.pg_log import PgLOG
+
+log = PgLOG()
+log.pglog("dsarch started", log.LOGWRN)
+
+# Database access (PgDBI inherits PgLOG, so you get logging too)
+from rda_python_common.pg_dbi import PgDBI
+
+db = PgDBI()
+rec = db.pgget('dataset', 'dsid, title', "dsid = 'd633000'")
+print(rec)
+```
+
+### 2. Subclassing a single common class
+
+```python
+# A small utility that needs date/record helpers plus logging.
+from rda_python_common.pg_util import PgUtil
+
+class DateReport(PgUtil):
+   def __init__(self):
+      super().__init__()           # initialise PgUtil (and PgLOG)
+      self.today = self.curtime()  # method inherited from PgUtil
+
+   def run(self):
+      self.pglog(f"report date: {self.today}", self.LOGWRN)
+
+DateReport().run()
+```
+
+### 3. Subclassing one of the multi-inheriting joins
+
+```python
+# A worker that needs file I/O (PgFile) and dscheck command tracking (PgCMD).
+# PgCMD already extends PgFile via PgLock, so a single base is enough.
+from rda_python_common.pg_cmd import PgCMD
+
+class Worker(PgCMD):
+   def __init__(self):
+      super().__init__()
+      self.jobs = []
+
+   def archive_one(self, src, dst):
+      # PgFile method, available through the inheritance chain
+      self.local_copy_local(src, dst)
+      # PgDBI method, available through PgCMD -> PgLock -> PgFile -> PgSIG -> PgDBI
+      self.pgupdt('wfile', {'status': 'A'}, f"wfile = '{dst}'")
+
+Worker().archive_one('/in/file', '/out/file')
+```
+
+### 4. Combining multiple common classes (application action class)
+
+This mirrors how RDA tools such as `dsarch` are structured.  The leaf class
+multi-inherits several common classes so a single object exposes options,
+command tracking, and wfile splitting.
+
+```python
+# Excerpt of the pattern used by rda_python_dsarch/dsarch.py
+from rda_python_common.pg_opt   import PgOPT
+from rda_python_common.pg_cmd   import PgCMD
+from rda_python_common.pg_split import PgSplit
+
+class PgArch(PgOPT, PgCMD, PgSplit):
+   """Shared state + helpers for a CLI archiving tool."""
+   def __init__(self):
+      super().__init__()
+      self.RTPATH = {}          # runtime path cache
+      self.OPTS   = {}          # option table (populated by subclass)
+
+class DsArch(PgArch):
+   def __init__(self):
+      super().__init__()
+      self.ALLCNT = self.ADDCNT = self.MODCNT = 0
+
+   def main(self):
+      self.read_parameters()    # from PgOPT
+      self.start_actions()      # dispatch
+
+if __name__ == "__main__":
+   DsArch().main()
+```
+
+### 5. Reading a PostgreSQL password from OpenBao or ~/.pgpass
+
+```python
+from rda_python_common.pgpassword import PgPassword
+
+pw = PgPassword()
+pw.default_scinfo('rdadb', 'dssdb', 'rda-pgdb', 'gdexweb', None, 5432)
+password = pw.get_baopassword() or pw.get_pgpassword()
+```
+
+In every case `super().__init__()` cooperates correctly across the
+multi-inheriting joins (`PgFile` and `PgSplit`), so subclasses only need
+to call it once.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rda_python_common"
-version = "2.1.7"
+version = "2.1.8"
 authors = [
   { name="Zaihua Ji", email="zji@ucar.edu" },
 ]
@@ -20,7 +20,8 @@ classifiers = [
 dependencies = [
   "psycopg2-binary",
   "rda-python-globus",
-  "unidecode"
+  "unidecode",
+  "hvac"
 ]
 
 [project.urls]

src/rda_python_common/__init__.py

@@ -0,0 +1,38 @@
+"""rda_python_common: shared utility package for RDA Python tools.
+
+This package exposes two parallel APIs:
+
+1. Legacy module-based API (back-compat). Import the capitalized submodules
+   and call their module-level functions, e.g.::
+
+       from rda_python_common import PgLOG
+       PgLOG.pglog("message", PgLOG.LOGWRN)
+
+2. Class-based API (preferred for new code). Import the class from the
+   lower-case module and either instantiate or subclass it, e.g.::
+
+       from rda_python_common.pg_log import PgLOG
+       log = PgLOG()
+       log.pglog("message", log.LOGWRN)
+
+The legacy submodules are eagerly imported below so that
+``from rda_python_common import PgLOG`` continues to return the module
+object that existing callers expect.
+"""
+
+from . import PgLOG, PgUtil, PgDBI, PgFile, PgLock, PgCMD, PgSIG, PgOPT, PgSplit
+
+__version__ = "2.1.8"
+
+__all__ = [
+   "PgLOG",
+   "PgUtil",
+   "PgDBI",
+   "PgFile",
+   "PgLock",
+   "PgCMD",
+   "PgSIG",
+   "PgOPT",
+   "PgSplit",
+   "__version__",
+]

src/rda_python_common/pg_dbi.py

@@ -62,12 +62,12 @@ def __init__(self):
       super().__init__()  # initialize parent class
 
       #  PostgreSQL specified query timestamp format
-      self.fmtyr = lambda fn=self: "extract(year from {})::int".format(fn)
-      self.fmtqt = lambda fn=self: "extract(quarter from {})::int".format(fn)
-      self.fmtmn = lambda fn=self: "extract(month from {})::int".format(fn)
-      self.fmtdt = lambda fn=self: "date({})".format(fn)
-      self.fmtym = lambda fn=self: "to_char({}, 'yyyy-mm')".format(fn)
-      self.fmthr = lambda fn=self: "extract(hour from {})::int".format(fn)
+      self.fmtyr = lambda fn: "extract(year from {})::int".format(fn)
+      self.fmtqt = lambda fn: "extract(quarter from {})::int".format(fn)
+      self.fmtmn = lambda fn: "extract(month from {})::int".format(fn)
+      self.fmtdt = lambda fn: "date({})".format(fn)
+      self.fmtym = lambda fn: "to_char({}, 'yyyy-mm')".format(fn)
+      self.fmthr = lambda fn: "extract(hour from {})::int".format(fn)
 
       self.pgdb = None    # reference to a connected database object
       self.curtran = 0    # 0 - no transaction, 1 - in transaction
@@ -577,7 +577,7 @@ def check_dberror(self, pgerr, pgcnt, sqlstr, ary, logact = None):
             self.qelog(dberror, 0, "Retry Connecting", ary, pgcnt, self.LOGWRN)
             self.pgconnect(1, pgcnt + 1)
             return (self.FAILURE if not self.pgdb else self.SUCCESS)
-         elif re.match(r'^55', pgcode):  #  try to lock again
+         elif pgcode.startswith('55'):  #  try to lock again
             self.qelog(dberror, 10, "Retry Locking", ary, pgcnt, self.LOGWRN)
             return self.SUCCESS
          elif pgcode == '25P02':   #  try to add table

src/rda_python_common/pg_file.py

@@ -2595,7 +2595,7 @@ def ftp_file_stat(self, line, opt):
       if opt&17:
          dy = int(items[6])
          mn = self.get_month(items[5])
-         if re.match(r'^\d+$', items[7]):
+         if items[7].isdigit():
             yr = int(items[7])
             mtime = "00:00:00"
          else:
@@ -2972,7 +2972,7 @@ def record_delete_directory(self, dir, val):
       if dir is None:
          if isinstance(val, int):
             self.DIRLVLS = val
-         elif re.match(r'^\d+$', val):
+         elif val.isdigit():
             self.DIRLVLS = int(val)
       elif dir and not re.match(r'^(\.|\./|/)$', dir) and dir not in self.DELDIRS:
          self.DELDIRS[dir] = val

src/rda_python_common/pg_lock.py

@@ -1,7 +1,7 @@
 ###############################################################################
 #     Title: pg_lock.py
 #    Author: Zaihua Ji,  zji@ucar.edu
-#      Date: 08/118/2020
+#      Date: 08/18/2020
 #             2025-01-10 transferred to package rda_python_common from
 #             https://github.com/NCAR/rda-shared-libraries.git
 #             2025-12-01 convert to class PgLock

src/rda_python_common/pg_log.py

@@ -252,7 +252,7 @@ def set_email(self, msg, logact=0):
                msg = self.PGLOG['PRGMSG'] + "\n" + msg
                self.PGLOG['PRGMSG'] = ""
             if self.PGLOG['ERRCNT'] == 0:
-               if not re.search(r'\n$', msg): msg += "!\n"
+               if not msg.endswith('\n'): msg += "!\n"
             else:
                if self.PGLOG['ERRCNT'] == 1:
                   msg += " with 1 Error:\n"
@@ -1423,7 +1423,7 @@ def set_common_pglog(self):
       try:
          self.PGLOG['RDAUID'] = self.PGLOG['GDEXUID'] = pwd.getpwnam(self.PGLOG['GDEXUSER']).pw_uid
          self.PGLOG['RDAGID'] = self.PGLOG['GDEXGID'] = grp.getgrnam(self.PGLOG['GDEXGRP']).gr_gid
-      except:
+      except KeyError:
          self.PGLOG['RDAUID'] = self.PGLOG['GDEXUID'] = 0
          self.PGLOG['RDAGID'] = self.PGLOG['GDEXGID'] = 0
       if self.PGLOG['CURUID'] == self.PGLOG['GDEXUSER']: self.PGLOG['SETUID'] = self.PGLOG['GDEXUSER']   
@@ -1624,8 +1624,8 @@ def set_specialist_environments(self, specialist):
       missthen = 0
       try:
          rf = open(resource, 'r')
-      except:
-         return   # skip if cannot open   
+      except OSError:
+         return   # skip if cannot open
       nline = rf.readline()
       while nline:
          line = self.pgtrim(nline)
@@ -1638,12 +1638,12 @@ def set_specialist_environments(self, specialist):
             missthen = 0
             if re.match(r'^then$', line): continue   # then on next line
             checkif = 0   # end of inline if
-         elif re.match(r'^endif', line):
+         elif line.startswith('endif'):
             checkif = 0   # end of if
             continue
          elif checkif == -1:   # skip the line
             continue
-         elif checkif == 2 and re.match(r'^else', line):
+         elif checkif == 2 and line.startswith('else'):
             checkif = -1   # done check envs in if
             continue
          if checkif == 1:

src/rda_python_common/pg_opt.py

@@ -1158,7 +1158,7 @@ def set_option_value(self, opt, val=None, cnl=0, lidx=0, line=None, infile=None)
          if self.OPTS[opt][2]&16:
             if not val:
                val = 0
-            elif re.match(r'^\d+$', val):
+            elif val.isdigit():
                val = int(val)
          elif val and (opt == 'DS' or opt == 'OD'):
             val = self.format_dataset_id(val)

src/rda_python_common/pg_sig.py

@@ -961,7 +961,7 @@ def check_pbs_status(self, bid, logact=None):
       lines = buf.split('\n')
       for line in lines:
          if chkt:
-            if re.match(r'^Job', line):
+            if line.startswith('Job'):
                line = re.sub(r'^Job ID', 'JobID', line, 1)
                line = re.sub(r'Finish Time', 'FinishTime', line, 1)
                line = re.sub(r'Req Mem', 'ReqMem', line, 1)