From 31efd7be504c96c65fc6c2210eb7b5c6280d2cdf Mon Sep 17 00:00:00 2001 From: zaihuaji Date: Thu, 14 May 2026 17:08:07 -0500 Subject: [PATCH 1/7] expand README.md and fix inheriting of class PgSplit() --- README.md | 209 ++++++++++++++++++++++++++++++ pyproject.toml | 2 +- src/rda_python_common/pg_split.py | 3 +- 3 files changed, 212 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d728563..f9102b6 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,211 @@ # rda-python-common + Python common library codes to be shared by other RDA python utility programs. + +## Modules + +All shared functionality lives under `src/rda_python_common/` and is organised as +a single-inheritance class hierarchy. Each module defines exactly one class; +later classes extend earlier ones, so an application that instantiates the +top-of-chain class (typically `PgOPT` or `PgCMD`) gets every helper through one +object. + +Inheritance tree (top-down; multi-inheritance shown as two arrows +converging on the same child): + +``` + PgLOG + ┌────┴────┐ + ▼ ▼ + PgUtil PgDBI + │ │ │ │ │ + │ └────┐ ┌─┘ │ └─► PgPassword + │ ▼ ▼ │ + │ PgSplit │ (multi-inherits + │ │ PgUtil + PgDBI) + │ ▼ + │ PgSIG + │ │ + │ ┌──────────┘ + ▼ ▼ + PgFile (multi-inherits + │ PgUtil + PgSIG) + ├─► PgOPT + │ + └─► PgLock + │ + └─► PgCMD +``` + +The tree is single inheritance everywhere except at two join points: + +- **`PgFile(PgUtil, PgSIG)`** — combines date/record utilities (`PgUtil` + via `PgLOG`) with daemon/signal/DB control (`PgSIG` → `PgDBI` → `PgLOG`), + so its descendants `PgOPT`, `PgLock`, and `PgCMD` inherit logging, DB, + util, signal, and file facilities through one MRO. +- **`PgSplit(PgUtil, PgDBI)`** — combines record-manipulation helpers + (`PgUtil`) with the `pgadd`/`pgget`/`pgmget`/`pgupdt`/`pgdel` DB + operations (`PgDBI`) it needs to keep the shared `wfile` table and the + per-dataset `wfile_` partitions in sync. + +- **`pg_log.py`** — `PgLOG`. Root of the hierarchy. Provides the central + logging facility (bit-mask `logact` flags such as `MSGLOG`, `WARNLG`, + `ERRLOG`, `EXITLG`), e-mail dispatch, system-command execution, process + metadata lookup, and the global `PGLOG` settings dictionary used by every + other module. + +- **`pg_util.py`** — `PgUtil(PgLOG)`. Miscellaneous date/time, dataset-ID, + and column-oriented record-manipulation helpers. Holds the `DATEFMTS` + regex table, `MONTHS`/`MNS`/`WDAYS`/`WDS` lookup lists, and the `MDAYS` + days-per-month array used for date arithmetic, formatting, parsing, and + record sort/search/classification across all RDA tools. + +- **`pg_file.py`** — `PgFile(PgUtil, PgSIG)`. Unified file-operation layer + spanning local file systems, remote hosts (rsync/ssh/scp), AWS S3 / object + store, and Globus endpoints. Used by `rdacp`, `dsarch`, `dsupdt`, and + related tools whenever data is moved, listed, or stat-ed. + +- **`pg_lock.py`** — `PgLock(PgFile)`. RDADB record-locking primitives for + the `dscheck`, `dsrqst`, `dlupdt`, `dcupdt`, `ptrqst`, and `dataset` + tables. Acquires, refreshes, and releases per-record locks so that + long-running batch jobs coordinate cleanly. + +- **`pg_dbi.py`** — `PgDBI(PgLOG)`. PostgreSQL database interface built on + `psycopg2`. Wraps connection management, batch `INSERT`/`SELECT`/ + `UPDATE`/`DELETE`, transaction control, and credential lookup from + `.pgpass` or OpenBao. All RDA tools talk to the `rdadb` database through + this class. + +- **`pg_sig.py`** — `PgSIG(PgDBI)`. Daemon process control, POSIX signal + handling, child/background-process management, and PBS/Torque batch-job + status queries. Provides the `PGSIG` runtime dictionary plus `VUSERS`, + `CPIDS`, `CBIDS`, and `SDUMP` tables that drive RDA daemon programs. + +- **`pg_cmd.py`** — `PgCMD(PgLock)`. Manages `dscheck` batch and delayed- + mode command tracking. Records, updates, and reaps the per-command rows + that let RDA utilities resume or be monitored across PBS batch jobs. + +- **`pg_split.py`** — `PgSplit(PgUtil, PgDBI)`. Synchronises `wfile` records + between the shared `wfile` table and the per-dataset `wfile_` + partition tables. Provides compare/add/update/delete helpers used when + archiving or reconciling dataset file inventories. + +- **`pg_opt.py`** — `PgOPT(PgFile)`. Command-line option parsing and + application configuration framework for RDA tools (`dsarch`, `dsupdt`, + `dsrqst`, ...). Holds the master `OPTS` definition table, parsed + `params`, command-line vs. input-file option tracking (`CMDOPTS`/ + `INOPTS`), output formatting, dataset/help/media/storage/backup type + maps, and the global `PGOPT` settings. + +- **`pgpassword.py`** — `PgPassword(PgDBI)`. Standalone CLI entry point + (`pgpassword`) that resolves a PostgreSQL login password from OpenBao + (`get_baopassword`) or `~/.pgpass` (`get_pg_pass`) given database/schema/ + host/port/user selectors via `-d`, `-c`, `-h`, `-p`, `-u`, `-l`, `-k`. + Prints the resolved password to stdout so shell scripts can capture it. + +## Usage examples + +Each class lives in its own submodule. Import the class you need, then +either instantiate it directly or subclass it to add application-specific +state and methods. + +### 1. Direct instantiation — use the helpers as-is + +```python +# Logging only +from rda_python_common.pg_log import PgLOG + +log = PgLOG() +log.pglog("dsarch started", log.LOGWRN) + +# Database access (PgDBI inherits PgLOG, so you get logging too) +from rda_python_common.pg_dbi import PgDBI + +db = PgDBI() +rec = db.pgget('dataset', 'dsid, title', "dsid = 'd633000'") +print(rec) +``` + +### 2. Subclassing a single common class + +```python +# A small utility that needs date/record helpers plus logging. +from rda_python_common.pg_util import PgUtil + +class DateReport(PgUtil): + def __init__(self): + super().__init__() # initialise PgUtil (and PgLOG) + self.today = self.curtime() # method inherited from PgUtil + + def run(self): + self.pglog(f"report date: {self.today}", self.LOGWRN) + +DateReport().run() +``` + +### 3. Subclassing one of the multi-inheriting joins + +```python +# A worker that needs file I/O (PgFile) and dscheck command tracking (PgCMD). +# PgCMD already extends PgFile via PgLock, so a single base is enough. +from rda_python_common.pg_cmd import PgCMD + +class Worker(PgCMD): + def __init__(self): + super().__init__() + self.jobs = [] + + def archive_one(self, src, dst): + # PgFile method, available through the inheritance chain + self.local_copy_local(src, dst) + # PgDBI method, available through PgCMD -> PgLock -> PgFile -> PgSIG -> PgDBI + self.pgupdt('wfile', {'status': 'A'}, f"wfile = '{dst}'") + +Worker().archive_one('/in/file', '/out/file') +``` + +### 4. Combining multiple common classes (application action class) + +This mirrors how RDA tools such as `dsarch` are structured. The leaf class +multi-inherits several common classes so a single object exposes options, +command tracking, and wfile splitting. + +```python +# Excerpt of the pattern used by rda_python_dsarch/dsarch.py +from rda_python_common.pg_opt import PgOPT +from rda_python_common.pg_cmd import PgCMD +from rda_python_common.pg_split import PgSplit + +class PgArch(PgOPT, PgCMD, PgSplit): + """Shared state + helpers for a CLI archiving tool.""" + def __init__(self): + super().__init__() + self.RTPATH = {} # runtime path cache + self.OPTS = {} # option table (populated by subclass) + +class DsArch(PgArch): + def __init__(self): + super().__init__() + self.ALLCNT = self.ADDCNT = self.MODCNT = 0 + + def main(self): + self.read_parameters() # from PgOPT + self.start_actions() # dispatch + +if __name__ == "__main__": + DsArch().main() +``` + +### 5. Reading a PostgreSQL password from OpenBao or ~/.pgpass + +```python +from rda_python_common.pgpassword import PgPassword + +pw = PgPassword() +pw.default_scinfo('rdadb', 'dssdb', 'rda-pgdb', 'gdexweb', None, 5432) +password = pw.get_baopassword() or pw.get_pg_pass() +``` + +In every case `super().__init__()` cooperates correctly across the +multi-inheriting joins (`PgFile` and `PgSplit`), so subclasses only need +to call it once. diff --git a/pyproject.toml b/pyproject.toml index 460f153..81668c9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "rda_python_common" -version = "2.1.7" +version = "2.1.8" authors = [ { name="Zaihua Ji", email="zji@ucar.edu" }, ] diff --git a/src/rda_python_common/pg_split.py b/src/rda_python_common/pg_split.py index 1f93206..e1e319d 100644 --- a/src/rda_python_common/pg_split.py +++ b/src/rda_python_common/pg_split.py @@ -12,8 +12,9 @@ import re from os import path as op from .pg_util import PgUtil +from .pg_dbi import PgDBI -class PgSplit(PgUtil): +class PgSplit(PgUtil, PgDBI): """Manages synchronisation of wfile records between shared and per-dataset tables. Handles compare, add, update, and delete operations between the shared From e754b2c5a8d92e15928e98d1f24a44082c83c090 Mon Sep 17 00:00:00 2001 From: zaihuaji Date: Thu, 14 May 2026 17:44:30 -0500 Subject: [PATCH 2/7] Fix bugs, add hvac dependency, and speed up hot-path string checks Bug fixes: - pgpassword.py: call get_pgpassword() (was get_pg_pass() which does not exist) - pg_log.py: replace two bare except: with KeyError / OSError - pg_lock.py, pg_split.py: correct header date typos - pgpassword.py: tighten arg regex so negative numbers are not parsed as flags Packaging: - pyproject.toml: declare hvac as a dependency (used by pg_dbi.py / pgpassword) API cleanup: - pg_dbi.py: drop meaningless lambda fn=self defaults on the fmt* helpers Efficiency (~30 sites across pg_util/pg_dbi/pg_log/pg_sig/pg_file/pg_opt): - re.match(r'^literal', x) -> x.startswith('literal') - re.match(r'^\d+$', s) -> s.isdigit() - re.search(r'\n$', s) -> s.endswith('\n') - re.match(r'^X', fmt, re.I) chain -> precomputed fmt[:1].upper() - chained range comparisons (a > 0 and a < N) -> 0 < a < N Co-Authored-By: Claude Opus 4.6 --- README.md | 2 +- pyproject.toml | 3 +- src/rda_python_common/pg_dbi.py | 14 ++++----- src/rda_python_common/pg_file.py | 4 +-- src/rda_python_common/pg_lock.py | 2 +- src/rda_python_common/pg_log.py | 12 ++++---- src/rda_python_common/pg_opt.py | 2 +- src/rda_python_common/pg_sig.py | 2 +- src/rda_python_common/pg_split.py | 2 +- src/rda_python_common/pg_util.py | 47 +++++++++++++++-------------- src/rda_python_common/pgpassword.py | 4 +-- 11 files changed, 48 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index f9102b6..5cf8d6f 100644 --- a/README.md +++ b/README.md @@ -203,7 +203,7 @@ from rda_python_common.pgpassword import PgPassword pw = PgPassword() pw.default_scinfo('rdadb', 'dssdb', 'rda-pgdb', 'gdexweb', None, 5432) -password = pw.get_baopassword() or pw.get_pg_pass() +password = pw.get_baopassword() or pw.get_pgpassword() ``` In every case `super().__init__()` cooperates correctly across the diff --git a/pyproject.toml b/pyproject.toml index 81668c9..33392ab 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,7 +20,8 @@ classifiers = [ dependencies = [ "psycopg2-binary", "rda-python-globus", - "unidecode" + "unidecode", + "hvac" ] [project.urls] diff --git a/src/rda_python_common/pg_dbi.py b/src/rda_python_common/pg_dbi.py index 193f7e7..096e612 100644 --- a/src/rda_python_common/pg_dbi.py +++ b/src/rda_python_common/pg_dbi.py @@ -62,12 +62,12 @@ def __init__(self): super().__init__() # initialize parent class # PostgreSQL specified query timestamp format - self.fmtyr = lambda fn=self: "extract(year from {})::int".format(fn) - self.fmtqt = lambda fn=self: "extract(quarter from {})::int".format(fn) - self.fmtmn = lambda fn=self: "extract(month from {})::int".format(fn) - self.fmtdt = lambda fn=self: "date({})".format(fn) - self.fmtym = lambda fn=self: "to_char({}, 'yyyy-mm')".format(fn) - self.fmthr = lambda fn=self: "extract(hour from {})::int".format(fn) + self.fmtyr = lambda fn: "extract(year from {})::int".format(fn) + self.fmtqt = lambda fn: "extract(quarter from {})::int".format(fn) + self.fmtmn = lambda fn: "extract(month from {})::int".format(fn) + self.fmtdt = lambda fn: "date({})".format(fn) + self.fmtym = lambda fn: "to_char({}, 'yyyy-mm')".format(fn) + self.fmthr = lambda fn: "extract(hour from {})::int".format(fn) self.pgdb = None # reference to a connected database object self.curtran = 0 # 0 - no transaction, 1 - in transaction @@ -577,7 +577,7 @@ def check_dberror(self, pgerr, pgcnt, sqlstr, ary, logact = None): self.qelog(dberror, 0, "Retry Connecting", ary, pgcnt, self.LOGWRN) self.pgconnect(1, pgcnt + 1) return (self.FAILURE if not self.pgdb else self.SUCCESS) - elif re.match(r'^55', pgcode): # try to lock again + elif pgcode.startswith('55'): # try to lock again self.qelog(dberror, 10, "Retry Locking", ary, pgcnt, self.LOGWRN) return self.SUCCESS elif pgcode == '25P02': # try to add table diff --git a/src/rda_python_common/pg_file.py b/src/rda_python_common/pg_file.py index 69502d9..d320a1c 100644 --- a/src/rda_python_common/pg_file.py +++ b/src/rda_python_common/pg_file.py @@ -2595,7 +2595,7 @@ def ftp_file_stat(self, line, opt): if opt&17: dy = int(items[6]) mn = self.get_month(items[5]) - if re.match(r'^\d+$', items[7]): + if items[7].isdigit(): yr = int(items[7]) mtime = "00:00:00" else: @@ -2972,7 +2972,7 @@ def record_delete_directory(self, dir, val): if dir is None: if isinstance(val, int): self.DIRLVLS = val - elif re.match(r'^\d+$', val): + elif val.isdigit(): self.DIRLVLS = int(val) elif dir and not re.match(r'^(\.|\./|/)$', dir) and dir not in self.DELDIRS: self.DELDIRS[dir] = val diff --git a/src/rda_python_common/pg_lock.py b/src/rda_python_common/pg_lock.py index fd592fc..5cb9036 100644 --- a/src/rda_python_common/pg_lock.py +++ b/src/rda_python_common/pg_lock.py @@ -1,7 +1,7 @@ ############################################################################### # Title: pg_lock.py # Author: Zaihua Ji, zji@ucar.edu -# Date: 08/118/2020 +# Date: 08/18/2020 # 2025-01-10 transferred to package rda_python_common from # https://github.com/NCAR/rda-shared-libraries.git # 2025-12-01 convert to class PgLock diff --git a/src/rda_python_common/pg_log.py b/src/rda_python_common/pg_log.py index 5488811..65256b9 100644 --- a/src/rda_python_common/pg_log.py +++ b/src/rda_python_common/pg_log.py @@ -252,7 +252,7 @@ def set_email(self, msg, logact=0): msg = self.PGLOG['PRGMSG'] + "\n" + msg self.PGLOG['PRGMSG'] = "" if self.PGLOG['ERRCNT'] == 0: - if not re.search(r'\n$', msg): msg += "!\n" + if not msg.endswith('\n'): msg += "!\n" else: if self.PGLOG['ERRCNT'] == 1: msg += " with 1 Error:\n" @@ -1423,7 +1423,7 @@ def set_common_pglog(self): try: self.PGLOG['RDAUID'] = self.PGLOG['GDEXUID'] = pwd.getpwnam(self.PGLOG['GDEXUSER']).pw_uid self.PGLOG['RDAGID'] = self.PGLOG['GDEXGID'] = grp.getgrnam(self.PGLOG['GDEXGRP']).gr_gid - except: + except KeyError: self.PGLOG['RDAUID'] = self.PGLOG['GDEXUID'] = 0 self.PGLOG['RDAGID'] = self.PGLOG['GDEXGID'] = 0 if self.PGLOG['CURUID'] == self.PGLOG['GDEXUSER']: self.PGLOG['SETUID'] = self.PGLOG['GDEXUSER'] @@ -1624,8 +1624,8 @@ def set_specialist_environments(self, specialist): missthen = 0 try: rf = open(resource, 'r') - except: - return # skip if cannot open + except OSError: + return # skip if cannot open nline = rf.readline() while nline: line = self.pgtrim(nline) @@ -1638,12 +1638,12 @@ def set_specialist_environments(self, specialist): missthen = 0 if re.match(r'^then$', line): continue # then on next line checkif = 0 # end of inline if - elif re.match(r'^endif', line): + elif line.startswith('endif'): checkif = 0 # end of if continue elif checkif == -1: # skip the line continue - elif checkif == 2 and re.match(r'^else', line): + elif checkif == 2 and line.startswith('else'): checkif = -1 # done check envs in if continue if checkif == 1: diff --git a/src/rda_python_common/pg_opt.py b/src/rda_python_common/pg_opt.py index 05168a6..6e5ffa6 100644 --- a/src/rda_python_common/pg_opt.py +++ b/src/rda_python_common/pg_opt.py @@ -1158,7 +1158,7 @@ def set_option_value(self, opt, val=None, cnl=0, lidx=0, line=None, infile=None) if self.OPTS[opt][2]&16: if not val: val = 0 - elif re.match(r'^\d+$', val): + elif val.isdigit(): val = int(val) elif val and (opt == 'DS' or opt == 'OD'): val = self.format_dataset_id(val) diff --git a/src/rda_python_common/pg_sig.py b/src/rda_python_common/pg_sig.py index f27901b..1b83da1 100644 --- a/src/rda_python_common/pg_sig.py +++ b/src/rda_python_common/pg_sig.py @@ -961,7 +961,7 @@ def check_pbs_status(self, bid, logact=None): lines = buf.split('\n') for line in lines: if chkt: - if re.match(r'^Job', line): + if line.startswith('Job'): line = re.sub(r'^Job ID', 'JobID', line, 1) line = re.sub(r'Finish Time', 'FinishTime', line, 1) line = re.sub(r'Req Mem', 'ReqMem', line, 1) diff --git a/src/rda_python_common/pg_split.py b/src/rda_python_common/pg_split.py index e1e319d..92bd238 100644 --- a/src/rda_python_common/pg_split.py +++ b/src/rda_python_common/pg_split.py @@ -1,7 +1,7 @@ ############################################################################### # Title: pg_split.py -- PostgreSQL DataBase Interface foe table wfile # Author: Zaihua Ji, zji@ucar.edu -# Date: 09/010/2024 +# Date: 09/10/2024 # 2025-01-10 transferred to package rda_python_common from # https://github.com/NCAR/rda-shared-libraries.git # 2025-12-01 convert to class PgSplit diff --git a/src/rda_python_common/pg_util.py b/src/rda_python_common/pg_util.py index e0ff151..8586e31 100644 --- a/src/rda_python_common/pg_util.py +++ b/src/rda_python_common/pg_util.py @@ -93,22 +93,22 @@ def get_month(self, mn, fmt = None): int | str: Numeric month (1-12) when fmt is None; formatted string otherwise. """ if not isinstance(mn, int): - if re.match(r'^\d+$', mn): + if mn.isdigit(): mn = int(mn) else: for m in range(12): if re.match(mn, self.MONTHS[m], re.I): mn = m + 1 break - if fmt and mn > 0 and mn < 13: + if fmt and 0 < mn < 13: slen = len(fmt) if slen == 2: smn = "{:02}".format(mn) - elif re.match(r'^mon', fmt, re.I): + elif fmt[:3].lower() == 'mon': smn = self.MNS[mn-1] if slen == 3 else self.MONTHS[mn-1] - if re.match(r'^Mon', fmt): + if fmt.startswith('Mon'): smn = smn.capitalize() - elif re.match(r'^MON', fmt): + elif fmt.startswith('MON'): smn = smn.upper() else: smn = str(mn) @@ -131,26 +131,26 @@ def get_wday(self, wday, fmt = None): formatted string otherwise. """ if not isinstance(wday, int): - if re.match(r'^\d+$', wday): + if wday.isdigit(): wday = int(wday) else: for w in range(7): if re.match(wday, self.WDAYS[w], re.I): wday = w break - if fmt and wday >= 0 and wday <= 6: + if fmt and 0 <= wday <= 6: slen = len(fmt) if slen == 4: swday = self.WDAYS[wday] - if re.match(r'^We', fmt): + if fmt.startswith('We'): swday = swday.capitalize() - elif re.match(r'^WE', fmt): + elif fmt.startswith('WE'): swday = swday.upper() elif slen == 3: swday = self.WDS[wday] - if re.match(r'^Ww', fmt): + if fmt.startswith('Ww'): swday = swday.capitalize() - elif re.match(r'^WW', fmt): + elif fmt.startswith('WW'): swday = swday.upper() else: swday = str(wday) @@ -179,7 +179,7 @@ def valid_online_file(file, type = None, exists = None): if exists is None or exists: if not op.exists(file): return '' # file does not exist bname = op.basename(file) - if re.match(r'^,.*', bname): return '' # hidden file + if bname.startswith(','): return '' # hidden file if re.search(r'index\.(htm|html|shtml)$', bname, re.I): return '' # index file if type and type != 'D': return type if re.search(r'\.(doc|php|html|shtml)(\.|$)', bname, re.I): return '' # file with special extention @@ -314,7 +314,7 @@ def check_datetime(date, default): """ if not date: return default if not isinstance(date, str): date = str(date) - if re.match(r'^0000', date): return default + if date.startswith('0000'): return default return date # fmt: date format, default to "YYYY-MM-DD" @@ -422,7 +422,7 @@ def split_datetime(sdt, sep = r'\D'): adt = re.split(sep, sdt) acnt = len(adt) for i in range(acnt): - if re.match(r'^\d+$', adt[i]): adt[i] = int(adt[i]) + if adt[i].isdigit(): adt[i] = int(adt[i]) return adt # date: given date in format of fromfmt @@ -490,19 +490,20 @@ def format_date(self, cdate, tofmt = None, fromfmt = None): if i >= mcnt: break fmt = formats[k] val = ms[0][i] - if re.match(r'^Y', fmt, re.I): + head = fmt[:1].upper() + if head == 'Y': dates[0] = int(val) if len(fmt) == 3: dates[0] *= 10 - elif re.match(r'^C', fmt, re.I): + elif head == 'C': dates[0] = 100 * int(val) # year at end of century - elif re.match(r'^M', fmt, re.I): - if re.match(r'^Mon', fmt, re.I): + elif head == 'M': + if fmt[:3].upper() == 'MON': dates[1] = self.get_month(val) else: dates[1] = int(val) - elif re.match(r'^Q', fmt, re.I): + elif head == 'Q': dates[1] = 3 * int(val) # month at end of quarter - elif re.match(r'^H', fmt, re.I): # hour + elif head == 'H': # hour dates.append(int(val)) else: # day dates[2] = int(val) @@ -653,11 +654,11 @@ def fmtdate(self, yr, mn, dy, tofmt = None): slen = len(fmt) if slen == 2: smn = "{:02}".format(m) - elif re.match(r'^mon', fmt, re.I): + elif fmt[:3].lower() == 'mon': smn = self.MNS[m-1] if slen == 3 else self.MONTHS[m-1] - if re.match(r'^Mo', fmt): + if fmt.startswith('Mo'): smn = smn.capitalize() - elif re.match(r'^MO', fmt): + elif fmt.startswith('MO'): smn = smn.upper() else: smn = str(m) diff --git a/src/rda_python_common/pgpassword.py b/src/rda_python_common/pgpassword.py index edfcc0c..ca33c4b 100644 --- a/src/rda_python_common/pgpassword.py +++ b/src/rda_python_common/pgpassword.py @@ -39,7 +39,7 @@ def read_parameters(self): opt = None dohelp = True for arg in argv: - if re.match(r'^-\w+$', arg): + if re.match(r'^-[a-zA-Z]\w*$', arg): opt = arg[1:] elif opt: if opt == 'l': @@ -72,7 +72,7 @@ def start_actions(self): self.default_scinfo(self.DBINFO['dbname'], self.DBINFO['scname'], self.DBINFO['dbhost'], self.DBINFO['lnname'], None, self.DBINFO['dbport']) self.password = self.get_baopassword() - if not self.password: self.password = self.get_pg_pass() + if not self.password: self.password = self.get_pgpassword() # main function to excecute this script def main(): From 52eb20c43c21f85368a354db86c2d81f6f38a7a1 Mon Sep 17 00:00:00 2001 From: zaihuaji Date: Thu, 14 May 2026 18:10:38 -0500 Subject: [PATCH 3/7] populate __init__.py with package docstring and __version__ Eagerly imports the legacy capitalized submodules so that ``from rda_python_common import PgLOG`` continues to return the module object that existing callers depend on. Documents both the legacy module-based API and the preferred class-based API. Co-Authored-By: Claude Opus 4.6 --- src/rda_python_common/__init__.py | 38 +++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/src/rda_python_common/__init__.py b/src/rda_python_common/__init__.py index e69de29..2914849 100644 --- a/src/rda_python_common/__init__.py +++ b/src/rda_python_common/__init__.py @@ -0,0 +1,38 @@ +"""rda_python_common: shared utility package for RDA Python tools. + +This package exposes two parallel APIs: + +1. Legacy module-based API (back-compat). Import the capitalized submodules + and call their module-level functions, e.g.:: + + from rda_python_common import PgLOG + PgLOG.pglog("message", PgLOG.LOGWRN) + +2. Class-based API (preferred for new code). Import the class from the + lower-case module and either instantiate or subclass it, e.g.:: + + from rda_python_common.pg_log import PgLOG + log = PgLOG() + log.pglog("message", log.LOGWRN) + +The legacy submodules are eagerly imported below so that +``from rda_python_common import PgLOG`` continues to return the module +object that existing callers expect. +""" + +from . import PgLOG, PgUtil, PgDBI, PgFile, PgLock, PgCMD, PgSIG, PgOPT, PgSplit + +__version__ = "2.1.8" + +__all__ = [ + "PgLOG", + "PgUtil", + "PgDBI", + "PgFile", + "PgLock", + "PgCMD", + "PgSIG", + "PgOPT", + "PgSplit", + "__version__", +] From 4d5118a312993232fe076338f1f41cb86043b5f8 Mon Sep 17 00:00:00 2001 From: Zaihua Ji Date: Fri, 15 May 2026 11:04:32 -0500 Subject: [PATCH 4/7] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- src/rda_python_common/pgpassword.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rda_python_common/pgpassword.py b/src/rda_python_common/pgpassword.py index ca33c4b..c1e31f2 100644 --- a/src/rda_python_common/pgpassword.py +++ b/src/rda_python_common/pgpassword.py @@ -39,7 +39,7 @@ def read_parameters(self): opt = None dohelp = True for arg in argv: - if re.match(r'^-[a-zA-Z]\w*$', arg): + if re.match(r'^-[a-zA-Z]$', arg): opt = arg[1:] elif opt: if opt == 'l': From a38b31c900750d7a16e6b6dcc08a126019260aa7 Mon Sep 17 00:00:00 2001 From: Zaihua Ji Date: Fri, 15 May 2026 11:04:48 -0500 Subject: [PATCH 5/7] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5cf8d6f..2f56d05 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ The tree is single inheritance everywhere except at two join points: - **`pgpassword.py`** — `PgPassword(PgDBI)`. Standalone CLI entry point (`pgpassword`) that resolves a PostgreSQL login password from OpenBao - (`get_baopassword`) or `~/.pgpass` (`get_pg_pass`) given database/schema/ + (`get_baopassword`) or `~/.pgpass` (`get_pgpassword()`) given database/schema/ host/port/user selectors via `-d`, `-c`, `-h`, `-p`, `-u`, `-l`, `-k`. Prints the resolved password to stdout so shell scripts can capture it. From 16272cfef5b6a5140e1732f8445840ab5e111bbb Mon Sep 17 00:00:00 2001 From: zaihuaji Date: Fri, 15 May 2026 11:12:52 -0500 Subject: [PATCH 6/7] add pgpassword.usg and docstrings; move help text out of pgpassword.py Moves the inline usage prints out of read_parameters() into a pgpassword.usg file displayed by show_usage(), matching the pattern used by other RDA console scripts. Also adds module, class, and method docstrings to pgpassword.py. Co-Authored-By: Claude Opus 4.6 --- src/rda_python_common/pgpassword.py | 65 +++++++++++++++++++++++----- src/rda_python_common/pgpassword.usg | 45 +++++++++++++++++++ 2 files changed, 98 insertions(+), 12 deletions(-) create mode 100644 src/rda_python_common/pgpassword.usg diff --git a/src/rda_python_common/pgpassword.py b/src/rda_python_common/pgpassword.py index c1e31f2..ff69e40 100644 --- a/src/rda_python_common/pgpassword.py +++ b/src/rda_python_common/pgpassword.py @@ -4,17 +4,42 @@ # Author: Zaihua Ji, zji@ucar.edu # Date: 2025-10-27 # 2025-12-02 convert to class PgPassword -# Purpose: python script to retrieve passwords for postgrsql login to connect a +# Purpose: python script to retrieve passwords for postgresql login to connect a # gdex database from inside an python application # Github: https://github.com/NCAR/rda-python-common.git ################################################################################## +""" +pgpassword.py - Command-line helper that retrieves a PostgreSQL password. + +Provides the PgPassword class and a ``main`` entry point used by the +``pgpassword`` console script. The password is looked up first from +OpenBao (using the URL/token configured in PgDBI) and, if not found, +from the user's ``.pgpass`` file. The result is printed to stdout so +that shell wrappers and other RDA utilities can capture it. +""" import sys import re from .pg_dbi import PgDBI class PgPassword(PgDBI): + """ + Command-line helper for retrieving a PostgreSQL login password. + + Inherits from PgDBI to reuse its database-connection metadata + (PGDBI), default schema handling, and password-lookup methods + (``get_baopassword`` / ``get_pgpassword``). + + Instance attributes set in __init__: + DBFLDS -- mapping of CLI option letters to PGDBI field names + DBINFO -- per-invocation overrides for dbname/scname/lnname/ + dbhost/dbport supplied via CLI options + dbopt -- True once at least one DB-override option is seen, + triggering ``default_scinfo`` before lookup + password -- the retrieved password (set by ``start_actions``) + """ def __init__(self): + """Initialize PgPassword with empty DB-override info and option maps.""" super().__init__() # initialize parent class self.DBFLDS = { 'd': 'dbname', @@ -34,7 +59,23 @@ def __init__(self): self.password = '' # read in command line parameters - def read_parameters(self): + def read_parameters(self): + """ + Parse ``sys.argv`` and apply CLI overrides. + + Recognized options: + -l URL -- OpenBao URL (stored in self.PGDBI['BAOURL']) + -k TOKEN -- OpenBao token name (stored in self.PGDBI['BAOTOKEN']) + -d NAME -- PostgreSQL database name + -c NAME -- PostgreSQL schema name + -u NAME -- PostgreSQL login user name + -h HOST -- PostgreSQL server host name + -p PORT -- PostgreSQL port number + + If no arguments are supplied a usage message is printed and the + process exits with status 0. Unknown options or stray values + cause an immediate error exit via ``self.pglog(..., LGEREX)``. + """ argv = sys.argv[1:] opt = None dohelp = True @@ -55,19 +96,18 @@ def read_parameters(self): else: self.pglog(arg + ": Value provided without option", self.LGEREX) if dohelp: - print("Usage: pgpassword [-l OpenBaoURL] [-k TokenName] [-d DBNAME] \\") - print(" [-c SCHEMA] [-u USName] [-h DBHOST] [-p DBPORT]") - print(" -l OpenBao URL to retrieve passwords") - print(" -k OpenBao Token Name to retrieve passwords") - print(" -d PostgreSQL Database Name") - print(" -c PostgreSQL Schema Name") - print(" -u PostgreSQL Login User Name") - print(" -h PostgreSQL Server Host Name") - print(" -p PostgreSQL Port Number") - sys.exit(0) + self.set_help_path(__file__) + self.show_usage("pgpassword") # get the pgpassword def start_actions(self): + """ + Look up the password and store it in ``self.password``. + + Applies any CLI-supplied DB overrides via ``default_scinfo``, then + tries OpenBao first (``get_baopassword``) and falls back to the + ``.pgpass`` file (``get_pgpassword``) if OpenBao returns nothing. + """ if self.dbopt: self.default_scinfo(self.DBINFO['dbname'], self.DBINFO['scname'], self.DBINFO['dbhost'], self.DBINFO['lnname'], None, self.DBINFO['dbport']) @@ -76,6 +116,7 @@ def start_actions(self): # main function to excecute this script def main(): + """Entry point for the ``pgpassword`` console script: print the retrieved password to stdout.""" object = PgPassword() object.read_parameters() object.start_actions() diff --git a/src/rda_python_common/pgpassword.usg b/src/rda_python_common/pgpassword.usg new file mode 100644 index 0000000..309c11f --- /dev/null +++ b/src/rda_python_common/pgpassword.usg @@ -0,0 +1,45 @@ + + Retrieve a PostgreSQL login password for use by RDA python applications. + The password is looked up first from OpenBao (using the URL and token + configured in PgDBI) and, if not found there, from the user's .pgpass + file. The retrieved password is printed to stdout so that shell + wrappers and other RDA utilities can capture it. + + Usage: pgpassword [-l OpenBaoURL] [-k TokenName] [-d DBNAME] \ + [-c SCHEMA] [-u USName] [-h DBHOST] [-p DBPORT] + + - Option -l, OpenBao URL used to retrieve passwords. Overrides + the BAOURL value from the PgDBI configuration; + + - Option -k, OpenBao token name used to authenticate the password + lookup. Overrides the BAOTOKEN value from the PgDBI + configuration; + + - Option -d, PostgreSQL database name; + + - Option -c, PostgreSQL schema name; + + - Option -u, PostgreSQL login user name; + + - Option -h, PostgreSQL server host name; + + - Option -p, PostgreSQL server port number (default: 5432); + + If any of -d, -c, -u, -h, or -p is supplied, the values are applied + via default_scinfo() before the password lookup is performed. With no + arguments, this usage information is displayed and the program exits. + + Examples: + + 1. Retrieve the password for the default database/user configured + in PgDBI: + + pgpassword + + 2. Retrieve the password for a specific database, schema and user: + + pgpassword -d rdadb -c dssdb -u metauser + + 3. Retrieve the password from a custom OpenBao endpoint: + + pgpassword -l https://bao.example.org -k my-token -d rdadb From 4256dec653d2f5dff0cb8f4b0b320eeafc50557d Mon Sep 17 00:00:00 2001 From: zaihuaji Date: Fri, 15 May 2026 11:16:57 -0500 Subject: [PATCH 7/7] add default values for db options in pgpassword.usg Documents the effective defaults inherited from PgDBI/PgLOG for the -l, -d, -c, -u, -h, and -p options so users know what they get when each option is omitted. Co-Authored-By: Claude Opus 4.6 --- src/rda_python_common/pgpassword.usg | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/rda_python_common/pgpassword.usg b/src/rda_python_common/pgpassword.usg index 309c11f..1661040 100644 --- a/src/rda_python_common/pgpassword.usg +++ b/src/rda_python_common/pgpassword.usg @@ -9,21 +9,28 @@ [-c SCHEMA] [-u USName] [-h DBHOST] [-p DBPORT] - Option -l, OpenBao URL used to retrieve passwords. Overrides - the BAOURL value from the PgDBI configuration; + the BAOURL value from the PgDBI configuration. + Default: https://bao.k8s.ucar.edu/ - Option -k, OpenBao token name used to authenticate the password lookup. Overrides the BAOTOKEN value from the PgDBI - configuration; + configuration. - - Option -d, PostgreSQL database name; + - Option -d, PostgreSQL database name. + Default: rdadb - - Option -c, PostgreSQL schema name; + - Option -c, PostgreSQL schema name. + Default: dssdb - - Option -u, PostgreSQL login user name; + - Option -u, PostgreSQL login user name. + Default: dssdb (same as the -c default) - - Option -h, PostgreSQL server host name; + - Option -h, PostgreSQL server host name. + Default: value of the DSSDBHOST environment variable, or + rda-db.ucar.edu if DSSDBHOST is not set. - - Option -p, PostgreSQL server port number (default: 5432); + - Option -p, PostgreSQL server port number. + Default: 5432 If any of -d, -c, -u, -h, or -p is supplied, the values are applied via default_scinfo() before the password lookup is performed. With no