diff --git a/README.md b/README.md index d728563..2f56d05 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,211 @@ # rda-python-common + Python common library codes to be shared by other RDA python utility programs. + +## Modules + +All shared functionality lives under `src/rda_python_common/` and is organised as +a single-inheritance class hierarchy. Each module defines exactly one class; +later classes extend earlier ones, so an application that instantiates the +top-of-chain class (typically `PgOPT` or `PgCMD`) gets every helper through one +object. + +Inheritance tree (top-down; multi-inheritance shown as two arrows +converging on the same child): + +``` + PgLOG + ┌────┴────┐ + ▼ ▼ + PgUtil PgDBI + │ │ │ │ │ + │ └────┐ ┌─┘ │ └─► PgPassword + │ ▼ ▼ │ + │ PgSplit │ (multi-inherits + │ │ PgUtil + PgDBI) + │ ▼ + │ PgSIG + │ │ + │ ┌──────────┘ + ▼ ▼ + PgFile (multi-inherits + │ PgUtil + PgSIG) + ├─► PgOPT + │ + └─► PgLock + │ + └─► PgCMD +``` + +The tree is single inheritance everywhere except at two join points: + +- **`PgFile(PgUtil, PgSIG)`** — combines date/record utilities (`PgUtil` + via `PgLOG`) with daemon/signal/DB control (`PgSIG` → `PgDBI` → `PgLOG`), + so its descendants `PgOPT`, `PgLock`, and `PgCMD` inherit logging, DB, + util, signal, and file facilities through one MRO. +- **`PgSplit(PgUtil, PgDBI)`** — combines record-manipulation helpers + (`PgUtil`) with the `pgadd`/`pgget`/`pgmget`/`pgupdt`/`pgdel` DB + operations (`PgDBI`) it needs to keep the shared `wfile` table and the + per-dataset `wfile_` partitions in sync. + +- **`pg_log.py`** — `PgLOG`. Root of the hierarchy. Provides the central + logging facility (bit-mask `logact` flags such as `MSGLOG`, `WARNLG`, + `ERRLOG`, `EXITLG`), e-mail dispatch, system-command execution, process + metadata lookup, and the global `PGLOG` settings dictionary used by every + other module. + +- **`pg_util.py`** — `PgUtil(PgLOG)`. Miscellaneous date/time, dataset-ID, + and column-oriented record-manipulation helpers. Holds the `DATEFMTS` + regex table, `MONTHS`/`MNS`/`WDAYS`/`WDS` lookup lists, and the `MDAYS` + days-per-month array used for date arithmetic, formatting, parsing, and + record sort/search/classification across all RDA tools. + +- **`pg_file.py`** — `PgFile(PgUtil, PgSIG)`. Unified file-operation layer + spanning local file systems, remote hosts (rsync/ssh/scp), AWS S3 / object + store, and Globus endpoints. Used by `rdacp`, `dsarch`, `dsupdt`, and + related tools whenever data is moved, listed, or stat-ed. + +- **`pg_lock.py`** — `PgLock(PgFile)`. RDADB record-locking primitives for + the `dscheck`, `dsrqst`, `dlupdt`, `dcupdt`, `ptrqst`, and `dataset` + tables. Acquires, refreshes, and releases per-record locks so that + long-running batch jobs coordinate cleanly. + +- **`pg_dbi.py`** — `PgDBI(PgLOG)`. PostgreSQL database interface built on + `psycopg2`. Wraps connection management, batch `INSERT`/`SELECT`/ + `UPDATE`/`DELETE`, transaction control, and credential lookup from + `.pgpass` or OpenBao. All RDA tools talk to the `rdadb` database through + this class. + +- **`pg_sig.py`** — `PgSIG(PgDBI)`. Daemon process control, POSIX signal + handling, child/background-process management, and PBS/Torque batch-job + status queries. Provides the `PGSIG` runtime dictionary plus `VUSERS`, + `CPIDS`, `CBIDS`, and `SDUMP` tables that drive RDA daemon programs. + +- **`pg_cmd.py`** — `PgCMD(PgLock)`. Manages `dscheck` batch and delayed- + mode command tracking. Records, updates, and reaps the per-command rows + that let RDA utilities resume or be monitored across PBS batch jobs. + +- **`pg_split.py`** — `PgSplit(PgUtil, PgDBI)`. Synchronises `wfile` records + between the shared `wfile` table and the per-dataset `wfile_` + partition tables. Provides compare/add/update/delete helpers used when + archiving or reconciling dataset file inventories. + +- **`pg_opt.py`** — `PgOPT(PgFile)`. Command-line option parsing and + application configuration framework for RDA tools (`dsarch`, `dsupdt`, + `dsrqst`, ...). Holds the master `OPTS` definition table, parsed + `params`, command-line vs. input-file option tracking (`CMDOPTS`/ + `INOPTS`), output formatting, dataset/help/media/storage/backup type + maps, and the global `PGOPT` settings. + +- **`pgpassword.py`** — `PgPassword(PgDBI)`. Standalone CLI entry point + (`pgpassword`) that resolves a PostgreSQL login password from OpenBao + (`get_baopassword`) or `~/.pgpass` (`get_pgpassword()`) given database/schema/ + host/port/user selectors via `-d`, `-c`, `-h`, `-p`, `-u`, `-l`, `-k`. + Prints the resolved password to stdout so shell scripts can capture it. + +## Usage examples + +Each class lives in its own submodule. Import the class you need, then +either instantiate it directly or subclass it to add application-specific +state and methods. + +### 1. Direct instantiation — use the helpers as-is + +```python +# Logging only +from rda_python_common.pg_log import PgLOG + +log = PgLOG() +log.pglog("dsarch started", log.LOGWRN) + +# Database access (PgDBI inherits PgLOG, so you get logging too) +from rda_python_common.pg_dbi import PgDBI + +db = PgDBI() +rec = db.pgget('dataset', 'dsid, title', "dsid = 'd633000'") +print(rec) +``` + +### 2. Subclassing a single common class + +```python +# A small utility that needs date/record helpers plus logging. +from rda_python_common.pg_util import PgUtil + +class DateReport(PgUtil): + def __init__(self): + super().__init__() # initialise PgUtil (and PgLOG) + self.today = self.curtime() # method inherited from PgUtil + + def run(self): + self.pglog(f"report date: {self.today}", self.LOGWRN) + +DateReport().run() +``` + +### 3. Subclassing one of the multi-inheriting joins + +```python +# A worker that needs file I/O (PgFile) and dscheck command tracking (PgCMD). +# PgCMD already extends PgFile via PgLock, so a single base is enough. +from rda_python_common.pg_cmd import PgCMD + +class Worker(PgCMD): + def __init__(self): + super().__init__() + self.jobs = [] + + def archive_one(self, src, dst): + # PgFile method, available through the inheritance chain + self.local_copy_local(src, dst) + # PgDBI method, available through PgCMD -> PgLock -> PgFile -> PgSIG -> PgDBI + self.pgupdt('wfile', {'status': 'A'}, f"wfile = '{dst}'") + +Worker().archive_one('/in/file', '/out/file') +``` + +### 4. Combining multiple common classes (application action class) + +This mirrors how RDA tools such as `dsarch` are structured. The leaf class +multi-inherits several common classes so a single object exposes options, +command tracking, and wfile splitting. + +```python +# Excerpt of the pattern used by rda_python_dsarch/dsarch.py +from rda_python_common.pg_opt import PgOPT +from rda_python_common.pg_cmd import PgCMD +from rda_python_common.pg_split import PgSplit + +class PgArch(PgOPT, PgCMD, PgSplit): + """Shared state + helpers for a CLI archiving tool.""" + def __init__(self): + super().__init__() + self.RTPATH = {} # runtime path cache + self.OPTS = {} # option table (populated by subclass) + +class DsArch(PgArch): + def __init__(self): + super().__init__() + self.ALLCNT = self.ADDCNT = self.MODCNT = 0 + + def main(self): + self.read_parameters() # from PgOPT + self.start_actions() # dispatch + +if __name__ == "__main__": + DsArch().main() +``` + +### 5. Reading a PostgreSQL password from OpenBao or ~/.pgpass + +```python +from rda_python_common.pgpassword import PgPassword + +pw = PgPassword() +pw.default_scinfo('rdadb', 'dssdb', 'rda-pgdb', 'gdexweb', None, 5432) +password = pw.get_baopassword() or pw.get_pgpassword() +``` + +In every case `super().__init__()` cooperates correctly across the +multi-inheriting joins (`PgFile` and `PgSplit`), so subclasses only need +to call it once. diff --git a/pyproject.toml b/pyproject.toml index 460f153..33392ab 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "rda_python_common" -version = "2.1.7" +version = "2.1.8" authors = [ { name="Zaihua Ji", email="zji@ucar.edu" }, ] @@ -20,7 +20,8 @@ classifiers = [ dependencies = [ "psycopg2-binary", "rda-python-globus", - "unidecode" + "unidecode", + "hvac" ] [project.urls] diff --git a/src/rda_python_common/__init__.py b/src/rda_python_common/__init__.py index e69de29..2914849 100644 --- a/src/rda_python_common/__init__.py +++ b/src/rda_python_common/__init__.py @@ -0,0 +1,38 @@ +"""rda_python_common: shared utility package for RDA Python tools. + +This package exposes two parallel APIs: + +1. Legacy module-based API (back-compat). Import the capitalized submodules + and call their module-level functions, e.g.:: + + from rda_python_common import PgLOG + PgLOG.pglog("message", PgLOG.LOGWRN) + +2. Class-based API (preferred for new code). Import the class from the + lower-case module and either instantiate or subclass it, e.g.:: + + from rda_python_common.pg_log import PgLOG + log = PgLOG() + log.pglog("message", log.LOGWRN) + +The legacy submodules are eagerly imported below so that +``from rda_python_common import PgLOG`` continues to return the module +object that existing callers expect. +""" + +from . import PgLOG, PgUtil, PgDBI, PgFile, PgLock, PgCMD, PgSIG, PgOPT, PgSplit + +__version__ = "2.1.8" + +__all__ = [ + "PgLOG", + "PgUtil", + "PgDBI", + "PgFile", + "PgLock", + "PgCMD", + "PgSIG", + "PgOPT", + "PgSplit", + "__version__", +] diff --git a/src/rda_python_common/pg_dbi.py b/src/rda_python_common/pg_dbi.py index 193f7e7..096e612 100644 --- a/src/rda_python_common/pg_dbi.py +++ b/src/rda_python_common/pg_dbi.py @@ -62,12 +62,12 @@ def __init__(self): super().__init__() # initialize parent class # PostgreSQL specified query timestamp format - self.fmtyr = lambda fn=self: "extract(year from {})::int".format(fn) - self.fmtqt = lambda fn=self: "extract(quarter from {})::int".format(fn) - self.fmtmn = lambda fn=self: "extract(month from {})::int".format(fn) - self.fmtdt = lambda fn=self: "date({})".format(fn) - self.fmtym = lambda fn=self: "to_char({}, 'yyyy-mm')".format(fn) - self.fmthr = lambda fn=self: "extract(hour from {})::int".format(fn) + self.fmtyr = lambda fn: "extract(year from {})::int".format(fn) + self.fmtqt = lambda fn: "extract(quarter from {})::int".format(fn) + self.fmtmn = lambda fn: "extract(month from {})::int".format(fn) + self.fmtdt = lambda fn: "date({})".format(fn) + self.fmtym = lambda fn: "to_char({}, 'yyyy-mm')".format(fn) + self.fmthr = lambda fn: "extract(hour from {})::int".format(fn) self.pgdb = None # reference to a connected database object self.curtran = 0 # 0 - no transaction, 1 - in transaction @@ -577,7 +577,7 @@ def check_dberror(self, pgerr, pgcnt, sqlstr, ary, logact = None): self.qelog(dberror, 0, "Retry Connecting", ary, pgcnt, self.LOGWRN) self.pgconnect(1, pgcnt + 1) return (self.FAILURE if not self.pgdb else self.SUCCESS) - elif re.match(r'^55', pgcode): # try to lock again + elif pgcode.startswith('55'): # try to lock again self.qelog(dberror, 10, "Retry Locking", ary, pgcnt, self.LOGWRN) return self.SUCCESS elif pgcode == '25P02': # try to add table diff --git a/src/rda_python_common/pg_file.py b/src/rda_python_common/pg_file.py index 69502d9..d320a1c 100644 --- a/src/rda_python_common/pg_file.py +++ b/src/rda_python_common/pg_file.py @@ -2595,7 +2595,7 @@ def ftp_file_stat(self, line, opt): if opt&17: dy = int(items[6]) mn = self.get_month(items[5]) - if re.match(r'^\d+$', items[7]): + if items[7].isdigit(): yr = int(items[7]) mtime = "00:00:00" else: @@ -2972,7 +2972,7 @@ def record_delete_directory(self, dir, val): if dir is None: if isinstance(val, int): self.DIRLVLS = val - elif re.match(r'^\d+$', val): + elif val.isdigit(): self.DIRLVLS = int(val) elif dir and not re.match(r'^(\.|\./|/)$', dir) and dir not in self.DELDIRS: self.DELDIRS[dir] = val diff --git a/src/rda_python_common/pg_lock.py b/src/rda_python_common/pg_lock.py index fd592fc..5cb9036 100644 --- a/src/rda_python_common/pg_lock.py +++ b/src/rda_python_common/pg_lock.py @@ -1,7 +1,7 @@ ############################################################################### # Title: pg_lock.py # Author: Zaihua Ji, zji@ucar.edu -# Date: 08/118/2020 +# Date: 08/18/2020 # 2025-01-10 transferred to package rda_python_common from # https://github.com/NCAR/rda-shared-libraries.git # 2025-12-01 convert to class PgLock diff --git a/src/rda_python_common/pg_log.py b/src/rda_python_common/pg_log.py index 5488811..65256b9 100644 --- a/src/rda_python_common/pg_log.py +++ b/src/rda_python_common/pg_log.py @@ -252,7 +252,7 @@ def set_email(self, msg, logact=0): msg = self.PGLOG['PRGMSG'] + "\n" + msg self.PGLOG['PRGMSG'] = "" if self.PGLOG['ERRCNT'] == 0: - if not re.search(r'\n$', msg): msg += "!\n" + if not msg.endswith('\n'): msg += "!\n" else: if self.PGLOG['ERRCNT'] == 1: msg += " with 1 Error:\n" @@ -1423,7 +1423,7 @@ def set_common_pglog(self): try: self.PGLOG['RDAUID'] = self.PGLOG['GDEXUID'] = pwd.getpwnam(self.PGLOG['GDEXUSER']).pw_uid self.PGLOG['RDAGID'] = self.PGLOG['GDEXGID'] = grp.getgrnam(self.PGLOG['GDEXGRP']).gr_gid - except: + except KeyError: self.PGLOG['RDAUID'] = self.PGLOG['GDEXUID'] = 0 self.PGLOG['RDAGID'] = self.PGLOG['GDEXGID'] = 0 if self.PGLOG['CURUID'] == self.PGLOG['GDEXUSER']: self.PGLOG['SETUID'] = self.PGLOG['GDEXUSER'] @@ -1624,8 +1624,8 @@ def set_specialist_environments(self, specialist): missthen = 0 try: rf = open(resource, 'r') - except: - return # skip if cannot open + except OSError: + return # skip if cannot open nline = rf.readline() while nline: line = self.pgtrim(nline) @@ -1638,12 +1638,12 @@ def set_specialist_environments(self, specialist): missthen = 0 if re.match(r'^then$', line): continue # then on next line checkif = 0 # end of inline if - elif re.match(r'^endif', line): + elif line.startswith('endif'): checkif = 0 # end of if continue elif checkif == -1: # skip the line continue - elif checkif == 2 and re.match(r'^else', line): + elif checkif == 2 and line.startswith('else'): checkif = -1 # done check envs in if continue if checkif == 1: diff --git a/src/rda_python_common/pg_opt.py b/src/rda_python_common/pg_opt.py index 05168a6..6e5ffa6 100644 --- a/src/rda_python_common/pg_opt.py +++ b/src/rda_python_common/pg_opt.py @@ -1158,7 +1158,7 @@ def set_option_value(self, opt, val=None, cnl=0, lidx=0, line=None, infile=None) if self.OPTS[opt][2]&16: if not val: val = 0 - elif re.match(r'^\d+$', val): + elif val.isdigit(): val = int(val) elif val and (opt == 'DS' or opt == 'OD'): val = self.format_dataset_id(val) diff --git a/src/rda_python_common/pg_sig.py b/src/rda_python_common/pg_sig.py index f27901b..1b83da1 100644 --- a/src/rda_python_common/pg_sig.py +++ b/src/rda_python_common/pg_sig.py @@ -961,7 +961,7 @@ def check_pbs_status(self, bid, logact=None): lines = buf.split('\n') for line in lines: if chkt: - if re.match(r'^Job', line): + if line.startswith('Job'): line = re.sub(r'^Job ID', 'JobID', line, 1) line = re.sub(r'Finish Time', 'FinishTime', line, 1) line = re.sub(r'Req Mem', 'ReqMem', line, 1) diff --git a/src/rda_python_common/pg_split.py b/src/rda_python_common/pg_split.py index 1f93206..92bd238 100644 --- a/src/rda_python_common/pg_split.py +++ b/src/rda_python_common/pg_split.py @@ -1,7 +1,7 @@ ############################################################################### # Title: pg_split.py -- PostgreSQL DataBase Interface foe table wfile # Author: Zaihua Ji, zji@ucar.edu -# Date: 09/010/2024 +# Date: 09/10/2024 # 2025-01-10 transferred to package rda_python_common from # https://github.com/NCAR/rda-shared-libraries.git # 2025-12-01 convert to class PgSplit @@ -12,8 +12,9 @@ import re from os import path as op from .pg_util import PgUtil +from .pg_dbi import PgDBI -class PgSplit(PgUtil): +class PgSplit(PgUtil, PgDBI): """Manages synchronisation of wfile records between shared and per-dataset tables. Handles compare, add, update, and delete operations between the shared diff --git a/src/rda_python_common/pg_util.py b/src/rda_python_common/pg_util.py index e0ff151..8586e31 100644 --- a/src/rda_python_common/pg_util.py +++ b/src/rda_python_common/pg_util.py @@ -93,22 +93,22 @@ def get_month(self, mn, fmt = None): int | str: Numeric month (1-12) when fmt is None; formatted string otherwise. """ if not isinstance(mn, int): - if re.match(r'^\d+$', mn): + if mn.isdigit(): mn = int(mn) else: for m in range(12): if re.match(mn, self.MONTHS[m], re.I): mn = m + 1 break - if fmt and mn > 0 and mn < 13: + if fmt and 0 < mn < 13: slen = len(fmt) if slen == 2: smn = "{:02}".format(mn) - elif re.match(r'^mon', fmt, re.I): + elif fmt[:3].lower() == 'mon': smn = self.MNS[mn-1] if slen == 3 else self.MONTHS[mn-1] - if re.match(r'^Mon', fmt): + if fmt.startswith('Mon'): smn = smn.capitalize() - elif re.match(r'^MON', fmt): + elif fmt.startswith('MON'): smn = smn.upper() else: smn = str(mn) @@ -131,26 +131,26 @@ def get_wday(self, wday, fmt = None): formatted string otherwise. """ if not isinstance(wday, int): - if re.match(r'^\d+$', wday): + if wday.isdigit(): wday = int(wday) else: for w in range(7): if re.match(wday, self.WDAYS[w], re.I): wday = w break - if fmt and wday >= 0 and wday <= 6: + if fmt and 0 <= wday <= 6: slen = len(fmt) if slen == 4: swday = self.WDAYS[wday] - if re.match(r'^We', fmt): + if fmt.startswith('We'): swday = swday.capitalize() - elif re.match(r'^WE', fmt): + elif fmt.startswith('WE'): swday = swday.upper() elif slen == 3: swday = self.WDS[wday] - if re.match(r'^Ww', fmt): + if fmt.startswith('Ww'): swday = swday.capitalize() - elif re.match(r'^WW', fmt): + elif fmt.startswith('WW'): swday = swday.upper() else: swday = str(wday) @@ -179,7 +179,7 @@ def valid_online_file(file, type = None, exists = None): if exists is None or exists: if not op.exists(file): return '' # file does not exist bname = op.basename(file) - if re.match(r'^,.*', bname): return '' # hidden file + if bname.startswith(','): return '' # hidden file if re.search(r'index\.(htm|html|shtml)$', bname, re.I): return '' # index file if type and type != 'D': return type if re.search(r'\.(doc|php|html|shtml)(\.|$)', bname, re.I): return '' # file with special extention @@ -314,7 +314,7 @@ def check_datetime(date, default): """ if not date: return default if not isinstance(date, str): date = str(date) - if re.match(r'^0000', date): return default + if date.startswith('0000'): return default return date # fmt: date format, default to "YYYY-MM-DD" @@ -422,7 +422,7 @@ def split_datetime(sdt, sep = r'\D'): adt = re.split(sep, sdt) acnt = len(adt) for i in range(acnt): - if re.match(r'^\d+$', adt[i]): adt[i] = int(adt[i]) + if adt[i].isdigit(): adt[i] = int(adt[i]) return adt # date: given date in format of fromfmt @@ -490,19 +490,20 @@ def format_date(self, cdate, tofmt = None, fromfmt = None): if i >= mcnt: break fmt = formats[k] val = ms[0][i] - if re.match(r'^Y', fmt, re.I): + head = fmt[:1].upper() + if head == 'Y': dates[0] = int(val) if len(fmt) == 3: dates[0] *= 10 - elif re.match(r'^C', fmt, re.I): + elif head == 'C': dates[0] = 100 * int(val) # year at end of century - elif re.match(r'^M', fmt, re.I): - if re.match(r'^Mon', fmt, re.I): + elif head == 'M': + if fmt[:3].upper() == 'MON': dates[1] = self.get_month(val) else: dates[1] = int(val) - elif re.match(r'^Q', fmt, re.I): + elif head == 'Q': dates[1] = 3 * int(val) # month at end of quarter - elif re.match(r'^H', fmt, re.I): # hour + elif head == 'H': # hour dates.append(int(val)) else: # day dates[2] = int(val) @@ -653,11 +654,11 @@ def fmtdate(self, yr, mn, dy, tofmt = None): slen = len(fmt) if slen == 2: smn = "{:02}".format(m) - elif re.match(r'^mon', fmt, re.I): + elif fmt[:3].lower() == 'mon': smn = self.MNS[m-1] if slen == 3 else self.MONTHS[m-1] - if re.match(r'^Mo', fmt): + if fmt.startswith('Mo'): smn = smn.capitalize() - elif re.match(r'^MO', fmt): + elif fmt.startswith('MO'): smn = smn.upper() else: smn = str(m) diff --git a/src/rda_python_common/pgpassword.py b/src/rda_python_common/pgpassword.py index edfcc0c..ff69e40 100644 --- a/src/rda_python_common/pgpassword.py +++ b/src/rda_python_common/pgpassword.py @@ -4,17 +4,42 @@ # Author: Zaihua Ji, zji@ucar.edu # Date: 2025-10-27 # 2025-12-02 convert to class PgPassword -# Purpose: python script to retrieve passwords for postgrsql login to connect a +# Purpose: python script to retrieve passwords for postgresql login to connect a # gdex database from inside an python application # Github: https://github.com/NCAR/rda-python-common.git ################################################################################## +""" +pgpassword.py - Command-line helper that retrieves a PostgreSQL password. + +Provides the PgPassword class and a ``main`` entry point used by the +``pgpassword`` console script. The password is looked up first from +OpenBao (using the URL/token configured in PgDBI) and, if not found, +from the user's ``.pgpass`` file. The result is printed to stdout so +that shell wrappers and other RDA utilities can capture it. +""" import sys import re from .pg_dbi import PgDBI class PgPassword(PgDBI): + """ + Command-line helper for retrieving a PostgreSQL login password. + + Inherits from PgDBI to reuse its database-connection metadata + (PGDBI), default schema handling, and password-lookup methods + (``get_baopassword`` / ``get_pgpassword``). + + Instance attributes set in __init__: + DBFLDS -- mapping of CLI option letters to PGDBI field names + DBINFO -- per-invocation overrides for dbname/scname/lnname/ + dbhost/dbport supplied via CLI options + dbopt -- True once at least one DB-override option is seen, + triggering ``default_scinfo`` before lookup + password -- the retrieved password (set by ``start_actions``) + """ def __init__(self): + """Initialize PgPassword with empty DB-override info and option maps.""" super().__init__() # initialize parent class self.DBFLDS = { 'd': 'dbname', @@ -34,12 +59,28 @@ def __init__(self): self.password = '' # read in command line parameters - def read_parameters(self): + def read_parameters(self): + """ + Parse ``sys.argv`` and apply CLI overrides. + + Recognized options: + -l URL -- OpenBao URL (stored in self.PGDBI['BAOURL']) + -k TOKEN -- OpenBao token name (stored in self.PGDBI['BAOTOKEN']) + -d NAME -- PostgreSQL database name + -c NAME -- PostgreSQL schema name + -u NAME -- PostgreSQL login user name + -h HOST -- PostgreSQL server host name + -p PORT -- PostgreSQL port number + + If no arguments are supplied a usage message is printed and the + process exits with status 0. Unknown options or stray values + cause an immediate error exit via ``self.pglog(..., LGEREX)``. + """ argv = sys.argv[1:] opt = None dohelp = True for arg in argv: - if re.match(r'^-\w+$', arg): + if re.match(r'^-[a-zA-Z]$', arg): opt = arg[1:] elif opt: if opt == 'l': @@ -55,27 +96,27 @@ def read_parameters(self): else: self.pglog(arg + ": Value provided without option", self.LGEREX) if dohelp: - print("Usage: pgpassword [-l OpenBaoURL] [-k TokenName] [-d DBNAME] \\") - print(" [-c SCHEMA] [-u USName] [-h DBHOST] [-p DBPORT]") - print(" -l OpenBao URL to retrieve passwords") - print(" -k OpenBao Token Name to retrieve passwords") - print(" -d PostgreSQL Database Name") - print(" -c PostgreSQL Schema Name") - print(" -u PostgreSQL Login User Name") - print(" -h PostgreSQL Server Host Name") - print(" -p PostgreSQL Port Number") - sys.exit(0) + self.set_help_path(__file__) + self.show_usage("pgpassword") # get the pgpassword def start_actions(self): + """ + Look up the password and store it in ``self.password``. + + Applies any CLI-supplied DB overrides via ``default_scinfo``, then + tries OpenBao first (``get_baopassword``) and falls back to the + ``.pgpass`` file (``get_pgpassword``) if OpenBao returns nothing. + """ if self.dbopt: self.default_scinfo(self.DBINFO['dbname'], self.DBINFO['scname'], self.DBINFO['dbhost'], self.DBINFO['lnname'], None, self.DBINFO['dbport']) self.password = self.get_baopassword() - if not self.password: self.password = self.get_pg_pass() + if not self.password: self.password = self.get_pgpassword() # main function to excecute this script def main(): + """Entry point for the ``pgpassword`` console script: print the retrieved password to stdout.""" object = PgPassword() object.read_parameters() object.start_actions() diff --git a/src/rda_python_common/pgpassword.usg b/src/rda_python_common/pgpassword.usg new file mode 100644 index 0000000..1661040 --- /dev/null +++ b/src/rda_python_common/pgpassword.usg @@ -0,0 +1,52 @@ + + Retrieve a PostgreSQL login password for use by RDA python applications. + The password is looked up first from OpenBao (using the URL and token + configured in PgDBI) and, if not found there, from the user's .pgpass + file. The retrieved password is printed to stdout so that shell + wrappers and other RDA utilities can capture it. + + Usage: pgpassword [-l OpenBaoURL] [-k TokenName] [-d DBNAME] \ + [-c SCHEMA] [-u USName] [-h DBHOST] [-p DBPORT] + + - Option -l, OpenBao URL used to retrieve passwords. Overrides + the BAOURL value from the PgDBI configuration. + Default: https://bao.k8s.ucar.edu/ + + - Option -k, OpenBao token name used to authenticate the password + lookup. Overrides the BAOTOKEN value from the PgDBI + configuration. + + - Option -d, PostgreSQL database name. + Default: rdadb + + - Option -c, PostgreSQL schema name. + Default: dssdb + + - Option -u, PostgreSQL login user name. + Default: dssdb (same as the -c default) + + - Option -h, PostgreSQL server host name. + Default: value of the DSSDBHOST environment variable, or + rda-db.ucar.edu if DSSDBHOST is not set. + + - Option -p, PostgreSQL server port number. + Default: 5432 + + If any of -d, -c, -u, -h, or -p is supplied, the values are applied + via default_scinfo() before the password lookup is performed. With no + arguments, this usage information is displayed and the program exits. + + Examples: + + 1. Retrieve the password for the default database/user configured + in PgDBI: + + pgpassword + + 2. Retrieve the password for a specific database, schema and user: + + pgpassword -d rdadb -c dssdb -u metauser + + 3. Retrieve the password from a custom OpenBao endpoint: + + pgpassword -l https://bao.example.org -k my-token -d rdadb