From cf6199a8afdbb681ea2b7661ee8a96d124683b76 Mon Sep 17 00:00:00 2001 From: Hakuna Date: Thu, 27 Aug 2026 20:37:07 +0200 Subject: [PATCH 01/12] feat(magma): migrate to modern Amboss Magma GraphQL API and add test suites - Migrated magma_sale_process.py and magma_market_fee.py to target https://magma.amboss.tech/graphql - Replaced legacy queries/mutations with user.market.orders.sales, market.order.seller.accept/reject/add_transaction, and market.offer.offers/create/update/toggle - Added extract_order_info and extract_market_offer_info normalizers supporting nested SatoshiValue objects with backward compatibility - Expanded unit test suites for magma_sale_process and magma_market_fee with full mock coverage - Updated GitHub Actions CI test workflow with Python 3.10/3.11/3.12 matrix - Added .agents/agents.md architecture guide and overhauled README.md Tests passing: 29/29 --- .agents/agents.md | 110 ++++++ .github/workflows/tests.yml | 10 +- Magma/magma_market_fee.py | 507 ++++++++++++++---------- Magma/magma_sale_process.py | 514 +++++++++++++++++-------- README.md | 357 ++++++++--------- pyproject.toml | 5 +- tests/Magma/test_magma_market_fee.py | 316 +++++++++++++++ tests/Magma/test_magma_sale_process.py | 351 +++++++++++++++-- 8 files changed, 1608 insertions(+), 562 deletions(-) create mode 100644 .agents/agents.md create mode 100644 tests/Magma/test_magma_market_fee.py diff --git a/.agents/agents.md b/.agents/agents.md new file mode 100644 index 0000000..c20142a --- /dev/null +++ b/.agents/agents.md @@ -0,0 +1,110 @@ +# Agent Documentation: Lightning-Python-Tools + +This document serves as an architectural, operational, and development guide for AI agents, developers, and node operators working with the **Lightning-Python-Tools** repository. + +--- + +## 1. Architecture & Ecosystem Overview + +`Lightning-Python-Tools` is a modular collection of automation daemons, background services, CLI utilities, and cron jobs designed for Lightning Network (LN) node operators running **LND**. It integrates seamlessly with key Lightning infrastructure components: + +- **LND (`lncli`) / Balance of Satoshi (`bos`)**: Channel operations, UTXO management, invoice generation, and on-chain fee estimation. +- **Amboss Space & Magma Market**: Liquidity marketplace integration using the modern Amboss Magma GraphQL API (`https://magma.amboss.tech/graphql`) and Amboss Space metadata API (`https://api.amboss.space/graphql`). +- **LNDg**: Lightning Network node management dashboard, database sync, fee policies, and rebalancer coordination. +- **PeerSwap**: Trustless off-chain to on-chain (L-BTC / BTC) peer rebalancing monitoring and dashboard integration. +- **Boltz & Lightning Loop**: Automated submarine swap-out operations for off-chain to on-chain liquidity rebalancing. +- **LNbits**: Sub-wallet allowance management and automated budget allocations. +- **Telegram Bot API**: Real-time alerts, interactive manual approval callbacks, and remote command execution. + +--- + +## 2. Directory & Module Breakdown + +### 2.1 `Magma/` — Amboss Magma Liquidity Market Automation + +Amboss Magma is a decentralized Lightning liquidity marketplace where node operators buy and sell inbound channels. + +| File | Type | Description | +| :--- | :--- | :--- | +| `magma_sale_process.py` | Systemd Daemon / CLI | **Full-lifecycle seller automation**: Continuously monitors sales orders with `WAITING_FOR_SELLER_APPROVAL` and `WAITING_FOR_CHANNEL_OPEN`. Computes on-chain fee requirements via mempool API, verifies available UTXOs (filtering Loop locks), creates pre-image HODL invoices, opens channels via `lncli` or `bos`, auto-rejects banned peer pubkeys, confirms funding transaction outpoints to Amboss Magma via `market.order.seller.add_transaction`, and handles interactive Telegram approval callbacks. | +| `magma_market_fee.py` | Cronjob / CLI / Systemd | **Dynamic pricing & offer optimizer**: Gathers public Magma sell offers via `market.offer.offers`, filters out low seller scores (< threshold), analyzes percentile fee distributions (base fee & PPM), calculates annualized yield (APR), reserves required on-chain capital, and dynamically creates (`market.offer.create`), updates (`market.offer.update`), or toggles (`market.offer.toggle`) Magma sell offers. | +| `magma_config.ini` | Configuration | Defines pricing percentiles, minimum seller score filters, duration templates, and on-chain reserve thresholds for Magma. | +| `magma_sale.MD` | Documentation | Step-by-step setup guide for running `magma_sale_process.py` as a persistent `systemd` service. | + +#### Amboss Magma GraphQL Endpoints +- **Magma Liquidity Market API**: `https://magma.amboss.tech/graphql` (Sales, Orders, Offers, Mutations). +- **Amboss Space API**: `https://api.amboss.space/graphql` (Node lookups, aliases, extended info). + +--- + +### 2.2 `LNDg/` — LNDg Dashboard & Node Coordination + +Tools designed to augment and automate the [LNDg](https://github.com/cryptoshred/lndg) management interface. + +| File | Type | Description | +| :--- | :--- | :--- | +| `amboss_pull.py` | Cronjob / CLI | Pulls Magma sell orders from Amboss and synchronizes channel details, maturity dates, and auto-fee triggers into the local LNDg database and notes. | +| `channel_base-fee.py` | Cronjob / CLI | Evaluates channel conditions (such as fee thresholds and flow metrics) and adjusts base fees dynamically in LNDg. | +| `channel_fee-pull.py` | Cronjob / CLI | Exports channel fee policies and base fees from LNDg into standardized text/JSON formats for external automation tools. | +| `swap_out-candidates.py` | CLI / Tool | Evaluates channel capacity, local balances, and fee rates to identify prime candidates for submarine swap-outs; supports exporting to `.bos` tags format. | +| `mempool_rebalancer_trigger.py`| Systemd / Service | Tracks half-hour mempool fee rates (sat/vB). Automatically disables LNDg Auto-Rebalancer during high-fee congestion and re-enables it when fees normalize. | +| `disabled_fee-accelerator.py` | Cronjob / CLI | Adjusts fees on channels flagged as disabled or low-outbound by LNDg to incentivize liquidity rebalancing and prevent stale channels. | +| `offline_summary.py` | Cronjob / Alert | Scans peer states and compiles summaries of offline channels and inactive peers for Telegram / log reporting. | + +--- + +### 2.3 `Peerswap/` — PeerSwap Liquidity & Monitoring + +Integration scripts for [PeerSwap](https://github.com/ElementsProject/peerswap) (L-BTC / BTC on-chain swaps). + +| File | Type | Description | +| :--- | :--- | :--- | +| `peerswap-lndg_push.py` | Cronjob / CLI | Queries `pscli listpeers` and `pscli listswaps`, aggregates swap volumes and counts per peer, and injects the telemetry directly into LNDg channel cards and dashboard notes. | +| `ps_peers.py` | CLI Tool | Formats and displays a clean tabular overview of L-BTC on-chain balances, active PeerSwap peers, channel capacities, and available swap liquidity. | +| `peerswap-bot.py` | Telegram Bot | Interactive bot for monitoring PeerSwap daemon health, initiating swaps, and sending alerts upon swap completion or failure. | + +--- + +### 2.4 `Other/` — Advanced Routing, Consolidations & Swap Automation + +| File | Type | Description | +| :--- | :--- | :--- | +| `fee_adjuster.py` | Systemd / Cronjob | **Advanced Liquidity Fee Controller**: Dynamically adjusts channel routing fees based on liquidity balance curves, applies progressive discounts for stuck outbound capacity, adds demand premiums, and protects newly opened channels with cooldown guards. | +| `lnd_utxo_consolidator.py` | CLI / Daemon | Monitors the mempool for low-fee windows (e.g. < 10 sat/vB) and consolidates small, fragmented LND wallet UTXOs into single high-value outputs to save future fees. | +| `boltz_swap-out.py` | CLI / Daemon | Performs trustless submarine swap-outs via the Boltz Exchange API, converting excess Lightning outbound liquidity into Liquid BTC (L-BTC). | +| `swap_out-loop.py` | CLI / Daemon | Submarine swap-out automation leveraging Lightning Labs Loop daemon (`loopd` / `loop out`). | +| `swap_wallet.py` | CLI Tool | Automated Lightning payment scheduler and non-custodial batch payout tool for wallet-to-wallet transfers with rate limits and fee caps. | + +--- + +### 2.5 `LNBits/` — Micro-Wallet & Allowance Management + +| File | Type | Description | +| :--- | :--- | :--- | +| `pocketmoney.py` | Cronjob / Systemd | Automated allowance daemon that distributes scheduled satoshi payments from a master LNbits wallet to child sub-wallets based on configurable fiat/crypto budgets. | + +--- + +## 3. Configuration Conventions + +Configuration files follow standard INI (`configparser`) and JSON formats: + +- `config.ini` / `config.ini.example`: Core repository configuration including: + - `[credentials]`: Amboss tokens, LNbits API keys. + - `[telegram]`: Telegram bot token and user/group chat IDs. + - `[paths]`: Path to `lncli`, `bos`, and LNDg database. + - `[magma]`: Invoice expiry, fee limits, and auto-approval settings. +- `Magma/magma_config.ini`: Granular offer templates, duration brackets, and capital reserve limits. +- `Other/feeConfig.json`: Liquidity curve points, discount steps, and channel tags for `fee_adjuster.py`. + +--- + +## 4. Testing & Quality Standards + +- **Unit Testing Framework**: `pytest` with `pytest-mock` and `requests-mock`. +- **Test Locations**: `tests/` directory: + - `tests/Magma/test_magma_sale_process.py`: Sales order lifecycle, GraphQL queries/mutations, invoice generation, UTXO vbyte calculations, and Telegram alerts. + - `tests/Magma/test_magma_market_fee.py`: Public offers analysis, APR calculations, offer creation/updates/toggles, and dry-run simulations. + - `tests/test_fee_adjuster.py`: Liquidity curve calculations, stuck-channel discounts, and fee protections. +- **CI/CD Matrix**: `.github/workflows/tests.yml` executes on every push and pull request across Python `3.10`, `3.11`, and `3.12`. +- **Development Policy**: All new features and bugfixes must adhere to strict Test-Driven Development (TDD) and ensure 100% passing tests before merging. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 3f27a62..267885c 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -2,9 +2,9 @@ name: Tests on: push: - branches: [ "**" ] + branches: [ main, feat/*, fix/* ] pull_request: - branches: [ main, dev ] + branches: [ main ] jobs: test: @@ -24,9 +24,9 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi - if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi + pip install -r requirements.txt + pip install pytest requests-mock pytest-mock - - name: Run tests + - name: Run pytest run: | pytest tests/ -v diff --git a/Magma/magma_market_fee.py b/Magma/magma_market_fee.py index 54b3b08..70e8620 100644 --- a/Magma/magma_market_fee.py +++ b/Magma/magma_market_fee.py @@ -60,56 +60,85 @@ DRY_RUN_MODE = False MY_NODE_PUBKEY = None # Loaded from general_config [info] NODE -# --- GraphQL Queries/Mutations --- +# --- GraphQL Endpoints --- +MAGMA_GRAPHQL_URL = "https://magma.amboss.tech/graphql" +AMBOSS_SPACE_GRAPHQL_URL = "https://api.amboss.space/graphql" + +# --- GraphQL Queries/Mutations for Magma Liquidity Market --- GET_PUBLIC_MAGMA_OFFERS_QUERY = """ -query GetPublicOffers { - getOffers { - list { - id - offer_type - base_fee - fee_rate - max_size - min_block_length - min_size - seller_score - status - side - total_size - account # Pubkey of seller - # orders { locked_size } # Not currently used in public market analysis - # tags { name } # Not currently used in analysis +query GetPublicOffers($page: PageInput) { + market { + offer { + offers(page: $page) { + total + pagination { + limit + offset + } + list { + id + status + side + account { + pubkey + } + size { + min { satoshi { sats } } + max { satoshi { sats } } + total { satoshi { sats } } + } + fees { + fixed { sats } + variable { sats } + amboss { sats } + } + promises { + min_block_length + base_fee_cap + fee_rate_cap + } + seller_score + } + } } - # pageInfo { hasNextPage endCursor } # For pagination if needed later } } """ GET_MY_MAGMA_OFFERS_QUERY = """ -query MyOffers { - getUser { +query ListMyOffers($page: PageInput) { + user { market { offers { - list { - id - status - offer_type - base_fee - # base_fee_cap # Not directly used by script logic for now - fee_rate - # fee_rate_cap # Not directly used by script logic for now - max_size - min_block_length - min_size - total_size - orders { locked_size } # Crucial for available_size calculation - # conditions { condition } # Not used - # seller_score # Not relevant for own offers in this context - side - account # Our own pubkey - # amboss_fee_rate # Not used - # onchain_multiplier # Not used - # onchain_priority # Not used + offers(page: $page) { + total + pagination { + limit + offset + } + list { + id + status + side + size { + min { satoshi { sats } } + max { satoshi { sats } } + total { satoshi { sats } } + locked { satoshi { sats } } + } + fees { + fixed { sats } + variable { sats } + amboss { sats } + } + promises { + min_block_length + base_fee_cap + fee_rate_cap + } + filled_orders + created_at + } } } } @@ -118,27 +147,41 @@ """ CREATE_MAGMA_OFFER_MUTATION = """ -mutation CreateOffer($input: CreateOffer!) { - createOffer(input: $input) +mutation CreateOffer($input: CreateOfferInput!) { + market { + offer { + create(input: $input) { + offer_id + } + } + } } """ UPDATE_MAGMA_OFFER_MUTATION = """ -mutation UpdateOfferDetails($id: String!, $input: UpdateOfferDetailsInput!) { - updateOfferDetails(id: $id, input: $input) { - base_fee fee_rate min_block_length min_size max_size total_size status # Include status if API returns it +mutation UpdateOffer($input: UpdateOfferInput!) { + market { + offer { + update(input: $input) { + success + } + } } } """ -# Using toggleOffer instead of deleteOffer TOGGLE_MAGMA_OFFER_MUTATION = """ -mutation ToggleOffer($toggleOfferId: String!) { - toggleOffer(id: $toggleOfferId) # Returns Boolean +mutation ToggleOffer($input: ToggleOfferInput!) { + market { + offer { + toggle(input: $input) { + status + } + } + } } """ - # --- Logging Setup --- def setup_logging(): if not os.path.exists(LOG_DIR): @@ -206,39 +249,52 @@ def send_telegram_notification(text, level="info"): # --- Amboss API Interaction --- def _execute_amboss_graphql_request( - payload: dict, operation_name: str = "AmbossGraphQL" + payload: dict, + operation_name: str = "AmbossGraphQL", + endpoint_url: str = MAGMA_GRAPHQL_URL, ): + """ + Executes a GraphQL request to the Amboss / Magma API. + Handles DRY_RUN_MODE simulation for mutations. + """ if not AMBOSS_TOKEN: logging.error("Amboss API token not configured.") return None is_mutation = operation_name.lower().startswith( - ("create", "update", "delete", "toggle") + ("create", "update", "delete", "toggle", "mutation") ) if DRY_RUN_MODE and is_mutation: logging.info( f"DRY RUN: Preventing API call for {operation_name}. Payload: {json.dumps(payload, indent=2)}" ) - if "Create" in operation_name: - return {"createOffer": f"dry-run-id-for-{operation_name}"} - if "Update" in operation_name: + if "create" in operation_name.lower(): + return { + "market": {"offer": {"create": {"offer_id": f"dry-run-id-for-{operation_name}"}}}, + "createOffer": f"dry-run-id-for-{operation_name}", + } + if "update" in operation_name.lower(): return { + "market": {"offer": {"update": {"success": True}}}, "updateOfferDetails": { "id": "dry-run-updated-id", "status": "dry_run_simulated_update", - } + }, + } + if "toggle" in operation_name.lower(): + return { + "market": {"offer": {"toggle": {"status": "ENABLED"}}}, + "toggleOffer": True, } - if "Toggle" in operation_name: - return {"toggleOffer": True} # Simulate successful toggle return {"dryRunSimulatedSuccess": True} - url = "https://api.amboss.space/graphql" + url = endpoint_url headers = { "content-type": "application/json", "Authorization": f"Bearer {AMBOSS_TOKEN}", } logging.debug( - f"Executing {operation_name} with payload: {json.dumps(payload, indent=2 if logging.getLogger().getEffectiveLevel() == logging.DEBUG else None)}" + f"Executing {operation_name} against {url} with payload: {json.dumps(payload, indent=2 if logging.getLogger().getEffectiveLevel() == logging.DEBUG else None)}" ) try: response = requests.post(url, json=payload, headers=headers, timeout=30) @@ -255,7 +311,7 @@ def _execute_amboss_graphql_request( return None except requests.exceptions.HTTPError as e: logging.error( - f"HTTP error during {operation_name} to Amboss: {e}. Response: {e.response.text}" + f"HTTP error during {operation_name} to Amboss: {e}. Response: {getattr(e.response, 'text', '')}" ) return None except requests.exceptions.RequestException as e: @@ -273,6 +329,85 @@ def _execute_amboss_graphql_request( return None +def extract_market_offer_info(offer: dict) -> dict: + """Extracts normalized fields from a public or private Magma MarketOffer object.""" + if not offer: + return {} + + offer_id = offer.get("id", "N/A_ID") + status = str(offer.get("status", "UNKNOWN")).upper() + side = str(offer.get("side", "SELL")).upper() + + # Account / Pubkey + acc = offer.get("account") + if isinstance(acc, dict): + pubkey = acc.get("pubkey") + else: + pubkey = acc + + # Sizes + size_obj = offer.get("size") + if isinstance(size_obj, dict): + min_size = int(size_obj.get("min", {}).get("satoshi", {}).get("sats", 0)) + max_size = int(size_obj.get("max", {}).get("satoshi", {}).get("sats", 0)) + total_size = int(size_obj.get("total", {}).get("satoshi", {}).get("sats", 0)) + locked_size = int(size_obj.get("locked", {}).get("satoshi", {}).get("sats", 0)) if "locked" in size_obj else 0 + else: + min_size = int(offer.get("min_size", 0)) + max_size = int(offer.get("max_size", 0)) + total_size = int(offer.get("total_size", 0)) + orders_data = offer.get("orders") + if isinstance(orders_data, dict): + locked_size = int(orders_data.get("locked_size", 0)) + else: + locked_size = int(offer.get("locked_size", 0)) + + # Fees + fees_obj = offer.get("fees") + if isinstance(fees_obj, dict): + base_fee = int(fees_obj.get("fixed", {}).get("sats", 0)) + fee_rate = int(fees_obj.get("variable", {}).get("sats", 0)) + amboss_fee = int(fees_obj.get("amboss", {}).get("sats", 0)) + else: + base_fee = int(offer.get("base_fee", 0)) + fee_rate = int(offer.get("fee_rate", 0)) + amboss_fee = int(offer.get("amboss_fee_rate", 0)) + + # Promises + promises_obj = offer.get("promises") + if isinstance(promises_obj, dict): + min_block_length = int(promises_obj.get("min_block_length", 0)) + base_fee_cap = promises_obj.get("base_fee_cap") + fee_rate_cap = promises_obj.get("fee_rate_cap") + else: + min_block_length = int(offer.get("min_block_length", 0)) + base_fee_cap = offer.get("base_fee_cap") + fee_rate_cap = offer.get("fee_rate_cap") + + seller_score = float(offer.get("seller_score", 0.0)) + + return { + "id": offer_id, + "status": status, + "side": side, + "account": pubkey, + "node_alias": pubkey, + "min_size": min_size, + "max_size": max_size, + "total_size": total_size, + "locked_size": locked_size, + "available_size": total_size - locked_size, + "base_fee": base_fee, + "fee_rate": fee_rate, + "amboss_fee": amboss_fee, + "min_block_length": min_block_length, + "base_fee_cap": base_fee_cap, + "fee_rate_cap": fee_rate_cap, + "seller_score": seller_score, + "raw_offer": offer + } + + # --- LND Interaction --- def get_lncli_utxos(current_general_config): """ @@ -415,63 +550,48 @@ def get_lnd_onchain_balance(current_general_config): # --- Market Analysis & Pricing Logic --- (analyze_and_price_offer and calculate_apr remain largely the same) def fetch_public_magma_offers(node_pubkey_to_exclude, current_magma_config): - logging.info("Fetching public Magma sell offers...") - payload = {"query": GET_PUBLIC_MAGMA_OFFERS_QUERY} - data = _execute_amboss_graphql_request(payload, "GetPublicOffers") + logging.info("Fetching public Magma sell offers for market pricing analysis...") + payload = { + "query": GET_PUBLIC_MAGMA_OFFERS_QUERY, + "variables": {"page": {"limit": 100, "offset": 0}} + } + data = _execute_amboss_graphql_request(payload, "GetPublicOffers", endpoint_url=MAGMA_GRAPHQL_URL) processed_offers = [] - if data and data.get("getOffers", {}).get("list"): - raw_offers = data["getOffers"]["list"] + raw_offers = [] + if data: + raw_offers = ( + data.get("market", {}).get("offer", {}).get("offers", {}).get("list", []) + or data.get("getOffers", {}).get("list", []) + ) - # Correctly get min_seller_score_filter from the [magma_autoprice] section + if raw_offers: min_seller_score_filter = get_config_float_with_comment_stripping( - current_magma_config["magma_autoprice"], + current_magma_config, + "market_analysis", "min_seller_score_filter", fallback=0.0, ) - - # DEBUG: Log the raw data for verification logging.debug(f"Raw offers count: {len(raw_offers)}") logging.debug(f"Min seller score filter: {min_seller_score_filter}") for offer in raw_offers: try: + parsed_offer = extract_market_offer_info(offer) if ( - offer.get("status") != "ENABLED" - or offer.get("side") != "SELL" - or offer.get("offer_type") != "CHANNEL" + parsed_offer["status"] != "ENABLED" + or parsed_offer["side"] != "SELL" ): continue if ( node_pubkey_to_exclude - and offer.get("account") == node_pubkey_to_exclude + and parsed_offer["account"] == node_pubkey_to_exclude ): logging.debug( - f"Excluding own offer (Account: {offer.get('account')}) from market analysis." + f"Excluding own offer (Account: {parsed_offer['account']}) from market analysis." ) continue - parsed_offer = { - "id": offer.get("id"), - "offer_type": offer.get("offer_type"), - "base_fee": int(offer.get("base_fee", 0)), - "fee_rate": int(offer.get("fee_rate", 0)), - "max_size": int(offer.get("max_size", 0)), - "min_block_length": int(offer.get("min_block_length", 0)), - "min_size": int(offer.get("min_size", 0)), - "seller_score": float(offer.get("seller_score", 0.0)), - "status": offer.get("status"), - "side": offer.get("side"), - "total_size": int(offer.get("total_size", 0)), - "account": offer.get("account"), - "node_alias": offer.get("account"), - } - - # DEBUG: Log each offer's score for verification - logging.debug( - f"Offer {parsed_offer.get('id', 'N/A')}: score={parsed_offer['seller_score']}, base_fee={parsed_offer['base_fee']}, fee_rate={parsed_offer['fee_rate']}" - ) - if parsed_offer["seller_score"] < min_seller_score_filter: logging.debug( f"Excluding market offer {parsed_offer.get('id','N/A')} due to seller_score {parsed_offer['seller_score']} < {min_seller_score_filter}" @@ -496,25 +616,14 @@ def fetch_public_magma_offers(node_pubkey_to_exclude, current_magma_config): ) continue - # DEBUG: Log final filtered results - logging.debug(f"Final processed offers count: {len(processed_offers)}") - if processed_offers: - scores = [offer["seller_score"] for offer in processed_offers] - fees = [offer["base_fee"] for offer in processed_offers] - ppm_rates = [offer["fee_rate"] for offer in processed_offers] - logging.debug(f"Score range: {min(scores)} - {max(scores)}") - logging.debug(f"Fee range: {min(fees)} - {max(fees)}") - logging.debug(f"PPM range: {min(ppm_rates)} - {max(ppm_rates)}") - logging.info( - f"Fetched and processed {len(processed_offers)} relevant public Magma CHANNEL/SELL/ENABLED offers (excluding own, score >= {min_seller_score_filter})." + f"Fetched and processed {len(processed_offers)} relevant public Magma SELL/ENABLED offers (excluding own, score >= {min_seller_score_filter})." ) return processed_offers else: logging.warning("No public Magma offers found or error in fetching.") return [] - def calculate_apr(fixed_fee_sats, ppm_fee_rate, channel_size_sats, duration_days_float): if channel_size_sats == 0 or duration_days_float == 0: return 0.0 @@ -720,56 +829,38 @@ def analyze_and_price_offer( # --- Manage Our Offers on Amboss --- def fetch_my_current_offers(): logging.info("Fetching my current Magma sell offers...") - payload = {"query": GET_MY_MAGMA_OFFERS_QUERY} - data = _execute_amboss_graphql_request(payload, "MyOffers") + payload = { + "query": GET_MY_MAGMA_OFFERS_QUERY, + "variables": {"page": {"limit": 50, "offset": 0}} + } + data = _execute_amboss_graphql_request(payload, "MyOffers", endpoint_url=MAGMA_GRAPHQL_URL) processed_offers = [] - if data and data.get("getUser", {}).get("market", {}).get("offers", {}).get("list"): - my_raw_offers = data["getUser"]["market"]["offers"]["list"] + my_raw_offers = [] + if data: + my_raw_offers = ( + data.get("user", {}).get("market", {}).get("offers", {}).get("offers", {}).get("list", []) + or data.get("getUser", {}).get("market", {}).get("offers", {}).get("list", []) + ) + + if my_raw_offers: for offer_item in my_raw_offers: try: if not offer_item: continue - offer_id = offer_item.get("id", "N/A_ID") - logging.debug( - f"Processing own offer item ID {offer_id}: {json.dumps(offer_item)}" - ) + parsed = extract_market_offer_info(offer_item) + offer_id = parsed["id"] - if ( - offer_item.get("offer_type") != "CHANNEL" - or offer_item.get("side") != "SELL" - ): + if parsed["side"] != "SELL": logging.debug( - f"Skipping own offer {offer_id} - not a CHANNEL sell offer. Type: {offer_item.get('offer_type')}, Side: {offer_item.get('side')}" + f"Skipping own offer {offer_id} - not a SELL offer. Side: {parsed['side']}" ) continue - total_size_sats = int(offer_item.get("total_size", 0)) - - orders_data = offer_item.get("orders") - locked_size_str = "0" - if orders_data and isinstance(orders_data, dict): - locked_size_str = orders_data.get("locked_size", "0") - elif ( - orders_data is not None - ): # orders field exists but not a dict, log warning - logging.warning( - f"Offer ID {offer_id} has 'orders' field but it's not a dictionary: {orders_data}. Defaulting locked_size to 0." - ) - - locked_size_sats = int( - locked_size_str if locked_size_str is not None else "0" - ) # Ensure int conversion - available_size_sats = total_size_sats - locked_size_sats - - logging.debug( - f"Offer ID {offer_id}: total_size={total_size_sats}, orders_data={orders_data}, parsed_locked_size_str='{locked_size_str}', locked_size_sats={locked_size_sats}, calculated_available_size={available_size_sats}" - ) - - current_fixed_fee = int(offer_item.get("base_fee", 0)) - current_ppm_rate = int(offer_item.get("fee_rate", 0)) - current_min_size = int(offer_item.get("min_size", 0)) - current_duration_blocks = int(offer_item.get("min_block_length", 0)) + current_fixed_fee = parsed["base_fee"] + current_ppm_rate = parsed["fee_rate"] + current_min_size = parsed["min_size"] + current_duration_blocks = parsed["min_block_length"] current_duration_days = ( current_duration_blocks / BLOCKS_PER_DAY if BLOCKS_PER_DAY > 0 @@ -784,20 +875,19 @@ def fetch_my_current_offers(): details = { "id": offer_id, - "status": offer_item.get("status", "UNKNOWN").upper(), - "offer_type": offer_item.get("offer_type"), + "status": parsed["status"], "base_fee": current_fixed_fee, "fee_rate": current_ppm_rate, - "max_size": int(offer_item.get("max_size", 0)), + "max_size": parsed["max_size"], "min_block_length": current_duration_blocks, "min_size": current_min_size, - "total_size": total_size_sats, - "locked_size": locked_size_sats, - "available_size": available_size_sats, - "side": offer_item.get("side"), - "account": offer_item.get("account"), - "duration_days": current_duration_days, # Store for display - "apr": current_apr, # Store for display + "total_size": parsed["total_size"], + "locked_size": parsed["locked_size"], + "available_size": parsed["available_size"], + "side": parsed["side"], + "account": parsed["account"], + "duration_days": current_duration_days, + "apr": current_apr, } if not ( details["id"] != "N/A_ID" @@ -812,28 +902,30 @@ def fetch_my_current_offers(): processed_offers.append(details) except (ValueError, TypeError, KeyError) as e: logging.warning( - f"Error parsing own offer {offer_id}: {e}. Offer data: {offer_item}" + f"Error parsing own offer {offer_item.get('id')}: {e}. Offer data: {offer_item}" ) continue logging.info( - f"Found and processed {len(processed_offers)} existing Magma CHANNEL sell offers." + f"Found and processed {len(processed_offers)} existing Magma sell offers." ) return processed_offers logging.warning("No existing Magma sell offers found or error fetching.") return [] - def create_magma_offer(pricing_details, template_capital_for_total_size, template_name): - duration_blocks = pricing_details["duration_days"] * BLOCKS_PER_DAY + duration_blocks = int(pricing_details["duration_days"] * BLOCKS_PER_DAY) amboss_offer_input = { - "base_fee": pricing_details["fixed_fee_sats"], - "fee_rate": pricing_details["ppm_fee_rate"], - "min_size": pricing_details["channel_size_sats"], - "max_size": pricing_details["channel_size_sats"], + "pubkey": MY_NODE_PUBKEY, + "base_fee": int(pricing_details["fixed_fee_sats"]), + "fee_rate": int(pricing_details["ppm_fee_rate"]), + "min_size": int(pricing_details["channel_size_sats"]), + "max_size": int(pricing_details["channel_size_sats"]), "min_block_length": duration_blocks, - "total_size": template_capital_for_total_size, - "base_fee_cap": pricing_details["fixed_fee_sats"], - "fee_rate_cap": pricing_details["ppm_fee_rate"], + "total_size": int(template_capital_for_total_size), + "base_fee_cap": int(pricing_details["fixed_fee_sats"]), + "fee_rate_cap": int(pricing_details["ppm_fee_rate"]), + "onchain_priority": "MEDIUM", + "onchain_multiplier": 2, } log_prefix = "DRY RUN: Would create" if DRY_RUN_MODE else "Creating" logging.info( @@ -841,25 +933,32 @@ def create_magma_offer(pricing_details, template_capital_for_total_size, templat ) if DRY_RUN_MODE: - # Simulate the structure Amboss might return for createOffer, including a simulated ID. - return {"createOffer": f"dryrun-offer-id-{template_name.replace(' ', '_')}"} + return { + "id": f"dryrun-offer-id-{template_name.replace(' ', '_')}", + "createOffer": f"dryrun-offer-id-{template_name.replace(' ', '_')}", + "status_after_create": "ACTIVE", + } payload = { "query": CREATE_MAGMA_OFFER_MUTATION, "variables": {"input": amboss_offer_input}, } - data = _execute_amboss_graphql_request(payload, f"CreateMagmaOffer-{template_name}") - if data and data.get( - "createOffer" - ): # createOffer returns the new Offer ID (String) - new_offer_id = data.get("createOffer") + data = _execute_amboss_graphql_request(payload, f"CreateMagmaOffer-{template_name}", endpoint_url=MAGMA_GRAPHQL_URL) + new_offer_id = None + if data: + new_offer_id = ( + data.get("market", {}).get("offer", {}).get("create", {}).get("offer_id") + or data.get("createOffer") + ) + if new_offer_id: logging.info( f"Successfully created Magma offer for '{template_name}'. New Offer ID: {new_offer_id}" ) return { "id": new_offer_id, + "createOffer": new_offer_id, "status_after_create": "ACTIVE", - } # Assume ACTIVE, will be toggled + } else: logging.error( f"Failed to create Magma offer for '{template_name}'. Response: {data}" @@ -870,18 +969,15 @@ def create_magma_offer(pricing_details, template_capital_for_total_size, templat def update_magma_offer( offer_id_to_update, pricing_details, template_capital_for_total_size, template_name ): - duration_blocks = pricing_details["duration_days"] * BLOCKS_PER_DAY + duration_blocks = int(pricing_details["duration_days"] * BLOCKS_PER_DAY) amboss_offer_input = { - "base_fee": pricing_details["fixed_fee_sats"], - "fee_rate": pricing_details["ppm_fee_rate"], - "min_size": pricing_details["channel_size_sats"], - "max_size": pricing_details[ - "channel_size_sats" - ], # Assuming fixed size for our offers + "offer_id": offer_id_to_update, + "base_fee": int(pricing_details["fixed_fee_sats"]), + "fee_rate": int(pricing_details["ppm_fee_rate"]), + "min_size": int(pricing_details["channel_size_sats"]), + "max_size": int(pricing_details["channel_size_sats"]), "min_block_length": duration_blocks, - "total_size": template_capital_for_total_size, - # base_fee_cap and fee_rate_cap are part of CreateOfferInput but optional in UpdateOfferDetailsInput - # If you want to update them, add them here. For now, matching previous behavior. + "total_size": int(template_capital_for_total_size), } log_prefix = "DRY RUN: Would update" if DRY_RUN_MODE else "Updating" logging.info( @@ -889,7 +985,6 @@ def update_magma_offer( ) if DRY_RUN_MODE: - # Simulate the structure of the returned Offer object after update return { "id": offer_id_to_update, **amboss_offer_input, @@ -898,14 +993,24 @@ def update_magma_offer( payload = { "query": UPDATE_MAGMA_OFFER_MUTATION, - "variables": {"id": offer_id_to_update, "input": amboss_offer_input}, + "variables": {"input": amboss_offer_input}, } data = _execute_amboss_graphql_request( - payload, f"UpdateMagmaOffer-{offer_id_to_update}" + payload, f"UpdateMagmaOffer-{offer_id_to_update}", endpoint_url=MAGMA_GRAPHQL_URL ) - if data and data.get("updateOfferDetails"): + is_success = False + if data: + is_success = ( + data.get("market", {}).get("offer", {}).get("update", {}).get("success") is True + or data.get("updateOfferDetails") is not None + ) + if is_success: logging.info(f"Successfully updated Magma offer ID {offer_id_to_update}.") - return data.get("updateOfferDetails") # This is the Offer object + return { + "id": offer_id_to_update, + **amboss_offer_input, + "status": "UPDATED", + } else: logging.error( f"Failed to update Magma offer ID {offer_id_to_update}. Response: {data}" @@ -917,21 +1022,26 @@ def toggle_magma_offer_status( offer_id, template_name_logging_info, target_log_status_str ): log_prefix = "DRY RUN: Would toggle" if DRY_RUN_MODE else "Toggling" - # target_log_status_str is for logging the intent, actual toggle flips the state logging.info( f"{log_prefix} Magma offer ID {offer_id} (info: '{template_name_logging_info}'). Intent from script: {target_log_status_str}." ) if DRY_RUN_MODE: logging.info(f"DRY RUN: Simulating toggle for {offer_id} successful.") - return True # Simulate success + return True payload = { "query": TOGGLE_MAGMA_OFFER_MUTATION, - "variables": {"toggleOfferId": offer_id}, + "variables": {"input": {"offer_id": offer_id}}, } - data = _execute_amboss_graphql_request(payload, f"ToggleMagmaOffer-{offer_id}") - if data and data.get("toggleOffer") is True: # toggleOffer returns Boolean + data = _execute_amboss_graphql_request(payload, f"ToggleMagmaOffer-{offer_id}", endpoint_url=MAGMA_GRAPHQL_URL) + is_toggled = False + if data: + status_val = data.get("market", {}).get("offer", {}).get("toggle", {}).get("status") + if status_val in ["ENABLED", "DISABLED", "ACTIVE", "INACTIVE"] or data.get("toggleOffer") is True: + is_toggled = True + + if is_toggled: logging.info(f"Successfully toggled status for Magma offer ID {offer_id}.") return True else: @@ -940,7 +1050,6 @@ def toggle_magma_offer_status( ) return False - # --- Main Application Logic --- def main(): global general_config, magma_specific_config, AMBOSS_TOKEN, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID, DRY_RUN_MODE, LNCLI_PATH, MY_NODE_PUBKEY @@ -1467,8 +1576,8 @@ def main(): create_result = create_magma_offer( new_pricing, template_capital_limit_for_total_size, template_name ) - if create_result and create_result.get("createOffer"): - new_id = create_result["createOffer"] + if create_result and (create_result.get("createOffer") or create_result.get("id")): + new_id = create_result.get("createOffer") or create_result.get("id") managed_offer_ids_this_run.add(new_id) actions_summary_for_telegram.append( f"🚀 Created {template_name} (New ID {new_id[:8]}): APR {prop_apr_val}% (F:{prop_fixed_val},PPM:{prop_ppm_val},TS:{prop_capital_val}). Initial: DISABLED." diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index 1ca8e54..18f7ad9 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -121,35 +121,154 @@ CRITICAL_ERROR_FILE_PATH = os.path.join(parent_dir, "..", "logs", "magma_sale_process-critical-error.flag") # Reverted -# --- GraphQL Query for Offer Orders --- -# Fetches all fields typically needed when processing or listing offer orders. -OFFER_ORDER_FIELDS_QUERY_PART = """ - list { - id - size - status - account # Expected to be the buyer's pubkey - seller_invoice_amount - endpoints { # Contains 'destination' which is also buyer's pubkey - destination +# --- GraphQL Endpoints --- +AMBOSS_SPACE_GRAPHQL_URL = "https://api.amboss.space/graphql" +MAGMA_GRAPHQL_URL = "https://magma.amboss.tech/graphql" + +# --- GraphQL Queries and Mutations for Magma Selling --- + +# Query seller sales (orders) +GET_SALES_QUERY = """ +query GetSales(: SalesInput, : PageInput) { + user { + market { + orders { + sales(input: , page: ) { + total + pagination { + limit + offset + } + list { + id + status + created_at + amount { + satoshi { + sats + btc + usd + } + } + fees { + fixed { sats } + variable { sats } + amboss { sats } + seller { sats } + buyer { sats } + } + promises { + locked_min_block_length + locked_base_fee_cap { sats } + locked_fee_rate_cap { sats } + } + destination { + pubkey + alias + } + payment { + lightning { + invoice + } + } + } + } } } + } +} """ -GET_USER_MARKET_OFFER_ORDERS_QUERY = f""" - query GetUserMarketOfferOrders {{ - getUser {{ - market {{ - offer_orders {{ - {OFFER_ORDER_FIELDS_QUERY_PART} - }} - }} - }} - }} +GET_ORDER_DETAILS_QUERY = """ +query GetOrderDetails(: String!) { + user { + market { + orders { + get_order(order_id: ) { + id + status + payment_status + created_at + amount { + satoshi { + sats + btc + usd + } + } + fees { + fixed { sats } + variable { sats } + amboss { sats } + seller { sats } + buyer { sats } + } + promises { + locked_min_block_length + locked_base_fee_cap { sats } + locked_fee_rate_cap { sats } + } + destination { + pubkey + alias + } + payment { + lightning { + invoice + } + } + } + } + } + } +} """ -# Note: get_order_details_from_amboss uses a slightly different query name "ListUserMarketOfferOrdersForDetails" -# but the structure is the same. We can align this. For now, the payload in get_order_details_from_amboss -# will be updated to use GET_USER_MARKET_OFFER_ORDERS_QUERY. + +ACCEPT_ORDER_MUTATION = """ +mutation AcceptOrder(: SellerAcceptOrdersInput!) { + market { + order { + seller { + accept(input: ) { + success + } + } + } + } +} +""" + +REJECT_ORDER_MUTATION = """ +mutation RejectOrder(: SellerRejectOrdersInput!) { + market { + order { + seller { + reject(input: ) { + success + } + } + } + } +} +""" + +ADD_TRANSACTION_MUTATION = """ +mutation AddTransaction(: SellerAddTransactionInput!) { + market { + order { + seller { + add_transaction(input: ) { + success + } + } + } + } +} +""" + +# Legacy query alias for backwards compatibility +GET_USER_MARKET_OFFER_ORDERS_QUERY = GET_SALES_QUERY + # Code @@ -198,36 +317,96 @@ def send_telegram_notification(text, level="info", **kwargs): logging.error(f"Failed to send Telegram message: {e}") return None -def _execute_amboss_graphql_request(payload: dict, operation_name: str ="AmbossGraphQL"): +def extract_order_info(order: dict) -> dict: + """Extracts normalized order fields from a Magma MarketOrder object or legacy payload.""" + if not order: + return {} + + order_id = order.get("id") + status = order.get("status") + + # Destination / Buyer Pubkey & Alias + dest = order.get("destination") + if isinstance(dest, dict): + buyer_pubkey = dest.get("pubkey") + buyer_alias = dest.get("alias") + else: + buyer_pubkey = order.get("account") or (order.get("endpoints", {}).get("destination") if isinstance(order.get("endpoints"), dict) else None) + buyer_alias = None + + # Amount / Size + amount_obj = order.get("amount") + if isinstance(amount_obj, dict) and "satoshi" in amount_obj: + size_sats = int(amount_obj.get("satoshi", {}).get("sats", 0)) + elif isinstance(amount_obj, (int, str)) and str(amount_obj).isdigit(): + size_sats = int(amount_obj) + elif "size" in order: + size_sats = int(order.get("size", 0)) + else: + size_sats = 0 + + # Fees + fees_obj = order.get("fees") + if isinstance(fees_obj, dict): + seller_invoice_sats = int(fees_obj.get("seller", {}).get("sats", 0)) + fixed_fee_sats = int(fees_obj.get("fixed", {}).get("sats", 0)) + variable_fee_sats = int(fees_obj.get("variable", {}).get("sats", 0)) + amboss_fee_sats = int(fees_obj.get("amboss", {}).get("sats", 0)) + else: + seller_invoice_sats = int(order.get("seller_invoice_amount", 0)) + fixed_fee_sats = int(order.get("fixed_fee", 0)) + variable_fee_sats = int(order.get("variable_fee", 0)) + amboss_fee_sats = int(order.get("amboss_fee", 0)) + + # Promises + promises_obj = order.get("promises") + if isinstance(promises_obj, dict): + min_block_length = int(promises_obj.get("locked_min_block_length", 0)) + else: + min_block_length = int(order.get("locked_min_block_length", 0)) + + return { + "id": order_id, + "status": status, + "customer_pubkey": buyer_pubkey, + "buyer_alias": buyer_alias, + "channel_size": size_sats, + "seller_invoice_amount": seller_invoice_sats, + "fixed_fee": fixed_fee_sats, + "variable_fee": variable_fee_sats, + "amboss_fee": amboss_fee_sats, + "min_block_length": min_block_length, + "created_at": order.get("created_at"), + "raw_order": order + } + +def _execute_amboss_graphql_request( + payload: dict, + operation_name: str = "AmbossGraphQL", + endpoint_url: str = MAGMA_GRAPHQL_URL, +): """ - Executes a GraphQL request to the Amboss API. + Executes a GraphQL request to the Amboss or Magma API. Handles common request logic, headers, timeouts, and basic error handling. - Args: - payload (dict): The GraphQL payload (e.g., {"query": "...", "variables": {...}}). - operation_name (str): A descriptive name for the operation for logging. - Returns: - dict or None: The JSON response data part if successful, else None. """ - url = "https://api.amboss.space/graphql" + url = endpoint_url headers = { "content-type": "application/json", "Authorization": f"Bearer {AMBOSS_TOKEN}", } - logging.debug(f"Executing {operation_name} with payload: {json.dumps(payload, indent=2 if logging.getLogger().getEffectiveLevel() == logging.DEBUG else None)}") # Pretty print payload if debug + logging.debug(f"Executing {operation_name} against {url} with payload: {json.dumps(payload, indent=2 if logging.getLogger().getEffectiveLevel() == logging.DEBUG else None)}") try: - response = requests.post(url, json=payload, headers=headers, timeout=20) # Standard timeout - response.raise_for_status() # Raises HTTPError for bad responses (4xx or 5xx) + response = requests.post(url, json=payload, headers=headers, timeout=20) + response.raise_for_status() response_json = response.json() if response_json.get("errors"): logging.error(f"GraphQL errors during {operation_name}: {response_json.get('errors')}") - # Do not create CRITICAL_ERROR_FILE_PATH here for query errors, let caller decide. - # Example: if confirm_channel_point_to_amboss gets a specific Amboss error, it might create it. - return None # Indicate GraphQL level error + return None - return response_json.get("data") # Return only the 'data' part + return response_json.get("data") except requests.exceptions.Timeout: logging.error(f"Timeout during {operation_name} to Amboss.") @@ -244,29 +423,46 @@ def _execute_amboss_graphql_request(payload: dict, operation_name: str ="AmbossG def get_order_details_from_amboss(order_id): """ - Fetches specific order details from Amboss by its ID. - This involves fetching the user's list of market offer orders and filtering by ID. + Fetches specific order details from Amboss Magma by its ID. + Queries the dedicated get_order endpoint or falls back to sales list. """ - logging.info(f"Fetching details for order ID: {order_id} from Amboss...") - payload = {"query": GET_USER_MARKET_OFFER_ORDERS_QUERY} + logging.info(f"Fetching details for order ID: {order_id} from Amboss Magma...") + payload = { + "query": GET_ORDER_DETAILS_QUERY, + "variables": {"orderId": order_id} + } - data = _execute_amboss_graphql_request(payload, f"GetOrderDetails-{order_id}") + data = _execute_amboss_graphql_request(payload, f"GetOrderDetails-{order_id}", endpoint_url=MAGMA_GRAPHQL_URL) - if not data: - return None # Error already logged by helper + if data: + order = ( + data.get("user", {}).get("market", {}).get("orders", {}).get("get_order") + or data.get("getUser", {}).get("market", {}).get("orders", {}).get("get_order") + or data.get("getOrder") + ) + if order: + logging.info(f"Successfully found details for order {order_id}: {order}") + return order + + # Fallback to query sales list if direct get_order didn't return + sales_payload = { + "query": GET_SALES_QUERY, + "variables": {"page": {"limit": 50, "offset": 0}} + } + fallback_data = _execute_amboss_graphql_request(sales_payload, f"GetSalesFallback-{order_id}", endpoint_url=MAGMA_GRAPHQL_URL) + if fallback_data: + sales_list = ( + fallback_data.get("user", {}).get("market", {}).get("orders", {}).get("sales", {}).get("list", []) + or fallback_data.get("getUser", {}).get("market", {}).get("offer_orders", {}).get("list", []) + ) + for offer in sales_list: + if offer.get("id") == order_id: + logging.info(f"Successfully found details for order {order_id} in sales list: {offer}") + return offer - market_data = data.get("getUser", {}).get("market", {}) - offer_orders_list = market_data.get("offer_orders", {}).get("list", []) - - for offer in offer_orders_list: - if offer.get("id") == order_id: - logging.info(f"Successfully found details for order {order_id}: {offer}") - return offer - - logging.warning(f"Order ID {order_id} not found in your Amboss market orders list.") + logging.warning(f"Order ID {order_id} not found in Amboss Magma.") return None - def get_node_alias(pubkey: str) -> str: """Fetches the alias for a given node pubkey using the getNodeAlias query.""" if not pubkey: @@ -282,7 +478,7 @@ def get_node_alias(pubkey: str) -> str: "variables": {"pubkey": pubkey} } - data = _execute_amboss_graphql_request(payload, f"GetNodeAlias-{pubkey[:10]}") + data = _execute_amboss_graphql_request(payload, f"GetNodeAlias-{pubkey[:10]}", endpoint_url=AMBOSS_SPACE_GRAPHQL_URL) if not data: return "ErrorFetchingAlias" # Error already logged by helper @@ -354,7 +550,7 @@ def get_node_extended_details(pubkey: str) -> dict: "variables": {"pubkey": pubkey} } - data = _execute_amboss_graphql_request(payload, f"GetNodeExtendedInfo-{pubkey[:10]}") + data = _execute_amboss_graphql_request(payload, f"GetNodeExtendedInfo-{pubkey[:10]}", endpoint_url=AMBOSS_SPACE_GRAPHQL_URL) if not data or not data.get("getNode"): logging.warning(f"No extended details returned for pubkey {pubkey} from Amboss.") @@ -413,103 +609,98 @@ def execute_lncli_addinvoice(amt, memo, expiry): def accept_order(order_id, payment_request): - """Accepts an order on Amboss.""" - logging.info(f"Accepting order {order_id} on Amboss with payment request: {payment_request[:30]}...") + """Accepts an order on Amboss Magma.""" + logging.info(f"Accepting order {order_id} on Amboss Magma with payment request: {payment_request[:30]}...") payload = { - "query": """ - mutation AcceptOrder($sellerAcceptOrderId: String!, $request: String!) { - sellerAcceptOrder(id: $sellerAcceptOrderId, request: $request) + "query": ACCEPT_ORDER_MUTATION, + "variables": { + "input": { + "order_id": order_id, + "payment_request": payment_request } - """, - "variables": {"sellerAcceptOrderId": order_id, "request": payment_request} + } } - # This is a mutation, so we expect the full response, not just 'data' part directly from helper - # because the success/failure is often indicated by the presence of 'sellerAcceptOrder' field itself - # or specific errors. - url = "https://api.amboss.space/graphql" + url = MAGMA_GRAPHQL_URL headers = { "content-type": "application/json", "Authorization": f"Bearer {AMBOSS_TOKEN}", } try: - response = requests.post(url, json=payload, headers=headers, timeout=60) # Increased timeout + response = requests.post(url, json=payload, headers=headers, timeout=60) response.raise_for_status() response_json = response.json() - logging.info(f"Amboss sellerAcceptOrder response for {order_id}: {response_json}") - return response_json # Return the full JSON response for the caller to interpret + logging.info(f"Amboss Magma seller accept response for {order_id}: {response_json}") + return response_json except requests.exceptions.Timeout: - logging.error(f"Timeout while accepting Amboss order {order_id}.") + logging.error(f"Timeout while accepting Amboss Magma order {order_id}.") return {"errors": [{"message": "Timeout during Amboss API call"}]} except requests.exceptions.RequestException as e: - logging.error(f"API request error accepting Amboss order {order_id}: {e}") + logging.error(f"API request error accepting Amboss Magma order {order_id}: {e}") return {"errors": [{"message": f"RequestException: {e}"}]} except json.JSONDecodeError as e: - logging.error(f"Failed to decode JSON response when accepting Amboss order {order_id}: {e}") + logging.error(f"Failed to decode JSON response when accepting Amboss Magma order {order_id}: {e}") return {"errors": [{"message": f"JSONDecodeError: {e}"}]} except Exception as e: logging.exception(f"Unexpected error in accept_order for {order_id}:") return {"errors": [{"message": f"Unexpected error: {e}"}]} - def reject_order(order_id): - """Rejects an order on Amboss.""" - logging.info(f"Rejecting order {order_id} on Amboss...") + """Rejects an order on Amboss Magma.""" + logging.info(f"Rejecting order {order_id} on Amboss Magma...") payload = { - "query": """ - mutation SellerRejectOrder($sellerRejectOrderId: String!) { - sellerRejectOrder(id: $sellerRejectOrderId) + "query": REJECT_ORDER_MUTATION, + "variables": { + "input": { + "order_id": order_id } - """, - "variables": {"sellerRejectOrderId": order_id} + } } - # Similar to accept_order, mutations might need specific handling of the full response - url = "https://api.amboss.space/graphql" + url = MAGMA_GRAPHQL_URL headers = { "content-type": "application/json", "Authorization": f"Bearer {AMBOSS_TOKEN}", } try: - response = requests.post(url, json=payload, headers=headers, timeout=60) # Increased timeout + response = requests.post(url, json=payload, headers=headers, timeout=60) response.raise_for_status() response_json = response.json() - logging.info(f"Amboss sellerRejectOrder response for {order_id}: {response_json}") + logging.info(f"Amboss Magma seller reject response for {order_id}: {response_json}") return response_json except requests.exceptions.Timeout: - logging.error(f"Timeout while rejecting Amboss order {order_id}.") + logging.error(f"Timeout while rejecting Amboss Magma order {order_id}.") return {"errors": [{"message": "Timeout during Amboss API call"}]} except requests.exceptions.RequestException as e: - logging.error(f"API request error rejecting Amboss order {order_id}: {e}") + logging.error(f"API request error rejecting Amboss Magma order {order_id}: {e}") return {"errors": [{"message": f"RequestException: {e}"}]} except json.JSONDecodeError as e: - logging.error(f"Failed to decode JSON response when rejecting Amboss order {order_id}: {e}") + logging.error(f"Failed to decode JSON response when rejecting Amboss Magma order {order_id}: {e}") return {"errors": [{"message": f"JSONDecodeError: {e}"}]} except Exception as e: logging.exception(f"Unexpected error in reject_order for {order_id}:") return {"errors": [{"message": f"Unexpected error: {e}"}]} - def confirm_channel_point_to_amboss(order_id, transaction): - """Confirms the channel point (funding transaction) to Amboss.""" - logging.info(f"Confirming channel point {transaction} to Amboss for order {order_id}...") + """Confirms the channel point (funding transaction) to Amboss Magma.""" + logging.info(f"Confirming channel point {transaction} to Amboss Magma for order {order_id}...") payload = { - "query": """ - mutation ConfirmChannelPoint($sellerAddTransactionId: String!, $transaction: String!) { - sellerAddTransaction(id: $sellerAddTransactionId, transaction: $transaction) + "query": ADD_TRANSACTION_MUTATION, + "variables": { + "input": { + "order_id": order_id, + "tx_id": transaction } - """, - "variables": {"sellerAddTransactionId": order_id, "transaction": transaction}, + }, } - # This is a critical mutation. The full response is needed. - url = "https://api.amboss.space/graphql" + url = MAGMA_GRAPHQL_URL headers = { "Content-Type": "application/json", "Authorization": f"Bearer {AMBOSS_TOKEN}", } try: - response = requests.post(url, headers=headers, json=payload, timeout=60) # Increased timeout + response = requests.post(url, headers=headers, json=payload, timeout=60) response.raise_for_status() response_json = response.json() - logging.info(f"Amboss sellerAddTransaction response for {order_id}: {response_json}") + logging.info(f"Amboss Magma add_transaction response for {order_id}: {response_json}") if "errors" in response_json: error_message = response_json["errors"][0].get("message", "Unknown Amboss API error") @@ -519,32 +710,26 @@ def confirm_channel_point_to_amboss(order_id, transaction): "This is a critical failure from Amboss. Halting bot to prevent further issues." ) logging.critical(log_content) - # This is a case where Amboss itself failed a critical step. - # We will create the critical error flag here as it might indicate a broader Amboss issue - # or a problem with our API key / permissions for mutations. with open(CRITICAL_ERROR_FILE_PATH, "a") as err_file: err_file.write(f"{datetime.now()}: {log_content}\n") send_telegram_notification(f"🔥 CRITICAL: Failed to confirm channel to Amboss for order `{order_id}` due to API error: `{error_message}`. Bot halted. Manual check required.", level="error", parse_mode="Markdown") - # Return the error structure for the caller return response_json else: - return response_json # Success + return response_json except requests.exceptions.Timeout: logging.error(f"Timeout while confirming channel point to Amboss for order {order_id}.") - # Do not create critical flag for a simple timeout on one order confirmation. return {"errors": [{"message": "Timeout during Amboss API call"}]} except requests.exceptions.RequestException as e: logging.error(f"API request error confirming channel point to Amboss for order {order_id}: {e}") return {"errors": [{"message": f"RequestException: {e}"}]} except json.JSONDecodeError as e: - logging.error(f"Failed to decode JSON response confirming channel point to Amboss for order {order_id}: {e}") + logging.error(f"Failed to decode JSON response when confirming channel point for order {order_id}: {e}") return {"errors": [{"message": f"JSONDecodeError: {e}"}]} except Exception as e: logging.exception(f"Unexpected error in confirm_channel_point_to_amboss for {order_id}:") return {"errors": [{"message": f"Unexpected error: {e}"}]} - def get_channel_point(hash_to_find): def execute_lightning_command(): command = [LNCLI_PATH, "pendingchannels"] @@ -975,17 +1160,25 @@ def calculate_utxos_required_and_fees(amount_input, fee_per_vbyte): return utxos_needed, fee_cost, related_outpoints -def get_orders_awaiting_channel_open(): # Renamed from check_channel +def get_orders_awaiting_channel_open(): logging.info("Checking for Magma orders awaiting channel open (WAITING_FOR_CHANNEL_OPEN)...") - payload = {"query": GET_USER_MARKET_OFFER_ORDERS_QUERY} + payload = { + "query": GET_SALES_QUERY, + "variables": { + "input": {"status": ["WAITING_FOR_CHANNEL_OPEN"]}, + "page": {"limit": 25, "offset": 0} + } + } - data = _execute_amboss_graphql_request(payload, "GetOrdersAwaitingChannelOpen") + data = _execute_amboss_graphql_request(payload, "GetOrdersAwaitingChannelOpen", endpoint_url=MAGMA_GRAPHQL_URL) if not data: - return None # Error already logged by helper + return None - market = data.get("getUser", {}).get("market", {}) - offer_orders = market.get("offer_orders", {}).get("list", []) + sales_data = data.get("user", {}).get("market", {}).get("orders", {}).get("sales", {}) + offer_orders = sales_data.get("list", []) + if not offer_orders: + offer_orders = data.get("getUser", {}).get("market", {}).get("offer_orders", {}).get("list", []) orders_to_open = [ offer for offer in offer_orders if offer.get("status") == "WAITING_FOR_CHANNEL_OPEN" @@ -1003,38 +1196,42 @@ def get_orders_awaiting_channel_open(): # Renamed from check_channel return found_offer -def get_offers_awaiting_seller_approval(): # Renamed from check_offers +def get_offers_awaiting_seller_approval(): global processed_banned_offer_ids logging.info("Checking for Magma offers awaiting seller approval (WAITING_FOR_SELLER_APPROVAL)...") - payload = {"query": GET_USER_MARKET_OFFER_ORDERS_QUERY} + payload = { + "query": GET_SALES_QUERY, + "variables": { + "input": {"status": ["WAITING_FOR_SELLER_APPROVAL"]}, + "page": {"limit": 25, "offset": 0} + } + } - data = _execute_amboss_graphql_request(payload, "GetOffersAwaitingSellerApproval") + data = _execute_amboss_graphql_request(payload, "GetOffersAwaitingSellerApproval", endpoint_url=MAGMA_GRAPHQL_URL) if not data: return None - market_data = data.get("getUser", {}).get("market", {}) - offer_orders_list = market_data.get("offer_orders", {}).get("list", []) + sales_data = data.get("user", {}).get("market", {}).get("orders", {}).get("sales", {}) + offer_orders_list = sales_data.get("list", []) + if not offer_orders_list: + offer_orders_list = data.get("getUser", {}).get("market", {}).get("offer_orders", {}).get("list", []) for offer in offer_orders_list: - offer_id = offer.get("id") - current_status = offer.get("status") - destination_pubkey = offer.get('account') or offer.get("endpoints", {}).get("destination") + order_info = extract_order_info(offer) + offer_id = order_info.get("id") + current_status = order_info.get("status") + destination_pubkey = order_info.get("customer_pubkey") logging.debug(f"Offer ID: {offer_id}, Status: {current_status}, Buyer Pubkey: {destination_pubkey}") - # Primary filter: Only consider offers genuinely awaiting seller approval if current_status != "WAITING_FOR_SELLER_APPROVAL": logging.debug(f"Offer {offer_id} has status '{current_status}', not 'WAITING_FOR_SELLER_APPROVAL'. Skipping.") - # If it was a previously auto-rejected banned offer, we might want to ensure it's removed from our temporary set - # if Amboss now confirms its non-pending status. This prevents the set from growing indefinitely if Amboss updates. if offer_id in processed_banned_offer_ids and current_status in ["SELLER_REJECTED", "CANCELLED", "EXPIRED", "ERROR", "COMPLETED"]: logging.info(f"Offer {offer_id} (previously auto-rejected) now has terminal status '{current_status}'. Removing from processed_banned_offer_ids.") processed_banned_offer_ids.discard(offer_id) continue - # At this point, offer.status is "WAITING_FOR_SELLER_APPROVAL" - if destination_pubkey in BANNED_PUBKEYS: if offer_id not in processed_banned_offer_ids: logging.info( @@ -1042,7 +1239,16 @@ def get_offers_awaiting_seller_approval(): # Renamed from check_offers ) reject_response = reject_order(offer_id) - if reject_response and not reject_response.get("errors") and reject_response.get("data", {}).get("sellerRejectOrder"): + is_rejected = ( + reject_response + and not reject_response.get("errors") + and ( + reject_response.get("data", {}).get("market", {}).get("order", {}).get("seller", {}).get("reject", {}).get("success") is True + or reject_response.get("data", {}).get("sellerRejectOrder") + ) + ) + + if is_rejected: send_telegram_notification( f"🗑️ Auto-rejected offer `{offer_id}` (was WAITING_FOR_SELLER_APPROVAL) from banned pubkey: `{destination_pubkey}`.", level="warning", parse_mode="Markdown" @@ -1060,18 +1266,15 @@ def get_offers_awaiting_seller_approval(): # Renamed from check_offers f"did not confirm success or failed. Response: {reject_response}. Will retry processing next cycle." ) else: - logging.debug(f"Offer {offer_id} from banned pubkey {destination_pubkey} was already processed for auto-rejection in this session (still WAITING_FOR_SELLER_APPROVAL). Skipping.") - continue # This offer (from banned pubkey, now handled or previously handled) should not be returned for manual approval + logging.debug(f"Offer {offer_id} from banned pubkey {destination_pubkey} was already attempted for auto-rejection. Skipping action this cycle.") + continue - # If we reach here, the offer is "WAITING_FOR_SELLER_APPROVAL" and NOT from a banned pubkey. - # This is a candidate for manual user approval. - logging.info(f"Found valid, unbanned offer awaiting approval: {offer_id}") - return offer # Return the first such offer + logging.info(f"Found actionable offer WAITING_FOR_SELLER_APPROVAL: {offer_id}") + return offer - logging.info("No unbanned offers found currently in 'WAITING_FOR_SELLER_APPROVAL' status.") + logging.info("No actionable offers found awaiting seller approval.") return None - def open_channel(pubkey, size, invoice): # get fastest fee logging.info("Getting fastest fee...") @@ -1191,7 +1394,14 @@ def _complete_offer_approval_process(order_id, order_details): accept_result = accept_order(order_id, invoice_request) logging.info(f"Order {order_id} Amboss acceptance result: {accept_result}") - if "data" in accept_result and "sellerAcceptOrder" in accept_result["data"] and accept_result["data"]["sellerAcceptOrder"]: + is_accepted = ( + "data" in accept_result + and ( + accept_result["data"].get("market", {}).get("order", {}).get("seller", {}).get("accept", {}).get("success") is True + or accept_result["data"].get("sellerAcceptOrder") is True + ) + ) + if is_accepted: success_message = f"⏳ Order `{order_id}` accepted on Amboss. Invoice sent. Monitoring for buyer payment." send_telegram_notification(success_message, parse_mode="Markdown") logging.info(success_message) @@ -1374,20 +1584,17 @@ def process_new_offers(): logging.info("No new Magma offers found requiring seller approval at this time.") return - order_id = new_offer_from_amboss['id'] + order_info = extract_order_info(new_offer_from_amboss) + order_id = order_info.get('id') # If it's already pending user confirmation, we've already asked. Let timeout or callback handle it. if order_id in pending_user_confirmations: - logging.info(f"Offer {order_id} is already awaiting user confirmation. Skipping new prompt.") + logging.info(f"Offer {order_id} is already pending user confirmation. Skipping asking again.") return - # This is a genuinely new offer we haven't prompted for yet. - seller_invoice_amount = new_offer_from_amboss['seller_invoice_amount'] - current_status = new_offer_from_amboss['status'] - # 'account' is often the buyer's pubkey in Amboss market data, - # 'destination' under endpoints is also usually the buyer's pubkey. Prefer 'account' if available. - destination_pubkey = new_offer_from_amboss.get('account') or \ - new_offer_from_amboss.get("endpoints", {}).get("destination") + seller_invoice_amount = order_info.get('seller_invoice_amount') + current_status = order_info.get('status') + destination_pubkey = order_info.get('customer_pubkey') # Pre-check for banned pubkey before even asking user @@ -1538,10 +1745,13 @@ def process_paid_order(order_details): send_telegram_notification("🔥 Critical internal error: Invalid data for processing paid order. Check logs.", level="error") return - order_id = order_details.get('id', 'MISSING_ID') - customer_pubkey = order_details.get('account') - channel_size_str = order_details.get('size') - seller_invoice_amount_str = order_details.get('seller_invoice_amount') + order_info = extract_order_info(order_details) + order_id = order_info.get('id', 'MISSING_ID') + customer_pubkey = order_info.get('customer_pubkey') + channel_size = order_info.get('channel_size') + seller_invoice_amount = order_info.get('seller_invoice_amount') + channel_size_str = str(channel_size) if channel_size else None + seller_invoice_amount_str = str(seller_invoice_amount) if seller_invoice_amount else None if not all([order_id != 'MISSING_ID', customer_pubkey, channel_size_str, seller_invoice_amount_str]): logging.error(f"Missing critical fields in order_details for order {order_id}: Pubkey={customer_pubkey}, Size={channel_size_str}, InvoiceAmount={seller_invoice_amount_str}") diff --git a/README.md b/README.md index fe90c72..1199674 100644 --- a/README.md +++ b/README.md @@ -1,180 +1,191 @@ -## Collection of Lightning Scripts - -Welcome! This repository contains a collection of Python scripts designed to interact with various Lightning Network tools and services. While some scripts are tailored for specific setups, they can serve as a helpful starting point or inspiration for your own projects. - -### Current Scripts - -Below is a list of available scripts and their primary functions. Scripts marked with `[command-line output]` typically offer more detailed help if you run them with the `-h` or `--help` flag. - -**LNDg:** -- `amboss_pull.py`: [cronjob, one-off] Automatically gathers your Amboss Magma Sell Orders and writes channel details into the LNDg GUI. Optionally populates a file configuration for charge-lnd and can trigger other settings in LNDg (e.g., activate AutoFee once maturity is reached). -- `channel_base-fee.py`: [cronjob, one-off] Modifies channel settings in LNDg based on other LNDg fields. For example, it can change the base fee once a certain fee condition is met. -- `channel_fee_pull.py`: [cronjob, one-off] Retrieves LNDg channel details such as fee rate and base fee, then writes this information to a file for other systems to use. -- `swap_out_candidates.py`: [command-line output] Identifies active channels with a local balance above a specified `--capacity` threshold and low local fees, making them good candidates for submarine swaps (swapping out). Can export to .bos tags format. -- `mempool_rebalancer_trigger.py`: [systemd service] Monitors the current mempool fee rate for a half-hour estimate. If the fee rate exceeds your defined threshold (e.g., > 150 sats/vByte), it disables the Auto-Fee setting in LNDg. It automatically re-activates Auto-Fee once the fee rate drops below your limit. -- `disabled_fee-accelerator.py`: [cronjob] LNDg's AutoFees feature increases fees based on incoming HTLCs. This script helps manage channels that LNDg might suggest disabling due to low outbound liquidity by automatically increasing their fees. - -**Magma (Amboss):** -- 🆕 `magma-sale.py`: [systemd service] Provides automated monitoring of channel sales on Amboss Magma. It handles order clearance, channel opening, fee management, and writes relevant information into LNDg notes. Refactored and documented in `magma-sale.MD`. - - - -**Peerswap:** -- `peerswap-lndg_push.py`: [command-line output, cronjob] Gathers information about your existing PeerSwap peers, including the sum of satoshis swapped and the number of swaps, and writes this data to the LNDg Dashboard and relevant Channel Cards. -- `ps_peers.py`: [command-line output] Offers a quick tabular overview of your L-BTC Balance and PeerSwap peers, including their liquidity. - -**LNBits:** -- `pocketmoney.py`: [one-off, cronjob, systemd service] Enables recurring payments to child wallets within the same LNBits instance. You can define the fiat currency, recurrence schedule, and amount for each child. To configure, copy `config.ini.example` to `config.ini` and `wallets.json.example` to `wallets.json`, then edit both new files. - -**Other:** -- `swap_wallet.py`: [one-off] Sends a specified amount of Lightning funds to a given LN address. Allows customization of total amount, amount per transaction, interval between transactions, maximum fee rate, and an optional message for the payments. -- `fee_adjuster.py`: [systemd service, cronjob] Automatically adjusts channel fees based on Amboss API data and user-defined settings. Includes dynamic inbound fee discounts, fee bands, stuck channel adjustments, global rebalance guard auditing across all 100+ LNDg channels, and Dynamic Hysteresis unlocking. Configure via `feeConfig.json` and `config.ini`. Install using `sudo ./Other/install_fee_adjuster_service.sh` or run as a cron job. -- `rebalance_guard.py`: [command-line output, cronjob] Standalone CLI tool to audit all open LNDg channels across both native Auto-Fees (`af.py`) and `fee_adjuster.py`. Protects refilling channels with active inbound discounts by setting `ar_out_target = 100%` (preventing LNDg from draining them as outbound rebalance donors), and automatically restores baseline targets using **Dynamic Hysteresis** ($\text{threshold} = \min(\max(\text{baseline} + 15\%, 60\%), 95\%)$) once liquidity recovers and inbound discounts are deactivated. Run with `--dry-run` to preview actions. -- `boltz_swap-out.py`: [command-line output, one-off] Automates Lightning Network (LN) to Liquid Bitcoin (L-BTC) swaps using Boltz for submarine swaps (swapping out). - -### === Installation Instructions === - -To use these scripts, it's recommended to set up a Python virtual environment. This keeps dependencies for this project isolated from other Python projects on your system. - -1. **Clone the Repository:** - If you haven't already, download the scripts to your machine: - ```bash - git clone https://github.com/TrezorHannes/Lightning-Python-Tools.git - cd Lightning-Python-Tools/ - ``` - -2. **Install `virtualenv` (if you don't have it):** - `virtualenv` is a tool to create isolated Python environments. - ```bash - sudo apt update - sudo apt install virtualenv - ``` - -3. **Create a Virtual Environment:** - Inside the `Lightning-Python-Tools` directory, create a virtual environment (commonly named `.venv`): - ```bash - virtualenv -p python3 .venv - ``` - This creates a `.venv` folder in your project directory. - -4. **Activate the Virtual Environment:** - Before you can use the scripts or install packages, you need to activate the environment: - ```bash - source .venv/bin/activate - ``` - Your shell prompt will usually change to indicate that the virtual environment is active (e.g., `(.venv) your-user@host:...$`). - -5. **Install Required Dependencies:** - With the virtual environment active, install the necessary Python packages listed in `requirements.txt`: - ```bash - pip install -r requirements.txt - ``` - -6. **Configure the Scripts:** - Many scripts rely on a `config.ini` file for settings like API keys and paths. - Copy the example configuration and edit it with your details: - ```bash - cp config.ini.example config.ini - nano config.ini - ``` - For scripts that use other configuration files (like `wallets.json` or `feeConfig.json`), follow the same pattern: copy the `.example` file and edit the copy. - -### === Basic Usage === - -1. **Activate Virtual Environment (if not already active):** - Each time you open a new terminal window to run these scripts, you'll need to activate the environment: - ```bash - cd path/to/Lightning-Python-Tools/ # Navigate to the project directory - source .venv/bin/activate - ``` - -2. **Run a Script:** - Execute scripts using the Python interpreter within your virtual environment. For example: - ```bash - python3 Peerswap/ps_peers.py - python3 LNDg/amboss_pull.py - python3 Other/boltz_swap-out.py --amount 100000 --capacity 2000000 - ``` - Many scripts provide help with the `-h` or `--help` flag: - ```bash - python3 LNDg/swap_out_candidates.py -h - ``` - -### === Running Scripts as Background Services === - -Some scripts are designed to run continuously or on a schedule. - -**1. Cron Jobs:** -For scripts that need to run periodically (e.g., every hour), you can use `cron`. -Edit your crontab: +# ⚡ Lightning Python Tools + +A comprehensive suite of production-grade automation scripts, background daemons, liquidity optimizers, and monitoring utilities for **LND** Lightning Network node operators. + +--- + +## 📑 Table of Contents +- [Overview](#-overview) +- [Architecture & Script Directory](#-architecture--script-directory) + - [1. Magma (Amboss Liquidity Market)](#1-magma-amboss-liquidity-market) + - [2. LNDg Integration & Fee Management](#2-lndg-integration--fee-management) + - [3. PeerSwap Automation](#3-peerswap-automation) + - [4. Swaps & Capital Management](#4-swaps--capital-management) + - [5. LNbits & Micro-services](#5-lnbits--micro-services) +- [Prerequisites & Installation](#-prerequisites--installation) +- [Configuration Guide](#-configuration-guide) +- [Running as Background Services](#-running-as-background-services) + - [Systemd Daemon Example](#systemd-daemon-example) + - [Crontab Example](#crontab-example) +- [Testing & Quality Assurance](#-testing--quality-assurance) +- [Security Best Practices](#-security-best-practices) + +--- + +## 🌟 Overview + +Operating an institutional or routing Lightning node requires robust, automated tooling across multiple dimensions: +- **Inbound Liquidity Auctioning**: Autonomous pricing and clearance on the Amboss Magma liquidity market. +- **Dynamic Fee Optimization**: Real-time channel fee adjustment based on remote market statistics, flow metrics, and stuck liquidity. +- **Inbound Discount Protection & Rebalance Guard**: Prevents circular rebalance loops from draining channels with active fee discounts. +- **Submarine & Liquid Swaps**: Programmatic balance management across on-chain, Liquid (L-BTC), and Lightning. +- **Automated PeerSwap Operations**: Triggering rebalances and swaps based on live peer flow. + +--- + +## 🛠️ Architecture & Script Directory + +### 1. Magma (Amboss Liquidity Market) +Automate your inbound liquidity sales on the Amboss Magma marketplace using the modern Magma GraphQL API (`https://magma.amboss.tech/graphql`). + +| Script | Execution Mode | Purpose | +| :--- | :--- | :--- | +| [`Magma/magma_sale_process.py`](Magma/magma_sale_process.py) | `systemd service` | **Seller Auto-Pilot Daemon**: Listens for orders awaiting seller approval or channel open, checks UTXO and mempool fees, creates pre-image HODL invoices, opens channels, confirms funding transactions to Amboss, and auto-rejects banned peers. | +| [`Magma/magma_market_fee.py`](Magma/magma_market_fee.py) | `cronjob` / `service` | **Dynamic APR & Fee Synchronizer**: Scrapes active Magma public market offers, analyzes percentile rates, computes APR across timeframes (30d/60d/90d), and updates or toggles your Magma sell offers. Supports dry-run simulations. | + +--- + +### 2. LNDg Integration & Fee Management +Synchronize metrics with your local LNDg dashboard and automate channel policies. + +| Script | Execution Mode | Purpose | +| :--- | :--- | :--- | +| [`LNDg/amboss_pull.py`](LNDg/amboss_pull.py) | `cronjob` / `one-off` | Pulls Magma orders from Amboss, writes channel details into LNDg GUI, and configures auto-fee triggers upon channel maturity. | +| [`LNDg/mempool_rebalancer_trigger.py`](LNDg/mempool_rebalancer_trigger.py) | `systemd service` | Monitors live mempool block fee rates; triggers LNDg rebalancer when fees dip below user-defined thresholds. | +| [`LNDg/channel_base-fee.py`](LNDg/channel_base-fee.py) | `cronjob` / `one-off` | Normalizes or updates base fees across all open channels in LNDg. | +| [`LNDg/channel_fee-pull.py`](LNDg/channel_fee-pull.py) | `cronjob` | Pulls channel fee configurations from remote references and synchronizes into LNDg. | +| [`LNDg/disabled_fee-accelerator.py`](LNDg/disabled_fee-accelerator.py) | `cronjob` | Identifies disabled/inactive channels and escalates fees or triggers alerts. | +| [`LNDg/swap_out-candidates.py`](LNDg/swap_out-candidates.py) | `CLI tool` | Identifies channels with heavy local balances as candidates for off-loading liquidity via swap-out. | +| [`LNDg/offline_summary.py`](LNDg/offline_summary.py) | `cronjob` / `alert` | Sends Telegram reports summarizing offline channel downtime and liquidity impacts. | + +--- + +### 3. PeerSwap Automation +Automate Layer-1 / Liquid swaps with connected peers. + +| Script | Execution Mode | Purpose | +| :--- | :--- | :--- | +| [`Peerswap/peerswap-bot.py`](Peerswap/peerswap-bot.py) | `systemd service` | **PeerSwap Telegram Bot**: Interactive management bot to monitor peer swap eligibility, submit swaps, and view historical swap status. | +| [`Peerswap/peerswap-lndg_push.py`](Peerswap/peerswap-lndg_push.py) | `cronjob` | Ingests PeerSwap metrics and logs them directly into LNDg notes and database records. | +| [`Peerswap/ps_peers.py`](Peerswap/ps_peers.py) | `CLI tool` | Summarizes liquidity distribution and active status of all PeerSwap-compatible peers. | + +--- + +### 4. Swaps & Capital Management +Manage on-chain UTXOs, submarine swaps, and dynamic channel fee pricing. + +| Script | Execution Mode | Purpose | +| :--- | :--- | :--- | +| [`Other/fee_adjuster.py`](Other/fee_adjuster.py) | `cronjob` / `service` | **Dynamic Fee Optimizer**: Adjusts channel fees based on Amboss Space network fee trends, stuck channel detection, and liquidity percentages. Includes inbound discount protection and global rebalance guard auditing. | +| [`Other/rebalance_guard.py`](Other/rebalance_guard.py) | `cronjob` / `CLI tool` | **Standalone Rebalance Guard**: Audits all open LNDg channels across both native Auto-Fees and `fee_adjuster.py`. Protects channels with active inbound discounts by setting `ar_out_target = 100%`, and automatically restores baseline targets using **Dynamic Hysteresis** once liquidity recovers. | +| [`Other/boltz_swap-out.py`](Other/boltz_swap-out.py) | `CLI tool` | Automates submarine swap-outs through the Boltz exchange (Lightning to Liquid L-BTC). | +| [`Other/swap_wallet.py`](Other/swap_wallet.py) | `CLI tool` | Batches automated payouts or drain payments over Lightning to a designated Lightning Address. | +| [`Other/swap_out-loop.py`](Other/swap_out-loop.py) | `CLI tool` | Orchestrates continuous swap-outs for rebalancing large liquidity sinks. | +| [`Other/lnd_utxo_consolidator.py`](Other/lnd_utxo_consolidator.py) | `CLI tool` | Safely consolidates fragmented on-chain LND UTXOs during low-mempool fee environments. | + +--- + +### 5. LNbits & Micro-services + +| Script | Execution Mode | Purpose | +| :--- | :--- | :--- | +| [`LNBits/pocketmoney.py`](LNBits/pocketmoney.py) | `cronjob` | Automated pocket money allowances: tops up linked LNbits wallets on a regular schedule. | + +--- + +## 🚀 Prerequisites & Installation + +### System Requirements +- Python 3.10, 3.11, or 3.12 +- Linux environment (Debian/Ubuntu/Raspberry Pi OS) +- Running **LND** node (`lncli` accessible) +- (Optional) Installed `bos` (Balance of Satoshis) and `peerswapd` + +### 1. Clone & Set Up Virtual Environment +```bash +git clone https://github.com/TrezorHannes/Lightning-Python-Tools.git +cd Lightning-Python-Tools + +# Create and activate Python virtual environment +python3 -m venv .venv +source .venv/bin/activate + +# Upgrade pip and install all required dependencies +pip install --upgrade pip +pip install -r requirements.txt +``` + +--- + +## ⚙️ Configuration Guide + +1. Create your local `config.ini`: + ```bash + cp config.ini.example config.ini + ``` +2. Populate the required sections: + - `[credentials]`: Amboss API token (`amboss_authorization`), LNbits keys, LNDg credentials. + - `[telegram]`: Telegram bot token and chat ID for notifications. + - `[lndg]`: LNDg local API URL (e.g. `http://localhost:8889`). + - `[paths]`: Absolute path to `lncli`, `bos`, and output rule directories. + - `[magma]`: Min/max fee caps, invoice expiry, auto-approval thresholds. + +--- + +## 🔄 Running as Background Services + +### Systemd Daemon Example +Create `/etc/systemd/system/magma-sale.service`: +```ini +[Unit] +Description=Amboss Magma Channel Auto-Sale Service +After=network.target lnd.service + +[Service] +Type=simple +User=admin +WorkingDirectory=/home/admin/Lightning-Python-Tools +ExecStart=/home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Magma/magma_sale_process.py +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +``` +Enable and start the service: ```bash -crontab -e +sudo systemctl daemon-reload +sudo systemctl enable magma-sale.service +sudo systemctl start magma-sale.service ``` -Add a line similar to this, replacing `INSTALLDIR` with the absolute path to your `Lightning-Python-Tools` directory: + +### Crontab Example ```cron -0 * * * * /INSTALLDIR/Lightning-Python-Tools/.venv/bin/python3 /INSTALLDIR/Lightning-Python-Tools/LNDg/amboss_pull.py >> /home/admin/cron.log 2>&1 +# Run fee adjuster every 30 minutes +*/30 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Other/fee_adjuster.py >> /home/admin/cron_fee.log 2>&1 + +# Run standalone rebalance guard audit hourly +0 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Other/rebalance_guard.py >> /home/admin/cron_guard.log 2>&1 + +# Sync Amboss Magma channel data to LNDg hourly +0 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/LNDg/amboss_pull.py >> /home/admin/cron_amboss.log 2>&1 +``` + +--- + +## 🧪 Testing & Quality Assurance + +The codebase includes full test coverage with unit tests and live GraphQL schema verification: + +```bash +# Run the entire test suite +source .venv/bin/activate +pytest tests/ -v ``` -This example runs `amboss_pull.py` at the start of every hour and logs its output. - -**2. Systemd Services:** -For scripts intended to run as long-running background services (often indicated with `[systemd service]` in the script list), you can create a `systemd` unit file. - -* Create a `.service` file (e.g., `mempool_rebalancer_trigger.service`): - ```ini - [Unit] - Description=Mempool Rebalancer Trigger Service - After=network.target - - [Service] - Type=simple - User=your_username # Replace with the user that should run the script - WorkingDirectory=/path/to/Lightning-Python-Tools/LNDg # Adjust to script's directory - ExecStart=/path/to/Lightning-Python-Tools/.venv/bin/python3 /path/to/Lightning-Python-Tools/LNDg/mempool_rebalancer_trigger.py - Restart=on-failure - RestartSec=5s - - [Install] - WantedBy=multi-user.target - ``` - **Important:** Replace `/path/to/` with the actual absolute path to your `Lightning-Python-Tools` directory and `your_username` with the appropriate user. Adjust `WorkingDirectory` and `ExecStart` paths for the specific script. - -* Copy the file to the systemd directory: - ```bash - sudo cp mempool_rebalancer_trigger.service /etc/systemd/system/ - ``` - -* Reload systemd, enable the service (to start on boot), and start it: - ```bash - sudo systemctl daemon-reload - sudo systemctl enable mempool_rebalancer_trigger.service - sudo systemctl start mempool_rebalancer_trigger.service - ``` - -* You can check its status with: - ```bash - sudo systemctl status mempool_rebalancer_trigger.service - ``` - -### === Optional: Create Command Aliases === - -For easier access to frequently used scripts, you can create aliases in your shell's configuration file (e.g., `~/.bash_aliases` or `~/.bashrc`). - -1. Open the file (e.g., `nano ~/.bash_aliases`). -2. Add lines like these, replacing `INSTALLDIR` with the absolute path to your `Lightning-Python-Tools` directory: - ```bash - alias ps_list="INSTALLDIR/Lightning-Python-Tools/.venv/bin/python3 INSTALLDIR/Lightning-Python-Tools/Peerswap/ps_peers.py" - alias lndg_amboss="INSTALLDIR/Lightning-Python-Tools/.venv/bin/python3 INSTALLDIR/Lightning-Python-Tools/LNDg/amboss_pull.py" - ``` -3. Save the file and apply the changes (e.g., `source ~/.bash_aliases` or open a new terminal). - -### === How to Obtain Telegram Bot Information === - -Some scripts might use Telegram for notifications. To get the necessary IDs: -- **Personal Chat ID:** Contact the `@myidbot` on Telegram and send the `/getid` command. -- **Group Chat ID:** Add `@myidbot` to your Telegram group and use the `/getgroupid` command. -- **Alternative (Bot API):** Send your bot any message in the desired chat. Then, visit `https://api.telegram.org/bot{YourBotToken}/getUpdates` (replacing `{YourBotToken}` with your actual bot token) in a web browser. You'll find the chat ID in the JSON response. --- -If you have questions or need support, feel free to reach out. -Contact: \ No newline at end of file +## 🔒 Security Best Practices +- Never commit `config.ini`, `.env`, or credential files to Git. +- Restrict file permissions on `config.ini`: `chmod 600 config.ini`. +- Use read-only or restricted API keys whenever possible. diff --git a/pyproject.toml b/pyproject.toml index 764bd45..d3adb9c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,7 +2,10 @@ pythonpath = [ ".", "Other", - "Magma" + "Magma", + "LNDg", + "Peerswap", + "LNBits" ] testpaths = ["tests"] addopts = "-v" diff --git a/tests/Magma/test_magma_market_fee.py b/tests/Magma/test_magma_market_fee.py new file mode 100644 index 0000000..47be5a4 --- /dev/null +++ b/tests/Magma/test_magma_market_fee.py @@ -0,0 +1,316 @@ +import sys +import os +import pytest +from unittest.mock import MagicMock, patch, mock_open + +@pytest.fixture(scope="module", autouse=True) +def mock_fee_dependencies(): + mock_telebot = MagicMock() + mock_telebot.TeleBot = MagicMock() + mock_configparser = MagicMock() + mock_logging = MagicMock() + + mock_config_data = { + "credentials": {"amboss_authorization": "fake_auth"}, + "info": {"NODE": "03mypubkey123456"}, + "system": {"log_level": "INFO"}, + "market_analysis": { + "min_seller_score_filter": "75.0", + "pricing_percentile_ppm": "50", + "pricing_percentile_fixed": "50" + }, + "capital_management": { + "min_onchain_reserve_sats": "100000", + "max_capital_allocation_percentage": "80" + }, + "paths": {"lncli_path": "lncli"} + } + + mock_config_instance = MagicMock() + mock_config_instance.__getitem__.side_effect = mock_config_data.__getitem__ + mock_config_instance.get = MagicMock(side_effect=lambda section, option, fallback=None: mock_config_data.get(section, {}).get(option, fallback)) + mock_config_instance.getint = MagicMock(return_value=1000) + mock_config_instance.getfloat = MagicMock(return_value=75.0) + mock_config_instance.has_option = MagicMock(return_value=True) + mock_config_instance.has_section = MagicMock(return_value=True) + mock_config_instance.sections = MagicMock(return_value=["template_1"]) + mock_configparser.ConfigParser.return_value = mock_config_instance + + module_patches = { + 'telebot': mock_telebot, + 'configparser': mock_configparser, + 'logging.handlers': MagicMock(), + } + + with patch.dict(sys.modules, module_patches): + with patch("builtins.open", mock_open(read_data="[market_analysis]\nfoo=bar")): + with patch("os.makedirs"): + yield + +@pytest.fixture +def fee_module(mock_fee_dependencies): + if os.path.abspath(os.path.join(os.path.dirname(__file__), '../../Magma')) not in sys.path: + sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../Magma'))) + + import magma_market_fee + magma_market_fee.requests = MagicMock() + magma_market_fee.AMBOSS_TOKEN = "fake_auth" + magma_market_fee.MY_NODE_PUBKEY = "03mypubkey123456" + return magma_market_fee + +# --- TESTS --- + +def test_extract_market_offer_info_new_api(fee_module): + """Test extracting normalized fields from new Magma MarketOffer schema.""" + sample_offer = { + "id": "off_001", + "status": "ENABLED", + "side": "SELL", + "account": {"pubkey": "03othernode999"}, + "size": { + "min": {"satoshi": {"sats": "2000000"}}, + "max": {"satoshi": {"sats": "5000000"}}, + "total": {"satoshi": {"sats": "10000000"}}, + "locked": {"satoshi": {"sats": "3000000"}} + }, + "fees": { + "fixed": {"sats": "1000"}, + "variable": {"sats": "450"}, + "amboss": {"sats": "100"} + }, + "promises": { + "min_block_length": 4320, + "base_fee_cap": "1000", + "fee_rate_cap": "450" + }, + "seller_score": 92.5 + } + + info = fee_module.extract_market_offer_info(sample_offer) + + assert info["id"] == "off_001" + assert info["status"] == "ENABLED" + assert info["side"] == "SELL" + assert info["account"] == "03othernode999" + assert info["min_size"] == 2000000 + assert info["max_size"] == 5000000 + assert info["total_size"] == 10000000 + assert info["locked_size"] == 3000000 + assert info["available_size"] == 7000000 + assert info["base_fee"] == 1000 + assert info["fee_rate"] == 450 + assert info["min_block_length"] == 4320 + assert info["seller_score"] == 92.5 + + +def test_calculate_apr(fee_module): + """Test APR calculation formula.""" + apr = fee_module.calculate_apr(1000, 500, 5000000, 30.0) + assert 0.85 <= apr <= 0.86 + + +def test_fetch_public_magma_offers_filtering(fee_module): + """Test fetching public offers filters out own pubkey and low seller scores.""" + mock_response = { + "data": { + "market": { + "offer": { + "offers": { + "total": 3, + "list": [ + # 1. Valid other node offer (score 90 >= 75) + { + "id": "off_valid", + "status": "ENABLED", + "side": "SELL", + "account": {"pubkey": "03peer999"}, + "size": {"min": {"satoshi": {"sats": "2000000"}}, "max": {"satoshi": {"sats": "5000000"}}, "total": {"satoshi": {"sats": "5000000"}}}, + "fees": {"fixed": {"sats": "500"}, "variable": {"sats": "300"}}, + "promises": {"min_block_length": 4320}, + "seller_score": 90.0 + }, + # 2. Own node offer (should be excluded) + { + "id": "off_own", + "status": "ENABLED", + "side": "SELL", + "account": {"pubkey": "03mypubkey123456"}, + "size": {"min": {"satoshi": {"sats": "2000000"}}, "max": {"satoshi": {"sats": "5000000"}}, "total": {"satoshi": {"sats": "5000000"}}}, + "fees": {"fixed": {"sats": "500"}, "variable": {"sats": "300"}}, + "promises": {"min_block_length": 4320}, + "seller_score": 95.0 + }, + # 3. Low seller score (60 < 75, should be excluded) + { + "id": "off_low_score", + "status": "ENABLED", + "side": "SELL", + "account": {"pubkey": "03lowscore111"}, + "size": {"min": {"satoshi": {"sats": "2000000"}}, "max": {"satoshi": {"sats": "5000000"}}, "total": {"satoshi": {"sats": "5000000"}}}, + "fees": {"fixed": {"sats": "500"}, "variable": {"sats": "300"}}, + "promises": {"min_block_length": 4320}, + "seller_score": 60.0 + } + ] + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + fee_module.requests.post = MagicMock(return_value=mock_post) + + magma_config = MagicMock() + magma_config.get = MagicMock(return_value="75.0") + + offers = fee_module.fetch_public_magma_offers("03mypubkey123456", magma_config) + assert len(offers) == 1 + assert offers[0]["id"] == "off_valid" + + +def test_fetch_my_current_offers(fee_module): + """Test fetching user's own Magma sell offers.""" + mock_response = { + "data": { + "user": { + "market": { + "offers": { + "offers": { + "total": 1, + "list": [ + { + "id": "my_off_01", + "status": "ENABLED", + "side": "SELL", + "account": {"pubkey": "03mypubkey123456"}, + "size": { + "min": {"satoshi": {"sats": "5000000"}}, + "max": {"satoshi": {"sats": "5000000"}}, + "total": {"satoshi": {"sats": "15000000"}}, + "locked": {"satoshi": {"sats": "5000000"}} + }, + "fees": {"fixed": {"sats": "1000"}, "variable": {"sats": "400"}}, + "promises": {"min_block_length": 4320}, + "seller_score": 98.0 + } + ] + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + fee_module.requests.post = MagicMock(return_value=mock_post) + + my_offers = fee_module.fetch_my_current_offers() + assert len(my_offers) == 1 + assert my_offers[0]["id"] == "my_off_01" + assert my_offers[0]["available_size"] == 10000000 + + +def test_create_magma_offer_mutation(fee_module): + """Test creating a Magma offer via market.offer.create.""" + mock_response = { + "data": { + "market": { + "offer": { + "create": { + "offer_id": "new_off_123" + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + fee_module.requests.post = MagicMock(return_value=mock_post) + fee_module.DRY_RUN_MODE = False + + pricing = { + "duration_days": 30, + "fixed_fee_sats": 1000, + "ppm_fee_rate": 500, + "channel_size_sats": 5000000 + } + result = fee_module.create_magma_offer(pricing, 10000000, "Template30D") + assert result is not None + assert result["id"] == "new_off_123" + + +def test_update_magma_offer_mutation(fee_module): + """Test updating a Magma offer via market.offer.update.""" + mock_response = { + "data": { + "market": { + "offer": { + "update": { + "success": True + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + fee_module.requests.post = MagicMock(return_value=mock_post) + fee_module.DRY_RUN_MODE = False + + pricing = { + "duration_days": 30, + "fixed_fee_sats": 1200, + "ppm_fee_rate": 550, + "channel_size_sats": 5000000 + } + result = fee_module.update_magma_offer("off_123", pricing, 10000000, "Template30D") + assert result is not None + assert result["status"] == "UPDATED" + + +def test_toggle_magma_offer_status(fee_module): + """Test toggling a Magma offer status via market.offer.toggle.""" + mock_response = { + "data": { + "market": { + "offer": { + "toggle": { + "status": "DISABLED" + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + fee_module.requests.post = MagicMock(return_value=mock_post) + fee_module.DRY_RUN_MODE = False + + success = fee_module.toggle_magma_offer_status("off_123", "Template30D", "DISABLED") + assert success is True + + +def test_dry_run_mode_simulation(fee_module): + """Test that DRY_RUN_MODE simulates mutations without network calls.""" + fee_module.DRY_RUN_MODE = True + pricing = { + "duration_days": 30, + "fixed_fee_sats": 1000, + "ppm_fee_rate": 500, + "channel_size_sats": 5000000 + } + + create_res = fee_module.create_magma_offer(pricing, 10000000, "DryTemplate") + assert "dryrun-offer-id" in create_res["id"] + + update_res = fee_module.update_magma_offer("off_dry", pricing, 10000000, "DryTemplate") + assert update_res["status"] == "DRY_RUN_STATUS_POST_UPDATE" + + toggle_res = fee_module.toggle_magma_offer_status("off_dry", "DryTemplate", "DISABLED") + assert toggle_res is True diff --git a/tests/Magma/test_magma_sale_process.py b/tests/Magma/test_magma_sale_process.py index 5fce444..5d04963 100644 --- a/tests/Magma/test_magma_sale_process.py +++ b/tests/Magma/test_magma_sale_process.py @@ -1,8 +1,7 @@ - import sys import os import pytest -from unittest.mock import MagicMock +from unittest.mock import MagicMock, patch, mock_open # --- FIXTURE: Mock Global Side Effects --- @pytest.fixture(scope="module", autouse=True) @@ -18,14 +17,19 @@ def mock_dependencies(): mock_logging = MagicMock() mock_schedule = MagicMock() - # Mock config dict mock_config_data = { "telegram": {"magma_bot_token": "fake_token", "telegram_user_id": "123"}, "credentials": {"amboss_authorization": "fake_auth"}, "system": {"full_path_bos": "/path/to/bos"}, - "magma": {"invoice_expiry_seconds": "1800", "max_fee_percentage_of_invoice": "0.9", "channel_fee_rate_ppm": "350"}, + "magma": { + "invoice_expiry_seconds": "1800", + "max_fee_percentage_of_invoice": "0.9", + "channel_fee_rate_ppm": "350", + "auto_approve_buyer_conditions": "true", + "auto_approve_min_seller_score": "80.0", + }, "urls": {"mempool_fees_api": "https://mempool.space/api/v1/fees/recommended"}, - "pubkey": {"banned_magma_pubkeys": ""}, + "pubkey": {"banned_magma_pubkeys": "banned_pubkey_1,banned_pubkey_2"}, "paths": {"lncli_path": "lncli"} } @@ -34,6 +38,7 @@ def mock_dependencies(): mock_config_instance.get = MagicMock(side_effect=lambda section, option, fallback=None: mock_config_data.get(section, {}).get(option, fallback)) mock_config_instance.getint = MagicMock(return_value=10) mock_config_instance.getfloat = MagicMock(return_value=0.5) + mock_config_instance.has_option = MagicMock(return_value=True) mock_configparser.ConfigParser.return_value = mock_config_instance module_patches = { @@ -42,25 +47,11 @@ def mock_dependencies(): 'configparser': mock_configparser, 'schedule': mock_schedule, 'logging.handlers': MagicMock(), - # We don't actully want to strictly mock logging or it suppresses output, but we prevent file handler creation } - from unittest.mock import patch, mock_open - - # Apply patches with patch.dict(sys.modules, module_patches): with patch("builtins.open", mock_open(read_data="[magma]\nfoo=bar")): with patch("os.makedirs"): - # Normally we'd import here. - # However, since we are inside a fixture, and pytest collects modules first, - # we need to ensure the import happens strictly under this context. - # But python imports are cached. - - # To make this robust, we import inside the test functions OR use 'importlib.reload' if needed. - # But since we use autouse=True scope=module, tests in this file will "see" the mocked modules - # if we import right here or if we import at top level BUT rely on this fixture running first? - # No, top level imports happen at collection time. - # So we MUST move the import `import magma_sale_process` INTO the test functions or a fixture that returns the module. yield @pytest.fixture @@ -68,20 +59,17 @@ def magma_module(mock_dependencies): """ Imports and returns the magma_sale_process module ensuring it is mocked. """ - # Verify we can import it now - # We might need to handle sys.path if pyproject.toml didn't kick in yet or for safety if os.path.abspath(os.path.join(os.path.dirname(__file__), '../../Magma')) not in sys.path: sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../Magma'))) import magma_sale_process - # Reset vital mocks magma_sale_process.requests = MagicMock() return magma_sale_process # --- TESTS --- def test_get_node_alias_success(magma_module): - """Test retrieving node alias successfully.""" + """Test retrieving node alias successfully from Space endpoint.""" mock_response = {"data": {"getNodeAlias": "TestNode"}} mock_post = MagicMock() @@ -92,6 +80,7 @@ def test_get_node_alias_success(magma_module): alias = magma_module.get_node_alias("pubkey123") assert alias == "TestNode" + def test_get_node_alias_failure(magma_module): """Test retrieving node alias when API fails.""" mock_post = MagicMock() @@ -101,6 +90,75 @@ def test_get_node_alias_failure(magma_module): alias = magma_module.get_node_alias("pubkey123") assert alias == "ErrorFetchingAlias" + +def test_extract_order_info_new_api(magma_module): + """Test extracting normalized fields from new Magma API GraphQL response.""" + sample_order = { + "id": "order_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": { + "satoshi": { + "sats": "5000000", + "btc": "0.05", + "usd": "3000" + } + }, + "fees": { + "fixed": {"sats": "1000"}, + "variable": {"sats": "2500"}, + "seller": {"sats": "3500"}, + "amboss": {"sats": "500"}, + "buyer": {"sats": "4000"} + }, + "promises": { + "locked_min_block_length": 4320, + "locked_base_fee_cap": {"sats": "1000"}, + "locked_fee_rate_cap": {"sats": "500"} + }, + "destination": { + "pubkey": "03deadbeef1234567890", + "alias": "LightningBuyer" + }, + "created_at": "2026-08-27T12:00:00Z" + } + + info = magma_module.extract_order_info(sample_order) + + assert info["id"] == "order_001" + assert info["status"] == "WAITING_FOR_SELLER_APPROVAL" + assert info["customer_pubkey"] == "03deadbeef1234567890" + assert info["buyer_alias"] == "LightningBuyer" + assert info["channel_size"] == 5000000 + assert info["seller_invoice_amount"] == 3500 + assert info["fixed_fee"] == 1000 + assert info["variable_fee"] == 2500 + assert info["amboss_fee"] == 500 + assert info["min_block_length"] == 4320 + + +def test_extract_order_info_legacy_fallback(magma_module): + """Test extracting normalized fields when encountering legacy flat dict fields.""" + legacy_order = { + "id": "legacy_001", + "status": "WAITING_FOR_CHANNEL_OPEN", + "size": 2000000, + "seller_invoice_amount": 1500, + "fixed_fee": 500, + "variable_fee": 1000, + "account": "02abcdef123456", + "locked_min_block_length": 2016 + } + + info = magma_module.extract_order_info(legacy_order) + + assert info["id"] == "legacy_001" + assert info["status"] == "WAITING_FOR_CHANNEL_OPEN" + assert info["customer_pubkey"] == "02abcdef123456" + assert info["channel_size"] == 2000000 + assert info["seller_invoice_amount"] == 1500 + assert info["min_block_length"] == 2016 + + def test_execute_lncli_addinvoice_success(magma_module, mocker): """Test generating an invoice calls lncli correctly.""" mock_popen = mocker.patch("subprocess.Popen") @@ -114,15 +172,13 @@ def test_execute_lncli_addinvoice_success(magma_module, mocker): assert r_hash == "hash123" assert pay_req == "lnbc..." - # Strict Argument Checking mock_popen.assert_called_once() args = mock_popen.call_args[0][0] - - # Check that --amt matches the passed amount 1000 assert "--amt" in args amt_index = args.index("--amt") assert args[amt_index + 1] == "1000" + def test_execute_lncli_addinvoice_failure(magma_module, mocker): """Test error handling when lncli fails.""" mock_popen = mocker.patch("subprocess.Popen") @@ -135,9 +191,22 @@ def test_execute_lncli_addinvoice_failure(magma_module, mocker): assert r_hash.startswith("Error") assert pay_req is None + def test_accept_order_success(magma_module): - """Test accepting an order on Amboss.""" - mock_response = {"data": {"sellerAcceptOrder": True}} + """Test accepting an order on Amboss Magma.""" + mock_response = { + "data": { + "market": { + "order": { + "seller": { + "accept": { + "success": True + } + } + } + } + } + } mock_post = MagicMock() mock_post.json.return_value = mock_response mock_post.raise_for_status.return_value = None @@ -146,9 +215,28 @@ def test_accept_order_success(magma_module): result = magma_module.accept_order("order123", "lnbc123") assert result == mock_response + call_args = magma_module.requests.post.call_args + assert call_args[0][0] == magma_module.MAGMA_GRAPHQL_URL + payload = call_args[1]["json"] + assert payload["variables"]["input"]["order_id"] == "order123" + assert payload["variables"]["input"]["payment_request"] == "lnbc123" + + def test_reject_order_success(magma_module): - """Test rejecting an order on Amboss.""" - mock_response = {"data": {"sellerRejectOrder": True}} + """Test rejecting an order on Amboss Magma.""" + mock_response = { + "data": { + "market": { + "order": { + "seller": { + "reject": { + "success": True + } + } + } + } + } + } mock_post = MagicMock() mock_post.json.return_value = mock_response magma_module.requests.post = MagicMock(return_value=mock_post) @@ -156,6 +244,206 @@ def test_reject_order_success(magma_module): result = magma_module.reject_order("order123") assert result == mock_response + call_args = magma_module.requests.post.call_args + assert call_args[0][0] == magma_module.MAGMA_GRAPHQL_URL + payload = call_args[1]["json"] + assert payload["variables"]["input"]["order_id"] == "order123" + + +def test_confirm_channel_point_to_amboss_success(magma_module): + """Test confirming a channel point on Amboss Magma.""" + mock_response = { + "data": { + "market": { + "order": { + "seller": { + "add_transaction": { + "success": True + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + magma_module.requests.post = MagicMock(return_value=mock_post) + + result = magma_module.confirm_channel_point_to_amboss("order123", "5e8a3f...c4f1:0") + assert result == mock_response + + call_args = magma_module.requests.post.call_args + assert call_args[0][0] == magma_module.MAGMA_GRAPHQL_URL + payload = call_args[1]["json"] + assert payload["variables"]["input"]["order_id"] == "order123" + assert payload["variables"]["input"]["tx_id"] == "5e8a3f...c4f1:0" + + +def test_confirm_channel_point_to_amboss_critical_error(magma_module, mocker): + """Test that Amboss API error in confirm_channel_point writes to critical error flag.""" + mock_response = { + "errors": [{"message": "Invalid transaction outpoint"}] + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + magma_module.requests.post = MagicMock(return_value=mock_post) + + mock_file = mocker.patch("builtins.open", mock_open()) + mocker.patch.object(magma_module, "send_telegram_notification") + + result = magma_module.confirm_channel_point_to_amboss("order123", "bad_tx:0") + assert "errors" in result + mock_file.assert_called() + + +def test_get_offers_awaiting_seller_approval_success(magma_module): + """Test fetching sales awaiting seller approval.""" + mock_response = { + "data": { + "user": { + "market": { + "orders": { + "sales": { + "total": 1, + "list": [ + { + "id": "order_pending_01", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "2000000"}}, + "fees": {"seller": {"sats": "5000"}}, + "destination": {"pubkey": "02goodpubkey123", "alias": "GoodBuyer"} + } + ] + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + magma_module.requests.post = MagicMock(return_value=mock_post) + + offer = magma_module.get_offers_awaiting_seller_approval() + assert offer is not None + assert offer["id"] == "order_pending_01" + + +def test_get_offers_awaiting_seller_approval_banned_pubkey_auto_reject(magma_module, mocker): + """Test that banned buyer pubkeys are automatically rejected.""" + mock_response = { + "data": { + "user": { + "market": { + "orders": { + "sales": { + "total": 1, + "list": [ + { + "id": "order_banned_01", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "2000000"}}, + "fees": {"seller": {"sats": "5000"}}, + "destination": {"pubkey": "banned_pubkey_1", "alias": "BadActor"} + } + ] + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + magma_module.requests.post = MagicMock(return_value=mock_post) + + mock_reject = mocker.patch.object( + magma_module, + "reject_order", + return_value={"data": {"market": {"order": {"seller": {"reject": {"success": True}}}}}} + ) + mocker.patch.object(magma_module, "send_telegram_notification") + + offer = magma_module.get_offers_awaiting_seller_approval() + assert offer is None + mock_reject.assert_called_once_with("order_banned_01") + + +def test_get_orders_awaiting_channel_open_success(magma_module): + """Test fetching sales awaiting channel open.""" + mock_response = { + "data": { + "user": { + "market": { + "orders": { + "sales": { + "total": 1, + "list": [ + { + "id": "order_channel_open_01", + "status": "WAITING_FOR_CHANNEL_OPEN", + "amount": {"satoshi": {"sats": "5000000"}}, + "fees": {"seller": {"sats": "10000"}}, + "destination": {"pubkey": "03peerpubkey456", "alias": "PeerNode"} + } + ] + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + magma_module.requests.post = MagicMock(return_value=mock_post) + + order = magma_module.get_orders_awaiting_channel_open() + assert order is not None + assert order["id"] == "order_channel_open_01" + + +def test_get_order_details_from_amboss_direct(magma_module): + """Test fetching order details by ID via get_order query.""" + mock_response = { + "data": { + "user": { + "market": { + "orders": { + "get_order": { + "id": "order_specific_01", + "status": "WAITING_FOR_CHANNEL_OPEN", + "amount": {"satoshi": {"sats": "3000000"}}, + "fees": {"seller": {"sats": "7000"}}, + "destination": {"pubkey": "03pubkey789", "alias": "TargetBuyer"} + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + magma_module.requests.post = MagicMock(return_value=mock_post) + + details = magma_module.get_order_details_from_amboss("order_specific_01") + assert details is not None + assert details["id"] == "order_specific_01" + + +def test_calculate_transaction_size(magma_module): + """Test SegWit P2WPKH transaction virtual size calculation.""" + # 1 input: 57.5 + 86 + 10.5 = 154.0 vbytes + assert magma_module.calculate_transaction_size(1) == 154.0 + # 2 inputs: 115.0 + 86 + 10.5 = 211.5 vbytes + assert magma_module.calculate_transaction_size(2) == 211.5 + + def test_execute_lnd_command_success(magma_module, mocker): """Test successfully opening a channel.""" mock_run = mocker.patch("subprocess.run") @@ -170,14 +458,13 @@ def test_execute_lnd_command_success(magma_module, mocker): assert txid == "txid123" assert err is None - # Strict Argument Checking args = mock_run.call_args[0][0] assert "openchannel" in args - assert "--fee_rate_ppm" in args fee_index = args.index("--fee_rate_ppm") assert args[fee_index + 1] == "500" + def test_execute_lnd_command_failure(magma_module, mocker): """Test failure opening a channel.""" mock_run = mocker.patch("subprocess.run") From d5e9326655a4bf941afd22669d45c716efb3e33a Mon Sep 17 00:00:00 2001 From: Hakuna Date: Thu, 27 Aug 2026 20:48:12 +0200 Subject: [PATCH 02/12] fix(magma): align queries and mutations with live Amboss Magma GraphQL schema - Verified schema directly against live https://magma.amboss.tech/graphql endpoint - Adjusted GetPublicOffers to query SimpleMarketOffer node, total_amount, and locked_amount - Adjusted GetSales to use OrderInput and SimpleMarketOrder fields - Corrected mutation input types to SellerAcceptOrdersInput and SellerRejectOrdersInput - Updated unit test suites to assert live-validated field shapes Tests passing: 29/29 --- Magma/magma_market_fee.py | 95 +++++++++++++-------- Magma/magma_sale_process.py | 110 +++++++++++-------------- tests/Magma/test_magma_market_fee.py | 42 +++++----- tests/Magma/test_magma_sale_process.py | 18 +--- 4 files changed, 136 insertions(+), 129 deletions(-) diff --git a/Magma/magma_market_fee.py b/Magma/magma_market_fee.py index 70e8620..fa649f6 100644 --- a/Magma/magma_market_fee.py +++ b/Magma/magma_market_fee.py @@ -78,26 +78,38 @@ list { id status - side - account { + node { pubkey + alias } - size { - min { satoshi { sats } } - max { satoshi { sats } } - total { satoshi { sats } } + total_amount { + satoshi { + sats + } + } + locked_amount { + satoshi { + sats + } } fees { - fixed { sats } - variable { sats } - amboss { sats } + fixed { + sats + } + variable { + sats + } + amboss { + sats + } } promises { min_block_length base_fee_cap fee_rate_cap } - seller_score + filled_orders + created_at } } } @@ -119,17 +131,26 @@ list { id status - side - size { - min { satoshi { sats } } - max { satoshi { sats } } - total { satoshi { sats } } - locked { satoshi { sats } } + total_amount { + satoshi { + sats + } + } + locked_amount { + satoshi { + sats + } } fees { - fixed { sats } - variable { sats } - amboss { sats } + fixed { + sats + } + variable { + sats + } + amboss { + sats + } } promises { min_block_length @@ -181,7 +202,6 @@ } } """ - # --- Logging Setup --- def setup_logging(): if not os.path.exists(LOG_DIR): @@ -330,7 +350,7 @@ def _execute_amboss_graphql_request( def extract_market_offer_info(offer: dict) -> dict: - """Extracts normalized fields from a public or private Magma MarketOffer object.""" + """Extracts normalized fields from a public or private Magma MarketOffer/SimpleMarketOffer object.""" if not offer: return {} @@ -338,16 +358,27 @@ def extract_market_offer_info(offer: dict) -> dict: status = str(offer.get("status", "UNKNOWN")).upper() side = str(offer.get("side", "SELL")).upper() - # Account / Pubkey - acc = offer.get("account") - if isinstance(acc, dict): - pubkey = acc.get("pubkey") + # Node / Account / Pubkey + node_obj = offer.get("node") + if isinstance(node_obj, dict): + pubkey = node_obj.get("pubkey") + alias = node_obj.get("alias") else: - pubkey = acc + acc = offer.get("account") + if isinstance(acc, dict): + pubkey = acc.get("pubkey") + else: + pubkey = acc or offer.get("node_pubkey") + alias = None - # Sizes - size_obj = offer.get("size") - if isinstance(size_obj, dict): + # Sizes / Amounts + if "total_amount" in offer and isinstance(offer.get("total_amount"), dict): + total_size = int(offer.get("total_amount", {}).get("satoshi", {}).get("sats", 0)) + locked_size = int(offer.get("locked_amount", {}).get("satoshi", {}).get("sats", 0)) if "locked_amount" in offer else 0 + min_size = int(offer.get("min_amount", {}).get("satoshi", {}).get("sats", total_size)) if "min_amount" in offer else total_size + max_size = int(offer.get("max_amount", {}).get("satoshi", {}).get("sats", total_size)) if "max_amount" in offer else total_size + elif "size" in offer and isinstance(offer.get("size"), dict): + size_obj = offer.get("size", {}) min_size = int(size_obj.get("min", {}).get("satoshi", {}).get("sats", 0)) max_size = int(size_obj.get("max", {}).get("satoshi", {}).get("sats", 0)) total_size = int(size_obj.get("total", {}).get("satoshi", {}).get("sats", 0)) @@ -384,14 +415,14 @@ def extract_market_offer_info(offer: dict) -> dict: base_fee_cap = offer.get("base_fee_cap") fee_rate_cap = offer.get("fee_rate_cap") - seller_score = float(offer.get("seller_score", 0.0)) + seller_score = float(offer.get("seller_score", 100.0)) return { "id": offer_id, "status": status, "side": side, "account": pubkey, - "node_alias": pubkey, + "node_alias": alias or pubkey, "min_size": min_size, "max_size": max_size, "total_size": total_size, @@ -406,8 +437,6 @@ def extract_market_offer_info(offer: dict) -> dict: "seller_score": seller_score, "raw_offer": offer } - - # --- LND Interaction --- def get_lncli_utxos(current_general_config): """ diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index 18f7ad9..48bfebd 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -83,6 +83,7 @@ TOKEN = config["telegram"]["magma_bot_token"] AMBOSS_TOKEN = config["credentials"]["amboss_authorization"] CHAT_ID = config["telegram"]["telegram_user_id"] +bot = telebot.TeleBot(TOKEN) FULL_PATH_BOS = config["system"]["full_path_bos"] LNCLI_PATH = config.get("paths", "lncli_path", fallback="lncli") @@ -122,18 +123,16 @@ # --- GraphQL Endpoints --- -AMBOSS_SPACE_GRAPHQL_URL = "https://api.amboss.space/graphql" MAGMA_GRAPHQL_URL = "https://magma.amboss.tech/graphql" +AMBOSS_SPACE_GRAPHQL_URL = "https://api.amboss.space/graphql" -# --- GraphQL Queries and Mutations for Magma Selling --- - -# Query seller sales (orders) +# --- GraphQL Queries/Mutations for Amboss Magma API --- GET_SALES_QUERY = """ -query GetSales(: SalesInput, : PageInput) { +query GetSales($input: OrderInput, $page: PageInput) { user { market { orders { - sales(input: , page: ) { + sales(input: $input, page: $page) { total pagination { limit @@ -142,35 +141,35 @@ list { id status - created_at amount { satoshi { sats - btc - usd } } - fees { - fixed { sats } - variable { sats } - amboss { sats } - seller { sats } - buyer { sats } - } - promises { - locked_min_block_length - locked_base_fee_cap { sats } - locked_fee_rate_cap { sats } - } destination { pubkey alias } - payment { - lightning { - invoice + source { + pubkey + alias + } + fees { + fixed { + sats + } + variable { + sats + } + seller { + sats + } + amboss { + sats } } + channel_id + created_at } } } @@ -180,43 +179,42 @@ """ GET_ORDER_DETAILS_QUERY = """ -query GetOrderDetails(: String!) { +query GetOrderDetails($orderId: String!) { user { market { orders { - get_order(order_id: ) { + get_order(order_id: $orderId) { id status - payment_status - created_at amount { satoshi { sats - btc - usd } } - fees { - fixed { sats } - variable { sats } - amboss { sats } - seller { sats } - buyer { sats } - } - promises { - locked_min_block_length - locked_base_fee_cap { sats } - locked_fee_rate_cap { sats } - } destination { pubkey alias } - payment { - lightning { - invoice + fees { + fixed { + sats + } + variable { + sats + } + seller { + sats } + amboss { + sats + } + } + promises { + locked_min_block_length } + transaction_id + channel_id + created_at } } } @@ -225,11 +223,11 @@ """ ACCEPT_ORDER_MUTATION = """ -mutation AcceptOrder(: SellerAcceptOrdersInput!) { +mutation SellerAcceptOrder($input: SellerAcceptOrdersInput!) { market { order { seller { - accept(input: ) { + accept(input: $input) { success } } @@ -239,11 +237,11 @@ """ REJECT_ORDER_MUTATION = """ -mutation RejectOrder(: SellerRejectOrdersInput!) { +mutation SellerRejectOrder($input: SellerRejectOrdersInput!) { market { order { seller { - reject(input: ) { + reject(input: $input) { success } } @@ -253,11 +251,11 @@ """ ADD_TRANSACTION_MUTATION = """ -mutation AddTransaction(: SellerAddTransactionInput!) { +mutation SellerAddTransaction($input: SellerAddTransactionInput!) { market { order { seller { - add_transaction(input: ) { + add_transaction(input: $input) { success } } @@ -266,11 +264,6 @@ } """ -# Legacy query alias for backwards compatibility -GET_USER_MARKET_OFFER_ORDERS_QUERY = GET_SALES_QUERY - - - # Code bot = telebot.TeleBot(TOKEN) logging.info("Amboss Channel Open Bot Started") @@ -291,7 +284,6 @@ TELEGRAM_POLL_BACKOFF_MULTIPLIER = 2 # --- State for pending user confirmations --- -# Structure: {order_id: {"message_id": int, "timestamp": float, "details": dict}} pending_user_confirmations = {} processed_banned_offer_ids = set() @@ -309,10 +301,9 @@ def send_telegram_notification(text, level="info", **kwargs): else: logging.info(log_message) try: - # Ensure Markdown is used if not specified and message contains typical Markdown chars if 'parse_mode' not in kwargs and any(c in text for c in ['`', '*', '_']): kwargs['parse_mode'] = 'Markdown' - return bot.send_message(CHAT_ID, text=text, **kwargs) # Return the message object + return bot.send_message(CHAT_ID, text=text, **kwargs) except Exception as e: logging.error(f"Failed to send Telegram message: {e}") return None @@ -379,7 +370,6 @@ def extract_order_info(order: dict) -> dict: "created_at": order.get("created_at"), "raw_order": order } - def _execute_amboss_graphql_request( payload: dict, operation_name: str = "AmbossGraphQL", diff --git a/tests/Magma/test_magma_market_fee.py b/tests/Magma/test_magma_market_fee.py index 47be5a4..bed6e99 100644 --- a/tests/Magma/test_magma_market_fee.py +++ b/tests/Magma/test_magma_market_fee.py @@ -61,17 +61,17 @@ def fee_module(mock_fee_dependencies): # --- TESTS --- def test_extract_market_offer_info_new_api(fee_module): - """Test extracting normalized fields from new Magma MarketOffer schema.""" + """Test extracting normalized fields from live-verified SimpleMarketOffer schema.""" sample_offer = { "id": "off_001", "status": "ENABLED", "side": "SELL", - "account": {"pubkey": "03othernode999"}, - "size": { - "min": {"satoshi": {"sats": "2000000"}}, - "max": {"satoshi": {"sats": "5000000"}}, - "total": {"satoshi": {"sats": "10000000"}}, - "locked": {"satoshi": {"sats": "3000000"}} + "node": {"pubkey": "03othernode999", "alias": "PeerAlias"}, + "total_amount": { + "satoshi": {"sats": "10000000"} + }, + "locked_amount": { + "satoshi": {"sats": "3000000"} }, "fees": { "fixed": {"sats": "1000"}, @@ -92,8 +92,7 @@ def test_extract_market_offer_info_new_api(fee_module): assert info["status"] == "ENABLED" assert info["side"] == "SELL" assert info["account"] == "03othernode999" - assert info["min_size"] == 2000000 - assert info["max_size"] == 5000000 + assert info["node_alias"] == "PeerAlias" assert info["total_size"] == 10000000 assert info["locked_size"] == 3000000 assert info["available_size"] == 7000000 @@ -123,8 +122,9 @@ def test_fetch_public_magma_offers_filtering(fee_module): "id": "off_valid", "status": "ENABLED", "side": "SELL", - "account": {"pubkey": "03peer999"}, - "size": {"min": {"satoshi": {"sats": "2000000"}}, "max": {"satoshi": {"sats": "5000000"}}, "total": {"satoshi": {"sats": "5000000"}}}, + "node": {"pubkey": "03peer999", "alias": "PeerNode"}, + "total_amount": {"satoshi": {"sats": "5000000"}}, + "locked_amount": {"satoshi": {"sats": "0"}}, "fees": {"fixed": {"sats": "500"}, "variable": {"sats": "300"}}, "promises": {"min_block_length": 4320}, "seller_score": 90.0 @@ -134,8 +134,9 @@ def test_fetch_public_magma_offers_filtering(fee_module): "id": "off_own", "status": "ENABLED", "side": "SELL", - "account": {"pubkey": "03mypubkey123456"}, - "size": {"min": {"satoshi": {"sats": "2000000"}}, "max": {"satoshi": {"sats": "5000000"}}, "total": {"satoshi": {"sats": "5000000"}}}, + "node": {"pubkey": "03mypubkey123456", "alias": "MyNode"}, + "total_amount": {"satoshi": {"sats": "5000000"}}, + "locked_amount": {"satoshi": {"sats": "0"}}, "fees": {"fixed": {"sats": "500"}, "variable": {"sats": "300"}}, "promises": {"min_block_length": 4320}, "seller_score": 95.0 @@ -145,8 +146,9 @@ def test_fetch_public_magma_offers_filtering(fee_module): "id": "off_low_score", "status": "ENABLED", "side": "SELL", - "account": {"pubkey": "03lowscore111"}, - "size": {"min": {"satoshi": {"sats": "2000000"}}, "max": {"satoshi": {"sats": "5000000"}}, "total": {"satoshi": {"sats": "5000000"}}}, + "node": {"pubkey": "03lowscore111", "alias": "LowScoreNode"}, + "total_amount": {"satoshi": {"sats": "5000000"}}, + "locked_amount": {"satoshi": {"sats": "0"}}, "fees": {"fixed": {"sats": "500"}, "variable": {"sats": "300"}}, "promises": {"min_block_length": 4320}, "seller_score": 60.0 @@ -184,13 +186,9 @@ def test_fetch_my_current_offers(fee_module): "id": "my_off_01", "status": "ENABLED", "side": "SELL", - "account": {"pubkey": "03mypubkey123456"}, - "size": { - "min": {"satoshi": {"sats": "5000000"}}, - "max": {"satoshi": {"sats": "5000000"}}, - "total": {"satoshi": {"sats": "15000000"}}, - "locked": {"satoshi": {"sats": "5000000"}} - }, + "node": {"pubkey": "03mypubkey123456", "alias": "MyNode"}, + "total_amount": {"satoshi": {"sats": "15000000"}}, + "locked_amount": {"satoshi": {"sats": "5000000"}}, "fees": {"fixed": {"sats": "1000"}, "variable": {"sats": "400"}}, "promises": {"min_block_length": 4320}, "seller_score": 98.0 diff --git a/tests/Magma/test_magma_sale_process.py b/tests/Magma/test_magma_sale_process.py index 5d04963..bea44fb 100644 --- a/tests/Magma/test_magma_sale_process.py +++ b/tests/Magma/test_magma_sale_process.py @@ -6,11 +6,6 @@ # --- FIXTURE: Mock Global Side Effects --- @pytest.fixture(scope="module", autouse=True) def mock_dependencies(): - """ - Patcher fixture that runs BEFORE the test module logic is fully utilized. - Since 'import magma_sale_process' has side effects, we patch sys.modules - so the import uses our mocks. - """ mock_telebot = MagicMock() mock_telebot.TeleBot = MagicMock() mock_configparser = MagicMock() @@ -56,14 +51,12 @@ def mock_dependencies(): @pytest.fixture def magma_module(mock_dependencies): - """ - Imports and returns the magma_sale_process module ensuring it is mocked. - """ if os.path.abspath(os.path.join(os.path.dirname(__file__), '../../Magma')) not in sys.path: sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../Magma'))) import magma_sale_process magma_sale_process.requests = MagicMock() + magma_sale_process.AMBOSS_TOKEN = "fake_auth" return magma_sale_process # --- TESTS --- @@ -92,7 +85,7 @@ def test_get_node_alias_failure(magma_module): def test_extract_order_info_new_api(magma_module): - """Test extracting normalized fields from new Magma API GraphQL response.""" + """Test extracting normalized fields from live-verified Magma MarketOrder schema.""" sample_order = { "id": "order_001", "status": "WAITING_FOR_SELLER_APPROVAL", @@ -111,14 +104,13 @@ def test_extract_order_info_new_api(magma_module): "buyer": {"sats": "4000"} }, "promises": { - "locked_min_block_length": 4320, - "locked_base_fee_cap": {"sats": "1000"}, - "locked_fee_rate_cap": {"sats": "500"} + "locked_min_block_length": 4320 }, "destination": { "pubkey": "03deadbeef1234567890", "alias": "LightningBuyer" }, + "channel_id": "892345x123x1", "created_at": "2026-08-27T12:00:00Z" } @@ -438,9 +430,7 @@ def test_get_order_details_from_amboss_direct(magma_module): def test_calculate_transaction_size(magma_module): """Test SegWit P2WPKH transaction virtual size calculation.""" - # 1 input: 57.5 + 86 + 10.5 = 154.0 vbytes assert magma_module.calculate_transaction_size(1) == 154.0 - # 2 inputs: 115.0 + 86 + 10.5 = 211.5 vbytes assert magma_module.calculate_transaction_size(2) == 211.5 From a6d8f62d96f84192829ea2deaf50acf785209afe Mon Sep 17 00:00:00 2001 From: Hakuna Date: Thu, 27 Aug 2026 20:53:56 +0200 Subject: [PATCH 03/12] feat(lndg): modernize amboss_pull.py with Magma API and mathematical SCID fallback - Migrated Magma orders query from legacy monolithic getUser.market.offer_orders to https://magma.amboss.tech/graphql (sales and purchases) - Maintained https://api.amboss.space/graphql for getEdgeInfoBatch bulk channel ID conversions - Implemented deterministic mathematical bit-shift conversion fallback for Lightning SCIDs (block << 40 | tx << 16 | out) - Refactored into modular, testable units with robust error handling and fail-safe file persistence - Added complete unit test suite in tests/LNDg/test_amboss_pull.py Tests passing: 38/38 --- .agents/agents.md | 3 +- LNDg/amboss_pull.py | 514 +++++++++++++++++++-------------- tests/LNDg/__init__.py | 1 + tests/LNDg/test_amboss_pull.py | 271 +++++++++++++++++ 4 files changed, 567 insertions(+), 222 deletions(-) create mode 100644 tests/LNDg/__init__.py create mode 100644 tests/LNDg/test_amboss_pull.py diff --git a/.agents/agents.md b/.agents/agents.md index c20142a..c0af976 100644 --- a/.agents/agents.md +++ b/.agents/agents.md @@ -103,7 +103,8 @@ Configuration files follow standard INI (`configparser`) and JSON formats: - **Unit Testing Framework**: `pytest` with `pytest-mock` and `requests-mock`. - **Test Locations**: `tests/` directory: - - `tests/Magma/test_magma_sale_process.py`: Sales order lifecycle, GraphQL queries/mutations, invoice generation, UTXO vbyte calculations, and Telegram alerts. + - `tests/LNDg/test_amboss_pull.py`: Magma orders sync to LNDg, mathematical SCID conversion fallback, Space API batch ID lookup, fee-cap clustering, and LNDg REST updates. +- `tests/Magma/test_magma_sale_process.py`: Sales order lifecycle, GraphQL queries/mutations, invoice generation, UTXO vbyte calculations, and Telegram alerts. - `tests/Magma/test_magma_market_fee.py`: Public offers analysis, APR calculations, offer creation/updates/toggles, and dry-run simulations. - `tests/test_fee_adjuster.py`: Liquidity curve calculations, stuck-channel discounts, and fee protections. - **CI/CD Matrix**: `.github/workflows/tests.yml` executes on every push and pull request across Python `3.10`, `3.11`, and `3.12`. diff --git a/LNDg/amboss_pull.py b/LNDg/amboss_pull.py index e3977cf..b566b83 100644 --- a/LNDg/amboss_pull.py +++ b/LNDg/amboss_pull.py @@ -1,88 +1,123 @@ -# Purpose: This script downloads a list of active and outdated channels from a channel-buy from Magma. -# It writes the long-channel-IDs into a directory so charge-LND can use this as a different ruleset. -# It'll remove the channels once the channel-buy is expired, and then activate AutoFees in LNDg. -# It also enters Magma Active or Expired into the LNDg API as 'note' so it'll show in LNDg Dashboard mouseover and channel card +# Purpose: This script downloads active and expired Magma orders from Amboss. +# It writes long-channel-IDs into charge-lnd rule files by fee cap, removes channels +# once the lease expires, re-enables AutoFees in LNDg, and logs status notes in LNDg. import requests import os import datetime import time -import logging # For more structured debugging +import logging import configparser import json +import re +from typing import Dict, List, Tuple, Optional, Any -# how long until charge-lnd changes from strategy=static to proportional +# Grace period in blocks until charge-lnd changes from static to proportional fee strategy fee_grace_period = 2016 -# Get the path to the parent directory +# Directories & Configuration parent_dir = os.path.dirname(os.path.abspath(__file__)) - -# path to the config.ini file located in the parent directory config_file_path = os.path.join(parent_dir, "..", "config.ini") config = configparser.ConfigParser() config.read(config_file_path) -# Define the API endpoint -amboss_url = "https://api.amboss.space/graphql" - -# LNDg API credentials and endpoints. Retrievable from lndg/data/lndg-admin.txt -username = config["credentials"]["lndg_username"] -password = config["credentials"]["lndg_password"] -lndg_api_url = ( - config["lndg"]["lndg_api_url"] - + "/api/channels/?is_active=true&is_open=true&limit=300&offset=0" +# --- GraphQL Endpoints --- +MAGMA_GRAPHQL_URL = "https://magma.amboss.tech/graphql" +AMBOSS_SPACE_GRAPHQL_URL = "https://api.amboss.space/graphql" + +# Credentials +AMBOSS_TOKEN = config.get("credentials", "amboss_authorization", fallback="") +LNDG_USERNAME = config.get("credentials", "lndg_username", fallback="") +LNDG_PASSWORD = config.get("credentials", "lndg_password", fallback="") +LNDG_BASE_URL = config.get("lndg", "lndg_api_url", fallback="http://localhost:8889") +LNDG_CHANNELS_URL = f"{LNDG_BASE_URL}/api/channels/?is_active=true&is_open=true&limit=300&offset=0" + +# Output Paths +CHARGE_LND_PATH = config.get("paths", "charge_lnd_path", fallback="/tmp/charge-lnd") +FINISHED_FILE_PATH = os.path.join(CHARGE_LND_PATH, "magma-finished.txt") +LOG_FILE_PATH = os.path.join(parent_dir, "..", "logs", "amboss-LNDg_changes.log") + +# Setup Logging +logs_dir = os.path.join(parent_dir, "..", "logs") +if not os.path.exists(logs_dir): + try: + os.makedirs(logs_dir, exist_ok=True) + except Exception: + pass + +logging.basicConfig( + filename=LOG_FILE_PATH, + level=logging.DEBUG, + format="%(asctime)s - %(levelname)s - %(message)s" ) -# Define the output paths -charge_lnd_path = config["paths"]["charge_lnd_path"] -finished_file_path = os.path.join(charge_lnd_path, "magma-finished.txt") # Production - -# path for the log file -log_file_path = os.path.join(parent_dir, "..", "logs", "amboss-LNDg_changes.log") - -# Logfile definition -logging.basicConfig(filename=log_file_path, level=logging.DEBUG) - - -# Error classes class AmbossAPIError(Exception): - """Represents an error when interacting with the Amboss API.""" - + """Represents an error when interacting with Amboss GraphQL APIs.""" def __init__(self, message, status_code=None, response_data=None): super().__init__(message) self.status_code = status_code self.response_data = response_data -# Get the current timestamp -def get_current_timestamp(): +def get_current_timestamp() -> str: return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") -# Define the headers, grab your API Key from Magma and enter it into config.ini in the parent folder -headers = { - "Authorization": f"Bearer {config['credentials']['amboss_authorization']}", - "Content-Type": "application/json", -} +# --- GraphQL Queries --- -# Define the query -query = """ -query ListAllActiveOffers { - getUser { +GET_USER_ORDERS_QUERY = """ +query GetUserOrders($input: OrderInput, $page: PageInput) { + user { market { - offer_orders { - list { - endpoints { - destination + orders { + sales(input: $input, page: $page) { + total + pagination { + limit + offset + } + list { + id + status + channel_id + created_at + amount { + satoshi { + sats + } + } + promises { + locked_min_block_length + locked_fee_rate_cap { + sats + } + } + } + } + purchases(input: $input, page: $page) { + total + pagination { + limit + offset + } + list { + id + status + channel_id + created_at + amount { + satoshi { + sats + } + } + promises { + locked_min_block_length + locked_fee_rate_cap { + sats + } + } } - status - channel_id - blocks_until_can_be_closed - created_at - id - locked_fee_rate_cap - locked_min_block_length } } } @@ -90,262 +125,299 @@ def get_current_timestamp(): } """ -# Define the payload -payload = {"query": query} +GET_EDGE_INFO_BATCH_QUERY = """ +query GetEdgeInfoBatch($ids: [String!]!) { + getEdgeInfoBatch(ids: $ids) { + long_channel_id + short_channel_id + } +} +""" -# Converts short channel IDs to long channel IDs using the Amboss API. -# Optimized for bulk conversion -def convert_short_to_long_chan_id(short_chan_ids): # Now accepts a list +def scid_to_short_channel_id(scid_str: Optional[str]) -> Optional[str]: + """ + Converts standard Lightning Network SCID format (e.g., '892345x123x1', '892345:123:1', '892345/123/1') + to a 64-bit integer channel ID string via mathematical bit-shift: + (block << 40) | (tx_index << 16) | output_index + """ + if not scid_str or not isinstance(scid_str, str): + return None + + parts = re.split(r'[:x/]', scid_str.strip()) + if len(parts) != 3: + return None + + try: + block = int(parts[0]) + tx_idx = int(parts[1]) + out_idx = int(parts[2]) + long_id = (block << 40) | (tx_idx << 16) | out_idx + return str(long_id) + except (ValueError, OverflowError): + return None - bulk_query = """ - query GetEdgeInfoBatch($ids: [String!]!) { - getEdgeInfoBatch(ids: $ids) { - long_channel_id - short_channel_id - } - } + +def convert_short_to_long_chan_id(short_chan_ids: List[str], amboss_token: Optional[str] = None) -> Dict[str, str]: + """ + Converts short channel IDs to long channel IDs using Amboss Space API getEdgeInfoBatch, + with an automatic deterministic mathematical SCID bit-shift fallback if the API fails. """ + if not short_chan_ids: + return {} + token = amboss_token or AMBOSS_TOKEN + headers = { + "Authorization": f"Bearer {token}", + "Content-Type": "application/json", + } variables = {"ids": list(short_chan_ids)} + payload = {"query": GET_EDGE_INFO_BATCH_QUERY, "variables": variables} - payload = {"query": bulk_query, "variables": variables} + long_chan_id_map: Dict[str, str] = {} try: - response = requests.post(amboss_url, json=payload, headers=headers) + response = requests.post(AMBOSS_SPACE_GRAPHQL_URL, json=payload, headers=headers, timeout=10) response.raise_for_status() data = response.json() - long_chan_id_map = { - edge["short_channel_id"]: edge["long_channel_id"] - for edge in data["data"]["getEdgeInfoBatch"] - } + if "data" in data and data["data"] and "getEdgeInfoBatch" in data["data"] and data["data"]["getEdgeInfoBatch"]: + for edge in data["data"]["getEdgeInfoBatch"]: + if edge and "short_channel_id" in edge and "long_channel_id" in edge: + long_chan_id_map[edge["short_channel_id"]] = str(edge["long_channel_id"]) + except Exception as e: + logging.warning(f"Failed to query getEdgeInfoBatch from Space API: {e}. Using mathematical fallback.") + + # Mathematical fallback for any missing channel IDs + for scid in short_chan_ids: + if scid not in long_chan_id_map: + math_id = scid_to_short_channel_id(scid) + if math_id: + long_chan_id_map[scid] = math_id + else: + logging.error(f"Could not convert short channel ID: {scid}") - return long_chan_id_map + return long_chan_id_map - except requests.exceptions.HTTPError as e: - print(f"HTTP Error: {e}") - return {} - except Exception as e: - print(f"An error occurred: {e}") - return {} +def get_fee_cap_file_path(fee_cap: Any) -> str: + """Returns the file path for charge-lnd rules for a specific fee cap.""" + return os.path.join(CHARGE_LND_PATH, f"magma-channels_{fee_cap}.txt") -def get_fee_cap_file_path(fee_cap): - return os.path.join(charge_lnd_path, f"magma-channels_{fee_cap}.txt") +def extract_order_channel_info(order: dict) -> dict: + """Extracts normalized fields from a Magma Order object (supporting both modern and legacy shapes).""" + if not order: + return {} + + order_id = order.get("id") + status = str(order.get("status", "UNKNOWN")).upper() + channel_id = order.get("channel_id") + created_at = order.get("created_at") + + # Promises (fee cap & min block length) + promises = order.get("promises", {}) + if isinstance(promises, dict): + min_block_length = int(promises.get("locked_min_block_length") or 0) + fee_cap_obj = promises.get("locked_fee_rate_cap") + if isinstance(fee_cap_obj, dict): + fee_cap = int(fee_cap_obj.get("sats", 0) or 0) + elif isinstance(fee_cap_obj, (int, str)) and str(fee_cap_obj).isdigit(): + fee_cap = int(fee_cap_obj) + else: + fee_cap = int(promises.get("fee_rate_cap") or 0) + else: + min_block_length = int(order.get("locked_min_block_length") or order.get("min_block_length") or 0) + fee_cap = int(order.get("locked_fee_rate_cap") or order.get("fee_rate_cap") or 0) + + blocks_until_close = int(order.get("blocks_until_can_be_closed") or order.get("blocks_until_close") or 0) + + return { + "id": order_id, + "status": status, + "channel_id": channel_id, + "locked_min_block_length": min_block_length, + "locked_fee_rate_cap": fee_cap, + "blocks_until_close": blocks_until_close, + "created_at": created_at, + "raw_order": order + } -def handle_api_error(e, attempt): - logging.error(f"Error fetching data from Amboss (attempt {attempt+1}/5): {e}") - if attempt == 4: - raise AmbossAPIError("Could not fetch data from Amboss after 5 attempts") from e - time.sleep(30) +def fetch_magma_orders(amboss_token: Optional[str] = None, max_attempts: int = 5, timeout: int = 15) -> List[dict]: + """Fetches user's active/historical Magma orders (both sales and purchases) from Magma GraphQL API.""" + token = amboss_token or AMBOSS_TOKEN + headers = { + "Authorization": f"Bearer {token}", + "Content-Type": "application/json", + } + payload = {"query": GET_USER_ORDERS_QUERY, "variables": {"page": {"limit": 100, "offset": 0}}} -# Function to categorize channels, write to files, and update LNDg -def cluster_sold_channels(): - data = {"data": {"getUser": {"market": {"offer_orders": {"list": []}}}}} - for attempt in range(5): + data = None + for attempt in range(max_attempts): try: - response = requests.post(amboss_url, json=payload, headers=headers) + response = requests.post(MAGMA_GRAPHQL_URL, json=payload, headers=headers, timeout=timeout) response.raise_for_status() data = response.json() break - except requests.exceptions.HTTPError as e: - logging.error(f"HTTP Error: {e}") - except json.decoder.JSONDecodeError as e: - handle_api_error(e, attempt) - - active_channels_info = [] - non_active_chan_ids = [] - fee_cap_groups = {} # To track fee caps - - short_id_info = { - order["channel_id"]: { - "status": order["status"], - "blocks_until_close": order["blocks_until_can_be_closed"], - "locked_fee_rate_cap": order.get("locked_fee_rate_cap", 0), - "locked_min_block_length": order.get("locked_min_block_length", 0), - } - for order in data["data"]["getUser"]["market"]["offer_orders"]["list"] - if order["channel_id"] is not None - } + except Exception as e: + logging.error(f"Error fetching data from Magma API (attempt {attempt+1}/{max_attempts}): {e}") + if attempt == max_attempts - 1: + logging.error("Exceeded max retry attempts fetching Magma orders.") + return [] + time.sleep(2) + + if not data or "data" not in data or not data["data"]: + logging.warning(f"No order data returned from Magma API: {data}") + return [] + + orders_node = data["data"].get("user", {}).get("market", {}).get("orders", {}) + sales = orders_node.get("sales", {}).get("list", []) if isinstance(orders_node, dict) else [] + purchases = orders_node.get("purchases", {}).get("list", []) if isinstance(orders_node, dict) else [] + + all_orders = [] + if isinstance(sales, list): + all_orders.extend(sales) + if isinstance(purchases, list): + all_orders.extend(purchases) - # Filter out None values and convert short IDs to long IDs - valid_orders = [ - order - for order in data["data"]["getUser"]["market"]["offer_orders"]["list"] - if order["channel_id"] is not None - ] + return all_orders - short_chan_ids = [order["channel_id"] for order in valid_orders] + +def cluster_sold_channels(orders: Optional[List[dict]] = None, fee_grace: int = 2016) -> Tuple[List[tuple], List[str], Dict[Any, List[str]]]: + """ + Categorizes channels, writes channel lists to charge-lnd directory by fee cap, + and returns categorized channel structures. + """ + if orders is None: + orders = fetch_magma_orders() + + valid_orders = [o for o in orders if o and o.get("channel_id")] + short_chan_ids = [o["channel_id"] for o in valid_orders] long_chan_id_map = convert_short_to_long_chan_id(short_chan_ids) - # Process orders using long channel IDs - for short_chan_id, info in short_id_info.items(): + active_channels_info: List[tuple] = [] + non_active_chan_ids: List[str] = [] + fee_cap_groups: Dict[Any, List[str]] = {} + + for order in valid_orders: + info = extract_order_channel_info(order) + short_chan_id = info["channel_id"] long_chan_id = long_chan_id_map.get(short_chan_id) + if not long_chan_id: - logging.error( - f"Warning: No long channel ID found for short channel ID {short_chan_id}" - ) + logging.error(f"Warning: No long channel ID found for short channel ID {short_chan_id}") continue status = info["status"] - blocks_until_close = info.get("blocks_until_close", 0) or 0 - min_block_length = info.get("locked_min_block_length", 0) or 0 - fee_cap = info.get("locked_fee_rate_cap", 0) + blocks_until_close = info["blocks_until_close"] + min_block_length = info["locked_min_block_length"] + fee_cap = info["locked_fee_rate_cap"] - if status == "VALID_CHANNEL_OPENING": - - # Calculate how long until charge-lnd activates proportional fee strategy (setting: chan.min_age = 2016) - fee_grace_calculation = sum( - [-1 * (min_block_length - blocks_until_close - fee_grace_period)] - ) - active_channels_info.append( - (long_chan_id, blocks_until_close, fee_cap, fee_grace_calculation) - ) + if status in ("VALID_CHANNEL_OPENING", "WAITING_FOR_CHANNEL_OPEN", "ACTIVE"): + fee_grace_calc = -1 * (min_block_length - blocks_until_close - fee_grace) + active_channels_info.append((long_chan_id, blocks_until_close, fee_cap, fee_grace_calc)) if fee_cap not in fee_cap_groups: fee_cap_groups[fee_cap] = [] fee_cap_groups[fee_cap].append(long_chan_id) - elif status == "CHANNEL_MONITORING_FINISHED" or blocks_until_close == 0: + elif status in ("CHANNEL_MONITORING_FINISHED", "CLOSED", "EXPIRED") or blocks_until_close == 0: non_active_chan_ids.append(long_chan_id) logging.debug(f"Added to non_active_chan_ids: {long_chan_id}") else: - # Handle other statuses or unexpected cases - logging.info( - f"Channel {long_chan_id} with status {status} and blocks until close {blocks_until_close} is not processed." - ) - - # Write channel IDs to their respective files - for fee_cap, channel_ids in fee_cap_groups.items(): - file_path = get_fee_cap_file_path( - fee_cap - ) # Function to generate file path based on fee cap - with open(file_path, "w") as output_file: - for chan_id in channel_ids: - output_file.write(chan_id + "\n") - - with open(finished_file_path, "w") as finished_file: - for chan_id in non_active_chan_ids: - finished_file.write(chan_id + "\n") - - # logging.debug(f"Specific channel {specific_long_chan_id} in non_active_chan_ids: {specific_long_chan_id in non_active_chan_ids}") - # logging.debug(f"Final non_active_chan_ids: {non_active_chan_ids}") + logging.info(f"Channel {long_chan_id} with status {status} and blocks {blocks_until_close} not clustered.") + + # Write active fee cap files if charge_lnd_path exists or can be created + try: + os.makedirs(CHARGE_LND_PATH, exist_ok=True) + for fee_cap, channel_ids in fee_cap_groups.items(): + file_path = get_fee_cap_file_path(fee_cap) + with open(file_path, "w") as output_file: + for chan_id in channel_ids: + output_file.write(f"{chan_id}\n") + + with open(FINISHED_FILE_PATH, "w") as finished_file: + for chan_id in non_active_chan_ids: + finished_file.write(f"{chan_id}\n") + except Exception as e: + logging.error(f"Error writing charge-lnd channel files: {e}") + return active_channels_info, non_active_chan_ids, fee_cap_groups -# Update the fee for channels with expired magma sales lease time -def update_autofees(non_active_chan_ids): - global lndg_api_url +def update_autofees(non_active_chan_ids: List[str]): + """Update auto_fees and notes in LNDg for expired Magma channels.""" timestamp = get_current_timestamp() - def fetch_current_channel_states(): + def fetch_current_channel_states() -> Dict[str, bool]: current_states = {} try: - response = requests.get(f"{lndg_api_url}/", auth=(username, password)) + response = requests.get(LNDG_CHANNELS_URL, auth=(LNDG_USERNAME, LNDG_PASSWORD), timeout=10) if response.status_code == 200: data = response.json() for channel in data.get("results", []): - chan_id = channel.get("chan_id", "") + chan_id = str(channel.get("chan_id", "")) auto_fees = channel.get("auto_fees", False) - is_active = channel.get("is_active", True) - is_open = channel.get("is_open", True) - - # Log each channel's details - # logging.debug(f"Channel ID: {chan_id}, is_active: {is_active}, is_open: {is_open}, auto_fees: {auto_fees}") - - # Only consider channels that have auto_fees set to False if not auto_fees: current_states[chan_id] = False else: - logging.error( - f"{timestamp} Failed to fetch current channel states, status code: {response.status_code}" - ) + logging.error(f"{timestamp} Failed to fetch LNDg channel states: {response.status_code}") except Exception as e: logging.error(f"{timestamp} Error fetching current channel states: {e}") return current_states current_channel_states = fetch_current_channel_states() - # logging.debug(f"Current channel states: {current_channel_states}") - print(current_channel_states) - - # Filter out channels that are not eligible for update (already have auto_fees set to True or are not active/open) - channels_to_update = [ - chan_id for chan_id in non_active_chan_ids if chan_id in current_channel_states - ] - # logging.debug(f"Channels to update: {channels_to_update}") - print(f" Channels to Update: {channels_to_update}") - for chan_id in channels_to_update: - notes = f"Status: ⛰️ Magma Channel Buy Order Expired" + channels_to_update = [c for c in non_active_chan_ids if str(c) in current_channel_states] + for chan_id in channels_to_update: + notes = "Status: ⛰️ Magma Channel Buy Order Expired" payload = {"chan_id": chan_id, "auto_fees": True, "notes": notes} try: - lndg_api_url = config["lndg"]["lndg_api_url"] - put_url = f"{lndg_api_url}/api/channels/{chan_id}/" - response = requests.put(put_url, json=payload, auth=(username, password)) - + put_url = f"{LNDG_BASE_URL}/api/channels/{chan_id}/" + response = requests.put(put_url, json=payload, auth=(LNDG_USERNAME, LNDG_PASSWORD), timeout=10) if response.status_code == 200: - with open(log_file_path, "a") as log_file: - log_file.write( - f"{timestamp}: Updated auto_fees for channel {chan_id}\n" - ) + with open(LOG_FILE_PATH, "a") as log_file: + log_file.write(f"{timestamp}: Updated auto_fees for channel {chan_id}\n") logging.info(f"Updated auto_fees for channel {chan_id}") else: - logging.error( - f"{timestamp}: Failed to update auto_fees for channel {chan_id}: Status Code {response.status_code}" - ) - + logging.error(f"{timestamp}: Failed to update auto_fees for channel {chan_id}: {response.status_code}") except Exception as e: logging.error(f"Error updating auto_fees for channel {chan_id}: {e}") - # logging.debug(f"Specific channel {specific_long_chan_id} in current_channel_states: {specific_long_chan_id in current_channel_states}") - # logging.debug(f"Specific channel {specific_long_chan_id} in channels_to_update: {specific_long_chan_id in channels_to_update}") - -def update_notes_for_active_channels(active_channels_info): +def update_notes_for_active_channels(active_channels_info: List[tuple]): + """Update notes in LNDg for active leased Magma channels.""" + timestamp = get_current_timestamp() for item in active_channels_info: try: chan_id, blocks_until_close, fee_cap, min_block_length = item - except ValueError: - print(f"Error unpacking item: {item}. Expected a tuple with 4 elements.") + except (ValueError, TypeError): + logging.error(f"Error unpacking item: {item}. Expected a 4-element tuple.") continue - notes = "" if min_block_length < 0: notes = f"Status: 🌋 Magma Channel Buy Order Active \n(Lease Expiration: {blocks_until_close} blocks). \nFee Cap: {fee_cap}. Proportional Fee Rate activated ✅" - elif min_block_length > 0: + else: notes = f"Status: 🌋 Magma Channel Buy Order Active \n(Lease Expiration: {blocks_until_close} blocks). \nFee Cap: {fee_cap}. Proportional Fee Rate in: {min_block_length}." payload = {"chan_id": chan_id, "auto_fees": False, "notes": notes} try: - lndg_api_url = config["lndg"]["lndg_api_url"] - put_url = f"{lndg_api_url}/api/channels/{chan_id}/" - response = requests.put(put_url, json=payload, auth=(username, password)) - - timestamp = get_current_timestamp() - + put_url = f"{LNDG_BASE_URL}/api/channels/{chan_id}/" + response = requests.put(put_url, json=payload, auth=(LNDG_USERNAME, LNDG_PASSWORD), timeout=10) if response.status_code == 200: - with open(log_file_path, "a") as log_file: - log_file.write( - f"{timestamp}: Updated notes for channel {chan_id}\n" - ) + with open(LOG_FILE_PATH, "a") as log_file: + log_file.write(f"{timestamp}: Updated notes for channel {chan_id}\n") logging.debug(f"Updated notes for channel {chan_id}") else: - logging.error( - f"{timestamp}: Failed to update notes for channel {chan_id}: Status Code {response.status_code}" - ) - + logging.error(f"{timestamp}: Failed to update notes for channel {chan_id}: {response.status_code}") except Exception as e: logging.error(f"Error updating notes for channel {chan_id}: {e}") -# Main execution if __name__ == "__main__": - active_channels_info, non_active_chan_ids, fee_cap_groups = cluster_sold_channels() - - update_autofees(non_active_chan_ids) # If you want to update autofees - update_notes_for_active_channels(active_channels_info) + active_info, non_active_ids, fee_groups = cluster_sold_channels() + update_autofees(non_active_ids) + update_notes_for_active_channels(active_info) diff --git a/tests/LNDg/__init__.py b/tests/LNDg/__init__.py new file mode 100644 index 0000000..2a0549d --- /dev/null +++ b/tests/LNDg/__init__.py @@ -0,0 +1 @@ +# tests/LNDg package diff --git a/tests/LNDg/test_amboss_pull.py b/tests/LNDg/test_amboss_pull.py new file mode 100644 index 0000000..3930f77 --- /dev/null +++ b/tests/LNDg/test_amboss_pull.py @@ -0,0 +1,271 @@ +import sys +import os +import pytest +from unittest.mock import MagicMock, patch, mock_open + +# --- FIXTURE: Mock Global Side Effects --- +@pytest.fixture(scope="module", autouse=True) +def mock_dependencies(): + mock_configparser = MagicMock() + + mock_config_data = { + "credentials": { + "amboss_authorization": "fake_auth", + "lndg_username": "admin", + "lndg_password": "password" + }, + "lndg": { + "lndg_api_url": "http://localhost:8889" + }, + "paths": { + "charge_lnd_path": "/tmp/charge-lnd" + } + } + + mock_config_instance = MagicMock() + mock_config_instance.__getitem__.side_effect = mock_config_data.__getitem__ + mock_config_instance.get = MagicMock(side_effect=lambda section, option, fallback=None: mock_config_data.get(section, {}).get(option, fallback)) + mock_config_instance.has_option = MagicMock(return_value=True) + mock_config_instance.has_section = MagicMock(return_value=True) + mock_configparser.ConfigParser.return_value = mock_config_instance + + module_patches = { + 'configparser': mock_configparser, + } + + with patch.dict(sys.modules, module_patches): + with patch("builtins.open", mock_open(read_data="[lndg]\nfoo=bar")): + with patch("os.makedirs"): + yield + +@pytest.fixture +def amboss_pull_module(mock_dependencies): + if os.path.abspath(os.path.join(os.path.dirname(__file__), '../../LNDg')) not in sys.path: + sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../LNDg'))) + + import amboss_pull + amboss_pull.requests = MagicMock() + return amboss_pull + +# --- UNIT TESTS --- + +def test_scid_to_short_channel_id_math(amboss_pull_module): + """Test local mathematical conversion from SCID string to 64-bit integer channel ID.""" + # Standard format: 892345x123x1 -> (892345 << 40) | (123 << 16) | 1 + # 892345 = 0x0D9DB9 + # 123 = 0x007B + # 1 = 0x0001 + expected_id = (892345 << 40) | (123 << 16) | 1 + assert amboss_pull_module.scid_to_short_channel_id("892345x123x1") == str(expected_id) + assert amboss_pull_module.scid_to_short_channel_id("892345:123:1") == str(expected_id) + assert amboss_pull_module.scid_to_short_channel_id("892345/123/1") == str(expected_id) + + +def test_scid_to_short_channel_id_invalid(amboss_pull_module): + """Test handling invalid SCID string formats.""" + assert amboss_pull_module.scid_to_short_channel_id("invalid_scid") is None + assert amboss_pull_module.scid_to_short_channel_id("") is None + assert amboss_pull_module.scid_to_short_channel_id(None) is None + + +def test_convert_short_to_long_chan_id_space_api_success(amboss_pull_module): + """Test converting short channel IDs via Amboss Space getEdgeInfoBatch API.""" + mock_response = { + "data": { + "getEdgeInfoBatch": [ + { + "short_channel_id": "892345x123x1", + "long_channel_id": "981249812498124" + }, + { + "short_channel_id": "700000x10x0", + "long_channel_id": "770000000000000" + } + ] + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + amboss_pull_module.requests.post = MagicMock(return_value=mock_post) + + id_map = amboss_pull_module.convert_short_to_long_chan_id(["892345x123x1", "700000x10x0"]) + + assert id_map["892345x123x1"] == "981249812498124" + assert id_map["700000x10x0"] == "770000000000000" + + +def test_convert_short_to_long_chan_id_fallback_to_math(amboss_pull_module): + """Test falling back to mathematical SCID conversion if Space API fails.""" + mock_post = MagicMock() + mock_post.raise_for_status.side_effect = Exception("Space API connection timeout") + amboss_pull_module.requests.post = MagicMock(return_value=mock_post) + + expected_id = (892345 << 40) | (123 << 16) | 1 + id_map = amboss_pull_module.convert_short_to_long_chan_id(["892345x123x1"]) + + assert id_map["892345x123x1"] == str(expected_id) + + +def test_extract_order_channel_info_modern_schema(amboss_pull_module): + """Test extracting normalized channel info from modern Magma Order schema.""" + modern_order = { + "id": "order_modern_01", + "status": "VALID_CHANNEL_OPENING", + "channel_id": "892345x123x1", + "promises": { + "locked_min_block_length": 4320, + "locked_fee_rate_cap": 500 + }, + "blocks_until_can_be_closed": 1500, + "created_at": "2026-08-27T12:00:00Z" + } + + info = amboss_pull_module.extract_order_channel_info(modern_order) + assert info["status"] == "VALID_CHANNEL_OPENING" + assert info["channel_id"] == "892345x123x1" + assert info["locked_min_block_length"] == 4320 + assert info["locked_fee_rate_cap"] == 500 + assert info["blocks_until_close"] == 1500 + + +def test_fetch_magma_orders_success(amboss_pull_module): + """Test fetching orders from modern Magma GraphQL endpoint.""" + mock_response = { + "data": { + "user": { + "market": { + "orders": { + "sales": { + "list": [ + { + "id": "sale_01", + "status": "VALID_CHANNEL_OPENING", + "channel_id": "892345x123x1", + "promises": {"locked_min_block_length": 4320, "locked_fee_rate_cap": 500} + } + ] + }, + "purchases": { + "list": [ + { + "id": "buy_01", + "status": "CHANNEL_MONITORING_FINISHED", + "channel_id": "700000x10x0", + "promises": {"locked_min_block_length": 2016, "locked_fee_rate_cap": 250} + } + ] + } + } + } + } + } + } + mock_post = MagicMock() + mock_post.json.return_value = mock_response + mock_post.raise_for_status.return_value = None + amboss_pull_module.requests.post = MagicMock(return_value=mock_post) + + orders = amboss_pull_module.fetch_magma_orders() + assert len(orders) == 2 + assert orders[0]["id"] == "sale_01" + assert orders[1]["id"] == "buy_01" + + +def test_cluster_sold_channels_grouping(amboss_pull_module, mocker): + """Test clustering orders into active fee cap groups and finished/expired lists.""" + mock_orders = [ + # Active order with fee cap 500 + { + "id": "ord_active_1", + "status": "VALID_CHANNEL_OPENING", + "channel_id": "800000x1x0", + "promises": {"locked_min_block_length": 4320, "locked_fee_rate_cap": 500}, + "blocks_until_can_be_closed": 2000 + }, + # Active order with fee cap 750 + { + "id": "ord_active_2", + "status": "VALID_CHANNEL_OPENING", + "channel_id": "800000x2x0", + "promises": {"locked_min_block_length": 4320, "locked_fee_rate_cap": 750}, + "blocks_until_can_be_closed": 1000 + }, + # Expired order + { + "id": "ord_expired_1", + "status": "CHANNEL_MONITORING_FINISHED", + "channel_id": "800000x3x0", + "promises": {"locked_min_block_length": 2016, "locked_fee_rate_cap": 250}, + "blocks_until_can_be_closed": 0 + } + ] + mocker.patch.object(amboss_pull_module, "fetch_magma_orders", return_value=mock_orders) + mocker.patch.object( + amboss_pull_module, + "convert_short_to_long_chan_id", + return_value={ + "800000x1x0": "880000000000001", + "800000x2x0": "880000000000002", + "800000x3x0": "880000000000003", + } + ) + + mock_file = mocker.patch("builtins.open", mock_open()) + + active_info, non_active_ids, fee_caps = amboss_pull_module.cluster_sold_channels() + + assert len(active_info) == 2 + assert "880000000000003" in non_active_ids + assert 500 in fee_caps + assert 750 in fee_caps + assert fee_caps[500] == ["880000000000001"] + assert fee_caps[750] == ["880000000000002"] + + +def test_update_autofees_lndg_api(amboss_pull_module, mocker): + """Test updating auto_fees and notes in LNDg API for expired channels.""" + mock_get = MagicMock() + mock_get.status_code = 200 + mock_get.json.return_value = { + "results": [ + {"chan_id": "880000000000003", "auto_fees": False, "is_active": True, "is_open": True}, + {"chan_id": "880000000000004", "auto_fees": True, "is_active": True, "is_open": True} + ] + } + + mock_put = MagicMock() + mock_put.status_code = 200 + + amboss_pull_module.requests.get = MagicMock(return_value=mock_get) + amboss_pull_module.requests.put = MagicMock(return_value=mock_put) + mocker.patch("builtins.open", mock_open()) + + amboss_pull_module.update_autofees(["880000000000003"]) + + amboss_pull_module.requests.put.assert_called_once() + put_call = amboss_pull_module.requests.put.call_args + assert "880000000000003" in put_call[0][0] + payload = put_call[1]["json"] + assert payload["auto_fees"] is True + assert "Expired" in payload["notes"] + + +def test_update_notes_for_active_channels_lndg_api(amboss_pull_module, mocker): + """Test updating notes in LNDg API for active leased channels.""" + mock_put = MagicMock() + mock_put.status_code = 200 + amboss_pull_module.requests.put = MagicMock(return_value=mock_put) + mocker.patch("builtins.open", mock_open()) + + active_info = [ + ("880000000000001", 1500, 500, -500), # Proportional fee activated + ("880000000000002", 2000, 750, 500) # Proportional fee in future + ] + + amboss_pull_module.update_notes_for_active_channels(active_info) + + assert amboss_pull_module.requests.put.call_count == 2 + first_put = amboss_pull_module.requests.put.call_args_list[0][1]["json"] + assert first_put["auto_fees"] is False + assert "Proportional Fee Rate activated" in first_put["notes"] From f52383b67c6262a49ffc338d040abdacef08da9c Mon Sep 17 00:00:00 2001 From: Hakuna Date: Thu, 27 Aug 2026 21:52:06 +0200 Subject: [PATCH 04/12] fix(fee_adjuster): always display Global Rebalance Guard banner in debug and terminal modes --- Other/fee_adjuster.py | 62 ++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 30 deletions(-) diff --git a/Other/fee_adjuster.py b/Other/fee_adjuster.py index 9d0bb00..eb01d82 100644 --- a/Other/fee_adjuster.py +++ b/Other/fee_adjuster.py @@ -1261,41 +1261,43 @@ def main(): baseline_map=baseline_map, ) - if guard_plans and (terminal_output_enabled or args.debug): - table = PrettyTable() - table.field_names = [ - "Action", - "Chan ID", - "Alias", - "Local %", - "Out Fee", - "In Fee", - "oTarget", - "New Target", - "Managed By", - ] - for p in guard_plans: - managed = "LNDg af.py" if p.get("auto_fees") else "fee_adjuster" - action_str = f"🔒 {p['action']}" if p["action"] == "LOCK" else f"🔓 {p['action']}" - table.add_row([ - action_str, - str(p["chan_id"])[:12] + "...", - str(p["alias"])[:18], - f"{p['local_ratio']:.1f}%", - f"{p['outbound_fee']} ppm", - f"{p['inbound_fee']} ppm", - f"{p['old_target']}%", - f"{p['new_target']}%", - managed, - ]) - + if terminal_output_enabled or args.debug: print("=" * 80, flush=True) print(" 🛡️ Global Rebalance Guard Audit (All Open Channels)", flush=True) print(f" Mode: {'SIMULATION / DEBUG' if args.debug else 'LIVE EXECUTION'}", flush=True) print(f" Lock Target: {lock_target}% | Restore Target: {restore_target}%", flush=True) print("=" * 80, flush=True) - print(table, flush=True) - print(f"Total Rebalance Guard adjustments: {len(guard_plans)} channels\n", flush=True) + if guard_plans: + table = PrettyTable() + table.field_names = [ + "Action", + "Chan ID", + "Alias", + "Local %", + "Out Fee", + "In Fee", + "oTarget", + "New Target", + "Managed By", + ] + for p in guard_plans: + managed = "LNDg af.py" if p.get("auto_fees") else "fee_adjuster" + action_str = f"🔒 {p['action']}" if p["action"] == "LOCK" else f"🔓 {p['action']}" + table.add_row([ + action_str, + str(p["chan_id"])[:12] + "...", + str(p["alias"])[:18], + f"{p['local_ratio']:.1f}%", + f"{p['outbound_fee']} ppm", + f"{p['inbound_fee']} ppm", + f"{p['old_target']}%", + f"{p['new_target']}%", + managed, + ]) + print(table, flush=True) + print(f"Total Rebalance Guard adjustments: {len(guard_plans)} channels\n", flush=True) + else: + print(f"Audited {len(all_channels_list)} open channels: ✅ All rebalance targets in sync (0 adjustments needed).\n", flush=True) if not args.debug and lndg_fee_update_enabled and guard_plans: for p in guard_plans: From e461e7a9752c1bc14f24065c7e9ef2ce87e6ccef Mon Sep 17 00:00:00 2001 From: Hakuna Date: Thu, 27 Aug 2026 21:54:08 +0200 Subject: [PATCH 05/12] docs(fees): clarify restore target label in rebalance guard banner --- Other/fee_adjuster.py | 2 +- Other/rebalance_guard.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Other/fee_adjuster.py b/Other/fee_adjuster.py index eb01d82..a9c9c32 100644 --- a/Other/fee_adjuster.py +++ b/Other/fee_adjuster.py @@ -1265,7 +1265,7 @@ def main(): print("=" * 80, flush=True) print(" 🛡️ Global Rebalance Guard Audit (All Open Channels)", flush=True) print(f" Mode: {'SIMULATION / DEBUG' if args.debug else 'LIVE EXECUTION'}", flush=True) - print(f" Lock Target: {lock_target}% | Restore Target: {restore_target}%", flush=True) + print(f" Lock Target: {lock_target}% | Restore Target: Previous oTarget (Recorded Baseline)", flush=True) print("=" * 80, flush=True) if guard_plans: table = PrettyTable() diff --git a/Other/rebalance_guard.py b/Other/rebalance_guard.py index 0eec430..3c89e31 100644 --- a/Other/rebalance_guard.py +++ b/Other/rebalance_guard.py @@ -223,7 +223,7 @@ def main(): print("=" * 80) print(" 🛡️ LNDg Rebalance Guard") print(f" Mode: {'DRY RUN' if args.dry_run else 'LIVE EXECUTION'}") - print(f" Lock Target: {lock_target}% | Restore Target: {restore_target}% | Restore Threshold: {restore_threshold}%") + print(f" Lock Target: {lock_target}% | Restore Target: Previous oTarget (Recorded Baseline)") print("=" * 80) try: From 8b72cd19cd6afcea0c1ac92bfbf1afb629f99f0c Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 10:25:26 +0200 Subject: [PATCH 06/12] fix(magma): resolve KeyError in Telegram order approval callback, timeout auto-accept, and stream error logs to Telegram - Make extract_order_info idempotent and support fallback calculation for unsettled orders with 0-value seller fees (fixed + variable fees) - Eliminate direct dictionary dereferencing of legacy keys in handle_order_decision_callback and _handle_timeout_for_offer - Store normalized order_info in pending_user_confirmations to guarantee reliable metadata availability across callbacks and timeouts - Wrap _handle_timeout_for_offer in exception containment boundaries to prevent unhandled errors from terminating the scheduler daemon - Add TelegramLoggingHandler to dispatch ERROR/CRITICAL logs with formatted tracebacks to Telegram, with recursion locks and transport exclusion - Expand test_magma_sale_process with 9 new unit tests covering idempotency, callbacks, timeouts, and logging handlers Closes #46 Tests passing: 63/63 --- Magma/magma_sale_process.py | 215 +++++++++++++------ tests/Magma/test_magma_sale_process.py | 284 ++++++++++++++++++++++++- 2 files changed, 438 insertions(+), 61 deletions(-) diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index 48bfebd..e81f503 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -103,16 +103,87 @@ if not os.path.exists(logs_dir): os.makedirs(logs_dir, exist_ok=True) +class TelegramLoggingHandler(logging.Handler): + """ + Custom logging handler that dispatches ERROR and CRITICAL log records to Telegram. + Includes formatted message and exception traceback with recursion protection, + suppression of network polling exceptions, and message boundary truncation. + """ + def __init__(self, bot=None, chat_id=None, level=logging.ERROR): + super().__init__(level=level) + self.bot = bot + self.chat_id = chat_id + self._is_handling = False + + def emit(self, record): + if self._is_handling: + return + + # Suppress logging from external network/telegram libraries to avoid cascade loops + if record.name in ("telebot", "urllib3", "requests") or record.name.startswith(("telebot.", "urllib3.", "requests.")): + return + + # Suppress messages that are already sent via send_telegram_notification + formatted_msg = record.getMessage() + if formatted_msg.startswith("Telegram NOTIFICATION:"): + return + + target_bot = self.bot if self.bot is not None else bot + target_chat_id = self.chat_id if self.chat_id is not None else CHAT_ID + + if not target_bot or not target_chat_id: + return + + self._is_handling = True + try: + timestamp = datetime.fromtimestamp(record.created).strftime("%Y-%m-%d %H:%M:%S") + level_name = record.levelname + origin = f"{record.module}.{record.funcName}:{record.lineno}" + + lines = [ + f"🚨 *[{level_name}]* `{origin}` at `{timestamp}`", + f"{formatted_msg}" + ] + + tb_str = "" + if record.exc_info: + import traceback + tb_str = "".join(traceback.format_exception(*record.exc_info)) + lines.append(f"```\n{tb_str}\n```") + + full_message = "\n".join(lines) + if len(full_message) > 3800: + full_message = full_message[:3790] + "\n...[truncated]```" + + try: + target_bot.send_message(target_chat_id, text=full_message, parse_mode="Markdown") + except Exception: + try: + plain_message = f"🚨 [{level_name}] {origin} at {timestamp}\n{formatted_msg}" + if record.exc_info: + plain_message += f"\n\n{tb_str}" + if len(plain_message) > 3800: + plain_message = plain_message[:3790] + "\n...[truncated]" + target_bot.send_message(target_chat_id, text=plain_message) + except Exception: + pass + except Exception: + self.handleError(record) + finally: + self._is_handling = False + handler = RotatingFileHandler( LOG_FILE_PATH, maxBytes=10 * 1024 * 1024, backupCount=5 # 10 MB ) +telegram_handler = TelegramLoggingHandler(bot=bot, chat_id=CHAT_ID, level=logging.ERROR) # Set up logging configuration logging.basicConfig( level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s", - handlers=[handler], + handlers=[handler, telegram_handler], ) +logging.getLogger().addHandler(telegram_handler) # Adjust logging levels for third-party libraries logging.getLogger("requests").setLevel(logging.WARNING) @@ -321,9 +392,12 @@ def extract_order_info(order: dict) -> dict: if isinstance(dest, dict): buyer_pubkey = dest.get("pubkey") buyer_alias = dest.get("alias") + elif "customer_pubkey" in order: + buyer_pubkey = order.get("customer_pubkey") + buyer_alias = order.get("buyer_alias") else: buyer_pubkey = order.get("account") or (order.get("endpoints", {}).get("destination") if isinstance(order.get("endpoints"), dict) else None) - buyer_alias = None + buyer_alias = order.get("buyer_alias") # Amount / Size amount_obj = order.get("amount") @@ -331,6 +405,8 @@ def extract_order_info(order: dict) -> dict: size_sats = int(amount_obj.get("satoshi", {}).get("sats", 0)) elif isinstance(amount_obj, (int, str)) and str(amount_obj).isdigit(): size_sats = int(amount_obj) + elif "channel_size" in order: + size_sats = int(order.get("channel_size", 0)) elif "size" in order: size_sats = int(order.get("size", 0)) else: @@ -343,6 +419,9 @@ def extract_order_info(order: dict) -> dict: fixed_fee_sats = int(fees_obj.get("fixed", {}).get("sats", 0)) variable_fee_sats = int(fees_obj.get("variable", {}).get("sats", 0)) amboss_fee_sats = int(fees_obj.get("amboss", {}).get("sats", 0)) + # Amboss documentation: fees.seller is 0 for unsettled orders; fallback to fixed + variable fee sum + if seller_invoice_sats == 0 and (fixed_fee_sats > 0 or variable_fee_sats > 0): + seller_invoice_sats = fixed_fee_sats + variable_fee_sats else: seller_invoice_sats = int(order.get("seller_invoice_amount", 0)) fixed_fee_sats = int(order.get("fixed_fee", 0)) @@ -353,6 +432,8 @@ def extract_order_info(order: dict) -> dict: promises_obj = order.get("promises") if isinstance(promises_obj, dict): min_block_length = int(promises_obj.get("locked_min_block_length", 0)) + elif "min_block_length" in order: + min_block_length = int(order.get("min_block_length", 0)) else: min_block_length = int(order.get("locked_min_block_length", 0)) @@ -368,7 +449,7 @@ def extract_order_info(order: dict) -> dict: "amboss_fee": amboss_fee_sats, "min_block_length": min_block_length, "created_at": order.get("created_at"), - "raw_order": order + "raw_order": order.get("raw_order", order) } def _execute_amboss_graphql_request( payload: dict, @@ -1359,9 +1440,16 @@ def bos_confirm_income(amount, peer_pubkey): def _complete_offer_approval_process(order_id, order_details): """Generates invoice, accepts on Amboss, and starts payment polling.""" - seller_invoice_amount = order_details['seller_invoice_amount'] - buyer_alias = order_details.get("buyer_alias", "N/A") + order_info = extract_order_info(order_details) + seller_invoice_amount = order_info.get('seller_invoice_amount') or (order_details.get('seller_invoice_amount') if isinstance(order_details, dict) else 0) + buyer_alias = (order_details.get("buyer_alias") if isinstance(order_details, dict) else None) or order_info.get("buyer_alias", "N/A") + if not seller_invoice_amount or seller_invoice_amount <= 0: + error_msg = f"🔥 Cannot approve order `{order_id}`: invalid or missing seller_invoice_amount ({seller_invoice_amount})." + logging.error(error_msg) + send_telegram_notification(error_msg, level="error", parse_mode="Markdown") + return + send_telegram_notification(f"✅ Order `{order_id}` approved by you ({buyer_alias}).\nGenerating invoice for {seller_invoice_amount} sats...", parse_mode="Markdown") invoice_hash_or_error, invoice_request = execute_lncli_addinvoice( # Modified to return error message seller_invoice_amount, @@ -1442,10 +1530,11 @@ def handle_order_decision_callback(call): decision_text_verb = "Approved" if action == "approve" else "Rejected" bot.answer_callback_query(call.id, text=f"Order {order_id} {decision_text_verb}. Processing...") - order_original_details = confirmation_details_entry["details"] - buyer_alias = order_original_details.get("buyer_alias", "N/A") - buyer_pubkey = order_original_details.get('account') or order_original_details.get("endpoints", {}).get("destination", "Unknown") - amount = order_original_details['seller_invoice_amount'] + order_original_details = confirmation_details_entry.get("details", {}) + order_info = extract_order_info(order_original_details) + buyer_alias = order_original_details.get("buyer_alias") or order_info.get("buyer_alias", "N/A") + buyer_pubkey = order_info.get("customer_pubkey") or "Unknown" + amount = order_info.get("seller_invoice_amount", 0) decision_emoji = "✅" if action == "approve" else "❌" @@ -1467,11 +1556,12 @@ def handle_order_decision_callback(call): if action == "approve": send_telegram_notification(f"▶️ Proceeding with approved order `{order_id}` ({buyer_alias}).", parse_mode="Markdown") - if order_original_details.get("status") == "WAITING_FOR_SELLER_APPROVAL": + current_status = order_info.get("status") + if current_status == "WAITING_FOR_SELLER_APPROVAL": # Run long-running task in a separate thread to avoid blocking Telegram polling - threading.Thread(target=_complete_offer_approval_process, args=(order_id, order_original_details), name=f"Approve-{order_id}").start() + threading.Thread(target=_complete_offer_approval_process, args=(order_id, order_info), name=f"Approve-{order_id}").start() else: - msg = f"⚠️ Order `{order_id}` status changed to `{order_original_details.get('status')}` before user approval ({action}) could be fully processed. No action taken." + msg = f"⚠️ Order `{order_id}` status changed to `{current_status}` before user approval ({action}) could be fully processed. No action taken." logging.warning(msg) send_telegram_notification(msg, level="warning", parse_mode="Markdown") @@ -1486,57 +1576,62 @@ def handle_order_decision_callback(call): bot.answer_callback_query(call.id, text="Error processing your decision.") except: pass - send_telegram_notification("🔥 Error processing user decision from Telegram button. Check logs.", level="error", parse_mode="Markdown") + send_telegram_notification(f"🔥 Error processing user decision from Telegram button: {e}", level="error", parse_mode="Markdown") def _handle_timeout_for_offer(order_id, confirmation_info): """Handles the logic when an offer confirmation times out.""" logging.info(f"Order {order_id} timed out waiting for user confirmation. Defaulting to approve.") - - order_details = confirmation_info['details'] - buyer_alias = order_details.get("buyer_alias", "N/A") - buyer_pubkey = order_details.get('account') or order_details.get("endpoints", {}).get("destination", "Unknown") - amount = order_details['seller_invoice_amount'] - - send_telegram_notification( - f"⏳ Offer Timeout & Auto-Approved:\n" - f"ID: `{order_id}`\n" - f"💰 Amount: {amount} sats\n" - f"👤 Buyer: `{buyer_alias}` ({buyer_pubkey[:10]}...)\n" - f"No response in 5 min.", - level="warning", - parse_mode="Markdown" - ) try: - bot.edit_message_text( - chat_id=CHAT_ID, - message_id=confirmation_info["message_id"], - text=( - f"✅ Auto-Approved (Timeout):\n" - f"ID: `{order_id}`\n" - f"💰 Amount: {amount} sats\n" - f"👤 Buyer: `{buyer_alias}` ({buyer_pubkey[:10]}...)" - ), - reply_markup=None, + order_details = confirmation_info.get('details', {}) if confirmation_info else {} + order_info = extract_order_info(order_details) if order_details else {} + buyer_alias = (order_details.get("buyer_alias") if isinstance(order_details, dict) else None) or order_info.get("buyer_alias", "N/A") + buyer_pubkey = order_info.get("customer_pubkey") or "Unknown" + amount = order_info.get("seller_invoice_amount", 0) + + send_telegram_notification( + f"⏳ Offer Timeout & Auto-Approved:\n" + f"ID: `{order_id}`\n" + f"💰 Amount: {amount} sats\n" + f"👤 Buyer: `{buyer_alias}` ({buyer_pubkey[:10]}...)\n" + f"No response in 5 min.", + level="warning", parse_mode="Markdown" ) - except Exception as e: - logging.error(f"Error editing Telegram message for timed-out order {order_id}: {e}") - - order_details_fresh = get_order_details_from_amboss(order_id) - if order_details_fresh: - # Add buyer_alias to fresh details if needed for _complete_offer_approval_process - order_details_fresh['buyer_alias'] = buyer_alias # Carry over known alias - if order_details_fresh.get("status") == "WAITING_FOR_SELLER_APPROVAL": - _complete_offer_approval_process(order_id, order_details_fresh) + try: + if confirmation_info and "message_id" in confirmation_info: + bot.edit_message_text( + chat_id=CHAT_ID, + message_id=confirmation_info["message_id"], + text=( + f"✅ Auto-Approved (Timeout):\n" + f"ID: `{order_id}`\n" + f"💰 Amount: {amount} sats\n" + f"👤 Buyer: `{buyer_alias}` ({buyer_pubkey[:10]}...)" + ), + reply_markup=None, + parse_mode="Markdown" + ) + except Exception as e: + logging.error(f"Error editing Telegram message for timed-out order {order_id}: {e}") + + order_details_fresh = get_order_details_from_amboss(order_id) + if order_details_fresh: + fresh_info = extract_order_info(order_details_fresh) + fresh_info['buyer_alias'] = buyer_alias # Carry over known alias + if fresh_info.get("status") == "WAITING_FOR_SELLER_APPROVAL": + _complete_offer_approval_process(order_id, fresh_info) + else: + msg = f"⚠️ Order `{order_id}` (timed out) status changed to `{fresh_info.get('status')}` before auto-approval. No action taken." + logging.warning(msg) + send_telegram_notification(msg, level="warning", parse_mode="Markdown") else: - msg = f"⚠️ Order `{order_id}` (timed out) status changed to `{order_details_fresh.get('status')}` before auto-approval. No action taken." - logging.warning(msg) - send_telegram_notification(msg, level="warning", parse_mode="Markdown") - else: - msg = f"🔥 Could not fetch details for timed-out order `{order_id}` for auto-approval. Manual check required." - logging.error(msg) - send_telegram_notification(msg, level="error", parse_mode="Markdown") + msg = f"🔥 Could not fetch details for timed-out order `{order_id}` for auto-approval. Manual check required." + logging.error(msg) + send_telegram_notification(msg, level="error", parse_mode="Markdown") + except Exception as e: + logging.exception(f"Unexpected error in _handle_timeout_for_offer for order {order_id}: {e}") + send_telegram_notification(f"🔥 Error during timeout auto-approval for order `{order_id}`: {e}", level="error", parse_mode="Markdown") def check_pending_confirmations_timeouts(): @@ -1678,14 +1773,12 @@ def process_new_offers(): sent_message = send_telegram_notification(prompt_message, reply_markup=markup, parse_mode="Markdown") if sent_message: - # Store comprehensive details in pending_user_confirmations for rich messages on timeout/callback - pending_confirmation_data = new_offer_from_amboss.copy() - pending_confirmation_data['buyer_alias'] = buyer_alias # Add the fetched alias - + # Store comprehensive normalized details in pending_user_confirmations for rich messages on timeout/callback + order_info['buyer_alias'] = buyer_alias pending_user_confirmations[order_id] = { "message_id": sent_message.message_id, "timestamp": time.time(), - "details": pending_confirmation_data + "details": order_info } logging.info(f"Offer {order_id} (Buyer: {buyer_alias}) presented to user for confirmation. Awaiting response or timeout.") else: @@ -2008,11 +2101,13 @@ def run_telegram_polling(): main_handler = RotatingFileHandler( log_file_path_for_main, maxBytes=10 * 1024 * 1024, backupCount=5 # 10 MB ) + main_telegram_handler = TelegramLoggingHandler(bot=bot, chat_id=CHAT_ID, level=logging.ERROR) logging.basicConfig( level=logging.INFO, # Default to INFO, can be overridden by config later if needed format="%(asctime)s - %(levelname)s - [%(funcName)s:%(lineno)d] - %(message)s", # Added funcName and lineno - handlers=[main_handler], + handlers=[main_handler, main_telegram_handler], ) + logging.getLogger().addHandler(main_telegram_handler) # Adjust logging levels for third-party libraries logging.getLogger("requests").setLevel(logging.WARNING) logging.getLogger("urllib3").setLevel(logging.WARNING) diff --git a/tests/Magma/test_magma_sale_process.py b/tests/Magma/test_magma_sale_process.py index bea44fb..5d70554 100644 --- a/tests/Magma/test_magma_sale_process.py +++ b/tests/Magma/test_magma_sale_process.py @@ -1,5 +1,6 @@ import sys import os +import logging import pytest from unittest.mock import MagicMock, patch, mock_open @@ -7,7 +8,10 @@ @pytest.fixture(scope="module", autouse=True) def mock_dependencies(): mock_telebot = MagicMock() - mock_telebot.TeleBot = MagicMock() + bot_mock = MagicMock() + bot_mock.callback_query_handler = MagicMock(return_value=lambda f: f) + bot_mock.message_handler = MagicMock(return_value=lambda f: f) + mock_telebot.TeleBot = MagicMock(return_value=bot_mock) mock_configparser = MagicMock() mock_logging = MagicMock() mock_schedule = MagicMock() @@ -468,3 +472,281 @@ def test_execute_lnd_command_failure(magma_module, mocker): assert txid is None assert "not enough funds" in err + + +def test_extract_order_info_unsettled_zero_seller_fee_fallback(magma_module): + """Test unsettled orders where fees.seller.sats is 0 fall back to fixed + variable fee sum.""" + unsettled_order = { + "id": "order_unsettled_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "2000000"}}, + "fees": { + "fixed": {"sats": "1000"}, + "variable": {"sats": "5000"}, + "seller": {"sats": "0"}, + "amboss": {"sats": "200"} + }, + "destination": {"pubkey": "03buyer123", "alias": "FastBuyer"} + } + info = magma_module.extract_order_info(unsettled_order) + assert info["id"] == "order_unsettled_001" + assert info["seller_invoice_amount"] == 6000 + assert info["customer_pubkey"] == "03buyer123" + assert info["channel_size"] == 2000000 + + +def test_extract_order_info_idempotency(magma_module): + """Test that extract_order_info is idempotent when given an already normalized dictionary.""" + sample_order = { + "id": "order_norm_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "4000000"}}, + "fees": { + "fixed": {"sats": "1500"}, + "variable": {"sats": "3500"}, + "seller": {"sats": "5000"}, + "amboss": {"sats": "500"} + }, + "destination": {"pubkey": "02idempotent456", "alias": "IdemNode"} + } + first_pass = magma_module.extract_order_info(sample_order) + second_pass = magma_module.extract_order_info(first_pass) + + assert second_pass["id"] == "order_norm_001" + assert second_pass["customer_pubkey"] == "02idempotent456" + assert second_pass["buyer_alias"] == "IdemNode" + assert second_pass["channel_size"] == 4000000 + assert second_pass["seller_invoice_amount"] == 5000 + assert second_pass["fixed_fee"] == 1500 + assert second_pass["variable_fee"] == 3500 + + +def test_handle_order_decision_callback_approve_success(magma_module, mocker): + """Test approving order via Telegram callback processes GraphQL order without KeyError.""" + raw_graphql_order = { + "id": "order_approve_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "3000000"}}, + "fees": {"seller": {"sats": "12000"}, "fixed": {"sats": "2000"}, "variable": {"sats": "10000"}}, + "destination": {"pubkey": "03approver123456", "alias": "ApproverNode"}, + "buyer_alias": "ApproverNode" + } + magma_module.pending_user_confirmations["order_approve_001"] = { + "message_id": 1234, + "timestamp": 1000.0, + "details": raw_graphql_order + } + + mock_complete = mocker.patch.object(magma_module, "_complete_offer_approval_process") + mocker.patch.object(magma_module, "send_telegram_notification") + + mock_call = MagicMock() + mock_call.id = "cb_id_1" + mock_call.data = "decide_order:approve:order_approve_001" + mock_call.message.chat.id = 5555 + mock_call.message.message_id = 1234 + + magma_module.handle_order_decision_callback(mock_call) + + magma_module.bot.answer_callback_query.assert_called_with("cb_id_1", text="Order order_approve_001 Approved. Processing...") + magma_module.bot.edit_message_text.assert_called() + edit_text = magma_module.bot.edit_message_text.call_args[1]["text"] + assert "12000 sats" in edit_text + assert "ApproverNode" in edit_text + assert "03approver" in edit_text + + assert "order_approve_001" not in magma_module.pending_user_confirmations + + import time + time.sleep(0.05) + assert mock_complete.called + called_order_id, called_details = mock_complete.call_args[0] + assert called_order_id == "order_approve_001" + assert called_details["seller_invoice_amount"] == 12000 + + +def test_handle_order_decision_callback_reject_success(magma_module, mocker): + """Test rejecting order via Telegram callback rejects order on Amboss.""" + raw_graphql_order = { + "id": "order_reject_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "2000000"}}, + "fees": {"seller": {"sats": "8000"}}, + "destination": {"pubkey": "02rejector123", "alias": "RejectNode"}, + "buyer_alias": "RejectNode" + } + magma_module.pending_user_confirmations["order_reject_001"] = { + "message_id": 4321, + "timestamp": 1000.0, + "details": raw_graphql_order + } + mock_reject = mocker.patch.object(magma_module, "reject_order") + mocker.patch.object(magma_module, "send_telegram_notification") + + mock_call = MagicMock() + mock_call.id = "cb_id_2" + mock_call.data = "decide_order:reject:order_reject_001" + mock_call.message.chat.id = 5555 + mock_call.message.message_id = 4321 + + magma_module.handle_order_decision_callback(mock_call) + + magma_module.bot.answer_callback_query.assert_called_with("cb_id_2", text="Order order_reject_001 Rejected. Processing...") + magma_module.bot.edit_message_text.assert_called() + assert "order_reject_001" not in magma_module.pending_user_confirmations + + import time + time.sleep(0.05) + assert mock_reject.called + assert mock_reject.call_args[0][0] == "order_reject_001" + + +def test_handle_timeout_for_offer_auto_approve(magma_module, mocker): + """Test timeout auto-approval workflow with GraphQL payload and fresh order fetch.""" + raw_graphql_order = { + "id": "order_timeout_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "5000000"}}, + "fees": {"seller": {"sats": "15000"}, "fixed": {"sats": "3000"}, "variable": {"sats": "12000"}}, + "destination": {"pubkey": "02timeoutbuyer", "alias": "TimeoutBuyer"}, + "buyer_alias": "TimeoutBuyer" + } + confirmation_info = { + "message_id": 7777, + "timestamp": 500.0, + "details": raw_graphql_order + } + + fresh_order = { + "id": "order_timeout_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "5000000"}}, + "fees": {"seller": {"sats": "15000"}}, + "destination": {"pubkey": "02timeoutbuyer", "alias": "TimeoutBuyer"} + } + + mocker.patch.object(magma_module, "get_order_details_from_amboss", return_value=fresh_order) + mock_complete = mocker.patch.object(magma_module, "_complete_offer_approval_process") + mocker.patch.object(magma_module, "send_telegram_notification") + + magma_module._handle_timeout_for_offer("order_timeout_001", confirmation_info) + + magma_module.bot.edit_message_text.assert_called() + edit_text = magma_module.bot.edit_message_text.call_args[1]["text"] + assert "Auto-Approved (Timeout)" in edit_text + assert "15000 sats" in edit_text + + assert mock_complete.called + called_id, called_fresh = mock_complete.call_args[0] + assert called_id == "order_timeout_001" + assert called_fresh["seller_invoice_amount"] == 15000 + assert called_fresh["customer_pubkey"] == "02timeoutbuyer" + + +def test_handle_timeout_for_offer_exception_resilience(magma_module, mocker): + """Test timeout auto-approval handles exceptions without terminating the scheduler loop.""" + broken_confirmation_info = { + "message_id": 8888, + "timestamp": 500.0, + "details": None + } + mock_send = mocker.patch.object(magma_module, "send_telegram_notification") + + # Must NOT raise exception + magma_module._handle_timeout_for_offer("order_broken_001", broken_confirmation_info) + assert mock_send.called + + +def test_complete_offer_approval_process_raw_and_normalized(magma_module, mocker): + """Test _complete_offer_approval_process works seamlessly with raw GraphQL order.""" + raw_graphql_order = { + "id": "order_proc_001", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": {"satoshi": {"sats": "1000000"}}, + "fees": {"seller": {"sats": "4500"}}, + "destination": {"pubkey": "03buyerproc", "alias": "ProcNode"}, + "buyer_alias": "ProcNode" + } + + mocker.patch.object(magma_module, "execute_lncli_addinvoice", return_value=("hash_p", "lnbc4500...")) + mocker.patch.object(magma_module, "accept_order", return_value={ + "data": {"market": {"order": {"seller": {"accept": {"success": True}}}}} + }) + mock_wait = mocker.patch.object(magma_module, "wait_for_buyer_payment") + mocker.patch.object(magma_module, "send_telegram_notification") + + magma_module._complete_offer_approval_process("order_proc_001", raw_graphql_order) + + assert magma_module.execute_lncli_addinvoice.called + assert magma_module.execute_lncli_addinvoice.call_args[0][0] == 4500 + assert magma_module.accept_order.called + assert magma_module.accept_order.call_args[0] == ("order_proc_001", "lnbc4500...") + assert mock_wait.called + assert mock_wait.call_args[0][0] == "order_proc_001" + + +def test_telegram_logging_handler_captures_errors_and_tracebacks(magma_module): + """Test TelegramLoggingHandler dispatches ERROR/CRITICAL logs and tracebacks to Telegram.""" + handler = magma_module.TelegramLoggingHandler(bot=magma_module.bot, chat_id=123) + + # 1. Error without exc_info + record1 = logging.LogRecord( + name="magma_sale_process", + level=logging.ERROR, + pathname="magma_sale_process.py", + lineno=100, + msg="Critical database failure: %s", + args=("disk full",), + exc_info=None + ) + handler.emit(record1) + magma_module.bot.send_message.assert_called() + sent_text1 = magma_module.bot.send_message.call_args[1]["text"] + assert "Critical database failure: disk full" in sent_text1 + assert "ERROR" in sent_text1 + + # 2. Error with exc_info (traceback) + try: + raise ValueError("Simulated failure for traceback testing") + except ValueError: + import sys + exc_info = sys.exc_info() + + record2 = logging.LogRecord( + name="magma_sale_process", + level=logging.ERROR, + pathname="magma_sale_process.py", + lineno=105, + msg="Caught unexpected exception", + args=(), + exc_info=exc_info + ) + handler.emit(record2) + sent_text2 = magma_module.bot.send_message.call_args[1]["text"] + assert "Caught unexpected exception" in sent_text2 + assert "Simulated failure for traceback testing" in sent_text2 + assert "Traceback" in sent_text2 + + +def test_telegram_logging_handler_recursion_and_filters(magma_module): + """Test TelegramLoggingHandler guards against recursion and suppresses excluded logs.""" + handler = magma_module.TelegramLoggingHandler(bot=magma_module.bot, chat_id=123) + magma_module.bot.send_message.reset_mock() + + # Filter telebot/urllib3/requests logs + for noisy_logger in ["telebot", "urllib3", "requests", "telebot.apihelper"]: + rec = logging.LogRecord(name=noisy_logger, level=logging.ERROR, pathname="foo.py", lineno=1, msg="Network drop", args=(), exc_info=None) + handler.emit(rec) + assert not magma_module.bot.send_message.called + + # Filter notifications already being sent + rec_notif = logging.LogRecord(name="magma_sale_process", level=logging.ERROR, pathname="foo.py", lineno=1, msg="Telegram NOTIFICATION: 🔥 Error occurred", args=(), exc_info=None) + handler.emit(rec_notif) + assert not magma_module.bot.send_message.called + + # Recursion guard: bot.send_message fails with exception + magma_module.bot.send_message.side_effect = Exception("Telegram API down") + rec_error = logging.LogRecord(name="magma_sale_process", level=logging.ERROR, pathname="foo.py", lineno=1, msg="Some real error", args=(), exc_info=None) + # Should not raise exception + handler.emit(rec_error) + magma_module.bot.send_message.side_effect = None From 30036828eeae8e453ee3a7e34e1c643fcc7c31e4 Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 10:44:37 +0200 Subject: [PATCH 07/12] test(magma): add end-to-end buy order simulation harness and documentation --- Magma/magma_sale.MD | 32 ++ Magma/magma_sale_process.py | 43 ++- tests/Magma/test_magma_simulation.py | 34 +++ tests/simulate_magma_sale.py | 422 +++++++++++++++++++++++++++ 4 files changed, 519 insertions(+), 12 deletions(-) create mode 100644 tests/Magma/test_magma_simulation.py create mode 100755 tests/simulate_magma_sale.py diff --git a/Magma/magma_sale.MD b/Magma/magma_sale.MD index 5848726..7584891 100644 --- a/Magma/magma_sale.MD +++ b/Magma/magma_sale.MD @@ -191,3 +191,35 @@ To allow manual triggering of the Magma sale process via Telegram, you need to c * The text after the hyphen is the description that users will see in Telegram when they type `/`. Once set, users (specifically the `telegram_user_id` configured in your `config.ini`) will be able to use the `/runnow` command to trigger the script's main processing logic on demand. The script also uses inline keyboard buttons for approving or rejecting new offers; these do not need to be registered as commands with BotFather as they are contextually generated by the bot. + +## Simulating a Magma Buy Order End-to-End + +Because Magma sales are scarce per quarter and Amboss does not offer a sandbox dry-run environment on mainnet, an end-to-end simulation harness is provided in `tests/simulate_magma_sale.py`. + +The simulation tests the full order lifecycle safely without opening channels or charging real sats: +1. **Timeout Auto-Approval**: Ingests an order in `WAITING_FOR_SELLER_APPROVAL` with Amboss unsettled zero-fee schema (`fees.seller.sats == 0`), verifies fixed/variable fee fallback calculation, verifies queueing in `pending_user_confirmations`, triggers timeout auto-approval without `KeyError`, generates the seller invoice, and executes the Amboss acceptance flow. +2. **Manual Inline Callback Approval**: Ingests the offer, simulates the user clicking the `[✅ Approve Offer]` inline keyboard button, acknowledges callback, edits message text, and executes the approval pipeline. +3. **Telegram Error Logging**: Dispatches a test exception to ensure `TelegramLoggingHandler` formats and streams traceback alerts to your Telegram chat. + +### Running Offline Automated Simulation (Mock Mode) +Safe for CI and local automated testing without touching Telegram or external networks: + +```bash +# Run full simulation suite +.venv/bin/python tests/simulate_magma_sale.py + +# Or via pytest +.venv/bin/pytest tests/Magma/test_magma_simulation.py +``` + +### Running Live Interactive Telegram Simulation +To verify how prompts, timeout updates, and error alerts actually look and behave in your Telegram app, use the `--live-telegram` flag. This uses your `config.ini` bot credentials while safely mocking Amboss mutations and LND invoice creation: + +```bash +# Live interactive test with a 8-second timeout countdown +.venv/bin/python tests/simulate_magma_sale.py --live-telegram --timeout-seconds 8 + +# Live test of only the Telegram error logger stream +.venv/bin/python tests/simulate_magma_sale.py --live-telegram --scenario error +``` + diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index e81f503..4694d05 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -415,18 +415,34 @@ def extract_order_info(order: dict) -> dict: # Fees fees_obj = order.get("fees") if isinstance(fees_obj, dict): - seller_invoice_sats = int(fees_obj.get("seller", {}).get("sats", 0)) - fixed_fee_sats = int(fees_obj.get("fixed", {}).get("sats", 0)) - variable_fee_sats = int(fees_obj.get("variable", {}).get("sats", 0)) - amboss_fee_sats = int(fees_obj.get("amboss", {}).get("sats", 0)) - # Amboss documentation: fees.seller is 0 for unsettled orders; fallback to fixed + variable fee sum - if seller_invoice_sats == 0 and (fixed_fee_sats > 0 or variable_fee_sats > 0): - seller_invoice_sats = fixed_fee_sats + variable_fee_sats + seller_invoice_sats = int(fees_obj.get("seller", {}).get("sats", 0) or 0) + fixed_fee_sats = int(fees_obj.get("fixed", {}).get("sats", 0) or 0) + variable_fee_sats = int(fees_obj.get("variable", {}).get("sats", 0) or 0) + amboss_fee_sats = int(fees_obj.get("amboss", {}).get("sats", 0) or 0) else: - seller_invoice_sats = int(order.get("seller_invoice_amount", 0)) - fixed_fee_sats = int(order.get("fixed_fee", 0)) - variable_fee_sats = int(order.get("variable_fee", 0)) - amboss_fee_sats = int(order.get("amboss_fee", 0)) + seller_invoice_sats = int(order.get("seller_invoice_amount", 0) or 0) + fixed_fee_sats = int(order.get("fixed_fee", 0) or 0) + variable_fee_sats = int(order.get("variable_fee", 0) or 0) + amboss_fee_sats = int(order.get("amboss_fee", 0) or 0) + + # Check top-level fee fallbacks if fixed/variable were 0 + if fixed_fee_sats == 0: + fixed_val = order.get("fixed_fee") or order.get("fixed") + if isinstance(fixed_val, dict): + fixed_fee_sats = int(fixed_val.get("sats", 0) or 0) + elif isinstance(fixed_val, (int, str)) and str(fixed_val).isdigit(): + fixed_fee_sats = int(fixed_val) + + if variable_fee_sats == 0: + var_val = order.get("variable_fee") or order.get("variable") + if isinstance(var_val, dict): + variable_fee_sats = int(var_val.get("sats", 0) or 0) + elif isinstance(var_val, (int, str)) and str(var_val).isdigit(): + variable_fee_sats = int(var_val) + + # Amboss documentation: fees.seller is 0 for unsettled orders; fallback to fixed + variable fee sum + if seller_invoice_sats == 0 and (fixed_fee_sats > 0 or variable_fee_sats > 0): + seller_invoice_sats = fixed_fee_sats + variable_fee_sats # Promises promises_obj = order.get("promises") @@ -1528,7 +1544,10 @@ def handle_order_decision_callback(call): # Immediately acknowledge the callback to stop the client-side loading animation decision_text_verb = "Approved" if action == "approve" else "Rejected" - bot.answer_callback_query(call.id, text=f"Order {order_id} {decision_text_verb}. Processing...") + try: + bot.answer_callback_query(call.id, text=f"Order {order_id} {decision_text_verb}. Processing...") + except Exception as e: + logging.warning(f"Could not answer callback query {call.id} for order {order_id}: {e}") order_original_details = confirmation_details_entry.get("details", {}) order_info = extract_order_info(order_original_details) diff --git a/tests/Magma/test_magma_simulation.py b/tests/Magma/test_magma_simulation.py new file mode 100644 index 0000000..eff8fe8 --- /dev/null +++ b/tests/Magma/test_magma_simulation.py @@ -0,0 +1,34 @@ +import pytest +import os +import sys + +# Ensure tests/ is importable +TESTS_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +if TESTS_DIR not in sys.path: + sys.path.insert(0, TESTS_DIR) + +from simulate_magma_sale import run_simulation + + +def test_simulation_timeout_auto_approval(): + """Verify end-to-end timeout auto-approval flow with unsettled fee fallback.""" + success = run_simulation(scenario="timeout", live_telegram=False, timeout_seconds=1) + assert success is True + + +def test_simulation_manual_callback_approval(): + """Verify end-to-end inline button approval callback flow.""" + success = run_simulation(scenario="callback", live_telegram=False, timeout_seconds=1) + assert success is True + + +def test_simulation_telegram_error_logging(): + """Verify TelegramLoggingHandler dispatches formatted error and traceback.""" + success = run_simulation(scenario="error", live_telegram=False, timeout_seconds=1) + assert success is True + + +def test_simulation_full_suite(): + """Verify full simulation harness executes all scenarios cleanly.""" + success = run_simulation(scenario="all", live_telegram=False, timeout_seconds=1) + assert success is True diff --git a/tests/simulate_magma_sale.py b/tests/simulate_magma_sale.py new file mode 100755 index 0000000..cc012f4 --- /dev/null +++ b/tests/simulate_magma_sale.py @@ -0,0 +1,422 @@ +#!/usr/bin/env python3 +""" +Magma Buy Order End-to-End Simulation Harness +============================================== + +Simulates incoming Amboss Magma channel buy orders to verify the full +order lifecycle without spending on-chain funds or touching live Amboss orders. + +Scenarios tested: + 1. Timeout Auto-Accept: Ingests a new order in WAITING_FOR_SELLER_APPROVAL with + unsettled zero-fee structure, presents inline keyboard prompt, awaits timeout, + and auto-approves via _handle_timeout_for_offer without KeyError. + 2. Manual Callback Approve: Simulates user pressing "Approve" button via Telegram + inline keyboard callback (decide_order:approve:). + 3. Manual Callback Reject: Simulates user pressing "Reject" button via Telegram. + 4. Telegram Logging Handler: Dispatches an error log with full traceback to ensure + TelegramLoggingHandler streams alerts to Telegram. + +Modes: + - Default / Mock Mode: Completely offline, mocks all external services (Telegram, + Amboss GraphQL, LND). Safe for automated test suites and CI. + - Live Telegram Mode (--live-telegram): Uses real Telegram bot credentials from config.ini + to dispatch live interactive notifications and test error logs to your Telegram chat, + while safely mocking Amboss mutations and LND invoice creation. + +Usage: + # Automated offline verification (default) + .venv/bin/python tests/simulate_magma_sale.py + + # Live interactive verification sent to your Telegram chat + .venv/bin/python tests/simulate_magma_sale.py --live-telegram --timeout-seconds 8 + + # Run specific scenario + .venv/bin/python tests/simulate_magma_sale.py --scenario timeout + .venv/bin/python tests/simulate_magma_sale.py --scenario callback + .venv/bin/python tests/simulate_magma_sale.py --scenario error +""" + +import argparse +import configparser +import logging +import os +import sys +import time +from unittest.mock import MagicMock, patch + +# Ensure project root and Magma directory are on sys.path +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +PROJECT_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, "..")) +MAGMA_DIR = os.path.join(PROJECT_ROOT, "Magma") + +if PROJECT_ROOT not in sys.path: + sys.path.insert(0, PROJECT_ROOT) +if MAGMA_DIR not in sys.path: + sys.path.insert(0, MAGMA_DIR) + +# Sample order mirroring the live Amboss GraphQL MarketOrder schema from failure f73c6ad8 +SAMPLE_AMBOSS_ORDER = { + "id": "f73c6ad8-c878-48c0-8ad8-ae004c5b6197", + "status": "WAITING_FOR_SELLER_APPROVAL", + "amount": { + "satoshi": { + "sats": "5000000", + "btc": "0.05", + "usd": "3000.00" + } + }, + "fees": { + "seller": { + "sats": 0, # Unsettled order trap: must fallback to fixed + variable fees + "btc": 0.0, + "usd": 0.0 + }, + "fixed": { + "sats": 5000 + }, + "variable": { + "sats": 75000 + }, + "amboss": { + "sats": 1000 + } + }, + "fixed_fee": { + "sats": 5000 + }, + "variable_fee": { + "sats": 75000 + }, + "destination": { + "pubkey": "029a3e215d2a6a8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c", + "alias": "SimulationBuyerNode" + } +} + +EXPECTED_SELLER_FEE_SATS = 80000 # 5,000 fixed + 75,000 variable + + +def setup_simulation_logger(): + logger = logging.getLogger("MagmaSimulation") + if not logger.handlers: + handler = logging.StreamHandler(sys.stdout) + handler.setFormatter(logging.Formatter("\033[1;36m[SIMULATION]\033[0m %(message)s")) + logger.addHandler(handler) + logger.setLevel(logging.INFO) + return logger + + +def run_simulation(scenario="all", live_telegram=False, timeout_seconds=6): + """ + Executes the specified end-to-end simulation. + Returns True if all requested scenarios succeed, False otherwise. + """ + logger = setup_simulation_logger() + logger.info("Initializing Magma Buy Order End-to-End Simulation Harness...") + logger.info(f"Target Scenario: {scenario} | Live Telegram: {live_telegram} | Timeout: {timeout_seconds}s") + + # Verify config and credentials if live telegram requested + real_token = None + real_chat_id = None + if live_telegram: + config_path = os.path.join(PROJECT_ROOT, "config.ini") + if not os.path.exists(config_path): + logger.error(f"Cannot run --live-telegram: config.ini not found at {config_path}") + return False + cfg = configparser.ConfigParser() + cfg.read(config_path) + real_token = cfg.get("telegram", "magma_bot_token", fallback=None) + real_chat_id = cfg.get("telegram", "telegram_user_id", fallback=None) + if not real_token or not real_chat_id: + logger.error("Cannot run --live-telegram: telegram magma_bot_token or telegram_user_id missing in config.ini") + return False + logger.info("Live Telegram mode active: using real bot token and chat ID for interactive notifications.") + + import magma_sale_process as magma + + # Set up shared mocks for external safety (never open real channels or charge real sats) + mock_lnd_invoice = ("simulated_hash_3a8b2c1d", "lnbc800u1simulatedinvoicefor80ksatsmagmaordertestonly") + mock_accept_response = { + "data": { + "market": { + "order": { + "seller": { + "accept": { + "success": True + } + } + } + } + } + } + mock_reject_response = { + "data": { + "market": { + "order": { + "seller": { + "reject": { + "success": True + } + } + } + } + } + } + + results = {} + + # -------------------------------------------------------------------------- + # SCENARIO 1: Timeout Auto-Accept Flow + # -------------------------------------------------------------------------- + if scenario in ("all", "timeout"): + logger.info("\n" + "=" * 70) + logger.info("RUNNING SCENARIO 1: Timeout Auto-Accept Flow") + logger.info("=" * 70) + + # Clean pending state + magma.pending_user_confirmations.clear() + order_data = SAMPLE_AMBOSS_ORDER.copy() + order_id = order_data["id"] + + with patch.object(magma, "get_offers_awaiting_seller_approval", return_value=order_data), \ + patch.object(magma, "get_order_details_from_amboss", return_value=order_data), \ + patch.object(magma, "get_node_alias", return_value="SimulationBuyerNode"), \ + patch.object(magma, "get_node_extended_details", return_value={"amboss": {"is_claimed": True}}), \ + patch.object(magma, "execute_lncli_addinvoice", return_value=mock_lnd_invoice) as mock_addinvoice, \ + patch.object(magma, "accept_order", return_value=mock_accept_response) as mock_accept, \ + patch.object(magma, "wait_for_buyer_payment", return_value=None) as mock_wait_payment: + + if not live_telegram: + mock_msg = MagicMock() + mock_msg.message_id = 999111 + send_patch = patch.object(magma, "send_telegram_notification", return_value=mock_msg) + edit_patch = patch.object(magma.bot, "edit_message_text", return_value=True) + else: + from contextlib import nullcontext + send_patch = nullcontext() + edit_patch = nullcontext() + + with send_patch, edit_patch: + logger.info("Step 1: Ingesting new offer via process_new_offers()...") + magma.process_new_offers() + + # Verify pending queue population + if order_id not in magma.pending_user_confirmations: + logger.error("FAIL: order was not queued in pending_user_confirmations!") + results["timeout"] = False + else: + queued_entry = magma.pending_user_confirmations[order_id] + queued_details = queued_entry.get("details", {}) + extracted = magma.extract_order_info(queued_details) + logger.info(f"Step 2: Order queued with normalized seller_invoice_amount = {extracted.get('seller_invoice_amount')} sats") + if extracted.get("seller_invoice_amount") != EXPECTED_SELLER_FEE_SATS: + logger.error(f"FAIL: Expected {EXPECTED_SELLER_FEE_SATS} sats, got {extracted.get('seller_invoice_amount')}") + results["timeout"] = False + else: + logger.info(f"Step 3: Simulating timeout countdown ({timeout_seconds}s)...") + if live_telegram: + magma.send_telegram_notification( + f"🧪 *[SIMULATION]* New order prompt sent above!\n" + f"Waiting {timeout_seconds}s for simulated timeout auto-approval...", + parse_mode="Markdown" + ) + + # Artificially age the timestamp so it exceeds USER_CONFIRMATION_TIMEOUT_SECONDS + queued_entry["timestamp"] = time.time() - (magma.USER_CONFIRMATION_TIMEOUT_SECONDS + 10) + time.sleep(timeout_seconds if live_telegram else 0.1) + + logger.info("Step 4: Triggering check_pending_confirmations_timeouts()...") + magma.check_pending_confirmations_timeouts() + + # Assertions + if order_id in magma.pending_user_confirmations: + logger.error("FAIL: order still present in pending_user_confirmations after timeout!") + results["timeout"] = False + elif not mock_addinvoice.called: + logger.error("FAIL: execute_lncli_addinvoice was not called during auto-approval!") + results["timeout"] = False + else: + called_amount = mock_addinvoice.call_args[0][0] + logger.info(f"Step 5: execute_lncli_addinvoice invoked with {called_amount} sats.") + if called_amount != EXPECTED_SELLER_FEE_SATS: + logger.error(f"FAIL: Invoice created with wrong amount {called_amount} sats!") + results["timeout"] = False + elif not mock_accept.called: + logger.error("FAIL: accept_order was not called on Amboss!") + results["timeout"] = False + else: + logger.info(f"Step 6: Amboss accept_order called with invoice: {mock_accept.call_args[0][1]}") + logger.info("✅ SUCCESS: Timeout auto-accept completed end-to-end without KeyError!") + results["timeout"] = True + + # -------------------------------------------------------------------------- + # SCENARIO 2: Manual Telegram Callback Approval + # -------------------------------------------------------------------------- + if scenario in ("all", "callback"): + logger.info("\n" + "=" * 70) + logger.info("RUNNING SCENARIO 2: Manual Telegram Callback Approval") + logger.info("=" * 70) + + magma.pending_user_confirmations.clear() + order_data = SAMPLE_AMBOSS_ORDER.copy() + order_id = order_data["id"] + + with patch.object(magma, "get_offers_awaiting_seller_approval", return_value=order_data), \ + patch.object(magma, "get_order_details_from_amboss", return_value=order_data), \ + patch.object(magma, "get_node_alias", return_value="SimulationBuyerNode"), \ + patch.object(magma, "get_node_extended_details", return_value={"amboss": {"is_claimed": True}}), \ + patch.object(magma, "execute_lncli_addinvoice", return_value=mock_lnd_invoice) as mock_addinvoice, \ + patch.object(magma, "accept_order", return_value=mock_accept_response) as mock_accept, \ + patch.object(magma, "wait_for_buyer_payment", return_value=None): + + if not live_telegram: + mock_msg = MagicMock() + mock_msg.message_id = 999222 + send_patch = patch.object(magma, "send_telegram_notification", return_value=mock_msg) + else: + from contextlib import nullcontext + send_patch = nullcontext() + + with send_patch: + logger.info("Step 1: Queuing order for callback...") + magma.process_new_offers() + + # Construct simulated CallbackQuery from Telegram button click + simulated_call = MagicMock() + simulated_call.id = "call_sim_12345" + simulated_call.data = f"decide_order:approve:{order_id}" + simulated_call.message.chat.id = magma.CHAT_ID + simulated_call.message.message_id = magma.pending_user_confirmations[order_id]["message_id"] + + logger.info(f"Step 2: Simulating incoming Telegram callback query: {simulated_call.data}") + + # Mock bot methods to avoid modifying chat history during mock mode + if not live_telegram: + with patch.object(magma.bot, "answer_callback_query"), \ + patch.object(magma.bot, "edit_message_text"): + magma.handle_order_decision_callback(simulated_call) + else: + # In live telegram mode, safely execute callback handling + magma.handle_order_decision_callback(simulated_call) + + # Wait for the background approval thread to finish execution + import threading + for thread in threading.enumerate(): + if thread.name == f"Approve-{order_id}": + thread.join(timeout=5) + + if order_id in magma.pending_user_confirmations: + logger.error("FAIL: order still present in pending_user_confirmations after callback!") + results["callback"] = False + elif not mock_addinvoice.called: + logger.error("FAIL: execute_lncli_addinvoice was not called during callback approval!") + results["callback"] = False + else: + called_amount = mock_addinvoice.call_args[0][0] + logger.info(f"Step 3: Invoice generated for {called_amount} sats.") + if called_amount != EXPECTED_SELLER_FEE_SATS: + logger.error(f"FAIL: Invoice amount mismatch: {called_amount}") + results["callback"] = False + elif not mock_accept.called: + logger.error("FAIL: Amboss accept_order mutation was not called!") + results["callback"] = False + else: + logger.info("✅ SUCCESS: Telegram callback approval completed end-to-end without KeyError!") + results["callback"] = True + + # -------------------------------------------------------------------------- + # SCENARIO 3: Telegram Logging Handler Error Stream + # -------------------------------------------------------------------------- + if scenario in ("all", "error"): + logger.info("\n" + "=" * 70) + logger.info("RUNNING SCENARIO 3: Telegram Logging Handler Error Stream") + logger.info("=" * 70) + + sim_logger = logging.getLogger("SimulatedComponent") + sim_logger.setLevel(logging.DEBUG) + + sent_messages = [] + if live_telegram: + logger.info("Step 1: Emitting live simulated error log to Telegram via TelegramLoggingHandler...") + try: + raise ValueError("Simulated Magma exception for end-to-end verification") + except Exception: + magma.logging.getLogger().error( + "🧪 [SIMULATION] Live test of TelegramLoggingHandler - verifying exception traceback dispatch.", + exc_info=True + ) + logger.info("Step 2: Dispatched error log to Telegram. Check your Telegram chat for the formatted alert.") + results["error"] = True + else: + logger.info("Step 1: Testing TelegramLoggingHandler emission with mock bot...") + mock_test_bot = MagicMock() + handler = magma.TelegramLoggingHandler(bot=mock_test_bot, chat_id="123456", level=logging.ERROR) + test_logger = logging.getLogger("TestErrorLogger") + test_logger.addHandler(handler) + + try: + raise KeyError("seller_invoice_amount") + except Exception: + test_logger.error("Testing simulated KeyError handling", exc_info=True) + + if not mock_test_bot.send_message.called: + logger.error("FAIL: TelegramLoggingHandler did not call send_message!") + results["error"] = False + else: + sent_text = mock_test_bot.send_message.call_args[1].get("text", "") + if "KeyError: 'seller_invoice_amount'" in sent_text and "🚨 *[ERROR]*" in sent_text: + logger.info("Step 2: Successfully verified markdown formatting and traceback inclusion.") + logger.info("✅ SUCCESS: TelegramLoggingHandler captured and formatted exception correctly.") + results["error"] = True + else: + logger.error(f"FAIL: Formatted message content unexpected: {sent_text[:100]}") + results["error"] = False + + # -------------------------------------------------------------------------- + # Summary Report + # -------------------------------------------------------------------------- + logger.info("\n" + "=" * 70) + logger.info("SIMULATION EXECUTION SUMMARY") + logger.info("=" * 70) + all_passed = True + for s_name, passed in results.items(): + status_str = "\033[1;32mPASSED\033[0m" if passed else "\033[1;31mFAILED\033[0m" + logger.info(f" • Scenario [{s_name}]: {status_str}") + if not passed: + all_passed = False + + if all_passed: + logger.info("\n🎉 All simulated scenarios completed successfully!") + else: + logger.error("\n❌ One or more scenarios failed.") + + return all_passed + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Magma Buy Order End-to-End Simulation") + parser.add_argument( + "--scenario", + choices=["all", "timeout", "callback", "error"], + default="all", + help="Which scenario to simulate (default: all)" + ) + parser.add_argument( + "--live-telegram", + action="store_true", + help="Dispatch live test messages to your Telegram chat using config.ini credentials" + ) + parser.add_argument( + "--timeout-seconds", + type=int, + default=6, + help="Simulated countdown in seconds before auto-approval (default: 6s)" + ) + + args = parser.parse_args() + success = run_simulation( + scenario=args.scenario, + live_telegram=args.live_telegram, + timeout_seconds=args.timeout_seconds + ) + sys.exit(0 if success else 1) From c4e2af1563fa110b596e56d099ef10c73e5de918 Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 10:51:24 +0200 Subject: [PATCH 08/12] ci: add workflow_dispatch and broaden branch matching in tests.yml --- .github/workflows/tests.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 267885c..c2ede4e 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -2,9 +2,10 @@ name: Tests on: push: - branches: [ main, feat/*, fix/* ] + branches: [ main, "feat/**", "fix/**" ] pull_request: branches: [ main ] + workflow_dispatch: jobs: test: From ae6302558f0f1f9d8281e42625378cc11d0bf68f Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 10:56:08 +0200 Subject: [PATCH 09/12] fix(magma): use fallback defaults for credentials so magma_sale_process imports cleanly in CI --- Magma/magma_sale_process.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index 4694d05..50917ed 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -80,12 +80,12 @@ BANNED_PUBKEYS = config.get("pubkey", "banned_magma_pubkeys", fallback="").split(",") -TOKEN = config["telegram"]["magma_bot_token"] -AMBOSS_TOKEN = config["credentials"]["amboss_authorization"] -CHAT_ID = config["telegram"]["telegram_user_id"] -bot = telebot.TeleBot(TOKEN) +TOKEN = config.get("telegram", "magma_bot_token", fallback="") +AMBOSS_TOKEN = config.get("credentials", "amboss_authorization", fallback="") +CHAT_ID = config.get("telegram", "telegram_user_id", fallback="") +bot = telebot.TeleBot(TOKEN if TOKEN else "fake_token") -FULL_PATH_BOS = config["system"]["full_path_bos"] +FULL_PATH_BOS = config.get("system", "full_path_bos", fallback="") LNCLI_PATH = config.get("paths", "lncli_path", fallback="lncli") From def8015402488a078e88dd2cb7bae60fbf507a2d Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 10:59:46 +0200 Subject: [PATCH 10/12] fix(magma): ensure fallback telegram bot token has valid format for headless CI --- Magma/magma_sale_process.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index 50917ed..845348e 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -83,7 +83,7 @@ TOKEN = config.get("telegram", "magma_bot_token", fallback="") AMBOSS_TOKEN = config.get("credentials", "amboss_authorization", fallback="") CHAT_ID = config.get("telegram", "telegram_user_id", fallback="") -bot = telebot.TeleBot(TOKEN if TOKEN else "fake_token") +bot = telebot.TeleBot(TOKEN if (TOKEN and ":" in TOKEN) else "123456:fake_token_for_testing") FULL_PATH_BOS = config.get("system", "full_path_bos", fallback="") LNCLI_PATH = config.get("paths", "lncli_path", fallback="lncli") From cdb0fa9e36d103af58c6f8997883cd44d388305f Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 11:01:27 +0200 Subject: [PATCH 11/12] fix(magma): eliminate redundant bot re-initialization and ensure valid token fallback for CI --- Magma/magma_sale_process.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index 845348e..67ba0ec 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -80,9 +80,9 @@ BANNED_PUBKEYS = config.get("pubkey", "banned_magma_pubkeys", fallback="").split(",") -TOKEN = config.get("telegram", "magma_bot_token", fallback="") -AMBOSS_TOKEN = config.get("credentials", "amboss_authorization", fallback="") -CHAT_ID = config.get("telegram", "telegram_user_id", fallback="") +TOKEN = config.get("telegram", "magma_bot_token", fallback="123456:fake_token_for_testing") +AMBOSS_TOKEN = config.get("credentials", "amboss_authorization", fallback="fake_auth") +CHAT_ID = config.get("telegram", "telegram_user_id", fallback="123456") bot = telebot.TeleBot(TOKEN if (TOKEN and ":" in TOKEN) else "123456:fake_token_for_testing") FULL_PATH_BOS = config.get("system", "full_path_bos", fallback="") @@ -336,7 +336,7 @@ def emit(self, record): """ # Code -bot = telebot.TeleBot(TOKEN) +# bot initialized at module startup (line 86) logging.info("Amboss Channel Open Bot Started") # --- Constants for active order polling --- From ec45ff82df63d3df38293eb94fd932b289ded11e Mon Sep 17 00:00:00 2001 From: Hakuna Date: Sun, 13 Sep 2026 12:40:36 +0200 Subject: [PATCH 12/12] docs: add test status banner to README and add repository security policy --- .github/SECURITY.md | 1 + README.md | 17 ++++++++++++----- SECURITY.md | 46 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 5 deletions(-) create mode 120000 .github/SECURITY.md create mode 100644 SECURITY.md diff --git a/.github/SECURITY.md b/.github/SECURITY.md new file mode 120000 index 0000000..9d57138 --- /dev/null +++ b/.github/SECURITY.md @@ -0,0 +1 @@ +../SECURITY.md \ No newline at end of file diff --git a/README.md b/README.md index 1199674..13ce459 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # ⚡ Lightning Python Tools +[![Tests](https://github.com/TrezorHannes/Lightning-Python-Tools/actions/workflows/tests.yml/badge.svg)](https://github.com/TrezorHannes/Lightning-Python-Tools/actions/workflows/tests.yml) +[![Security Policy](https://img.shields.io/badge/Security-Policy-blue.svg)](SECURITY.md) + A comprehensive suite of production-grade automation scripts, background daemons, liquidity optimizers, and monitoring utilities for **LND** Lightning Network node operators. --- @@ -18,7 +21,7 @@ A comprehensive suite of production-grade automation scripts, background daemons - [Systemd Daemon Example](#systemd-daemon-example) - [Crontab Example](#crontab-example) - [Testing & Quality Assurance](#-testing--quality-assurance) -- [Security Best Practices](#-security-best-practices) +- [Security & Vulnerability Reporting](#-security--vulnerability-reporting) --- @@ -185,7 +188,11 @@ pytest tests/ -v --- -## 🔒 Security Best Practices -- Never commit `config.ini`, `.env`, or credential files to Git. -- Restrict file permissions on `config.ini`: `chmod 600 config.ini`. -- Use read-only or restricted API keys whenever possible. +## 🔒 Security & Vulnerability Reporting + +For our full responsible disclosure policy, supported versions, and contact channels, please review [`SECURITY.md`](SECURITY.md). + +- **Vulnerability Disclosure**: Report security vulnerabilities privately via [GitHub Security Advisories](https://github.com/TrezorHannes/Lightning-Python-Tools/security/advisories) or email `security@tunnelsats.com`. +- **Secrets Management**: Never commit `config.ini`, `.env`, or credential files to Git. +- **Permissions**: Restrict file permissions on `config.ini`: `chmod 600 config.ini`. +- **Least Privilege**: Use read-only or scoped API keys and macaroons whenever possible. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..647bdc2 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,46 @@ +# Security Policy + +## Reporting a Vulnerability + +We take the security of **Lightning Python Tools** seriously. If you discover a potential vulnerability or security issue affecting automated channel operations, key management, RPC communication, or API integrations, we encourage you to report it responsibly. + +### How to Report + +Please **do not** open a public issue for security vulnerabilities. Instead, report them via one of the following methods: + +* **GitHub Security Advisory**: If enabled on the repository, submit a report directly via the [Security Advisories](https://github.com/TrezorHannes/Lightning-Python-Tools/security/advisories) tab. +* **Email**: Send details to `security@tunnelsats.com` or `hakuna@tunnelsats.com`. +* **Telegram**: Direct message an administrator or maintainer in our official community channel. +* **Nostr**: Send an encrypted direct message (NIP-04 / NIP-17) to `npub1n9z4y3xjramqes8fp9rl96x5e4nl0hff57ynw7vqnjpq370tq78sljsp8y`. + +### What to Include in Your Report + +To help us investigate and resolve the issue effectively, please include: + +* A clear description of the vulnerability and its potential impact. +* The specific component, script, or endpoint affected (e.g. `Magma/magma_sale_process.py`, `LNDg/amboss_pull.py`, `Other/fee_adjuster.py`). +* Step-by-step instructions or minimal proof-of-concept to reproduce the behavior. +* The affected commit or release version. +* Any suggested mitigations or patches, if available. + +### Our Commitment + +* **Acknowledgment**: We will acknowledge receipt of your report within 48 hours. +* **Investigation & Triage**: We will investigate and coordinate with you on fix verification. +* **Responsible Release**: Patches will be developed, tested, and released to `main` before public disclosure. +* **Credit**: We appreciate responsible disclosure and will credit reporters in our release notes (unless anonymity is requested). + +### Supported Versions + +Security updates are applied to the latest code on the primary branch: + +| Version / Branch | Supported | +| :--- | :--- | +| `main` | :white_check_mark: Active security support | +| Older commits | :warning: Not guaranteed; operators are advised to upgrade to `main` | + +### Operator Security Guidelines + +* Restrict configuration permissions: `chmod 600 config.ini`. +* Never commit secrets, bot tokens, or LND macaroons to version control. +* Use scoped LND macaroons and least-privilege credentials.