diff --git a/README.rst b/README.rst
index 800c5df..a64f422 100644
--- a/README.rst
+++ b/README.rst
@@ -8,48 +8,12 @@ Python C API compatibility
The ``pythoncapi-compat`` project can be used to write a C or C++ extension
supporting a wide range of Python versions with a single code base. It is made
-of the ``pythoncapi_compat.h`` header file and the ``upgrade_pythoncapi.py``
-script.
-
-``upgrade_pythoncapi.py`` requires Python 3.6 or newer.
+of the ``pythoncapi_compat.h`` header file.
See the `documentation at ReadTheDocs
`_
for more details.
-Getting started
-===============
-
-To upgrade a specific file::
-
- python3 upgrade_pythoncapi.py module.c
-
-To upgrade all C/C++ files in a directory::
-
- python3 upgrade_pythoncapi.py src/
-
-Select operations
------------------
-
-To only replace ``op->ob_type`` with ``Py_TYPE(op)``, select the ``Py_TYPE``
-operation with::
-
- python3 upgrade_pythoncapi.py -o Py_TYPE module.c
-
-Or the opposite, to apply all operations but leave ``op->ob_type`` unchanged,
-deselect the ``Py_TYPE`` operation with::
-
- python3 upgrade_pythoncapi.py -o all,-Py_TYPE module.c
-
-Download pythoncapi_compat.h
-----------------------------
-
-If you want to ``pythoncapi_compat.h`` to your code base, use the
-``upgrade_pythoncapi.py`` tool to fetch it::
-
- python3 upgrade_pythoncapi.py --download PATH
-
-
This project is distributed under the `Zero Clause BSD (0BSD) license
`_ and is covered by the `PSF Code of
Conduct `_.
diff --git a/docs/changelog.rst b/docs/changelog.rst
index 02cdd72..86dcbed 100644
--- a/docs/changelog.rst
+++ b/docs/changelog.rst
@@ -1,6 +1,7 @@
Changelog
=========
+* 2026-10-11: Remove the ``upgrade_pythoncapi.py`` script.
* 2026-10-10: Add ``PyObject_GenericHash()`` and
``PyUnicodeWriter_WriteUCS4()`` functions. runtests.py now runs tests in
parallel.
diff --git a/docs/index.rst b/docs/index.rst
index 484f7cf..a093204 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -4,8 +4,7 @@ Python C API compatibility
The ``pythoncapi-compat`` project can be used to write a C or C++ extension
supporting a wide range of Python versions with a single code base. It is made
-of the ``pythoncapi_compat.h`` header file and the ``upgrade_pythoncapi.py``
-script.
+of the ``pythoncapi_compat.h`` header file.
* Homepage: `GitHub pythoncapi-compat project
`_.
@@ -18,7 +17,6 @@ Documentation:
:maxdepth: 2
api
- upgrade
users
tests
changelog
diff --git a/docs/upgrade.rst b/docs/upgrade.rst
deleted file mode 100644
index 10aa97a..0000000
--- a/docs/upgrade.rst
+++ /dev/null
@@ -1,168 +0,0 @@
-++++++++++++++++++++++++++++
-upgrade_pythoncapi.py script
-++++++++++++++++++++++++++++
-
-``upgrade_pythoncapi.py`` requires Python 3.6 or newer.
-
-Usage
-=====
-
-Run the with no arguments to display command line options and list available
-operations::
-
- python3 upgrade_pythoncapi.py
-
-Select files and directories
-----------------------------
-
-To upgrade ``module.c`` file, type::
-
- python3 upgrade_pythoncapi.py module.c
-
-To upgrade all C and C++ files (``.c``, ``.h``, ``.cc``, ``.cpp``, ``.cxx`` and
-``.hpp`` files) in the ``directory/`` directory, type::
-
- python3 upgrade_pythoncapi.py directory/
-
-Multiple filenames an directories can be specified on the command line.
-
-Files are modified in-place! If a file is modified, a copy of the original file
-is created with the ``.old`` suffix.
-
-Select operations
------------------
-
-To only replace ``op->ob_type`` with ``Py_TYPE(op)``, select the ``Py_TYPE``
-operation with::
-
- python3 upgrade_pythoncapi.py -o Py_TYPE module.c
-
-Or the opposite, to apply all operations but leave ``op->ob_type`` unchanged,
-deselect the ``Py_TYPE`` operation with::
-
- python3 upgrade_pythoncapi.py -o all,-Py_TYPE module.c
-
-Download pythoncapi_compat.h
-----------------------------
-
-Most ``upgrade_pythoncapi.py`` operations add ``#include
-"pythoncapi_compat.h"``. You may have to download the ``pythoncapi_compat.h``
-header file to your project. It can be downloaded with::
-
- python3 upgrade_pythoncapi.py --download PATH
-
-
-Upgrade Operations
-==================
-
-``upgrade_pythoncapi.py`` implements the following operations:
-
-Py_TYPE
--------
-
-* Replace ``op->ob_type`` with ``Py_TYPE(op)``.
-
-Py_SIZE
--------
-
-* Replace ``op->ob_size`` with ``Py_SIZE(op)``.
-
-Py_REFCNT
----------
-
-* Replace ``op->ob_refcnt`` with ``Py_REFCNT(op)``.
-
-Py_SET_TYPE
------------
-
-* Replace ``obj->ob_type = type;`` with ``Py_SET_TYPE(obj, type);``.
-* Replace ``Py_TYPE(obj) = type;`` with ``Py_SET_TYPE(obj, type);``.
-
-Py_SET_SIZE
------------
-
-* Replace ``obj->ob_size = size;`` with ``Py_SET_SIZE(obj, size);``.
-* Replace ``Py_SIZE(obj) = size;`` with ``Py_SET_SIZE(obj, size);``.
-
-Py_SET_REFCNT
--------------
-
-* Replace ``obj->ob_refcnt = refcnt;`` with ``Py_SET_REFCNT(obj, refcnt);``.
-* Replace ``Py_REFCNT(obj) = refcnt;`` with ``Py_SET_REFCNT(obj, refcnt);``.
-
-Py_Is
------
-
-* Replace ``x == Py_None`` with ``Py_IsNone(x)``.
-* Replace ``x == Py_True`` with ``Py_IsTrue(x)``.
-* Replace ``x == Py_False`` with ``Py_IsFalse(x)``.
-* Replace ``x != Py_None`` with ``!Py_IsNone(x)``.
-* Replace ``x != Py_True`` with ``!Py_IsTrue(x)``.
-* Replace ``x != Py_False`` with ``!Py_IsFalse(x)``.
-
-PyObject_NEW
-------------
-
-* Replace ``PyObject_NEW(...)`` with ``PyObject_New(...)``.
-* Replace ``PyObject_NEW_VAR(...)`` with ``PyObject_NewVar(...)``.
-
-PyMem_MALLOC
-------------
-
-* Replace ``PyMem_MALLOC(n)`` with ``PyMem_Malloc(n)``.
-* Replace ``PyMem_REALLOC(ptr, n)`` with ``PyMem_Realloc(ptr, n)``.
-* Replace ``PyMem_FREE(ptr)``, ``PyMem_DEL(ptr)`` and ``PyMem_Del(ptr)`` .
- with ``PyMem_Free(n)``.
-
-PyObject_MALLOC
----------------
-
-* Replace ``PyObject_MALLOC(n)`` with ``PyObject_Malloc(n)``.
-* Replace ``PyObject_REALLOC(ptr, n)`` with ``PyObject_Realloc(ptr, n)``.
-* Replace ``PyObject_FREE(ptr)``, ``PyObject_DEL(ptr)``
- and ``PyObject_Del(ptr)`` . with ``PyObject_Free(n)``.
-
-PyFrame_GetBack
----------------
-
-* Replace ``frame->f_back`` with ``_PyFrame_GetBackBorrow(frame)``.
-
-PyFrame_GetCode
----------------
-
-* Replace ``frame->f_code`` with ``_PyFrame_GetCodeBorrow(frame)``.
-
-PyThreadState_GetInterpreter
-----------------------------
-
-* Replace ``tstate->interp`` with ``PyThreadState_GetInterpreter(tstate)``.
-
-PyThreadState_GetFrame
-----------------------
-
-* Replace ``tstate->frame`` with ``_PyThreadState_GetFrameBorrow(tstate)``.
-
-Experimental operations
------------------------
-
-The following operations are experimental (ex: can introduce compiler warnings)
-and so not included in the ``all`` group, they have to be selected explicitly.
-Example: ``-o all,Py_SETREF``.
-
-Experimental operations:
-
-* ``Py_NewRef``:
-
- * Replace ``Py_INCREF(res); return res;`` with ``return Py_NewRef(res);``
- * Replace ``x = y; Py_INCREF(x);`` with ``x = Py_NewRef(y);``
- * Replace ``x = y; Py_INCREF(y);`` with ``x = Py_NewRef(y);``
- * Replace ``Py_INCREF(y); x = y;`` with ``x = Py_NewRef(y);``
-
-* ``Py_CLEAR``:
-
- * Replace ``Py_XDECREF(var); var = NULL;`` with ``Py_CLEAR(var);``
-
-* ``Py_SETREF``:
-
- * Replace ``Py_DECREF(x); x = y;`` with ``Py_SETREF(x, y);``
- * Replace ``Py_XDECREF(x); x = y;`` with ``Py_XSETREF(x, y);``
diff --git a/runtests.py b/runtests.py
index 04ea8a3..d18d320 100755
--- a/runtests.py
+++ b/runtests.py
@@ -20,7 +20,6 @@
TEST_DIR = os.path.join(os.path.dirname(__file__), 'tests')
TEST_COMPAT = os.path.join(TEST_DIR, "test_pythoncapi_compat.py")
-TEST_UPGRADE = os.path.join(TEST_DIR, "test_upgrade_pythoncapi.py")
PYTHONS = (
# CPython
@@ -123,16 +122,6 @@ def worker(executable):
print(f"Tested: {len(jobs)} Python executables")
-def test_upgrade_pythoncapi(args):
- # upgrade_pythoncapi.py requires Python 3.6 or newer
- print(f"Run {TEST_UPGRADE}")
- cmd = [sys.executable, "-u", TEST_UPGRADE]
- if args.verbose:
- cmd.append('-v')
- run_command(cmd)
- print()
-
-
def test_current_version(args):
with tempfile.TemporaryDirectory() as build_dir:
cmd = get_test_command(sys.executable, args.verbose, build_dir)
@@ -144,8 +133,6 @@ def main():
start_time = time.perf_counter()
args = parse_args()
- test_upgrade_pythoncapi(args)
-
if not args.current:
run_tests_parallel(args)
else:
diff --git a/tests/test_upgrade_pythoncapi.py b/tests/test_upgrade_pythoncapi.py
deleted file mode 100644
index 48bad12..0000000
--- a/tests/test_upgrade_pythoncapi.py
+++ /dev/null
@@ -1,962 +0,0 @@
-#!/usr/bin/env python3
-import io
-import os
-import sys
-import tempfile
-import textwrap
-import unittest
-
-# Get upgrade_pythoncapi.py of the parent directory
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
-import upgrade_pythoncapi # noqa
-
-
-def operations(disable=None):
- if isinstance(disable, str):
- disable = (disable,)
- elif not disable:
- disable = ()
- operations = ["all"]
- for op in upgrade_pythoncapi.EXCLUDE_FROM_ALL:
- if op.NAME in disable:
- continue
- operations.append(op.NAME)
- for name in disable:
- operations.append(f'-{name}')
- operations = ','.join(operations)
- return operations
-
-
-def patch(source, no_compat=False, disable=None):
- args = ['script', 'mod.c', '-o', operations(disable=disable)]
- if no_compat:
- args.append('--no-compat')
-
- patcher = upgrade_pythoncapi.Patcher(args)
- return patcher.patch(source)
-
-
-def reformat(source):
- return textwrap.dedent(source).strip()
-
-
-class Tests(unittest.TestCase):
- maxDiff = 80 * 30
-
- def _patch_file(self, source, tmp_dir=None):
- # test Patcher.patcher()
- filename = tempfile.mktemp(suffix='.c', dir=tmp_dir)
- old_filename = filename + ".old"
- try:
- with open(filename, "w", encoding="utf-8", newline="") as fp:
- fp.write(source)
-
- old_stderr = sys.stderr
- old_argv = list(sys.argv)
- try:
- # redirect stderr
- sys.stderr = io.StringIO()
-
- if tmp_dir is not None:
- arg = tmp_dir
- else:
- arg = filename
- sys.argv = ['script', arg]
- try:
- upgrade_pythoncapi.Patcher().main()
- except SystemExit as exc:
- self.assertEqual(exc.code, 0)
- else:
- self.fail("SystemExit not raised")
- finally:
- sys.stderr = old_stderr
- sys.argv = old_argv
-
- with open(filename, encoding="utf-8", newline="") as fp:
- new_contents = fp.read()
-
- with open(old_filename, encoding="utf-8", newline="") as fp:
- old_contents = fp.read()
- finally:
- try:
- os.unlink(filename)
- except FileNotFoundError:
- pass
- try:
- os.unlink(old_filename)
- except FileNotFoundError:
- pass
-
- self.assertEqual(old_contents, source)
- return new_contents
-
- def test_patch_file(self):
- source = """
- PyTypeObject*
- test_type(PyObject *obj, PyTypeObject *type)
- {
- Py_TYPE(obj) = type;
- return Py_TYPE(obj);
- }
- """
- expected = """
- #include "pythoncapi_compat.h"
-
- PyTypeObject*
- test_type(PyObject *obj, PyTypeObject *type)
- {
- Py_SET_TYPE(obj, type);
- return Py_TYPE(obj);
- }
- """
- source = reformat(source)
- expected = reformat(expected)
-
- new_contents = self._patch_file(source)
- self.assertEqual(new_contents, expected)
-
- with tempfile.TemporaryDirectory() as tmp_dir:
- new_contents = self._patch_file(source, tmp_dir)
- self.assertEqual(new_contents, expected)
-
- def test_patch_file_preserve_newlines(self):
- source = """
- Py_ssize_t get_size(PyVarObject *obj)\r\n\
- \n\
- { return obj->ob_size; }\r\
- """
- expected = """
- Py_ssize_t get_size(PyVarObject *obj)\r\n\
- \n\
- { return Py_SIZE(obj); }\r\
- """
- source = reformat(source)
- expected = reformat(expected)
- new_contents = self._patch_file(source)
- self.assertEqual(new_contents, expected)
-
- def check_replace(self, source, expected, **kwargs):
- source = reformat(source)
- expected = reformat(expected)
- self.assertEqual(patch(source, **kwargs), expected)
-
- def check_dont_replace(self, source, disable=None):
- source = reformat(source)
- self.assertEqual(patch(source, disable=disable), source)
-
- def test_expr_regex(self):
- # Test EXPR_REGEX
- self.check_replace("a->b->ob_type", "Py_TYPE(a->b)")
- self.check_replace("a.b->ob_type", "Py_TYPE(a.b)")
- self.check_replace("array[2]->ob_type", "Py_TYPE(array[2])")
-
- # Don't match function calls
- self.check_dont_replace("func()->ob_type")
-
- def test_pythoncapi_compat(self):
- # If pythoncapi_compat.h is included, avoid compatibility includes
- # and macros.
- #
- # Otherise, Py_SET_TYPE() requires a macro and PyFrame_GetBack()
- # requires 2 macros and an include.
- HEADERS = (
- '',
- '"pythoncapi_compat.h"',
- )
- for header in HEADERS:
- # There is no empty line between the include and the function
- # on purpose.
- self.check_replace("""
- #include %s
- void test_set_type(PyObject *obj, PyTypeObject *type)
- {
- Py_TYPE(obj) = type;
- }
- PyFrameObject* frame_back_borrowed(PyFrameObject *frame)
- {
- return frame->f_back;
- }
- """ % header, """
- #include %s
- void test_set_type(PyObject *obj, PyTypeObject *type)
- {
- Py_SET_TYPE(obj, type);
- }
- PyFrameObject* frame_back_borrowed(PyFrameObject *frame)
- {
- return _PyFrame_GetBackBorrow(frame);
- }
- """ % header)
-
- def test_py_type(self):
- source = """
- PyTypeObject* get_type(PyObject *obj)
- { return obj->ob_type; }
- """
- expected = """
- PyTypeObject* get_type(PyObject *obj)
- { return Py_TYPE(obj); }
- """
- self.check_replace(source, expected)
-
- def test_py_size(self):
- source = """
- Py_ssize_t get_size(PyVarObject *obj)
- { return obj->ob_size; }
- """
- expected = """
- Py_ssize_t get_size(PyVarObject *obj)
- { return Py_SIZE(obj); }
- """
- self.check_replace(source, expected)
-
- def test_py_refcnt(self):
- source = """
- Py_ssize_t get_refcnt(PyObject *obj)
- { return obj->ob_refcnt; }
- """
- expected = """
- Py_ssize_t get_refcnt(PyObject *obj)
- { return Py_REFCNT(obj); }
- """
- self.check_replace(source, expected)
-
- def test_py_set_type(self):
- source = """
- void test_type(PyObject *obj, PyTypeObject *type)
- {
- obj->ob_type = type;
- Py_TYPE(obj) = type;
- }
- """
- expected = """
- #include "pythoncapi_compat.h"
-
- void test_type(PyObject *obj, PyTypeObject *type)
- {
- Py_SET_TYPE(obj, type);
- Py_SET_TYPE(obj, type);
- }
- """
- self.check_replace(source, expected)
-
- self.check_dont_replace("""
- PyTypeObject* get_type(PyObject *obj, PyTypeObject *check_type)
- {
- assert(Py_TYPE(args) == check_type);
- return Py_TYPE(obj);
- }
- """)
-
- def test_py_set_size(self):
- source = """\
- void test_size(PyVarObject *obj)
- {
- obj->ob_size = 3;
- Py_SIZE(obj) = 4;
- }
- """
- expected = """\
- #include "pythoncapi_compat.h"
-
- void test_size(PyVarObject *obj)
- {
- Py_SET_SIZE(obj, 3);
- Py_SET_SIZE(obj, 4);
- }
- """
- self.check_replace(source, expected)
-
- self.check_dont_replace("""
- Py_ssize_t
- get_size(PyObject *obj)
- {
- assert(Py_SIZE(args) == 1);
- return Py_SIZE(obj);
- }
- """)
-
- def test_py_set_refcnt(self):
- source = """\
- void set_refcnt(PyObject *obj)
- {
- obj->ob_refcnt = 1;
- Py_REFCNT(obj) = 2;
- }
- """
- expected = """\
- #include "pythoncapi_compat.h"
-
- void set_refcnt(PyObject *obj)
- {
- Py_SET_REFCNT(obj, 1);
- Py_SET_REFCNT(obj, 2);
- }
- """
- self.check_replace(source, expected)
-
- self.check_dont_replace("""
- Py_ssize_t
- get_refcnt(PyObject *obj)
- {
- assert(Py_REFCNT(args) == 1);
- return Py_REFCNT(obj);
- }
- """)
-
- def test_pyobject_new(self):
- source = """\
- capsule = PyObject_NEW(PyCapsule, &PyCapsule_Type);
- pattern = PyObject_NEW_VAR(PatternObject, Pattern_Type, n);
- """
- expected = """\
- capsule = PyObject_New(PyCapsule, &PyCapsule_Type);
- pattern = PyObject_NewVar(PatternObject, Pattern_Type, n);
- """
- self.check_replace(source, expected)
-
- self.check_dont_replace("""
- func = PyObject_NEW;
- capsule2 = PyObject_NEW2(PyCapsule, &PyCapsule_Type);
-
- func2 = PyObject_NEW_VAR;
- pattern2 = PyObject_NEW_VAR2(PatternObject, Pattern_Type, n);
- """)
-
- def test_pymem_malloc(self):
- source = """\
- void *ptr = PyMem_MALLOC(10);
- ptr = PyMem_REALLOC(ptr, 20);
- PyMem_FREE(ptr);
- PyMem_DEL(ptr);
- PyMem_Del(ptr);
- """
- expected = """\
- void *ptr = PyMem_Malloc(10);
- ptr = PyMem_Realloc(ptr, 20);
- PyMem_Free(ptr);
- PyMem_Free(ptr);
- PyMem_Free(ptr);
- """
- self.check_replace(source, expected)
-
- def test_pyobject_malloc(self):
- source = """\
- void *ptr = PyObject_MALLOC(10);
- ptr = PyObject_REALLOC(ptr, 20);
- PyObject_FREE(ptr);
- PyObject_DEL(ptr);
- PyObject_Del(ptr);
- """
- expected = """\
- void *ptr = PyObject_Malloc(10);
- ptr = PyObject_Realloc(ptr, 20);
- PyObject_Free(ptr);
- PyObject_Free(ptr);
- PyObject_Free(ptr);
- """
- self.check_replace(source, expected)
-
- def test_pyframe_getback(self):
- source = """\
- PyFrameObject* frame_back_borrowed(PyFrameObject *frame)
- {
- return frame->f_back;
- }
- """
- expected = """\
- #include "pythoncapi_compat.h"
-
- PyFrameObject* frame_back_borrowed(PyFrameObject *frame)
- {
- return _PyFrame_GetBackBorrow(frame);
- }
- """
- self.check_replace(source, expected)
-
- def test_pyframe_getcode(self):
- self.check_replace("""\
- PyCodeObject* frame_code_borrowed(PyFrameObject *frame)
- {
- return frame->f_code;
- }
- """, """\
- #include "pythoncapi_compat.h"
-
- PyCodeObject* frame_code_borrowed(PyFrameObject *frame)
- {
- return _PyFrame_GetCodeBorrow(frame);
- }
- """)
-
- def test_get_member_regex(self):
- # Use PyFrame_GetCode() to test get_member_regex()
- self.check_dont_replace("""
- void frame_set_code(PyFrameObject *frame, PyCodeObject *code)
- {
- frame->f_code = code;
- }
-
- void frame_clear_code(PyFrameObject *frame)
- {
- Py_CLEAR(frame->f_code);
- }
- """)
-
- def test_pythreadstate_getinterpreter(self):
- self.check_replace("""
- PyInterpreterState* get_interp(PyThreadState *tstate)
- { return tstate->interp; }
- """, """
- #include "pythoncapi_compat.h"
-
- PyInterpreterState* get_interp(PyThreadState *tstate)
- { return PyThreadState_GetInterpreter(tstate); }
- """)
-
- def test_pythreadstate_getframe(self):
- self.check_replace("""
- PyFrameObject* get_frame(PyThreadState *tstate)
- { return tstate->frame; }
- """, """
- #include "pythoncapi_compat.h"
-
- PyFrameObject* get_frame(PyThreadState *tstate)
- { return _PyThreadState_GetFrameBorrow(tstate); }
- """)
-
- def test_py_newref_return(self):
- self.check_replace("""
- PyObject* new_ref(PyObject *obj) {
- Py_INCREF(obj);
- return obj;
- }
-
- PyObject* same_line(PyObject *obj) {
- Py_INCREF(obj); return obj;
- }
-
- PyObject* new_xref(PyObject *obj) {
- Py_XINCREF(obj);
- return obj;
- }
-
- PyObject* cast(PyLongObject *obj) {
- Py_XINCREF(obj);
- return (PyObject *)obj;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- PyObject* new_ref(PyObject *obj) {
- return Py_NewRef(obj);
- }
-
- PyObject* same_line(PyObject *obj) {
- return Py_NewRef(obj);
- }
-
- PyObject* new_xref(PyObject *obj) {
- return Py_XNewRef(obj);
- }
-
- PyObject* cast(PyLongObject *obj) {
- return Py_XNewRef(obj);
- }
- """)
-
- def test_py_newref(self):
- # INCREF, assign
- self.check_replace("""
- void set_attr(MyStruct *obj, PyObject *value, int test)
- {
- // 1
- Py_INCREF(value);
- obj->attr = value;
- // 2
- obj->attr = value;
- Py_INCREF(value);
- // 3
- obj->attr = value;
- Py_INCREF(obj->attr);
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set_attr(MyStruct *obj, PyObject *value, int test)
- {
- // 1
- obj->attr = Py_NewRef(value);
- // 2
- obj->attr = Py_NewRef(value);
- // 3
- obj->attr = Py_NewRef(value);
- }
- """)
-
- # Same line
- self.check_replace("""
- void set_attr(MyStruct *obj, PyObject *value, int test)
- {
- // same line 1
- obj->attr = value; Py_INCREF(value);
- // same line 2
- if (test) { obj->attr = value; Py_INCREF(obj->attr); }
- // same line 3
- if (test) { Py_INCREF(value); obj->attr = value; }
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set_attr(MyStruct *obj, PyObject *value, int test)
- {
- // same line 1
- obj->attr = Py_NewRef(value);
- // same line 2
- if (test) { obj->attr = Py_NewRef(value); }
- // same line 3
- if (test) { obj->attr = Py_NewRef(value); }
- }
- """)
-
- # Cast
- self.check_replace("""
- void set_attr(MyStruct *obj, PyObject *value, int test)
- {
- // cast 1
- Py_INCREF(value);
- obj->attr = (PyObject*)value;
- // cast 2
- obj->attr = (PyObject*)value;
- Py_INCREF(value);
-
- // assign var, incref
- PyCodeObject *code_obj = (PyCodeObject *)code;
- Py_INCREF(code_obj);
- // assign var, incref
- PyCodeObject* code_obj = (PyCodeObject *)code;
- Py_INCREF(code);
- // assign var, xincref
- PyCodeObject * code_obj = (PyCodeObject *)code;
- Py_XINCREF(code_obj);
-
- // incref, assign var
- Py_INCREF(code);
- PyCodeObject* code_obj = (PyCodeObject *)code;
- // xincref, assign var
- Py_XINCREF(code);
- PyCodeObject *code_obj = (PyCodeObject *)code;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set_attr(MyStruct *obj, PyObject *value, int test)
- {
- // cast 1
- obj->attr = Py_NewRef(value);
- // cast 2
- obj->attr = Py_NewRef(value);
-
- // assign var, incref
- PyCodeObject *code_obj = (PyCodeObject *)Py_NewRef(code);
- // assign var, incref
- PyCodeObject* code_obj = (PyCodeObject *)Py_NewRef(code);
- // assign var, xincref
- PyCodeObject * code_obj = (PyCodeObject *)Py_XNewRef(code);
-
- // incref, assign var
- PyCodeObject* code_obj = (PyCodeObject *)Py_NewRef(code);
- // xincref, assign var
- PyCodeObject *code_obj = (PyCodeObject *)Py_XNewRef(code);
- }
- """)
-
- # Py_XINCREF
- self.check_replace("""
- void set_xattr(MyStruct *obj, PyObject *value)
- {
- // 1
- Py_XINCREF(value);
- obj->attr = value;
- // 2
- obj->attr = value;
- Py_XINCREF(value);
- // 3
- obj->attr = value;
- Py_XINCREF(obj->attr);
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set_xattr(MyStruct *obj, PyObject *value)
- {
- // 1
- obj->attr = Py_XNewRef(value);
- // 2
- obj->attr = Py_XNewRef(value);
- // 3
- obj->attr = Py_XNewRef(value);
- }
- """)
-
- # the first Py_INCREF should be replaced before the second one,
- # otherwise the first Py_INCREF is not replaced.
- self.check_replace("""
- void set(void)
- {
- PyObject *x, *y;
- Py_INCREF(Py_None);
- x = Py_None;
- Py_INCREF(Py_None);
- x = Py_None;
- Py_DECREF(x);
- Py_DECREF(y);
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set(void)
- {
- PyObject *x, *y;
- x = Py_NewRef(Py_None);
- x = Py_NewRef(Py_None);
- Py_DECREF(x);
- Py_DECREF(y);
- }
- """)
-
- # Indentation matters for conditional code
- self.check_dont_replace("""
- void test1(int test)
- {
- PyObject *res;
- if (test)
- res = Py_True;
- else
- res = Py_False;
- Py_INCREF(res);
-
- Py_DECREF(res);
- }
-
- int test2(struct datetime* result, PyObject *tzinfo)
- {
- int res = 0;
- if (test)
- res = 1;
- else
- Py_INCREF(tzinfo);
- result->tzinfo = tzinfo;
- return res;
- }
- """)
-
- def test_py_clear(self):
- self.check_replace("""
- void clear(int test)
- {
- PyObject *obj;
-
- // two lines
- Py_XDECREF(obj);
- obj = NULL;
-
- // inside if
- if (test) { Py_XDECREF(obj); obj = NULL; }
- }
- """, """
- void clear(int test)
- {
- PyObject *obj;
-
- // two lines
- Py_CLEAR(obj);
-
- // inside if
- if (test) { Py_CLEAR(obj); }
- }
- """)
-
- # Don't replace Py_DECREF()
- self.check_dont_replace("""
- void dont_clear(void)
- {
- PyObject *obj;
- Py_DECREF(obj);
- obj = NULL;
- }
- """, disable="Py_SETREF")
-
- def test_py_setref(self):
- self.check_replace("""
- void set(PyObject **obj, PyObject *t)
- {
- // DECREF
- Py_DECREF(*obj);
- *obj = t;
-
- // XDECREF
- Py_XDECREF(*obj);
- *obj = t;
-
- // DECREF, INCREF
- Py_DECREF(*obj);
- Py_INCREF(t);
- *obj = t;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set(PyObject **obj, PyObject *t)
- {
- // DECREF
- Py_SETREF(*obj, t);
-
- // XDECREF
- Py_XSETREF(*obj, t);
-
- // DECREF, INCREF
- Py_SETREF(*obj, Py_NewRef(t));
- }
- """)
-
- self.check_replace("""
- void set(PyObject **obj, PyObject *value)
- {
- // 1
- PyObject *old = *obj;
- *obj = value;
- Py_DECREF(old);
- // 2
- PyObject *old = *obj;
- *obj = Py_XNewRef(value);
- Py_DECREF(old);
- // 3
- PyObject *old = *obj;
- *obj = value;
- Py_XDECREF(old);
- // 4
- PyObject *old = *obj;
- *obj = Py_NewRef(value);
- Py_XDECREF(old);
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set(PyObject **obj, PyObject *value)
- {
- // 1
- Py_SETREF(*obj, value);
- // 2
- Py_SETREF(*obj, Py_XNewRef(value));
- // 3
- Py_XSETREF(*obj, value);
- // 4
- Py_XSETREF(*obj, Py_NewRef(value));
- }
- """)
-
- # INCREF, DECREF, assign
- self.check_replace("""
- void set(void)
- {
- // 1
- Py_INCREF(value);
- Py_DECREF(obj);
- obj = value;
- // 2
- Py_INCREF(value);
- Py_XDECREF(obj);
- obj = value;
- // 3
- Py_XINCREF(value);
- Py_DECREF(obj);
- obj = value;
- // 4
- Py_XINCREF(value);
- Py_XDECREF(obj);
- obj = value;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set(void)
- {
- // 1
- Py_SETREF(obj, Py_NewRef(value));
- // 2
- Py_XSETREF(obj, Py_NewRef(value));
- // 3
- Py_SETREF(obj, Py_XNewRef(value));
- // 4
- Py_XSETREF(obj, Py_XNewRef(value));
- }
- """)
-
- # old variable
- self.check_replace("""
- void set(PyObject **obj, PyObject *value)
- {
- // 1
- PyObject *old_next = (PyObject*)self->tb_next;
- self->tb_next = (PyTracebackObject *)Py_XNewRef(new_next);
- Py_XDECREF(old_next);
- // 2
- old_next = (PyObject*)self->tb_next;
- self->tb_next = (PyTracebackObject *)Py_XNewRef(new_next);
- Py_XDECREF(old_next);
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set(PyObject **obj, PyObject *value)
- {
- // 1
- Py_XSETREF(self->tb_next, (PyTracebackObject *)Py_XNewRef(new_next));
- // 2
- Py_XSETREF(self->tb_next, (PyTracebackObject *)Py_XNewRef(new_next));
- }
- """)
-
- # Py_CLEAR
- self.check_replace("""
- void set(PyObject **obj, PyObject *value)
- {
- // 1
- Py_CLEAR(self->tb_next);
- self->tb_next = value;
- // 2
- Py_INCREF(value);
- Py_CLEAR(self->tb_next);
- self->tb_next = value;
- // 3
- Py_XINCREF(value);
- Py_CLEAR(self->tb_next);
- self->tb_next = value;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void set(PyObject **obj, PyObject *value)
- {
- // 1
- Py_XSETREF(self->tb_next, value);
- // 2
- Py_XSETREF(self->tb_next, Py_NewRef(value));
- // 3
- Py_XSETREF(self->tb_next, Py_XNewRef(value));
- }
- """)
-
- def test_py_is(self):
- self.check_replace("""
- void test_py_is(PyObject *x)
- {
- if (x == Py_None) {
- return 1;
- }
- if (x == Py_True) {
- return 2;
- }
- if (x == Py_False) {
- return 3;
- }
- return 0;
- }
-
- void test_py_is_not(PyObject *x)
- {
- if (x != Py_None) {
- return 1;
- }
- if (x != Py_True) {
- return 2;
- }
- if (x != Py_False) {
- return 3;
- }
- return 0;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void test_py_is(PyObject *x)
- {
- if (Py_IsNone(x)) {
- return 1;
- }
- if (Py_IsTrue(x)) {
- return 2;
- }
- if (Py_IsFalse(x)) {
- return 3;
- }
- return 0;
- }
-
- void test_py_is_not(PyObject *x)
- {
- if (!Py_IsNone(x)) {
- return 1;
- }
- if (!Py_IsTrue(x)) {
- return 2;
- }
- if (!Py_IsFalse(x)) {
- return 3;
- }
- return 0;
- }
- """)
-
- self.check_replace("""
- void test_expr(struct MyStruct *obj, PyObject **obj2)
- {
- if (obj->attr1 == Py_None) {
- return 1;
- }
- if (obj->attr2.name == Py_None) {
- return 1;
- }
- if (*obj2 == Py_None) {
- return 1;
- }
- return 0;
- }
- """, """
- #include "pythoncapi_compat.h"
-
- void test_expr(struct MyStruct *obj, PyObject **obj2)
- {
- if (Py_IsNone(obj->attr1)) {
- return 1;
- }
- if (Py_IsNone(obj->attr2.name)) {
- return 1;
- }
- if (Py_IsNone(*obj2)) {
- return 1;
- }
- return 0;
- }
- """)
-
-
- def test_no_compat(self):
- # Don't add "#include "pythoncapi_compat.h"
- source = """
- void test_type(PyObject *obj, PyTypeObject *type)
- {
- obj->ob_type = type;
- }
- """
- expected = """
- void test_type(PyObject *obj, PyTypeObject *type)
- {
- Py_SET_TYPE(obj, type);
- }
- """
- self.check_replace(source, expected, no_compat=True)
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/upgrade_pythoncapi.py b/upgrade_pythoncapi.py
deleted file mode 100755
index 4348e70..0000000
--- a/upgrade_pythoncapi.py
+++ /dev/null
@@ -1,752 +0,0 @@
-#!/usr/bin/env python3
-import argparse
-import os
-import re
-import urllib.request
-import sys
-
-
-MIN_PYTHON = (2, 7)
-
-
-PYTHONCAPI_COMPAT_URL = ('https://raw.githubusercontent.com/python/'
- 'pythoncapi-compat/main/pythoncapi_compat.h')
-PYTHONCAPI_COMPAT_H = 'pythoncapi_compat.h'
-INCLUDE_PYTHONCAPI_COMPAT = f'#include "{PYTHONCAPI_COMPAT_H}"'
-INCLUDE_PYTHONCAPI_COMPAT2 = f'#include <{PYTHONCAPI_COMPAT_H}>'
-
-C_FILE_EXT = (
- # C language
- ".c", ".h",
- # C++ language
- ".cc", ".cpp", ".cxx", ".hpp",
-)
-IGNORE_DIRS = (".git", ".tox")
-
-
-# Match spaces but not newline characters.
-# Similar to \s but exclude newline characters and only look for ASCII spaces
-SPACE_REGEX = r'[ \t\f\v]'
-# Match the end of a line: newline characters of a single line
-NEWLINE_REGEX = r'(?:\n|\r|\r\n)'
-# Match the indentation at the beginning of a line
-INDENTATION_REGEX = fr'^{SPACE_REGEX}*'
-
-
-# Match a C identifier: 'identifier', 'var_3', 'NameCamelCase', '_var'
-# Use \b to only match a full word: match "a_b", but not just "b" in "a_b".
-ID_REGEX = r'\b[a-zA-Z_][a-zA-Z0-9_]*\b'
-# Match 'array[3]'
-SUBEXPR_REGEX = fr'{ID_REGEX}(?:\[[^]]+\])*'
-# Match a C expression like "frame", "frame.attr", "obj->attr" or "*obj".
-# Don't match functions calls like "func()".
-EXPR_REGEX = (fr"\*?" # "*" prefix
- fr"{SUBEXPR_REGEX}" # "var"
- fr"(?:(?:->|\.){SUBEXPR_REGEX})*") # "->attr" or ".attr"
-
-# # Match 'PyObject *var' and 'struct MyStruct* var'
-TYPE_PTR_REGEX = fr'{ID_REGEX} *\*'
-
-# Match '(PyObject*)' and nothing
-OPT_CAST_REGEX = fr'(?:\({TYPE_PTR_REGEX} *\){SPACE_REGEX}*)?'
-
-
-def same_indentation(group):
- # the regex must have re.MULTILINE flag
- return fr'{SPACE_REGEX}*(?:{NEWLINE_REGEX}{group})?'
-
-
-def get_member_regex_str(member):
- # Match "var->member".
- return fr'\b({EXPR_REGEX}) *-> *{member}\b'
-
-
-def get_member_regex(member):
- # Match "var->member" (get).
- # Don't match "var->member = value" (set).
- # Don't match "Py_CLEAR(var->member)".
- # Only "Py_CLEAR(" exact string is excluded.
- regex = (r'(?member = expr;".
- regex = assign_regex_str(get_member_regex_str(member), r'([^=].*)')
- return re.compile(regex)
-
-
-def call_assign_regex(name):
- # Match "Py_TYPE(expr) = expr;".
- # Don't match "assert(Py_TYPE(expr) == expr);".
- # Tolerate spaces
- regex = fr'{name} *\( *(.+) *\) *= *([^=].*) *;'
- return re.compile(regex)
-
-
-def is_c_filename(filename):
- return filename.endswith(C_FILE_EXT)
-
-
-class Operation:
- NAME = ""
- REPLACE = ()
- NEED_PYTHONCAPI_COMPAT = False
-
- def __init__(self, patcher):
- self.patcher = patcher
-
- def patch(self, content):
- old_content = content
- for regex, replace in self.REPLACE:
- content = regex.sub(replace, content)
- if content != old_content and self.NEED_PYTHONCAPI_COMPAT:
- content = self.patcher.add_pythoncapi_compat(content)
- return content
-
-
-class Py_TYPE(Operation):
- NAME = "Py_TYPE"
- REPLACE = (
- (get_member_regex('ob_type'), r'Py_TYPE(\1)'),
- )
- # Py_TYPE() was added to Python 2.6.
-
-
-class Py_SIZE(Operation):
- NAME = "Py_SIZE"
- REPLACE = (
- (get_member_regex('ob_size'), r'Py_SIZE(\1)'),
- )
- # Py_SIZE() was added to Python 2.6.
-
-
-class Py_REFCNT(Operation):
- NAME = "Py_REFCNT"
- REPLACE = (
- (get_member_regex('ob_refcnt'), r'Py_REFCNT(\1)'),
- )
- # Py_REFCNT() was added to Python 2.6.
-
-
-class Py_SET_TYPE(Operation):
- NAME = "Py_SET_TYPE"
- REPLACE = (
- (call_assign_regex('Py_TYPE'), r'Py_SET_TYPE(\1, \2);'),
- (set_member_regex('ob_type'), r'Py_SET_TYPE(\1, \2);'),
- )
- # Need Py_SET_TYPE(): new in Python 3.9.
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class Py_SET_SIZE(Operation):
- NAME = "Py_SET_SIZE"
- REPLACE = (
- (call_assign_regex('Py_SIZE'), r'Py_SET_SIZE(\1, \2);'),
- (set_member_regex('ob_size'), r'Py_SET_SIZE(\1, \2);'),
- )
- # Need Py_SET_SIZE(): new in Python 3.9.
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class Py_SET_REFCNT(Operation):
- NAME = "Py_SET_REFCNT"
- REPLACE = (
- (call_assign_regex('Py_REFCNT'), r'Py_SET_REFCNT(\1, \2);'),
- (set_member_regex('ob_refcnt'), r'Py_SET_REFCNT(\1, \2);'),
- )
- # Need Py_SET_REFCNT(): new in Python 3.9.
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class PyObject_NEW(Operation):
- NAME = "PyObject_NEW"
- # In Python 3.9, the PyObject_NEW() macro becomes an alias to the
- # PyObject_New() macro, and the PyObject_NEW_VAR() macro becomes an alias
- # to the PyObject_NewVar() macro.
- REPLACE = (
- (re.compile(r"\bPyObject_NEW\b( *\()"), r'PyObject_New\1'),
- (re.compile(r"\bPyObject_NEW_VAR\b( *\()"), r'PyObject_NewVar\1'),
- )
-
-
-class PyMem_MALLOC(Operation):
- NAME = "PyMem_MALLOC"
- # In Python 3.9, the PyObject_NEW() macro becomes an alias to the
- # PyObject_New() macro, and the PyObject_NEW_VAR() macro becomes an alias
- # to the PyObject_NewVar() macro.
-
- REPLACE = (
- (re.compile(r"\bPyMem_MALLOC\b( *\()"), r'PyMem_Malloc\1'),
- (re.compile(r"\bPyMem_REALLOC\b( *\()"), r'PyMem_Realloc\1'),
- (re.compile(r"\bPyMem_FREE\b( *\()"), r'PyMem_Free\1'),
- (re.compile(r"\bPyMem_Del\b( *\()"), r'PyMem_Free\1'),
- (re.compile(r"\bPyMem_DEL\b( *\()"), r'PyMem_Free\1'),
- )
-
-
-class PyObject_MALLOC(Operation):
- NAME = "PyObject_MALLOC"
- # In Python 3.9, the PyObject_NEW() macro becomes an alias to the
- # PyObject_New() macro, and the PyObject_NEW_VAR() macro becomes an alias
- # to the PyObject_NewVar() macro.
-
- REPLACE = (
- (re.compile(r"\bPyObject_MALLOC\b( *\()"), r'PyObject_Malloc\1'),
- (re.compile(r"\bPyObject_REALLOC\b( *\()"), r'PyObject_Realloc\1'),
- (re.compile(r"\bPyObject_FREE\b( *\()"), r'PyObject_Free\1'),
- (re.compile(r"\bPyObject_Del\b( *\()"), r'PyObject_Free\1'),
- (re.compile(r"\bPyObject_DEL\b( *\()"), r'PyObject_Free\1'),
- )
-
-
-class PyFrame_GetBack(Operation):
- NAME = "PyFrame_GetBack"
- REPLACE = (
- (get_member_regex('f_back'), r'_PyFrame_GetBackBorrow(\1)'),
- )
- # Need _PyFrame_GetBackBorrow() (PyFrame_GetBack() is new in Python 3.9)
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class PyFrame_GetCode(Operation):
- NAME = "PyFrame_GetCode"
-
- REPLACE = (
- (get_member_regex('f_code'), r'_PyFrame_GetCodeBorrow(\1)'),
- )
- # Need _PyFrame_GetCodeBorrow() (PyFrame_GetCode() is new in Python 3.9)
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class PyThreadState_GetInterpreter(Operation):
- NAME = "PyThreadState_GetInterpreter"
- REPLACE = (
- (get_member_regex('interp'), r'PyThreadState_GetInterpreter(\1)'),
- )
- # Need PyThreadState_GetInterpreter() (new in Python 3.9)
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class PyThreadState_GetFrame(Operation):
- NAME = "PyThreadState_GetFrame"
- REPLACE = (
- (get_member_regex('frame'), r'_PyThreadState_GetFrameBorrow(\1)'),
- )
- # Need _PyThreadState_GetFrameBorrow()
- # (PyThreadState_GetFrame() is new in Python 3.9)
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 9))
-
-
-class Py_NewRef(Operation):
- NAME = "Py_NewRef"
- REPLACE = (
- # "Py_INCREF(x); return x;" => "return Py_NewRef(x);"
- # "Py_XINCREF(x); return x;" => "return Py_XNewRef(x);"
- # The two statements must be at the same indentation, otherwise the
- # regex does not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_(X?)INCREF\(({EXPR_REGEX})\)\s*;'
- + same_indentation(r'\1')
- + fr'return {OPT_CAST_REGEX}\3;',
- re.MULTILINE),
- r'\1return Py_\2NewRef(\3);'),
-
- # Same regex than the previous one,
- # but the two statements are on the same line.
- (re.compile(fr'Py_(X?)INCREF\(({EXPR_REGEX})\)\s*;'
- + fr'{SPACE_REGEX}*'
- + fr'return {OPT_CAST_REGEX}\2;',
- re.MULTILINE),
- r'return Py_\1NewRef(\2);'),
-
- # "Py_INCREF(x); y = x;" must be replaced before
- # "y = x; Py_INCREF(y);", to not miss consecutive
- # "Py_INCREF; assign; Py_INCREF; assign; ..." (see unit tests).
-
- # "Py_INCREF(x); y = x;" => "y = Py_NewRef(x)"
- # "Py_XINCREF(x); y = x;" => "y = Py_XNewRef(x)"
- # The two statements must have the same indentation, otherwise the
- # regex does not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_(X?)INCREF\(({EXPR_REGEX})\);'
- + same_indentation(r'\1')
- + assign_regex_str(fr'({EXPR_REGEX})',
- fr'{OPT_CAST_REGEX}\3'),
- re.MULTILINE),
- r'\1\4 = Py_\2NewRef(\3);'),
-
- # Same regex than the previous one,
- # but the two statements are on the same line.
- (re.compile(fr'Py_(X?)INCREF\(({EXPR_REGEX})\);'
- + fr'{SPACE_REGEX}*'
- + assign_regex_str(fr'({EXPR_REGEX})',
- fr'{OPT_CAST_REGEX}\2')),
- r'\3 = Py_\1NewRef(\2);'),
-
- # "y = x; Py_INCREF(x);" => "y = Py_NewRef(x);"
- # "y = x; Py_INCREF(y);" => "y = Py_NewRef(x);"
- # "y = x; Py_XINCREF(x);" => "y = Py_XNewRef(x);"
- # "y = x; Py_XINCREF(y);" => "y = Py_XNewRef(x);"
- # "y = (PyObject*)x; Py_XINCREF(y);" => "y = Py_XNewRef(x);"
- # The two statements must have the same indentation, otherwise the
- # regex does not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + assign_regex_str(fr'({EXPR_REGEX})',
- fr'{OPT_CAST_REGEX}({EXPR_REGEX})')
- + same_indentation(r'\1')
- + r'Py_(X?)INCREF\((?:\2|\3)\);',
- re.MULTILINE),
- r'\1\2 = Py_\4NewRef(\3);'),
-
- # Same regex than the previous one,
- # but the two statements are on the same line.
- (re.compile(assign_regex_str(fr'({EXPR_REGEX})',
- fr'{OPT_CAST_REGEX}({EXPR_REGEX})')
- + fr'{SPACE_REGEX}*'
- + r'Py_(X?)INCREF\((?:\1|\2)\);'),
- r'\1 = Py_\3NewRef(\2);'),
-
- # "PyObject *var = x; Py_INCREF(x);" => "PyObject *var = Py_NewRef(x);"
- # The two statements must have the same indentation, otherwise the
- # regex does not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- # "type* var = expr;"
- + assign_regex_str(fr'({TYPE_PTR_REGEX} *)({EXPR_REGEX})',
- fr'({OPT_CAST_REGEX})({EXPR_REGEX})')
- + same_indentation(r'\1')
- # "Py_INCREF(var);"
- + r'Py_(X?)INCREF\((?:\3|\5)\);',
- re.MULTILINE),
- r'\1\2\3 = \4Py_\6NewRef(\5);'),
-
- # "Py_INCREF(x); PyObject *var = x;" => "PyObject *var = Py_NewRef(x);"
- # The two statements must have the same indentation, otherwise the
- # regex does not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- # "Py_INCREF(var);"
- + fr'Py_(X?)INCREF\(({EXPR_REGEX})\);'
- + same_indentation(r'\1')
- # "type* var = expr;"
- + assign_regex_str(fr'({TYPE_PTR_REGEX} *{EXPR_REGEX})',
- fr'({OPT_CAST_REGEX})\3'),
- re.MULTILINE),
- r'\1\4 = \5Py_\2NewRef(\3);'),
- )
- # Need Py_NewRef(): new in Python 3.10
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 10))
-
-
-class Py_CLEAR(Operation):
- NAME = "Py_CLEAR"
- REPLACE = (
- # "Py_XDECREF(x); x = NULL;" => "Py_CLEAR(x)";
- # The two statements must have the same indentation, otherwise the
- # regex does not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_XDECREF\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + assign_regex_str(r'\2', r'NULL'),
- re.MULTILINE),
- r'\1Py_CLEAR(\2);'),
-
- # "Py_XDECREF(x); x = NULL;" => "Py_CLEAR(x)";
- (re.compile(fr'Py_XDECREF\(({EXPR_REGEX})\) *;'
- + fr'{SPACE_REGEX}*'
- + assign_regex_str(r'\1', r'NULL')),
- r'Py_CLEAR(\1);'),
- )
-
-
-SETREF_VALUE = fr'{OPT_CAST_REGEX}(?:{EXPR_REGEX}|Py_X?NewRef\({EXPR_REGEX}\))'
-
-
-class Py_SETREF(Operation):
- NAME = "Py_SETREF"
- REPLACE = (
- # "Py_INCREF(y); Py_CLEAR(x); x = y;" => "Py_XSETREF(x, y)";
- # Statements must have the same indentation, otherwise the regex does
- # not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_(X?)INCREF\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + fr'Py_CLEAR\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + assign_regex_str(r'\4', r'\3'),
- re.MULTILINE),
- r'\1Py_XSETREF(\4, Py_\2NewRef(\3));'),
-
- # "Py_CLEAR(x); x = y;" => "Py_XSETREF(x, y)";
- # Statements must have the same indentation, otherwise the regex does
- # not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_CLEAR\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + assign_regex_str(r'\2',
- fr'({SETREF_VALUE})'),
- re.MULTILINE),
- r'\1Py_XSETREF(\2, \3);'),
-
- # "Py_INCREF(y); Py_DECREF(x); x = y;" => "Py_SETREF(x, y)";
- # Statements must have the same indentation, otherwise the regex does
- # not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_(X?)INCREF\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + fr'Py_(X?)DECREF\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + assign_regex_str(r'\5', r'\3'),
- re.MULTILINE),
- r'\1Py_\4SETREF(\5, Py_\2NewRef(\3));'),
-
- # "Py_DECREF(x); x = y;" => "Py_SETREF(x, y)";
- # "Py_DECREF(x); x = Py_NewRef(y);" => "Py_SETREF(x, Py_NewRef(y))";
- # Statements must have the same indentation, otherwise the regex does
- # not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'Py_(X?)DECREF\(({EXPR_REGEX})\) *;'
- + same_indentation(r'\1')
- + assign_regex_str(r'\3',
- fr'({SETREF_VALUE})'),
- re.MULTILINE),
- r'\1Py_\2SETREF(\3, \4);'),
-
- # "old = var; var = new; Py_DECREF(old);" => "Py_SETREF(var, new);"
- # "PyObject *old = var; var = new; Py_DECREF(old);" => "Py_SETREF(var, new);"
- # Statements must have the same indentation, otherwise the regex does
- # not match.
- (re.compile(fr'({INDENTATION_REGEX})'
- + fr'(?:{ID_REGEX} *\* *)?({ID_REGEX}) *= *{OPT_CAST_REGEX}({EXPR_REGEX}) *;'
- + same_indentation(r'\1')
- + assign_regex_str(r'\3',
- fr'({SETREF_VALUE})')
- + same_indentation(r'\1')
- + fr'Py_(X?)DECREF\(\2\) *;',
- re.MULTILINE),
- r'\1Py_\5SETREF(\3, \4);'),
- )
- # Need Py_NewRef(): new in Python 3.5
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 5))
-
-
-class Py_Is(Operation):
- NAME = "Py_Is"
-
- def replace2(regs):
- x = regs.group(1)
- y = regs.group(2)
- if y == 'NULL':
- return regs.group(0)
- return f'{x} = _Py_StealRef({y});'
-
- REPLACE = []
- expr = fr'({EXPR_REGEX})'
- for name in ('None', 'True', 'False'):
- REPLACE.extend((
- (re.compile(fr'{expr} == Py_{name}\b'),
- fr'Py_Is{name}(\1)'),
- (re.compile(fr'{expr} != Py_{name}\b'),
- fr'!Py_Is{name}(\1)'),
- ))
-
- # Need Py_IsNone(), Py_IsTrue(), Py_IsFalse(): new in Python 3.10
- NEED_PYTHONCAPI_COMPAT = (MIN_PYTHON < (3, 10))
-
-
-OPERATIONS = (
- Py_SET_TYPE,
- Py_SET_SIZE,
- Py_SET_REFCNT,
- # Py_SET_xxx must be run before Py_xxx
- Py_TYPE,
- Py_SIZE,
- Py_REFCNT,
-
- Py_Is,
-
- PyObject_NEW,
- PyMem_MALLOC,
- PyObject_MALLOC,
-
- PyFrame_GetBack,
- PyFrame_GetCode,
-
- PyThreadState_GetInterpreter,
- PyThreadState_GetFrame,
-
- # Code style: excluded from "all"
- Py_NewRef,
- Py_CLEAR,
- Py_SETREF,
-)
-
-EXCLUDE_FROM_ALL = (
- Py_NewRef,
- Py_CLEAR,
- Py_SETREF,
-)
-
-
-def all_operations():
- return set(operation_class.NAME for operation_class in OPERATIONS
- if operation_class not in EXCLUDE_FROM_ALL)
-
-
-class Patcher:
- def __init__(self, args=None):
- self.exitcode = 0
- self.pythoncapi_compat_added = 0
- self.want_pythoncapi_compat = False
- self.operations = None
- self.applied_operations = set()
-
- # Set temporarily by patch()
- self._has_pythoncapi_compat = None
- self._applied_operations = None
-
- self._parse_options(args)
-
- def log(self, msg=''):
- print(msg, file=sys.stderr, flush=True)
-
- def warning(self, msg):
- self.log(f"WARNING: {msg}")
-
- def _get_operations(self, parser):
- args_names = self.args.operations.split(',')
-
- wanted = set()
- for name in args_names:
- name = name.strip()
- if not name:
- continue
-
- if name == "all":
- wanted |= all_operations()
- elif name.startswith("-"):
- name = name[1:]
- wanted.discard(name)
- else:
- wanted.add(name)
-
- operations = []
- for operation_class in OPERATIONS:
- name = operation_class.NAME
- if name not in wanted:
- continue
- wanted.discard(name)
- operation = operation_class(self)
- operations.append(operation)
-
- if wanted:
- print(f"invalid operations: {','.join(wanted)}")
- print()
- self.usage(parser)
- sys.exit(1)
-
- return operations
-
- def add_line(self, content, line):
- # Use the first matching newline
- match = re.search(r'(?:\r\n|\n|\r)', content)
- newline = match.group(0) if match else '\n'
-
- line = line + newline
- # FIXME: tolerate trailing spaces
- if line not in content:
- # FIXME: add macro after the first header comment
- # FIXME: add macro after includes
- # FIXME: add macro after: #define PY_SSIZE_T_CLEAN
- return line + newline + content
- else:
- return content
-
- def add_pythoncapi_compat(self, content):
- if self._has_pythoncapi_compat:
- return content
- content = self.add_line(content, INCLUDE_PYTHONCAPI_COMPAT)
- self._has_pythoncapi_compat = True
- self.pythoncapi_compat_added += 1
- return content
-
- def _patch(self, content):
- try:
- has = (self.args.no_compat
- or INCLUDE_PYTHONCAPI_COMPAT in content
- or INCLUDE_PYTHONCAPI_COMPAT2 in content)
- self._has_pythoncapi_compat = has
- self._applied_operations = []
- for operation in self.operations:
- new_content = operation.patch(content)
- if new_content != content:
- self._applied_operations.append(operation.NAME)
- content = new_content
- applied_operations = self._applied_operations
- finally:
- self._has_pythoncapi_compat = None
- self._applied_operations = None
- return (content, applied_operations)
-
- def patch(self, content):
- return self._patch(content)[0]
-
- def patch_file(self, filename):
- if os.path.basename(filename) == PYTHONCAPI_COMPAT_H:
- self.log(f"Skip {filename}")
- return
-
- encoding = "utf-8"
- errors = "surrogateescape"
-
- with open(filename, encoding=encoding, errors=errors, newline="") as fp:
- old_contents = fp.read()
-
- new_contents, operations = self._patch(old_contents)
-
- if self.args.to_stdout:
- print(new_contents, end="")
- return (new_contents != old_contents)
-
- # Don't rewrite if the filename for in-place replacement,
- # to avoid changing the file modification time.
- if new_contents == old_contents:
- return False
-
- if not self.args.no_backup:
- old_filename = filename + ".old"
- # If old_filename already exists, replace it
- os.replace(filename, old_filename)
-
- with open(filename, "w", encoding=encoding, errors=errors, newline="") as fp:
- fp.write(new_contents)
-
- self.applied_operations |= set(operations)
- operations = ', '.join(operations)
- self.log(f"Patched file: {filename} ({operations})")
- return True
-
- def _walk_dir(self, path):
- empty = True
-
- for dirpath, dirnames, filenames in os.walk(path):
- # Don't walk into .tox
- for ignore_name in IGNORE_DIRS:
- try:
- dirnames.remove(ignore_name)
- except ValueError:
- pass
- for filename in filenames:
- if is_c_filename(filename):
- yield os.path.join(dirpath, filename)
- empty = False
-
- if empty:
- self.warning(f"Directory {path} doesn't contain any C file")
- self.exitcode = 1
-
- def walk(self, paths):
- for path in paths:
- if os.path.isdir(path):
- for filename in self._walk_dir(path):
- yield filename
- elif os.path.exists(path):
- yield path
- else:
- self.warning(f"Path {path} does not exist")
- self.exitcode = 1
-
- def get_latest_header(self, base_dir):
- target = os.path.join(base_dir, PYTHONCAPI_COMPAT_H)
- self.log(f"Download the file from {PYTHONCAPI_COMPAT_URL} to {target}.")
- urllib.request.urlretrieve(PYTHONCAPI_COMPAT_URL, target)
-
- @staticmethod
- def usage(parser):
- parser.print_help()
- print()
- print("Operations:")
- print()
- for operation in sorted(OPERATIONS,
- key=lambda operation: operation.NAME.lower()):
- print(f"- {operation.NAME}")
- print()
- print("If a directory is passed, search for .c and .h files "
- "in subdirectories.")
-
- def _parse_dir_path(self, path):
- if os.path.isdir(path):
- return path
- else:
- raise argparse.ArgumentTypeError(f"{path} is not a valid path")
-
- def _parse_options(self, args):
- parser = argparse.ArgumentParser(
- description="Upgrade C extension modules to newer Python C API")
- parser.add_argument(
- '-o', '--operations', action="store",
- default="all",
- help='Space separated list of operation names to apply')
- parser.add_argument(
- '-q', '--quiet', action="store_true",
- help='Quiet mode')
- parser.add_argument(
- '-c', '--to-stdout', action="store_true",
- help='Write output into stdout instead of modifying files '
- 'in-place (imply quiet mode)')
- parser.add_argument(
- '-B', '--no-backup', action="store_true",
- help="Don't create .old backup files")
- parser.add_argument(
- '-C', '--no-compat', action="store_true",
- help=f"Don't add: {INCLUDE_PYTHONCAPI_COMPAT}")
- parser.add_argument(
- '-d', '--download', metavar='PATH',
- help=f'Download latest pythoncapi_compat.h file to designated PATH',
- type=self._parse_dir_path)
- parser.add_argument(
- metavar='file_or_directory', dest="paths", nargs='*')
-
- args = parser.parse_args(args)
- if not args.paths and not args.download:
- self.usage(parser)
- sys.exit(1)
-
- if args.to_stdout:
- args.quiet = True
-
- self.args = args
- self.operations = self._get_operations(parser)
-
- def main(self):
- if self.args.paths:
- for filename in self.walk(self.args.paths):
- self.patch_file(filename)
-
- if self.applied_operations:
- nops = len(self.applied_operations)
- ops = ', '.join(sorted(self.applied_operations))
- self.log()
- self.log(f"Applied operations ({nops}): {ops}")
-
- if self.args.download:
- path = self.args.download
- self.get_latest_header(path)
-
- if self.pythoncapi_compat_added and not self.args.quiet:
- self.log()
- self.log(f"{INCLUDE_PYTHONCAPI_COMPAT} added: you may have "
- f"to copy {PYTHONCAPI_COMPAT_H} to your project")
- self.log("Run 'python upgrade_pythoncapi.py --download '")
-
- sys.exit(self.exitcode)
-
-
-if __name__ == "__main__":
- Patcher().main()