Dropping GIL when calling Julia from Python (JuliaCall) #343

Activity

1. Moelf

   added

   enhancementNew feature or requestNew feature or request

   on Jul 25, 2023
2. cjdoris commented on Jul 26, 2023

   

   cjdoris

   on Jul 26, 2023

   Member

   More actions

   Repeating what I said in #108

   Yeah calling into Julia from Julia doesn't unlock the GIL. This is because the Julia code could well call back into Python, which would require automatically relocking the GIL, which is all quite complicated and probably slow in general. If you can guarantee your Julia code doesn't call back into Python, you can release the GIL yourself. A future version of JuliaCall will allow something like some_julia_function.jl_call_nogil(x, y, z) to make this a bit easier.

   Specifically, you can release the GIL yourself in the Julia code you call like:

   pythread = PythonCall.C.PyEval_SaveThread()
   try
       # code which doesn't touch Python
   finally
       PythonCall.C.PyEval_RestoreThread(pythread)
   end

   👍React with 👍2Reacted by Mark Kittisopikul and Tyler Thomas
3. github-actions commented on Aug 26, 2023

   

   github-actionsBot

   on Aug 26, 2023

   Contributor

   More actions

   This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.
4. github-actions

   added

   staleIssues about to be auto-closedIssues about to be auto-closed

   on Aug 26, 2023
5. PallHaraldsson commented on Aug 29, 2023

   

   PallHaraldsson

   on Aug 29, 2023

   Last edited by PallHaraldsson

   Contributor

   More actions

   In case relevant, new in 3.12rc1:

   https://peps.python.org/pep-0684/

   https://docs.python.org/3.12/whatsnew/3.12.html#pep-684-a-per-interpreter-gil

   PEP 684: A Per-Interpreter GIL
   Sub-interpreters may now be created with a unique GIL per interpreter. This allows Python programs to take full advantage of multiple CPU cores.

   Use the new Py_NewInterpreterFromConfig() function to create an interpreter with its own GIL:

   PyInterpreterConfig config = {
       .check_multi_interp_extensions = 1,
       .gil = PyInterpreterConfig_OWN_GIL,
   };
   PyThreadState *tstate = NULL;
   PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
   if (PyStatus_Exception(status)) {
       return -1;
   }
   /* The new interpeter is now active in the current thread. */
   For further examples how to use the C-API for sub-interpreters with a per-interpreter GIL, see Modules/_xxsubinterpretersmodule.c.
   

   New release candidate 2 expected 2023-09-04 then

   3.12.0 final: Monday, 2023-10-02

   See also new in 3.13:

   https://peps.python.org/pep-0554/

   PEP 554 – Multiple Interpreters in the Stdlib

   A Disclaimer about the GIL
   To avoid any confusion up front: This PEP is meant to be independent of any efforts to stop sharing the GIL between interpreters (PEP 684). At most this proposal will allow users to take advantage of any GIL-related work.

   Copied from: JuliaPy/PyCall.jl#1046 (comment)

   👍React with 👍1Reacted by Christopher Rowley
6. github-actions

   removed

   staleIssues about to be auto-closedIssues about to be auto-closed

   on Aug 30, 2023
7. Moelf commented on Aug 9, 2024

   

   Moelf

   on Aug 9, 2024

   Author

   More actions

   is this closed by #535?
8. cjdoris commented on Aug 13, 2024

   

   cjdoris

   on Aug 13, 2024

   Member

   More actions

   Yep thanks.
9. cjdoris

   closed this as completedon Aug 13, 2024