PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?)

I'm writing down the issue so that it is easy to find by the users.

At the moment, for PyJulia to load Julia >= 0.7, two conditions have to be met:

Python interpreter configured for PyCall.jl and Python interpreter used to import PyJulia have to use identical libpython (once Support virtualenv/venv without separate cache #190 is merged) ~~have exactly identical paths~~.
The Python interpreter have to be dynamically linked (which is usually the case in Windows and macOS according to this comment)

To check if condition 2 is the case, run:

$ ldd /usr/bin/python | grep libpython
        libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f17c12c4000)

where /usr/bin/python is the appropriate Python interpreter.

If it does not print anything, you can't use PyJulia with this Python interpreter.

@tkf

This was very helpful; thank you, @tkf

I must say I was very-very surprised by not finding libpython3 in my ldd $(which python3) output.

Debian/Ubuntu packagers are the known strong advocates against linking things statically; but doing so in python/3?!?!

Yeah, I was surprised too. I thought it was just conda (in which static compilation maybe makes sense).

@tkf

@tkf Thanks for your reply. But there are still some questions:

Currently we do not have any method to call Julia in conda based/Ubuntu/Debian python3, right? Even dirty/tricky method would be helpful.
Could we use custom compiled python for it? If true, is this a practical solution?
Will this issue be solved in near future or we should not expect that?

Thanks.

I think there are three ways to use Python with Julia 1.0 on Ubuntu/Debian/etc. at the moment:

If you just want to use your Python module, you can import it from PyCall.jl and call the functions in it. See, e.g., https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L31 and https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L20 in IPython.jl/src/core.jl.
If you just want an interactive Python session, you can use my IPython.jl to drop into IPython through Julia REPL.
If you want fully functional PyJulia in Python, one way to do it is though docker. Check out: https://github.com/tkf/docker-ipyjulia

Will this issue be solved in near future

I think I know how to fix it #173. But likely not in next weeks. I can't grant anything :) If I were to do it I want to do it after dropping Julia 0.6 (which requires PyJulia to be released soon).

I made an experimental package julia-venv: https://github.com/tkf/julia-venv

It implements the private DEPOT_PATH hack mentioned in #173 but does so per-virtualenv basis. This let us use different Python executables without recompiling PyCall.jl all the time. On top of creating a private DEPOT_PATH, it installs a CLI julia-venv which acts like the normal julia interpreter (by invoking it via C API from Python). This way, we can run PyCall.jl always in "PyJulia-mode" so that the issue with Python in Ubuntu/Debian and Conda can be avoided (https://github.com/tkf/julia-venv#how-it-works). As a nice side-effect, you can also use PyCall.jl with different Python executable and different set of Python libraries.

~~Edit: Actually, julia-venv probably still doesn't work with Conda (and probably also with Ubuntu): tkf/julia-venv#2~~ Edit^2: It works now.

@tkf

@tkf ，thanks for your awesome work. The new package works on Ubuntu 16.04 with anaconda. It really helps.

There are some minor suggestions (or some misunderstanding of me).

third-party packages need to be re-compiled/re-installed

for test purpose, I test from julia import NPZ, while NPZ is already added in original julia environment. But it failed.

and the following code works:

from julia import Pkg
Pkg.add("NPZ")
#... some install outputs
from julia import NPZ
data = NPZ.npzread("path")

It seems that pyjulia and julia of system is using different package cache? I think re-install packages is not a problem and it might be a valuable feature. I only suggest add some doc for this

convenient broadcast call

broadcast is commonly used in julia, but Main.abs.([1,2,3]) is a SyntaxError in python.
in README.md of pyjulia, there is

Main.eval("sin.(xs)")

which seems that pyjulia will not expose broadcast method to python level. I'm not sure if I am right, but if it is true, I'm thinking is it possible to adding some special syntax like Main.abs.broadcast([1,2,3]) or in short Main.abs.b([1,2,3]) (although may introduce some confusing to sum_b for python version sum!).

`InterruptException()` handling

When I press ctrl-C in python REPL, there would be a small problem.
If I'm using a pure python REPL, it should work like this

>>> # presss ctrl-C here
KeyboardInterrupt
>>>

But in a python REPL which loaded julia, it would work like this:

>>> # press ctrl-C here

and nothing happens, after that if I enter next command

>>> # Ctrl - C, and nothing happens
>>> Main.xs = [1,2,3] # Enter, change line and type new command and Enter I got:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 140, in __setattr__
    self._julia.eval(setter)(value)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 606, in eval
    ans = self._call(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'InterruptException' occurred while calling julia code:

            PyCall.pyfunctionret(
                (x) -> eval(Main, :(xs = $x)),
                Any,
                PyCall.PyAny)

It seems the InterruptException is send to julia in one delayed statement?

@Hong-Xiang

@Hong-Xiang Hey, thanks for the feedback. Are you using julia-venv in Ubuntu or Debian? Did it actually work?

I think re-install packages is not a problem and it might be a valuable feature.

Yeah, something like virtualenv's --system-site-packages would be nice. I actually did that in first versions but stopped doing that since I encountered with some quirks in julia's package handling. It it actually as "easy" as replacing depots = filter(!isequal(default_depot1), __py_julia_venv_orig_DEPOT_PATH) with depots = copy(__py_julia_venv_orig_DEPOT_PATH).

convenient broadcast call

I started experimenting more numpy-like interface on top of PyJulia (i.e., broadcast operations on arrays by default):

https://ipyjulia-hacks.readthedocs.io/en/latest/

Not sure if it is suitable for PyJulia, though.

But if you want to leverage Julia, I think it's better to just use Julia syntax. If you don't want to use global variable (of course), you can do:

Main.eval("(xs) -> sin.(exp.(xs) .+ 1)")(xs)

or equivalently

Main.eval("(xs) -> @. sin(exp(xs) + 1)")(xs)

Unlike numpy.exp etc., it doesn't allocate intermediate arrays.

InterruptException

Hmm... I didn't notice that. But yes, I can reproduce it. Actually, I just noticed that it's reported in #189. Let's track the issue there.

It looks like IPython 7 (development version) handle ctrl-C differently and it doesn't have this problem. I've been using it so I didn't notice the problem.

@tkf

@tkf my system/environment info is:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial

and actually it's Ubuntu Mate.

conda:

$ conda --version
conda 4.3.30

python(in conda virutal env, with julia-venv)

$ source activate juliaenv

(juliaenv) $ which python
(home path)/anaconda3/envs/juliaenv/bin/python

(juliaenv)  $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia_venv import get_julia
>>> get_julia()
<julia.core.Julia object at 0x7f66c6cc7f28>
>>> from julia import Main

it worked, without error.

python(in conda virtual env, without julia-venv)

(juliaenv)  $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import Main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 172, in load_module
    JuliaMainModule(self, fullname))
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 77, in __init__
    self._julia = loader.julia
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 164, in julia
    self.__class__.julia = julia = Julia()
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 496, in __init__
    "\"lib\", \"pyjulia%s-v$(VERSION.major).$(VERSION.minor)\"))" % sys.version_info[0])
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'UndefVarError' occurred while calling julia code:
unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot(),"lib", "pyjulia3-v$(VERSION.major).$(VERSION.minor)"))
>>>

tests like using standard libraries, basic array operations and even third-party packages like NPZ is performed, and it seems just worked, as mentioned above.
Is there any further tests I can do for this?

Thanks. Interesting... Actually, the problem I thought I had was already fixed tkf/julia-venv#2. My guess was that it was some kind of IO caching issue (compilation file was not properly written before exiting the problem). It then maybe explains why it's machine-dependent.

Anyway, I'm glad that it worked in your machine!

For Ubuntu/Debian users, probably the cleanest and relatively easy way to use PyJulia at the moment is to install (build) Python using pyenv. But you need to pass --enable-shared manually.

$ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6
Downloading Python-3.6.6.tar.xz...
-> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz
Installing Python-3.6.6...
Installed Python-3.6.6 to /home/takafumi/.pyenv/versions/3.6.6

  200.54s user 16.69s system 271% cpu 1:20.10 total
$ ldd ~/.pyenv/versions/3.6.6/bin/python3.6
        linux-vdso.so.1 =>  (0x00007fff6933c000)
        libpython3.6m.so.1.0 => /home/takafumi/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca44a6e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca446a4000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca444a0000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fca4429d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca43f94000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fca451c0000)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

Or you can switch to Arch Linux 😆

It uses --enable-shared: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python&id=68e188a1f975dcc6798072bcda4ef35abfe162ed#n57

@ExpandingMan

@ExpandingMan @mprogram So it turned out static linking is just the default behavior of Python's ./configure script (see my message above #185 (comment)). I guess Debian is not doing anything particularly evil here.

@tkf

Thanks, @tkf for your last two comments. I'm a fan of deb-based system and my thought would be that Debian needs to look at those. If I'll be up to recompile Python (for the sake of optimization with MKL, for example as Julia does), I would include the --enable-shared option in my build.

@mprogram

@mprogram It looks like pyenv does something more than passing --enable-shared to ./configure:

I didn't check if ./configure --enable-shared was enough when building without pyenv.

@tkf

This problem is not going to go away, Debian is not going to change it and lots and lots of people are going to try this as debian distros are the most popular (though I prefer Arch I still have to use them often myself). My first question is how other languages solve this problem. We can't be the first people ever to want to use a dynamically linked Python executable. It would be good to see how Scala and Java solve this, though I haven't started looking yet.

I think at a minimum what we need is a very clear set of the simplest possible steps that one would take to get pyjulia running on debian in a very visible location right in the packages readme or documentation. Thanks to @tkf to taking the first steps to this with his pyenv setup, I'm going to try it myself when I get a chance.

If you are using Conda you are already not using the system Python binaries. In the other cases, I agree there is still need for python-jl. Also, there can still be issues with python-jl #208.

Agreed, let's just aim to document as many options as possible and people will be able to select whatever solution works best for them.

@bstellato

@bstellato Thanks!. This is a good news! So I guess conda-forge distributes a Python executable dynamically linked to libpython. I'll add it to the documentation.

@ExpandingMan Yeah, I think python-jl should stay. Another benefit of python-jl is that signal handler is slightly well behaved this way (libjulia "steals" signal handler from python #211).

It's a bit strange. 3.7.0 from conda-forge is statically linked:

$ conda create --prefix conda-forge-python conda-forge::python
...
The following NEW packages will be INSTALLED:
...
    python:          3.7.0-h5001a0f_4        conda-forge
...
$ conda-forge-python/bin/python
Python 3.7.0 | packaged by conda-forge | (default, Sep 30 2018, 14:56:18)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ ldd conda-forge-python/bin/python
        linux-vdso.so.1 (0x00007ffec5dce000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f86eb290000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f86eb28b000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f86eb286000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f86eb27c000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f86eb0f7000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f86eaf33000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f86eb2e4000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
None

But conda-forge::python=3.6 is dynamically linked.

$ rm -r conda-forge-python
$ conda create --prefix conda-forge-python conda-forge::python=3.6
...
The following NEW packages will be INSTALLED:
...
    python:          3.6.6-h5001a0f_3        conda-forge
...
$ ldd conda-forge-python/bin/python
        linux-vdso.so.1 (0x00007ffff0f7e000)
        libpython3.6m.so.1.0 => /home/takafumi/repos/pyjulia/conda-forge-python/bin/../lib/libpython3.6m.so.1.0 (0x00007f1f758a8000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1f75856000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f1f75851000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f1f7584c000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f1f75842000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f1f756bd000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1f754f7000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1f75de7000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/home/takafumi/repos/pyjulia/conda-forge-python/lib/libpython3.6m.so.1.0

Version 3.7.0 is still dynamically linked for me on mac os.

conda create -n testpyjulia37 conda-forge::python
conda activate testpyjulia37
otool -L $(which python)

I get

/Users/xxx/miniconda3/envs/testpyjulia37/bin/python:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)
	@rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Also,

> python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/Users/xxxx/miniconda3/envs/testpyjulia37/lib/libpython3.7m.dylib

Interesting. I asked the question in conda-forge: conda-forge/python-feedstock#222

@ExpandingMan

I wrote README in #210 and you can read it here: https://github.com/tkf/pyjulia/blob/readme/README.md

@ExpandingMan If you are interested, have a look at Troubleshooting and pre-compilation mechanism sections:
https://github.com/tkf/pyjulia/blob/readme/README.md#troubleshooting
https://github.com/tkf/pyjulia/blob/readme/README.md#pre-compilation-mechanism-in-julia-10

I just merged #256 to master. I think PyJulia is now usable with many CPython executables if you can compile a custom system image with python3 -m julia.sysimage command. The bonus point is that this makes PyJulia startup virtually instantaneous. See: https://pyjulia.readthedocs.io/en/latest/sysimage.html

@tkf

Hi @tkf - it's annoying having to deal with this issue. Do you know which Linux distro (ideally docker container) that provides a dynamically linked Python?

It works in Arch Linux :)

IIRC, the official python docker works. It's a bit old, but here is an example for setting up PyJulia in Docker: https://github.com/tkf/docker-pyjulia

Also, just FYI, I started working on compile-time preference integration for PyCall JuliaPy/PyCall.jl#835. If it works as I expected, PyJulia should just work without any hacks in Julia 1.6 and above. Obviously, there are still some unknowns as I haven't finished implementing all. So, let's see ;)

What's the status of this issue? I am trying to run PyJulia in Ubuntu 22 but python is statically linked.

Is it possible to build my own Python with --enable-shared in Conda?

I think that might be a question for conda-forge. If you are going to compile it yourself, do you actually need conda though?

Perhaps we could use https://github.com/JuliaBinaryWrappers/Python_jll.jl

Just FYI, it seems that also python-build-standalone is considering switching to static builds of python for performance gains: astral-sh/python-build-standalone#540 .

Just FYI, it seems that also python-build-standalone is considering switching to static builds of python for performance gains: astral-sh/python-build-standalone#540 .

Ah, it seems that this package is now deprecated, so it may not be a big problem: #556 .

Navigation Menu

Repository navigation

PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?) #185

third-party packages need to be re-compiled/re-installed

convenient broadcast call

`InterruptException()` handling

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?) #185

Description

Activity

mprogram commented on Aug 18, 2018

tkf commented on Aug 19, 2018

Hong-Xiang commented on Aug 21, 2018

tkf commented on Aug 21, 2018

tkf commented on Sep 5, 2018

Hong-Xiang commented on Sep 5, 2018

third-party packages need to be re-compiled/re-installed

convenient broadcast call

InterruptException() handling

tkf commented on Sep 5, 2018

Hong-Xiang commented on Sep 5, 2018

tkf commented on Sep 5, 2018

tkf commented on Sep 12, 2018

tkf commented on Sep 12, 2018

tkf commented on Sep 12, 2018

mprogram commented on Sep 12, 2018

tkf commented on Sep 12, 2018

ExpandingMan commented on Sep 12, 2018

8 remaining items

bstellato commented on Oct 25, 2018

ExpandingMan commented on Oct 25, 2018

tkf commented on Oct 25, 2018

tkf commented on Oct 26, 2018

bstellato commented on Oct 26, 2018

tkf commented on Oct 26, 2018

tkf commented on Oct 26, 2018

tkf commented on Apr 12, 2019

tk3369 commented on Sep 25, 2020

tkf commented on Sep 25, 2020

Ge0rges commented on Nov 1, 2022

EmanuelCastanho commented on Aug 16, 2023

mkitti commented on Aug 16, 2023

traversaro commented on Feb 27, 2025

traversaro commented on Feb 27, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`InterruptException()` handling