Importing a third party python library


#1

Hi there,

What is the way to include third party python libraries like numpy in NAP? Is there a way to include these libraries in the packaged NAP application so they are accessible from python scripts within the project, on any machine?

I tried placing an installed numpy build directly in the /data folder of the project but this resulted in various ImportErrors or EXC_BAD_INSTRUCTION errors.

Casimir


#2

Heya,

I don’t think there is a mechanism for this in place right now. Placing modules inside the data folder won’t help because the python interpreter doesn’t look in that directory.

From what I know you must extend the python path (PYTHONPATH) to add other locations. To check the current python path use:

import sys
print(sys.path)

You can also print the individual locations in the path. Every element in the array points to a location python looks for additional modules to import. You can add your own search path as required.


#3

Hey Coen,

Thank you. I managed to get it to work in the end, but not using the proposed method.

Firstly, appending the sys.path with an existing installed version of numpy on my computer didn’t work, I think because of the difference in the versions of python 3.7 on my computer and python 3.6 that NAP uses (I got a verbose numpy error basically saying that I’m importing numpy from the wrong python install).

Secondly, this solution will only make the library accessible on my current machine where this library installed on this hardcoded path, which means that the python scripts will break when we distribute our software to other users and other machines.

I figured I had to install numpy from within the python build in NAP. I did this in the following way:

Inside the /thirdparty/python/bin folder in the NAP directory, I executed this command to install pip:

wget -qO- https://bootstrap.pypa.io/get-pip.py | sudo ./python3.6

And then I installed numpy from within the local python build, into the local directory, like this:

sudo ./python3 -m pip install --target “…/lib/python3.6/site-packages” numpy

And now numpy can be imported within any python script in our project, and is included in the packages of our software.

What do you think of this method?
For future users that want to achieve the same, maybe it would be useful to have a simple NAP command that installs a python module for you in this manner. Would this be possible?

Greetings,
Casimir


#4

Hey Casimir,

This sounds like a viable option using PIP. It makes the python package available to all projects in your current nap project. The problem is that you will have to execute the command every time you download a new package.

In my opinion it would be better to declare per project what kind of additional python modules you want to use and install those somewhere in the data folder of your project. The build system should automatically pick up on those. This can be done using the PYTHONPATH, specific to the python interpreter that ships with NAP. This is something me and Chris should take a look at. Would that work for you as well? Preferably using pip but specific to a project.


#5

Hey Coen,

Making it specific to the project sounds like a good idea in essence. However, I can imagine the process might get a little bit confusing in the case you would want to use the same package for multiple projects. I don’t know if it’s possible to install the same python package from the same python build on multiple locations, it certainly doesn’t seem to be common practice. So you would still first have to install it on one location, then find the installed files and copy them over to each project. Something to think about.

Casi


#6

Hey Casi,

Python is rather flexible on that end. I have successfully in the past extended the python path for a specific python interpreter to point to multiple locations that had python specific scripts / modules. For example: I had multiple versions of houdini, all of them shipping with their own python interpreter. My custom python modules lived somewhere external from the package but every Houdini had access to those modules and scripts.

What Houdini does is try to locate a python script the moment it launches a new session (123.py) or loads a new scene (456.py). By extending the python path in there (by declaring the script) I was successfully bringing in external scripts / packages into the Houdini Hython Runtime environment. You could also use it to intall new packages using pip etc, but I always prefer to be more explicit.

That’s maybe the best of both worlds. NAP executes a script (launch.py) which can be placed in:

  • a module - launchmodule.py
  • a package - launchapi.py
  • a project - launchproject.py

You can configure from there what you want project, module or package specific. Why limit ourselves to only 1 option? Do as you please, configure the entire pipe the way you want. If a python package depends on a pre-compiled component (such as numpy etc.) it’s best to find a pre-compiled version and package it alongside the python interface, or, use pip instead. The problem with pip (as with all package managers) is that versions could differ, have already been installed on the machine and defined in some sort of path, etc. We run into this a lot, especailly on OSX and has been the most major source of package / build system related issues we encountered.

Source of reference: http://www.sidefx.com/docs/houdini/hom/locations.html

Cheers Coen