Importing wrong packages when calling a python script from anoth

ghz 10hours ago ⋅ 2 views

Importing wrong packages when calling a python script from another python script via subprocess

I am trying to automate executable creation for my program in 32 and 64bit with cx_Freeze. I have two separate python 3.4 installations (32 and 64 bit) with all required packages, which all correspond to their appropriate architectures.

However, when I execute the script that calls the different python installations via subprocess, the called python subprocesses import the wrong packages. They both import the packages, the original script was called from and not the packages the python installation, that was called in the subprocess.

Minimal example called with a 64bit python:

#! coding=utf-8
import subprocess as sp
sp.call(["python34-32","test.py"])
sp.call(["python34","test.py"])

The test.py contains only the line:

import cx_Freeze

The second process, that also calls the 64bit python won't have a problem. But the 32bit will throw the following error:

ImportError: DLL load failed: %1 is not a valid Win32 application.

This is because the subprocess, which is calling 32bit python, imports cx_freeze from the 64bit python. The situation can also be reversed. When the main script is executed with a 32bit python, the same error occurs, however, now it is because the 64bit python called in the suprocess is importing the 32bit freeze package.

How can I stop this behavior and tell it to import from the appropriate source?

I am using Windows 7 x64 and PyDev as IDE, in case it is relevant, which it probably is.

EDIT: So apparently it works when the Main Script is executed from command line inside the source directory. Thanks to the comments below, I guess the problem has to with how PyDev sets the environment variables.

Answer

The issue you're facing arises because the subprocesses you spawn are inheriting the environment of the parent process (your main script). Since your parent process is using one Python installation (e.g., 64-bit), the subprocesses, when invoked, are picking up environment variables related to the parent environment (like the Python path, etc.) and using the wrong Python package (32-bit or 64-bit).

Explanation of the Problem:

  1. Python Environment Variables: The subprocess calls spawn new processes, but they inherit environment variables from the parent process, such as PYTHONPATH and PATH. These environment variables determine which Python libraries are loaded. If the parent process uses a 64-bit Python installation, the subprocesses will also pick up the 64-bit environment, even if you explicitly call the 32-bit Python executable.

  2. Wrong Architecture Loading: Since the subprocess is using the wrong environment, it's loading the wrong version of packages, such as the 64-bit version of cx_Freeze when the subprocess expects the 32-bit version, and vice versa. This leads to the error you are seeing: ImportError: DLL load failed: %1 is not a valid Win32 application.

Solution: Ensure Correct Environment for Each Python Version

To solve this, you'll need to explicitly set the environment variables for each subprocess to ensure that the correct Python installation (32-bit or 64-bit) is used.

Here’s a way to set the correct environment for each subprocess call:

Approach 1: Modify PATH and PYTHONHOME Environment Variables

You can modify the environment variables for each subprocess to point to the appropriate Python installation before calling the subprocess.call() function.

import subprocess
import os

def call_subprocess(python_executable, script):
    # Get the current environment variables
    env = os.environ.copy()

    # Set the correct Python executable and related environment variables for subprocess
    env["PATH"] = os.path.dirname(python_executable) + ";" + env.get("PATH", "")
    env["PYTHONHOME"] = os.path.dirname(python_executable)  # Optional: helps to control Python environment

    # Call the subprocess with the adjusted environment
    subprocess.call([python_executable, script], env=env)

# Main script that calls 32-bit and 64-bit Python
call_subprocess("python34-32", "test.py")  # Calling 32-bit Python
call_subprocess("python34", "test.py")     # Calling 64-bit Python

Explanation:

  1. Set PATH: By adjusting the PATH environment variable, you ensure that the subprocess will use the correct Python binary (32-bit or 64-bit). The os.path.dirname(python_executable) will give you the directory where the Python executable resides, which should contain the correct python.dll (or similar).

  2. Set PYTHONHOME (Optional): Sometimes setting the PYTHONHOME environment variable can help ensure that the subprocess uses the correct Python installation's libraries. However, it's not always required, so you can try it both with and without this line to see if it makes a difference.

  3. Use subprocess.call(): The env=env argument ensures that the subprocess uses the modified environment.

Approach 2: Using Full Path to Python Executable

Alternatively, you can use the full path to the Python executables (python34-32 or python34) to avoid issues with environment variables. This method doesn't rely on manipulating the environment variables.

import subprocess

# Full path to the 32-bit and 64-bit Python executables
python_32bit = r"C:\path\to\32bit\python34.exe"  # Full path to 32-bit python
python_64bit = r"C:\path\to\64bit\python34.exe"  # Full path to 64-bit python

def call_subprocess(python_executable, script):
    subprocess.call([python_executable, script])

# Main script that calls 32-bit and 64-bit Python
call_subprocess(python_32bit, "test.py")  # Calling 32-bit Python
call_subprocess(python_64bit, "test.py")  # Calling 64-bit Python

In this case, you directly specify the full path to the Python executable. The subprocess will not inherit the environment, and this avoids the issue of conflicting cx_Freeze installations.

Approach 3: Running Subprocesses with Different Virtual Environments

Another solution (especially if you have different Python environments for 32-bit and 64-bit) is to use virtualenv or venv to isolate the two Python environments. Each virtual environment will have its own set of installed packages, and this can prevent cross-contamination.

  1. Create two virtual environments for your 32-bit and 64-bit installations.
  2. Activate the respective environment when running subprocesses.

For example:

python32 -m venv venv_32bit
python64 -m venv venv_64bit

Then, run your subprocesses within the appropriate virtual environment, ensuring isolation between the two architectures.

Conclusion:

  • The main problem is that subprocesses inherit the environment of the calling script.
  • To ensure that each subprocess uses the correct Python installation (32-bit or 64-bit), you can either modify the environment variables (PATH, PYTHONHOME) or use the full path to the correct Python executables.
  • Additionally, using virtual environments is a good strategy to ensure isolation between the different Python versions.
#py