How does one install torchmeta for a ppc64le architecture in pytorch?
I was trying to use torchmeta in a ppc64le architecture. Unfortunately it's not been easy to install since ppc64le requires special binaries to work.
I eventually managed to get the right binaries for pytorch and torchvision by following these instructions (that prepend the right ibm channel with the conda binaries, plus installs all the required files too):
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
conda create -n my_new_env python=3.7 powerai=1.7.0
conda activate my_new_env
after that I proceeded to install the right version of torchmeta, which was 1.3.1 since ppc64le only has pytorch 1.3.1 and torchvision 0.4.2. So I did:
pip install torchmeta==1.3.1
but now I have a new error that it cannot find the right version of h5py compatible with what I want to do. The error message is to large to paste but I will paste what I hope are useful part of it:
(my_new_env) [miranda9@hal-login ~]$ pip install torchmeta==1.3.1
Collecting torchmeta==1.3.1
Using cached torchmeta-1.3.1-py3-none-any.whl (144 kB)
Requirement already satisfied: requests in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (2.22.0)
Requirement already satisfied: torchvision<0.6.0,>=0.4.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (0.4.2)
Requirement already satisfied: torch<1.5.0,>=1.3.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.3.1)
Processing ./.cache/pip/wheels/87/f5/ad/9f04a48453875e8054c19f9fe3f50cbbe0c09b956835555019/Pillow-6.2.2-cp37-cp37m-linux_ppc64le.whl
Requirement already satisfied: numpy>=1.14.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.17.4)
Requirement already satisfied: tqdm>=4.0.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (4.36.1)
Collecting h5py~=2.9.0
Using cached h5py-2.9.0.tar.gz (287 kB)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2020.6.20)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (1.25.10)
Requirement already satisfied: idna<2.9,>=2.5 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2.8)
Requirement already satisfied: six in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchvision<0.6.0,>=0.4.0->torchmeta==1.3.1) (1.13.0)
Building wheels for collected packages: h5py
Building wheel for h5py (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ccg1oj0n
cwd: /tmp/pip-install-bpmeop26/h5py/
Complete output (1321 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-ppc64le-3.7
creating build/lib.linux-ppc64le-3.7/h5py
copying h5py/__init__.py -> build/lib.linux-ppc64le-3.7/h5py
copying h5py/h5py_warnings.py -> build/lib.linux-ppc64le-3.7/h5py
copying h5py/highlevel.py -> build/lib.linux-ppc64le-3.7/h5py
copying h5py/ipy_completer.py -> build/lib.linux-ppc64le-3.7/h5py
copying h5py/version.py -> build/lib.linux-ppc64le-3.7/h5py
creating build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/attrs.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/base.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/compat.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/dataset.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/datatype.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/dims.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/files.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/filters.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/group.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/selections.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/selections2.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
copying h5py/_hl/vds.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
creating build/lib.linux-ppc64le-3.7/h5py/tests
copying h5py/tests/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests
copying h5py/tests/common.py -> build/lib.linux-ppc64le-3.7/h5py/tests
creating build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_attrs.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_base.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_dataset.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_file_image.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_group.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_h5.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_h5f.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_h5p.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_h5t.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_objects.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_selections.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
copying h5py/tests/old/test_slicing.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
creating build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_filters.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
copying h5py/tests/hl/test_threads.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
creating build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
running build_ext
Autodetected HDF5 1.10.2
********************************************************************************
Summary of the h5py configuration
Path to HDF5: None
HDF5 Version: '1.10.2'
MPI Enabled: False
Rebuild Required: True
********************************************************************************
Executing api_gen rebuild of defs
Executing cythonize()
[ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
/tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
...
/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^
In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
/tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for h5py
Running setup.py clean for h5py
Failed to build h5py
DEPRECATION: Could not build wheels for h5py which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: Pillow, h5py, torchmeta
Attempting uninstall: Pillow
Found existing installation: Pillow 7.1.2
Uninstalling Pillow-7.1.2:
Successfully uninstalled Pillow-7.1.2
Attempting uninstall: h5py
Found existing installation: h5py 2.8.0
Uninstalling h5py-2.8.0:
Successfully uninstalled h5py-2.8.0
Running setup.py install for h5py ... error
ERROR: Command errored out with exit status 1:
command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py
...
copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
running build_ext
Autodetected HDF5 1.10.2
********************************************************************************
Summary of the h5py configuration
Path to HDF5: None
HDF5 Version: '1.10.2'
MPI Enabled: False
Rebuild Required: True
********************************************************************************
Executing cythonize()
[ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
/tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd
...
warning: h5py/api_types_hdf5.pxd:730:6: 'H5Z_ERROR_EDC' redeclared
warning: h5py/api_types_hdf5.pxd:731:6: 'H5Z_DISABLE_EDC' redeclared
warning: h5py/api_types_hdf5.pxd:732:6: 'H5Z_ENABLE_EDC' redeclared
warning: h5py/api_types_hdf5.pxd:733:6: 'H5Z_NO_EDC' redeclared
building 'h5py.defs' extension
creating build/temp.linux-ppc64le-3.7
creating build/temp.linux-ppc64le-3.7/tmp
creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26
creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py
creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py
gcc -pthread -B /home/miranda9/.conda/envs/my_new_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-bpmeop26/h5py/lzf -I/opt/local/include -I/usr/local/include -I/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include -I/home/miranda9/.conda/envs/my_new_env/include/python3.7m -c /tmp/pip-install-bpmeop26/h5py/h5py/defs.c -o build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py/defs.o
In file included from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:26,
from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:
/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^
In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
/tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Rolling back uninstall of h5py
Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py
Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py-2.8.0-py3.7.egg-info
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py-2.8.0-py3.7.egg-info
ERROR: Command errored out with exit status 1: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py Check the logs for full command output.
anyone know how I can successfully install a working torchmeta version in a ppc64le (using wmcle 1.7.0)?
Answer
The issue you're facing with installing h5py on the ppc64le architecture likely stems from a lack of prebuilt binaries (wheels) for this platform. Instead, pip tries to build the package from source, which often fails due to missing dependencies, incompatible versions, or architecture-specific issues.
Here’s a systematic way to resolve the problem:
1. Install h5py via Conda
h5py is available as a prebuilt package through Conda, which should work seamlessly on ppc64le. Use the IBM Conda channel (already configured in your setup):
conda install h5py=2.9.0
2. Ensure Dependencies are Met
If the Conda installation doesn't work or you need pip, ensure the following dependencies are installed:
- HDF5 library compatible with
h5py. - Development tools (
gcc,make, etc.) for building C extensions.
Install the HDF5 library and development tools:
conda install -c conda-forge hdf5=1.10.4
conda install -c conda-forge gcc_linux-ppc64le
3. Build h5py from Source
If you must build h5py from source, explicitly point it to the HDF5 installation:
pip install --no-binary=h5py h5py
You can specify the HDF5 library path if necessary:
HDF5_DIR=/path/to/hdf5 pip install --no-binary=h5py h5py
Replace /path/to/hdf5 with the actual installation directory of the HDF5 library.
4. Verify Installed Packages
Ensure all versions of related libraries (e.g., numpy, torch, torchvision) are compatible. Since you're using torchmeta 1.3.1, verify it aligns with torch 1.3.1 and torchvision 0.4.2, which seems to be the case.
5. Debug Installation Issues
If the installation still fails, check for detailed error messages in the logs. Focus on:
- Missing libraries or headers (e.g., HDF5).
- GCC or compilation-related errors.
Alternative Workaround
If none of these solutions work, consider contacting the maintainers of h5py or the IBM AI Conda channel to check for prebuilt binaries specific to ppc64le.
Let me know if you encounter further issues—I can assist with debugging or exploring other solutions.
