I'm just trying to run access GCP from python in a shared gitlab runner. I suspect there is something about the runner's OS that I don't understand. I can't seem to import google.cloud. I get ImportError: No module named 'google.cloud' no matter what I try.
My .gitlab-ci.yml
---
before_script:
- chmod -v -v +x build/*.sh
test_in_test_job:
environment:
name: test
script:
- bash -x build/test.sh
The test shell script:
#!/usr/bin/env bash
# create verbose logging output for remote execution
function verbose() {
# Where is the executable that is being called
command -v "$1"
# What version is the executable being called
"$1" --version
# And make the call
"${@}"
}
function install_python {
verbose apt-get install python3 python3-pip -y
verbose pip3 install virtualenv
VENV_NAME="gitlabci"
verbose virtualenv "${VENV_NAME}"
source "${VENV_NAME}/bin/activate"
verbose pip3 install --requirement requirements.txt
}
function run_tests {
verbose python3 build/test_google_import.py
}
apt-get update
install_python
run_tests
# finally exit virtualenv
deactivate
The python code:
#!/usr/bin/env python3
def import_info(module):
print(module)
print(module.__class__)
print(dir(module))
import google
import_info(google)
import google.cloud
import_info(google.cloud)
And the log output
$ bash -x build/test.sh
+ apt-get update
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:6 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [461 kB]
Get:7 http://deb.debian.org/debian stretch-updates/main amd64 Packages [5152 B]
Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7089 kB]
Fetched 7861 kB in 1s (4528 kB/s)
Reading package lists...
+ install_python
+ verbose apt-get install python3 python3-pip -y
+ command -v apt-get
/usr/bin/apt-get
+ apt-get --version
apt 1.4.8 (amd64)
Supported modules:
*Ver: Standard .deb
*Pkg: Debian dpkg interface (Priority 30)
Pkg: Debian APT solver interface (Priority -1000)
Pkg: Debian APT planner interface (Priority -1000)
S.L: 'deb' Debian binary tree
S.L: 'deb-src' Debian source tree
Idx: Debian Source Index
Idx: Debian Package Index
Idx: Debian Translation Index
Idx: Debian dpkg status file
Idx: Debian deb file
Idx: Debian dsc file
Idx: Debian control file
Idx: EDSP scenario file
Idx: EIPP scenario file
+ apt-get install python3 python3-pip -y
Reading package lists...
Building dependency tree...
Reading state information...
python3 is already the newest version (3.5.3-1).
python3 set to manually installed.
The following additional packages will be installed:
build-essential dbus dpkg-dev fakeroot libalgorithm-diff-perl
libalgorithm-diff-xs-perl libalgorithm-merge-perl libapparmor1
libdbus-glib-1-2 libfakeroot libpython3-dev libpython3.5 libpython3.5-dev
python-pip-whl python3-cffi-backend python3-crypto python3-cryptography
python3-dbus python3-dev python3-gi python3-idna python3-keyring
python3-keyrings.alt python3-pkg-resources python3-pyasn1
python3-secretstorage python3-setuptools python3-six python3-wheel
python3-xdg python3.5-dev
Suggested packages:
default-dbus-session-bus | dbus-session-bus debian-keyring
python3-crypto-dbg python-crypto-doc python-cryptography-doc
python3-cryptography-vectors python-dbus-doc python3-dbus-dbg gnome-keyring
libkf5wallet-bin gir1.2-gnomekeyring-1.0 python3-pykde4 doc-base
python-secretstorage-doc python-setuptools-doc
The following NEW packages will be installed:
build-essential dbus dpkg-dev fakeroot libalgorithm-diff-perl
libalgorithm-diff-xs-perl libalgorithm-merge-perl libapparmor1
libdbus-glib-1-2 libfakeroot libpython3-dev libpython3.5 libpython3.5-dev
python-pip-whl python3-cffi-backend python3-crypto python3-cryptography
python3-dbus python3-dev python3-gi python3-idna python3-keyring
python3-keyrings.alt python3-pip python3-pkg-resources python3-pyasn1
python3-secretstorage python3-setuptools python3-six python3-wheel
python3-xdg python3.5-dev
0 upgraded, 32 newly installed, 0 to remove and 13 not upgraded.
Need to get 45.2 MB of archives.
After this operation, 72.7 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 libapparmor1 amd64 2.11.0-3+deb9u2 [78.9 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 dbus amd64 1.10.26-0+deb9u1 [210 kB]
Get:3 http://deb.debian.org/debian stretch/main amd64 dpkg-dev all 1.18.25 [1595 kB]
Get:4 http://deb.debian.org/debian stretch/main amd64 build-essential amd64 12.3 [7346 B]
Get:5 http://deb.debian.org/debian stretch/main amd64 libfakeroot amd64 1.21-3.1 [45.7 kB]
Get:6 http://deb.debian.org/debian stretch/main amd64 fakeroot amd64 1.21-3.1 [85.6 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 libalgorithm-diff-perl all 1.19.03-1 [48.7 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 libalgorithm-diff-xs-perl amd64 0.04-4+b2 [11.6 kB]
Get:9 http://deb.debian.org/debian stretch/main amd64 libalgorithm-merge-perl all 0.08-3 [12.7 kB]
Get:10 http://deb.debian.org/debian stretch/main amd64 libdbus-glib-1-2 amd64 0.108-2 [206 kB]
Get:11 http://deb.debian.org/debian stretch/main amd64 libpython3.5 amd64 3.5.3-1+deb9u1 [1372 kB]
Get:12 http://deb.debian.org/debian stretch/main amd64 libpython3.5-dev amd64 3.5.3-1+deb9u1 [37.7 MB]
Get:13 http://deb.debian.org/debian stretch/main amd64 libpython3-dev amd64 3.5.3-1 [18.7 kB]
Get:14 http://deb.debian.org/debian stretch/main amd64 python-pip-whl all 9.0.1-2 [1406 kB]
Get:15 http://deb.debian.org/debian stretch/main amd64 python3-cffi-backend amd64 1.9.1-2 [70.1 kB]
Get:16 http://deb.debian.org/debian stretch/main amd64 python3-crypto amd64 2.6.1-7 [259 kB]
Get:17 http://deb.debian.org/debian stretch/main amd64 python3-idna all 2.2-1 [32.7 kB]
Get:18 http://deb.debian.org/debian stretch/main amd64 python3-pyasn1 all 0.1.9-2 [34.5 kB]
Get:19 http://deb.debian.org/debian stretch/main amd64 python3-pkg-resources all 33.1.1-1 [137 kB]
Get:20 http://deb.debian.org/debian stretch/main amd64 python3-setuptools all 33.1.1-1 [215 kB]
Get:21 http://deb.debian.org/debian stretch/main amd64 python3-six all 1.10.0-3 [14.4 kB]
Get:22 http://deb.debian.org/debian stretch/main amd64 python3-cryptography amd64 1.7.1-3 [210 kB]
Get:23 http://deb.debian.org/debian stretch/main amd64 python3-dbus amd64 1.2.4-1+b1 [184 kB]
Get:24 http://deb.debian.org/debian stretch/main amd64 python3.5-dev amd64 3.5.3-1+deb9u1 [413 kB]
Get:25 http://deb.debian.org/debian stretch/main amd64 python3-dev amd64 3.5.3-1 [1154 B]
Get:26 http://deb.debian.org/debian stretch/main amd64 python3-gi amd64 3.22.0-2 [473 kB]
Get:27 http://deb.debian.org/debian stretch/main amd64 python3-secretstorage all 2.3.1-2 [14.2 kB]
Get:28 http://deb.debian.org/debian stretch/main amd64 python3-keyring all 10.1-1 [36.8 kB]
Get:29 http://deb.debian.org/debian stretch/main amd64 python3-keyrings.alt all 1.3-1 [16.2 kB]
Get:30 http://deb.debian.org/debian stretch/main amd64 python3-pip all 9.0.1-2 [142 kB]
Get:31 http://deb.debian.org/debian stretch/main amd64 python3-wheel all 0.29.0-2 [51.8 kB]
Get:32 http://deb.debian.org/debian stretch/main amd64 python3-xdg all 0.25-4 [35.7 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 45.2 MB in 0s (69.3 MB/s)
Selecting previously unselected package libapparmor1:amd64.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 29375 files and directories currently installed.)
Preparing to unpack .../00-libapparmor1_2.11.0-3+deb9u2_amd64.deb ...
Unpacking libapparmor1:amd64 (2.11.0-3+deb9u2) ...
Selecting previously unselected package dbus.
Preparing to unpack .../01-dbus_1.10.26-0+deb9u1_amd64.deb ...
Unpacking dbus (1.10.26-0+deb9u1) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../02-dpkg-dev_1.18.25_all.deb ...
Unpacking dpkg-dev (1.18.25) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../03-build-essential_12.3_amd64.deb ...
Unpacking build-essential (12.3) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../04-libfakeroot_1.21-3.1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.21-3.1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../05-fakeroot_1.21-3.1_amd64.deb ...
Unpacking fakeroot (1.21-3.1) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../06-libalgorithm-diff-perl_1.19.03-1_all.deb ...
Unpacking libalgorithm-diff-perl (1.19.03-1) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../07-libalgorithm-diff-xs-perl_0.04-4+b2_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-4+b2) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../08-libalgorithm-merge-perl_0.08-3_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-3) ...
Selecting previously unselected package libdbus-glib-1-2:amd64.
Preparing to unpack .../09-libdbus-glib-1-2_0.108-2_amd64.deb ...
Unpacking libdbus-glib-1-2:amd64 (0.108-2) ...
Selecting previously unselected package libpython3.5:amd64.
Preparing to unpack .../10-libpython3.5_3.5.3-1+deb9u1_amd64.deb ...
Unpacking libpython3.5:amd64 (3.5.3-1+deb9u1) ...
Selecting previously unselected package libpython3.5-dev:amd64.
Preparing to unpack .../11-libpython3.5-dev_3.5.3-1+deb9u1_amd64.deb ...
Unpacking libpython3.5-dev:amd64 (3.5.3-1+deb9u1) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../12-libpython3-dev_3.5.3-1_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.5.3-1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../13-python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) ...
Selecting previously unselected package python3-cffi-backend.
Preparing to unpack .../14-python3-cffi-backend_1.9.1-2_amd64.deb ...
Unpacking python3-cffi-backend (1.9.1-2) ...
Selecting previously unselected package python3-crypto.
Preparing to unpack .../15-python3-crypto_2.6.1-7_amd64.deb ...
Unpacking python3-crypto (2.6.1-7) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../16-python3-idna_2.2-1_all.deb ...
Unpacking python3-idna (2.2-1) ...
Selecting previously unselected package python3-pyasn1.
Preparing to unpack .../17-python3-pyasn1_0.1.9-2_all.deb ...
Unpacking python3-pyasn1 (0.1.9-2) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../18-python3-pkg-resources_33.1.1-1_all.deb ...
Unpacking python3-pkg-resources (33.1.1-1) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../19-python3-setuptools_33.1.1-1_all.deb ...
Unpacking python3-setuptools (33.1.1-1) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../20-python3-six_1.10.0-3_all.deb ...
Unpacking python3-six (1.10.0-3) ...
Selecting previously unselected package python3-cryptography.
Preparing to unpack .../21-python3-cryptography_1.7.1-3_amd64.deb ...
Unpacking python3-cryptography (1.7.1-3) ...
Selecting previously unselected package python3-dbus.
Preparing to unpack .../22-python3-dbus_1.2.4-1+b1_amd64.deb ...
Unpacking python3-dbus (1.2.4-1+b1) ...
Selecting previously unselected package python3.5-dev.
Preparing to unpack .../23-python3.5-dev_3.5.3-1+deb9u1_amd64.deb ...
Unpacking python3.5-dev (3.5.3-1+deb9u1) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../24-python3-dev_3.5.3-1_amd64.deb ...
Unpacking python3-dev (3.5.3-1) ...
Selecting previously unselected package python3-gi.
Preparing to unpack .../25-python3-gi_3.22.0-2_amd64.deb ...
Unpacking python3-gi (3.22.0-2) ...
Selecting previously unselected package python3-secretstorage.
Preparing to unpack .../26-python3-secretstorage_2.3.1-2_all.deb ...
Unpacking python3-secretstorage (2.3.1-2) ...
Selecting previously unselected package python3-keyring.
Preparing to unpack .../27-python3-keyring_10.1-1_all.deb ...
Unpacking python3-keyring (10.1-1) ...
Selecting previously unselected package python3-keyrings.alt.
Preparing to unpack .../28-python3-keyrings.alt_1.3-1_all.deb ...
Unpacking python3-keyrings.alt (1.3-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../29-python3-pip_9.0.1-2_all.deb ...
Unpacking python3-pip (9.0.1-2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../30-python3-wheel_0.29.0-2_all.deb ...
Unpacking python3-wheel (0.29.0-2) ...
Selecting previously unselected package python3-xdg.
Preparing to unpack .../31-python3-xdg_0.25-4_all.deb ...
Unpacking python3-xdg (0.25-4) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-cffi-backend (1.9.1-2) ...
Setting up python3-crypto (2.6.1-7) ...
Setting up libdbus-glib-1-2:amd64 (0.108-2) ...
Setting up libpython3.5:amd64 (3.5.3-1+deb9u1) ...
Setting up python3-idna (2.2-1) ...
Setting up python3-xdg (0.25-4) ...
Setting up python3-keyrings.alt (1.3-1) ...
Setting up python3-six (1.10.0-3) ...
Setting up python3-wheel (0.29.0-2) ...
Setting up python3-pkg-resources (33.1.1-1) ...
Setting up python3-gi (3.22.0-2) ...
Setting up libpython3.5-dev:amd64 (3.5.3-1+deb9u1) ...
Setting up python3-pyasn1 (0.1.9-2) ...
Setting up dpkg-dev (1.18.25) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libapparmor1:amd64 (2.11.0-3+deb9u2) ...
Setting up libfakeroot:amd64 (1.21-3.1) ...
Setting up python3-pip (9.0.1-2) ...
Setting up libalgorithm-diff-perl (1.19.03-1) ...
Setting up dbus (1.10.26-0+deb9u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up python3-setuptools (33.1.1-1) ...
Setting up python3.5-dev (3.5.3-1+deb9u1) ...
Setting up libpython3-dev:amd64 (3.5.3-1) ...
Setting up python3-cryptography (1.7.1-3) ...
Setting up python3-dbus (1.2.4-1+b1) ...
Setting up build-essential (12.3) ...
Setting up python3-dev (3.5.3-1) ...
Setting up fakeroot (1.21-3.1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up libalgorithm-diff-xs-perl (0.04-4+b2) ...
Setting up python3-secretstorage (2.3.1-2) ...
Setting up python3-keyring (10.1-1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
+ verbose pip3 install virtualenv
+ command -v pip3
/usr/bin/pip3
+ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)
+ pip3 install virtualenv
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/7c/17/9b7b6cddfd255388b58c61e25b091047f6814183e1d63741c8df8dcd65a2/virtualenv-16.1.0-py2.py3-none-any.whl (1.9MB)
Installing collected packages: virtualenv
Successfully installed virtualenv-16.1.0
+ VENV_NAME=gitlabci
+ verbose virtualenv gitlabci
+ command -v virtualenv
/usr/local/bin/virtualenv
+ virtualenv --version
16.1.0
+ virtualenv gitlabci
Using base prefix '/usr'
New python executable in /builds/powerco-analytics/poco-analytics-terraform/gitlabci/bin/python3
Also creating executable in /builds/powerco-analytics/poco-analytics-terraform/gitlabci/bin/python
Installing setuptools, pip, wheel...
done.
+ source gitlabci/bin/activate
++ deactivate nondestructive
++ unset -f pydoc
++ '[' -z '' ']'
++ '[' -z '' ']'
++ '[' -n /bin/bash ']'
++ hash -r
++ '[' -z '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/builds/powerco-analytics/poco-analytics-terraform/gitlabci
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/usr/local/bundle/bin:/usr/local/bundle/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/builds/powerco-analytics/poco-analytics-terraform/gitlabci/bin:/usr/local/bundle/bin:/usr/local/bundle/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' -z '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' x '!=' x ']'
+++ basename /builds/powerco-analytics/poco-analytics-terraform/gitlabci
++ PS1='(gitlabci) '
++ export PS1
++ alias pydoc
++ '[' -n /bin/bash ']'
++ hash -r
+ verbose pip3 install --requirement requirements.txt
+ command -v pip3
/builds/powerco-analytics/poco-analytics-terraform/gitlabci/bin/pip3
+ pip3 --version
pip 18.1 from /builds/powerco-analytics/poco-analytics-terraform/gitlabci/lib/python3.5/site-packages/pip (python 3.5)
+ pip3 install --requirement requirements.txt
Collecting pytest>=3.7 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/19/80/1ac71d332302a89e8637456062186bf397abc5a5b663c1919b73f4d68b1b/pytest-4.0.2-py2.py3-none-any.whl (217kB)
Collecting behave>=1.2 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/a8/6c/ec9169548b6c4cb877aaa6773408ca08ae2a282805b958dbc163cb19822d/behave-1.2.6-py2.py3-none-any.whl (136kB)
Collecting google-cloud>=0.34 (from -r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/ba/b1/7c54d1950e7808df06642274e677dbcedba57f75307adf2e5ad8d39e5e0e/google_cloud-0.34.0-py2.py3-none-any.whl
Collecting google-auth>=1.6 (from -r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/4e/85/71b2dfbf5b4241cd031cc333ed71f90a271074a97cb2c517bb65f07a1a90/google_auth-1.6.2-py2.py3-none-any.whl (73kB)
Requirement already satisfied: setuptools in ./gitlabci/lib/python3.5/site-packages (from pytest>=3.7->-r requirements.txt (line 1)) (40.6.3)
Collecting more-itertools>=4.0.0 (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/79/b1/eace304ef66bd7d3d8b2f78cc374b73ca03bc53664d78151e9df3b3996cc/more_itertools-4.3.0-py3-none-any.whl (48kB)
Collecting attrs>=17.4.0 (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/3a/e1/5f9023cc983f1a628a8c2fd051ad19e76ff7b142a0faf329336f9a62a514/attrs-18.2.0-py2.py3-none-any.whl
Collecting py>=1.5.0 (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/3e/c7/3da685ef117d42ac8d71af525208759742dd235f8094221fdaafcd3dba8f/py-1.7.0-py2.py3-none-any.whl (83kB)
Collecting pathlib2>=2.2.0; python_version < "3.6" (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/2a/46/c696dcf1c7aad917b39b875acdc5451975e3a9b4890dca8329983201c97a/pathlib2-2.3.3-py2.py3-none-any.whl
Collecting pluggy>=0.7 (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/1c/e7/017c262070af41fe251401cb0d0e1b7c38f656da634cd0c15604f1f30864/pluggy-0.8.0-py2.py3-none-any.whl
Collecting atomicwrites>=1.0 (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/3a/9a/9d878f8d885706e2530402de6417141129a943802c084238914fa6798d97/atomicwrites-1.2.1-py2.py3-none-any.whl
Collecting six>=1.10.0 (from pytest>=3.7->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting parse-type>=0.4.2 (from behave>=1.2->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/a2/c9/e6fd8092a5a06f2519ec434ca8e9e42238384f64c9b659456d98b0593b89/parse_type-0.4.2-py2.py3-none-any.whl
Collecting parse>=1.8.2 (from behave>=1.2->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/b6/98/809e53e5778c59c4af9eb920605e7a8ab439407efbe89a6d51a46efd1937/parse-1.9.0.tar.gz
Collecting pyasn1-modules>=0.2.1 (from google-auth>=1.6->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl (62kB)
Collecting rsa>=3.1.4 (from google-auth>=1.6->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
Collecting cachetools>=2.0.0 (from google-auth>=1.6->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/76/7e/08cd3846bebeabb6b1cfc4af8aae649d90249b4aeed080bddb5297f1d73b/cachetools-3.0.0-py2.py3-none-any.whl
Collecting pyasn1<0.5.0,>=0.4.1 (from pyasn1-modules>=0.2.1->google-auth>=1.6->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl (72kB)
Building wheels for collected packages: parse
Running setup.py bdist_wheel for parse: started
Running setup.py bdist_wheel for parse: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/ef/db/c6/18568a2cc574848f3996ac4552241fbec046b7be29feb2077d
Successfully built parse
Installing collected packages: six, more-itertools, attrs, py, pathlib2, pluggy, atomicwrites, pytest, parse, parse-type, behave, google-cloud, pyasn1, pyasn1-modules, rsa, cachetools, google-auth
Successfully installed atomicwrites-1.2.1 attrs-18.2.0 behave-1.2.6 cachetools-3.0.0 google-auth-1.6.2 google-cloud-0.34.0 more-itertools-4.3.0 parse-1.9.0 parse-type-0.4.2 pathlib2-2.3.3 pluggy-0.8.0 py-1.7.0 pyasn1-0.4.4 pyasn1-modules-0.2.2 pytest-4.0.2 rsa-4.0 six-1.12.0
+ run_tests
+ verbose python3 build/test_google_import.py
+ command -v python3
/builds/powerco-analytics/poco-analytics-terraform/gitlabci/bin/python3
+ python3 --version
Python 3.5.3
+ python3 build/test_google_import.py
<module 'google' (namespace)>
<class 'module'>
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
Traceback (most recent call last):
File "build/test_google_import.py", line 11, in <module>
import google.cloud
ImportError: No module named 'google.cloud'
For what little this is worth, the python works on my machine (OS X 10.14.2 and Python 3.7.1).
I've tried with and without virtualenv, and I've installed the GCP SDK too. None of which changed the import error.
Answer
It looks like your google.cloud import issue is likely due to missing dependencies or incorrect setup in the environment. Here's a checklist and potential solutions to resolve the ImportError: No module named 'google.cloud':
1. Ensure that google-cloud is installed
You need to make sure that the google-cloud package is installed in the virtual environment. You might be installing google-cloud under the wrong Python environment or not installing it at all. Add this to your requirements.txt file to ensure it's installed during the setup:
google-cloud
2. Recheck Virtual Environment Setup
Make sure you're activating the virtual environment properly and that the pip command is installing packages into the correct environment. In your script, ensure you're activating the virtual environment with:
source "${VENV_NAME}/bin/activate"
Then run the following command to confirm that the google-cloud package is installed in the environment:
pip list
Check if google-cloud is in the list.
3. Install google-cloud explicitly
In case it's not listed in requirements.txt, you can explicitly install it using pip:
pip install google-cloud
4. Check Python Version Compatibility
Ensure that the Python version you're using in the GitLab Runner (Python 3.5 in your case) is compatible with the version of google-cloud. The google-cloud library supports Python 3.5 and above, so this shouldn't be an issue, but it's good to check.
5. Verify Correct Python Interpreter
In your CI/CD pipeline, make sure that the correct Python interpreter is being used. Since you're using virtualenv, ensure that the python3 interpreter from the virtual environment is being used, not the system Python. Double-check that you're running the script inside the virtual environment by using:
which python3
This should return the path to the Python interpreter inside the virtual environment (something like /path/to/your/virtualenv/bin/python3).
6. Modify requirements.txt if needed
If you're using google-cloud for specific services (like storage or Pub/Sub), you might need to install the corresponding submodule. For example, for Google Cloud Storage, you can add this to your requirements.txt:
google-cloud-storage
For other services, you might need specific modules like google-cloud-bigquery, etc.
7. Add Debugging Output
To debug where the issue is coming from, you can modify your test script to print out the paths of the installed packages and confirm where Python is loading modules from:
import sys
print(sys.path)
import google
import_info(google)
This will show the directories Python is searching for modules and might help you spot if something is wrong with the module installation or environment setup.
8. Rebuild the Virtual Environment
If you're still facing issues, try deleting the virtual environment and starting fresh:
rm -rf "${VENV_NAME}"
virtualenv "${VENV_NAME}"
source "${VENV_NAME}/bin/activate"
pip install --requirement requirements.txt
Final Suggestions:
If the error persists after these steps, try printing the Python environment details (e.g., python3 --version, pip freeze, and which python3) to ensure that the setup is correct. If the runner is using a containerized environment, also verify that the environment has internet access for downloading the dependencies.
Let me know if you need further assistance!
