Ansible is a great tool used by many engineers to automate their work. Also, Ansible is growing very, very fast. It means, that there are many bugs inside that tool. Sometimes bug is not a problem – especially when it doesn’t exist – but not fully written documentation. This is probably the reason for the problem I want to describe. If you use Ansible for GCP deployment, you may face the failure reason saying “Please install google-auth library”. But what if you are sure, that you have this library installed?
TL;DR – just give me a solution!
Run your playbook with an additional parameter:
1 |
-e 'ansible_python_interpreter=/PATH/TO/PYTHON/INTERPRETER’ |
or add it to ansible.cfg file:
1 2 |
[defaults] interpreter_python=/PATH/TO/PYTHON/INTERPRETER |
When I faced “Please install google-auth library”?
This is my case I had few days ago. A bunch of information:
- I use macOS on a daily basis
- I have the default python version installed (Python 2.7), but I don’t use it
- Pyenv is my tool to switching between Python versions
- Almost all my projects (including Ansible) are built with virtualenv
Ok, so what happened to me is:
Normally, I use Python 3.7.4 (via pyenv) and Ansible 2.7.13 installed for that Python:
1 2 |
$ python --version Python 3.7.4 |
1 2 |
$ pyenv version 3.7.4 (set by /Users/amph/.python-version) |
1 2 |
$ which python /Users/amph/.pyenv/shims/python |
1 2 |
$ pip --version pip 19.0.3 from /Users/amph/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip (python 3.7) |
1 2 3 4 5 6 7 |
$ ansible --version ansible 2.7.13 config file = None configured module search path = ['/Users/amph/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /Users/amph/.pyenv/versions/3.7.4/lib/python3.7/site-packages/ansible executable location = /Users/amph/.pyenv/versions/3.7.4/bin/ansible python version = 3.7.4 (default, Aug 12 2019, 17:29:05) [Clang 10.0.1 (clang-1001.0.46.4)] |
So everything looks great, right? Yup. And Ansible had been working without any problem. Until I wanted to use GCP modules. And in this scenario, I decided to use virtualenv with Ansible 2.8 (some of the modules require Ansible 2.8, so I just wanted to have the newest, possible version). So I created a new directories structure, new virtualenv with Ansible, requirements file with requests and google-auth library and wrote the simple playbook (gcp_sql_instance module). And when I tried to run the playbook…
1 2 |
TASK [Create a Cloud SQL instance] ********************************************************************************************************************************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": false, "msg": "Please install the google-auth library”} |
But, wait. I have installed this library.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
(venv) $ pip list Package Version -------------- --------- ansible 2.8.5 asn1crypto 0.24.0 cachetools 3.1.1 certifi 2019.9.11 cffi 1.12.3 chardet 3.0.4 cryptography 2.7 google-auth 1.6.3 idna 2.8 Jinja2 2.10.1 MarkupSafe 1.1.1 pip 19.2.3 pyasn1 0.4.7 pyasn1-modules 0.2.6 pycparser 2.19 PyYAML 5.1.2 requests 2.22.0 rsa 4.0 setuptools 41.2.0 six 1.12.0 urllib3 1.25.6 wheel 0.33.6 |
So what the hell is wrong with that? I found this issue on GitHub, but it didn’t help me at all. But, you know, it’s Python, and I use virtualenv, hence I can make some changes in Ansible files without any fear, right? Yup, right!
What is the problem?
After some research, I found out that conditional responsible for my message is placed in lib/ansible/module_utils/gcp_utils.py. Please take a look at this code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
try: import google.auth import google.auth.compute_engine from google.oauth2 import service_account from google.auth.transport.requests import AuthorizedSession HAS_GOOGLE_LIBRARIES = True except ImportError: HAS_GOOGLE_LIBRARIES = False … def _validate(self): if not HAS_REQUESTS: self.module.fail_json(msg="Please install the requests library") if not HAS_GOOGLE_LIBRARIES: self.module.fail_json(msg="Please install the google-auth library") |
Ok, it makes sense… I was sure, that everything on my side is ok, but I created a simple python script to confirm that:
1 2 3 4 5 6 7 8 9 10 11 12 |
import sys try: import google.auth import google.auth.compute_engine from google.oauth2 import service_account from google.auth.transport.requests import AuthorizedSession HAS_GOOGLE_LIBRARIES = True except ImportError: HAS_GOOGLE_LIBRARIES = False print(HAS_GOOGLE_LIBRARIES) print(sys.path) |
As I expected, the output looked that:
1 2 |
True ['/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python37.zip', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python3.7', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python3.7/site-packages'] |
According to Python documentation, sys.path:
A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default.
Everything looks fine, what now?
Not good, everything looks fine. But the problem still exists. So I thought, that maybe I should print sys.path in the gcp_utils.py file and see what will happen. As I thought, I did. I modified gcp_utils.py one more time and change
1 2 |
if not HAS_GOOGLE_LIBRARIES: self.module.fail_json(msg="Please install the google-auth library") |
to this:
1 2 |
if not HAS_GOOGLE_LIBRARIES: self.module.fail_json(msg="Please install the google-auth library "+str(sys.path)) |
And also import sys library. After that, I run my playbook and the output changed to this one:
1 |
{"changed": false, "msg": "Please install the google-auth library ['/var/folders/7c/8z4_2y4971g82k82jktlf1sh0000gn/T/ansible_gcp_sql_instance_payload_756wpR/ansible_gcp_sql_instance_payload.zip', '/Library/Python/2.7/site-packages/jenkins_python-1.1-py2.7.egg', '/Library/Python/2.7/site-packages/jenkinsapi-0.3.6-py2.7.egg', '/Library/Python/2.7/site-packages/six-1.11.0-py2.7.egg', '/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg', '/Library/Python/2.7/site-packages/pytz-2017.3-py2.7.egg', '/Library/Python/2.7/site-packages/certifi-2018.1.18-py2.7.egg', '/Library/Python/2.7/site-packages/urllib3-1.22-py2.7.egg', '/Library/Python/2.7/site-packages/chardet-3.0.4-py2.7.egg', '/Library/Python/2.7/site-packages/PyForms-0.1.7.3-py2.7.egg', '/Library/Python/2.7/site-packages/pysettings-1.0.0-py2.7.egg', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Python/2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC']”} |
YEAH! Bingo! Wrong module paths. Ansible takes my default installation of Python to look for modules. But I don’t have installed google-auth for this Python version, so everything is clear right now.
But why?!
The answer is simple and we can find it in the documentation. Please take a look at this fragment:
Detects the target OS platform, distribution, and version, then consults a table listing the correct Python interpreter and path for each platform/distribution/version. If an entry is found, and /usr/bin/python is absent, uses the discovered interpreter (and path). If an entry is found, and /usr/bin/python is present, uses /usr/bin/python and issues a warning. This exception provides temporary compatibility with previous versions of Ansible that always defaulted to /usr/bin/python, so if you have installed Python and other dependencies at usr/bin/python on some hosts, Ansible will find and use them with this setting. If no entry is found, or the listed Python is not present on the target host, searches a list of common Python interpreter paths and uses the first one found; also issues a warning that future installation of another Python interpreter could alter the one chosen.
And I have /usr/bin/python binary.
1 2 |
$ ls -al /usr/bin/python -rwxr-xr-x 1 root wheel 66880 May 4 09:04 /usr/bin/python |
Ok, so how can I solve “Please install google-auth library”?
It’s more workaround than a solution (from my perspective), but you can use one of the following methods:
- Run your playbooks with -e ‘ansible_python_interpreter=/PATH/TO/YOUR/INTERPRETER’
- Add python_interpreter=/PATH/TO/YOUR/INTERPRETER to your ansible.cfg in [defaults] section.
To confirm that this solution works, I made another change in gcp_utils.py:
1 2 |
def _validate(self): self.module.fail_json(msg=str(sys.path)) |
Thanks to that, it should fail at validation and print me the modules’ paths. And it did that. And look what I saw after I run my playbook with -e ‘ansible_python_interpreter=/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/bin/python’:
1 2 |
TASK [Create a Cloud SQL instance] ********************************************************************************************************************************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": false, "msg": "System paths: ['/var/folders/7c/8z4_2y4971g82k82jktlf1sh0000gn/T/ansible_gcp_sql_instance_payload_nfvo45wi/ansible_gcp_sql_instance_payload.zip', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python37.zip', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python3.7', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv/lib/python3.7/site-packages']"} |
As you see, there is no “Please install google-auth library” anymore!
/Users/amph/Documents/Nauka/php-k8s-poc/ansible-setup/venv is of course a path to my virtualenv. After I had removed unnecessary changes in Ansible files, module worked as expected.
So if you face this problem, try to solve that in that way. But if you have a better solution, I will be more than grateful if you can share it with me (and all persons reading this post).