Sphinx autodoc gets stuck on random.choice() using mock

ghz 11hours ago ⋅ 1 views

This is a continuation of a question I originally asked in this post: random.choice error due to np.linspace and np.logspace

In this question, I have stripped down the problem to the bare minimum and was able to reproduce the problem where Sphinx is hanging up on the random.choice() function.

Here is the Python code in the file randor_test.py; it runs in PyCharm:

import random
import numpy as np

def rand_test(svr_C, svr_gamma):
    """This is test docstring

    #. item one
    #. item two
    """
    ml_params = {'C': random.choice(svr_C), 'gamma': random.choice(svr_gamma)}
    return ml_params

svr_C = list(np.linspace(50, 300, 10))
svr_gamma = list(np.logspace(-4, -2, 3))

rand_result = rand_test(svr_C, svr_gamma)

for i in rand_result:
    print(i, rand_result[i])

I set up the Sphinx directory and followed all instructions in this post: Getting Started with Sphinx...

After running make html I receive the following error:

WARNING: autodoc: failed to import module 'randor_test'; the following exception was raised:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sphinx/ext/autodoc/importer.py", line 32, in import_module
    return importlib.import_module(modname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/kellihed/PAC_projects/basic_0629/src/randor_test.py", line 19, in <module>
    rand_result = rand_test(svr_C, svr_gamma)
  File "/Users/kellihed/PAC_projects/basic_0629/src/randor_test.py", line 10, in rand_test
    ml_params = {'C': random.choice(svr_C), 'gamma': random.choice(svr_gamma)}
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/random.py", line 261, in choice
    raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence

My file structure follows the suggested layout:

|--docs
|  |
|  |--Makefile
|  |--build
|  |  |
|  |  |--doctrees
|  |  |--html
|  |  |--_static
|  |  |--genindex.html
|  |  |--index.html 
|  |  |--modules.html
|  |--make.bat
|  |--source
|     |
|     |--_static
|     |--_templates
|     |--conf.py
|     |--index.rst
|     |--modules.rst
|     |--randor_test.rst
|
|--src
|  |__pychache__
|  |  |
|  |  |--randor_test.cpython-37.pyc
|  |
|  |--randor_test.py

I have the following ignore imports statement in conf.py:

autodoc_mock_imports = ["random", "numpy"]

Below is my conf.py file:

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))


# -- Project information -----------------------------------------------------

project = 'random_test'
copyright = '2020, DK'
author = 'DK'

# The full version, including alpha/beta/rc tags
release = '0.1'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc']
autodoc_mock_imports = ["random", "numpy"]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

Answer

The issue you're encountering is due to the fact that when Sphinx is trying to generate documentation and execute the code, it's hitting a problem with the random.choice() function. Specifically, it seems that Sphinx is trying to run the code, but it's unable to find a valid sequence from which to randomly select an item, leading to the IndexError: Cannot choose from an empty sequence error.

Let's go over the main parts of the problem and potential solutions:

1. Mocking random and numpy Imports

The problem is that Sphinx cannot import the random and numpy modules (which are being mocked in conf.py using autodoc_mock_imports). This may cause Sphinx to attempt to run code that relies on these modules but fails due to their unavailability in the context of the documentation build.

The mock imports you’ve defined in the conf.py file prevent Sphinx from importing these libraries but still attempt to run code that relies on them. The mock prevents any real code execution and can lead to empty lists being passed to random.choice().

2. Fixing the Issue

To resolve the issue, we have a few options:

  • Ensure Mocking of Imports is Correct: The autodoc_mock_imports directive in conf.py is correct for mocking imports, but it may not be enough if the code execution is still expected. You might need to adjust how or when these imports are mocked or ensure the random.choice() doesn't get executed when generating documentation.

  • Conditionally Mock Code Execution: A better approach is to ensure that the code inside randor_test.py doesn't get executed during the Sphinx documentation generation. One way to do this is to wrap the code in a check that ensures it doesn't run during the Sphinx build.

3. Proposed Solution: Modify randor_test.py to Avoid Execution During Sphinx Build

You can add a conditional block to ensure that the code does not run when Sphinx is building the documentation. Here’s how you can modify your randor_test.py file:

import random
import numpy as np

def rand_test(svr_C, svr_gamma):
    """This is test docstring

    #. item one
    #. item two
    """
    ml_params = {'C': random.choice(svr_C), 'gamma': random.choice(svr_gamma)}
    return ml_params

# Ensure that the code only runs when the script is executed directly, not during Sphinx build
if __name__ == "__main__":
    svr_C = list(np.linspace(50, 300, 10))
    svr_gamma = list(np.logspace(-4, -2, 3))

    rand_result = rand_test(svr_C, svr_gamma)

    for i in rand_result:
        print(i, rand_result[i])

This modification uses the if __name__ == "__main__": construct, which ensures that the code block inside it (including calls to random.choice()) is only executed when the script is run directly, and not when Sphinx tries to import the module during documentation build.

4. Verify Sphinx Configuration for Mocking

You mentioned that autodoc_mock_imports = ["random", "numpy"] is set in conf.py. This is correct for ensuring that Sphinx doesn't try to import these libraries. However, make sure that Sphinx is properly handling these mock imports and not accidentally executing code that requires them.

5. Additional Debugging

If the error persists, you can add additional debugging statements in your Sphinx build process:

  • Check if Sphinx is Importing randor_test.py Correctly: You can add print statements inside your randor_test.py to check if it is being loaded correctly by Sphinx.
  • Ensure Correct Sphinx Extensions and Path Setup: Verify that the path setup and extensions in conf.py are correct and that the Sphinx environment can properly locate randor_test.py.

Summary of Fixes:

  1. Modify randor_test.py to ensure that code execution only happens when running the script directly (using if __name__ == "__main__":).
  2. Verify Sphinx Configuration to ensure that autodoc_mock_imports is correctly set and that the code is not executed during Sphinx's build process.
  3. Test and Debug to confirm that no unexpected execution of code happens during the build, especially involving the random.choice() function.

This should resolve the issue where Sphinx attempts to execute the random.choice() function on empty sequences, leading to the IndexError.

#py