Azure function failing after successfull deployment with OSError

ghz 12hours ago ⋅ 1 views

Azure function failing after successfull deployment with OSError: [Errno 107]

The function was working before the last deployment. It is used for NLC and we are running two functions for two different languages. For the new deployment we removed some code which was only used for one language and mirrored the code to be equal to the other language. Azure CLI tells us:

Upload completed successfully.
Deployment completed successfully.

But when trying to invoke the function afterwards it either returns 500 without a body or it returns 404 with:

<html>
<head>
    <title>404 Not Found</title>
</head>
<body bgcolor="white">
    <center>
        <h1>404 Not Found</h1>
    </center>
    <hr>
    <center>nginx</center>
</body>
</html>



The stacktrace we can see in the monitoring is the following:
Exception while executing function: Functions.classify Result: Failure
Exception: OSError: [Errno 107] Transport endpoint is not connected: '/home/site/wwwroot/.python_packages/lib/site-packages'
Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 271, in _handle__function_load_request
func = loader.load_function(
File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 32, in call
return func(*args, **kwargs)
File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/loader.py", line 76, in load_function
mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/classify/__init__.py", line 4, in <module>
File "/home/site/wwwroot/classify/predict.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/text/all.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/basics.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/data/all.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/torch_basics.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/torch/__init__.py", line 482, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/torch/serialization.py", line 8, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 914, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1342, in find_spec
File "<frozen importlib._bootstrap_external>", line 1314, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1443, in find_spec
File "<frozen importlib._bootstrap_external>", line 1483, in _fill_cache

Commands used for deployment were the following, which worked before:

pip3 install  --target="./.python_packages/lib/site-packages" -r requirements.txt 
func azure functionapp publish [functionapp name] --no-build
  • Deleting the python packages and rebuilding etc. did not change anything.
  • I tried creating a new function app from scratch but it shows the same errors.
  • I tried to deploy from the folder of the other language with one file swapped (classifier language model) but it showed the same error. I fear trying to redeploy the working language for testing as it is working and I don't want to kill it.

Answer

The issue you're experiencing seems related to an OSError: [Errno 107] Transport endpoint is not connected error when loading the Python packages in your Azure Function. This error typically indicates that there is a problem with the filesystem or with the path to the Python packages, specifically the location where your site-packages are stored.

Given that you've already tried deleting the Python packages and rebuilding without success, as well as redeploying from a different folder, let's walk through some troubleshooting steps to isolate and resolve the issue.

1. Check Azure Function App Disk Space

The OSError might be related to running out of disk space in your Azure Function App. The .python_packages directory is where Azure stores Python dependencies, and if there isn't enough space, it could cause issues loading the packages.

  • Action: Check the disk space usage in the Azure portal. If disk space is low, try cleaning up unnecessary files, or try increasing the storage limit if possible.

2. Inspect Python Dependencies

There might be a conflict or issue with the Python dependencies you're using, especially since the error happens in the site-packages directory.

  • Action: Ensure all dependencies in requirements.txt are compatible with each other, and that there are no conflicting versions.

    • For example, check if torch, fastai, and other deep learning dependencies are compatible.
    • Try installing them manually in a local environment to see if any errors occur:
    pip install --target="./.python_packages/lib/site-packages" -r requirements.txt
    • Then, check for any potential errors that may be related to specific dependencies.

3. Check Function App Logs

Review the detailed logs for your Azure Function to identify if there are any other errors or warnings that might give you more information about what's going wrong.

  • Action: Go to the Azure portal, find your function app, and look at the App Service logs (under Monitor > Logs). Check if there are any logs that indicate issues with loading dependencies or file system access.

4. Verify Deployment Process

It looks like you're using the func azure functionapp publish command. However, the error indicates issues with loading the Python packages. Here's a refined process to follow:

  • Action: After running the command, verify that the packages are correctly deployed to your function app.

    pip3 install --target="./.python_packages/lib/site-packages" -r requirements.txt
func azure functionapp publish [functionapp_name] --no-build

    After deploying, try accessing the /home/site/wwwroot/.python_packages/lib/site-packages directory through the Kudu console in the Azure portal. Ensure that the necessary dependencies are correctly installed in the target directory. Sometimes, deployment may succeed, but some dependencies might be missing or misplaced.

5. Rebuild Environment (Clean Deployment)

You mentioned you tried deleting the packages, but it didn't change anything. Try the following steps for a more thorough clean deployment:

  • Action: Run the following commands in sequence to clear and rebuild the environment:

    1. Delete the .python_packages directory from the Azure function app. You can do this using the Kudu console or by SSHing into the Azure function instance.

    2. Re-deploy the function app:

    pip install --target="./.python_packages/lib/site-packages" -r requirements.txt
func azure functionapp publish [functionapp_name] --no-build
    1. After deploying, check if the dependencies are properly installed in the Azure App Service.

6. Inspect Python and Azure Runtime Configuration

The issue might also be related to a mismatch between the Python version you're using locally and the Python runtime in Azure.

  • Action: Verify that your Azure Function App is using the correct Python runtime version (Python 3.8 or 3.9 as per your deployment configuration). You can do this by checking the settings in the Azure portal under Configuration > General settings.

    • If necessary, update the Python version to match your local environment.

7. Function Timeout or Network Issue

The 500 and 404 errors you’re seeing could also be indicative of a timeout or network issue when the function is attempting to load its dependencies. This could be a transient error, or there might be a deeper issue with network connectivity.

  • Action: Verify that there are no timeout issues in the Azure settings, and check the network settings if your function is accessing external resources.

8. Test Deployment from Another Region

If you're still encountering issues, it could be worth trying to deploy the function app in a different Azure region or creating a new function app in a clean environment.

  • Action: Try creating a new Azure Function App in a different region, redeploy the function, and see if the issue persists.

Conclusion

The issue seems to be with the Python environment in the Azure function app, specifically related to how the function is attempting to access Python dependencies. Here are the critical actions to take:

  1. Ensure sufficient disk space on the Azure Function App.
  2. Check for Python dependency conflicts by manually installing them and testing locally.
  3. Check deployment logs in Azure to find any hidden issues.
  4. Perform a clean rebuild of the deployment.
  5. Ensure your Python version matches between local and Azure environments.
  6. Consider redeploying to a new region or clean function app.

By following these steps, you should be able to isolate and resolve the issue preventing your function from working correctly after the deployment.

#py