python subprocess.call output is not interleaved

ghz 15hours ago ⋅ 13 views

I have a python (v3.3) script that runs other shell scripts. My python script also prints message like "About to run script X" and "Done running script X".

When I run my script I'm getting all the output of the shell scripts separate from my print statements. I see something like this:

All of script X's output
All of script Y's output
All of script Z's output
About to run script X
Done running script X
About to run script Y
Done running script Y
About to run script Z
Done running script Z

My code that runs the shell scripts looks like this:

print( "running command: " + cmnd )
ret_code = subprocess.call( cmnd, shell=True )
print( "done running command")

I wrote a basic test script and do not see this behaviour. This code does what I would expect:

print("calling")
ret_code = subprocess.call("/bin/ls -la", shell=True )
print("back")

Any idea on why the output is not interleaved?

Answer

The issue you're encountering, where the output of the shell scripts is appearing separately from your print statements, is most likely due to the way buffering works in Python and the subprocess module. Specifically, the output of the shell scripts may be buffered, causing it to be printed all at once after the script finishes, rather than being interleaved with the Python print statements.

Understanding the Problem

When you use subprocess.call(), it runs the command, waits for it to complete, and then returns. During this time, the output from the shell command might not be flushed immediately, which can cause it to appear all at once after the script finishes running.

By default, output from subprocesses (like stdout) is buffered, meaning it might not be immediately written to the terminal or interleaved with the Python print statements.

Solutions

Here are several ways to ensure that the output from the shell scripts is printed immediately and is interleaved with your Python print statements.

1. Using subprocess.call() with stdout and stderr Piping

You can explicitly capture the stdout and stderr streams and ensure they are printed line by line as the command runs.

import subprocess

cmnd = "your_shell_command_here"
print("running command: " + cmnd)

# Open a subprocess and capture the output in real-time
process = subprocess.Popen(cmnd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1, universal_newlines=True)

# Print output as it arrives
for line in process.stdout:
    print(line, end='')  # Print stdout line by line as it comes

# Also print stderr if needed
for line in process.stderr:
    print(line, end='')  # Print stderr line by line as it comes

process.wait()  # Wait for the process to finish
print("done running command")

Explanation:

  • subprocess.Popen(): This method allows you to have more control over how the command is run, including capturing stdout and stderr.
  • bufsize=1: This sets line buffering, so the output is printed line-by-line.
  • universal_newlines=True: This ensures the output is decoded into a string, making it easier to work with.
  • stdout=subprocess.PIPE and stderr=subprocess.PIPE: These capture the standard output and standard error streams.
  • process.wait(): Waits for the command to finish running before moving on.

2. Using subprocess.run() (Python 3.5+)

In Python 3.5 and above, you can use subprocess.run() with the capture_output=True argument to capture both stdout and stderr, and print them as they come in:

import subprocess

cmnd = "your_shell_command_here"
print("running command: " + cmnd)

# Run the command and capture output in real-time
result = subprocess.run(cmnd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

# Print stdout and stderr as they come in
print(result.stdout, end='')
print(result.stderr, end='')

print("done running command")

3. Flushing Python’s Standard Output

If you want to ensure that Python's own print statements are flushed immediately (in case you're seeing delays with the prints themselves), you can use flush=True with the print() function.

print("About to run script X", flush=True)

This will ensure that the print statements themselves are immediately flushed to the console, rather than being buffered.

4. Disabling Output Buffering for Subprocess

You can also disable buffering in the shell script itself by setting the PYTHONUNBUFFERED environment variable or passing stdbuf (on Unix-like systems) to disable buffering:

import subprocess

cmnd = "stdbuf -oL your_shell_command_here"  # Disable buffering on stdout
print("running command: " + cmnd)

# Run the command with no output buffering
ret_code = subprocess.call(cmnd, shell=True)
print("done running command")

This will ensure that the output from the shell script is immediately flushed.

Summary

To ensure that the output of your shell commands is printed line-by-line and interleaved with Python’s print statements, the key is to control the buffering behavior. Using subprocess.Popen() to capture stdout and stderr in real time is the most flexible approach, but you can also achieve similar results with subprocess.run() in newer versions of Python. Additionally, using flush=True in Python’s print() calls ensures that Python’s own output doesn’t get buffered.

#py