Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method.
In this article I will show how to invoke a process from Python and show stdout live without waiting for the process to complete. This is a better end-user experience for longer running jobs.
Example long-running command
As an example of a long running command, consider the Bash script loopWithSleep.sh which I’ve uploaded to github. Below is the full script:
#!/bin/bash for i in $(seq 1 5); do echo "iteration" $i sleep 1 done
Every second it displays a line of output and takes a total of 5 seconds to run.
iteration 1 iteration 2 iteration 3 iteration 4 iteration 5
Python Popen.poll()
If you start a process using process.call() or process.check_output(), then you cannot get the output until the process is compete. However if you use subprocess.Popen along with Popen.poll() to check for new output, then you see a live view of the stdout.
# invoke process
process = subprocess.Popen(shlex.split(command),shell=False,stdout=process.PIPE)
# Poll process.stdout to show stdout live
while True:
output = process.stdout.readline()
if process.poll() is not None:
break
if output:
print output.strip()
rc = process.poll()
Example Program
For an example that pulls all this together, see my runProcessWithLiveOutput.py on github. Download this into the same directory as the Bash loopWithSleep.sh as an example program.
$ ./runProcessWithLiveOutput.py Execute which commmand [./loopWithSleep.sh]: == invoke_process_popen_blocking ============== iteration 1 iteration 2 iteration 3 iteration 4 iteration 5 == invoke_process_popen_poll_live ============== iteration 1 iteration 2 iteration 3 iteration 4 iteration 5
You then get an example of how subprocess.communicate() does not show the output until the subprocess is complete. However using poll, you get the ouput in real-time.
REFERENCES
endpoint, realtime output from subprocess
stackoverflow, Popen and streaming
pythonspot, subprocess.popen, call, and check_output