python testing strategy to develop and auto-grader
<p>I have a list of input files and an expected output file, I want to write an auto-grader that does the job of accepting a python program, running it on the input files, and comparing its output to the output file. The approach I have used is to use the <code>os</code> module of python to run the program using <code>os.system('python program.py &gt; actual.out')</code> and then perform a diff between the output and expected.out again using os.system().</p>

<p>The problem which I am currently facing is reading the input from the file because the program which is given is reading from the console. So, how should I redirect the input from a file such that it is readable by sys.stdin in program.py.</p>

<pre><code>import os
def grade(program_py_file_handler,input_dir,output_dir):
   #create temporary file for program.py using program_py_file_handler
   #one by one read all files from input_dir
   #run program.py using os.system generating a temp file
   #do diff be temp file and expected file
</code></pre>

<p>Is there a better way to perform a diff without using the diff command?</p>

<p>To redirect output from program.py to a file I used <code>python program.py&gt;tem.out</code>. What equivalent should I use to redirect an input file to progam.py such that wherever I have used sys.stdin in program.py it will instead read from the passed file? (Modifying program.py is not an option.)</p>

<p>You can be doing everything using builtin modules in Python 3.3+, since you are effectively spinning up a <a href="https://docs.python.org/3/library/subprocess.html" rel="nofollow"><code>subprocess</code></a> and doing a <a href="https://docs.python.org/3/library/difflib.html" rel="nofollow"><code>diff</code></a> on the expected output.  Simple minimum example:</p>

<p><em>check.py</em></p>

<pre><code>import sys
from subprocess import check_output
from difflib import ndiff

def split(s):
    return s.splitlines(keepends=True)

def check(program_name, expected):
    output = check_output([sys.executable, program_name]).decode('utf8')
    diff = ndiff(split(output), split(expected))
    print(''.join(diff), end="")

def main():
    check('hello.py', 'Good Morning!\n')

if __name__ == '__main__':
    main()
</code></pre>

<p><em>hello.py</em></p>

<pre><code>print('Good Evening!')
</code></pre>

<p>Example run</p>

<pre><code>$ python check.py 
- Good Evening!
?      ^^^
+ Good Morning!
?      ^^^
</code></pre>

<p>Modify as you see fit, with other methods/functions in the libraries linked.  If you need stdin for the subprocess you probably will need to create Popen object and call communicate, but please read documentations first for future reference.</p>

python file python-3.4 file-handling