SciATH Tutorial

SciATH is designed to test programs written in any language, by working at the level of executables and expected output.

This tutorial uses simple executables which should work from all POSIX-compatible shells.

In typical usage, one would of course run scientific application executables (or scripts), to test them.

It may also help to read Helpful concepts for SciATH users.

Installing SciATH

Clone SciATH from the upstream Git repository, and make sure it is on your Python path, e.g.

cd $HOME
git clone https://github.com/sciath/sciath
cd sciath
export PYTHONPATH=$PYTHONPATH:$HOME/sciath

You need a working installation of Python. Test that the above worked by running python and then import sciath - if this command succeeds without error, the the sciath module can be used as expected.

SciATH is a Python module which can be run as an executable. This is achieved by using the syntax python -m sciath, once SciATH has been successfully installed.

The simplest test

The simplest test is to run an executable and check the exit code. If the code is 0, we declare success. Otherwise, failure.

Create a scratch directory to work in

mkdir scratch
cd scratch

Create a file, tutorial1.yml, with the following content

tests:
  - name: first
    command: printf "Hello, World!\n"
    type: exit_code

The file uses a subset of the YAML format. Blank lines and lines beginning with # are ignored.

Execute the SciATH module to run the test

python -m sciath tutorial1.yml

You will be prompted to describe information about how to run jobs on your system. This tutorial can be used on a local machine or a cluster. To begin with, enter local and none for the first two questions.

This information is stored in a file called SciATH_launcher.conf.

Once you’ve defined this information, you’ll see output like the following

[*** Cleanup ***]
[SciATH] Removing output for Test: first

[*** Executing Tests ***]
[SciATH] Batch queueing system configuration [SciATH_launcher.conf]
  Version:           0.13.0
  MPI launcher:      none
  Submit command:    sh
  Blocking:          True
  Job-level ranks:   False
[Executing first]from /Users/patrick/scratch/first_output/sandbox
sh /Users/patrick/scratch/first_output/first.sh

[*** Summary ***]
[first]  pass

SUCCESS

Report written to file:
  /Users/patrick/scratch/sciath_test_report.txt

Note that you are given the full path for the “sandbox” from which your command is run.

Your output will contain colors, by default. If you don’t want the colors (or see strange codes like ^[[35m), use the --no-colors option.

SciATH accepts other command line arguments. Use the -h argument to see them.

python -m sciath -h

To (re-)verify your test results, without re-running the tests, use -v:

python -m sciath tutorial1.yml -v

Take a look in the directory that was created, called first_output. You will see several files and directories, including

  • first.sh, a shell script that was used to run your test
  • first.exitcode which contains the exitcode (0)
  • first.stdout which contains Hello, World!
  • first.stderr, which should be an empty file
  • sandbox, which should be an empty directory

Running a subset of Tests

Create a new file, tutorial.yml, with the following contents. Don’t worry that you don’t understand everything yet. Just note that it contains several tests, each with a unique name.

tests:
  - name: first
    command: printf "Hello, World!\n"
    type: exit_code
  - name: second
    command: printf "Hello Again, World!\n"
    type: exit_code
  - name: failing
    command: grep foo bar
    type: exit_code
  - name: text_diff
    command: printf "a line of text\n"
    expected: text_diff.expected
  - name: text_diff_failing
    command: printf "a line of text\n"
    expected: HERE/text_diff_failing.expected

You can use the -l command line option to list all available tests.

python -m sciath tutorial.yml -l

You can run one or more of the tests, by name, with the -t option. For example, run

python -m sciath tutorial.yml -t first

to generate output

[*** Cleanup ***]
[SciATH] Removing output for Test: first

[*** Executing Tests ***]
[SciATH] Batch queueing system configuration [SciATH_launcher.conf]
  Version:           0.13.0
  MPI launcher:      none
  Submit command:    sh
  Blocking:          True
  Job-level ranks:   False
[Executing first]from /Users/patrick/scratch/first_output/sandbox
sh /Users/patrick/scratch/first_output/first.sh

[*** Summary ***]
[first]  pass
[second]  deactivated
[failing]  deactivated
[text_diff]  deactivated
[text_diff_failing]  deactivated

SUCCESS

Report written to file:
  /Users/patrick/scratch/sciath_test_report.txt

or

python -m sciath tutorial.yml -t first -t second

to generate output

[*** Cleanup ***]
[SciATH] Removing output for Test: first
[SciATH] Removing output for Test: second

[*** Executing Tests ***]
[SciATH] Batch queueing system configuration [SciATH_launcher.conf]
  Version:           0.13.0
  MPI launcher:      none
  Submit command:    sh
  Blocking:          True
  Job-level ranks:   False
[Executing first]from /Users/patrick/scratch/first_output/sandbox
sh /Users/patrick/scratch/first_output/first.sh
[Executing second]from /Users/patrick/scratch/second_output/sandbox
sh /Users/patrick/scratch/second_output/second.sh

[*** Summary ***]
[first]  pass
[second]  pass
[failing]  deactivated
[text_diff]  deactivated
[text_diff_failing]  deactivated

SUCCESS

Report written to file:
  /Users/patrick/scratch/sciath_test_report.txt

A Failing Test

The test failing attempts to run grep on a non-existent file, which prints an error message to stderr and returns a non-zero exit code (2 on BSD and GNU systems). It is defined in the input file with

  - name: failing
    command: grep foo bar
    type: exit_code

Run it with

python -m sciath tutorial.yml -t failing

To produce output

[*** Cleanup ***]
[SciATH] Removing output for Test: failing

[*** Executing Tests ***]
[SciATH] Batch queueing system configuration [SciATH_launcher.conf]
  Version:           0.13.0
  MPI launcher:      none
  Submit command:    sh
  Blocking:          True
  Job-level ranks:   False
[Executing failing]from /Users/patrick/scratch/failing_output/sandbox
sh /Users/patrick/scratch/failing_output/failing.sh

[*** Verification Reports ***]
[Report for failing]
[ExitCodeDiff] Expected exit code(s): [0]
[ExitCodeDiff] Output exit code(s)  : [2]
check non-empty stderr file:
    less /Users/patrick/scratch/failing_output/failing.stderr

[*** Summary ***]
[first]  deactivated
[second]  deactivated
[failing]  fail
[text_diff]  deactivated
[text_diff_failing]  deactivated

FAILURE
To re-run failed tests, use e.g.
  -t failing

Report written to file:
  /Users/patrick/scratch/sciath_test_report.txt

Note that you are given information about the location of the output files at the end of the report for the failing test. You can copy-paste this line to see the error message:

check non-empty stderr file:
    less /Users/patrick/scratch/failing_output/failing.stderr

A Comparison-Based Test

The test text_diff compares output to stdout against a reference file.

It is defined in the input file with

  - name: text_diff
    command: printf "a line of text\n"
    expected: text_diff.expected

Note that type: is not specified, so the default is used. This is a text-based comparison.

Run the test with

python -m sciath tutorial.yml -t text_diff

It will fail because the expected file is not found:

[*** Cleanup ***]
[SciATH] Removing output for Test: text_diff

[*** Executing Tests ***]
[SciATH] Batch queueing system configuration [SciATH_launcher.conf]
  Version:           0.13.0
  MPI launcher:      none
  Submit command:    sh
  Blocking:          True
  Job-level ranks:   False
[Executing text_diff]from /Users/patrick/scratch/text_diff_output/sandbox
sh /Users/patrick/scratch/text_diff_output/text_diff.sh

[*** Verification Reports ***]
[Report for text_diff]
[Comparison] Expected file missing: text_diff.expected
check stdout file:
    less /Users/patrick/scratch/text_diff_output/text_diff.stdout

[*** Summary ***]
[first]  deactivated
[second]  deactivated
[failing]  deactivated
[text_diff]  fail (expected file not found)
[text_diff_failing]  deactivated

FAILURE
To re-run failed tests, use e.g.
  -t text_diff

Report written to file:
  /Users/patrick/scratch/sciath_test_report.txt

Note that the full path where SciATH expected the file is printed. It looked for it in the same location as tutorial.yml.

You can generate the expected file with the -u flag. Note that this will overwrite this file, if it already exists!

python -m sciath tutorial.yml -t text_diff -u

You will now notice that the expected file, text_diff.expected has appeared and now the test will pass if you re-run

python -m sciath tutorial.yml -t text_diff

Edit the newly created file to make the test fail, for example changing it to

a line of text???

and examine the output again.

Additional features

Additional features not (yet) included in this tutorial include

  • Numerical tests
  • MPI-parallel tests
  • Tests with multiple commands
  • Using replacements in the input file (HERE and environment variables)