API Reference

The SciATH API is based on a few key abstractions:

• A Test consisting of:

  • a unique name;
  • a Job, containing one or more Tasks, which consist of a command and a set of required resources (typically, a number of MPI ranks).
  • a method to determine success (e.g., checking an error code or an output file)

• A Launcher to manage launching a serial or MPI Jobs locally or via a batch/queuing system

• A Harness object to quickly define and process a set of Test objects.

• A set of Verifier implementations to ascertain test success (e.g. by comparing output to a refernce).

Task

SciATH Task class

class sciath.task.Task(command, **kwargs)

A class which encapsulates a command and a set of required resources

A “command”, here, is a list of strings, the first of which names an executable and the rest of which provide arguments. For example ['printf','Hello, World!'] corresponds to a local command like printf 'Hello, World!'. This corresponds to the recommended way to pass commands to Python’s subprocess.run (click link for information).

Note that relative paths (not recommended) in the command will be interpreted relative to wherever it is actually executed from, which is determined when a Launcher is requested to execute a Job.

A “resource” here is a specification of a computational resource required, for example the number of (MPI) ranks or (OpenMP) threads, or an amount of time to allocate.

create_execute_command()

Returns a command, resource tuple.

get_resource(key)

Returns the value for a given resource, specified by a string

Returns None if resource not defined.

set_resources(**kwargs)

Define task resources (e.g. number of mpi ranks)

via “resource_name”=number keyword=value pairs. The keywords used to identify the number mpi ranks iare [‘ranks’, ‘Ranks’, ‘mpiranks’, ‘MPIRanks’].

The keywords used to identify the number of threads are [‘threads’, ‘ompthreads’].

Unrecognized resource types will produce an error.

Job

SciATH Job class

class sciath.job.Job(task_or_tasks, name=None, minimum_wall_time=None)

Job describes a ordered list of Tasks.

It describes the Tasks themselves, not information about any particular “run” (handled within Harness) or how one might interpret the results of such a run (handled by Test). A Launcher object executes the Tasks described by a Job.

Data include

• A name
• An ordered list of Tasks
• A minimum wall time required (can be None)

It also defines how the name may be converted to various standard filenames.

If a minimum wall time is specified, the wall time allocated is the maximum of this value and the sum of values of wall times for all included Tasks that defined them. If no minimum wall time is specified, the Job’s wall time is only defined if all included Tasks specify a wall time, in which case the sum is used.

complete_filename

Returns a filename for a sentinel signifying completion

create_execute_command()

Returns a list containing (command, resource) tuples for the Job.

exitcode_filename

Returns a filename to use for exit codes

get_max_resources()

Returns a dict() defining the maximum required counts / values

launched_filename

Returns a filename for a sentinel signifying launching

number_tasks()

Returns the number of tasks within the Job

resource_max(key)

Returns the maximum value of a resource over all Tasks

Returns None if no Tasks define that resource.

stderr_filename

Returns a filename to use for stderr

stdout_filename

Returns a filename to use for stdout

Launcher

SciATH Launcher class

class sciath.launcher.Launcher(conf_filename=None)

Launcher is responsible for executing Task`s specified by a :class:`Job, depending on its system-dependent configuration.

Thus, it is:

• The exclusive location for system-specific information
• The exclusive reader of system-specific configuration file

Launcher include methods to operate on a combination of a Job and a path:

• Run the job from that path. If not on a batch system, blocks until the job completes.
• Check the status of the job as run from that path
• Clean up after a job, removing configuration-specific generated files

Launcher does not know about Test or Harness, and it should be possible to use Launcher and a collection of Job objects as a convenience to execute sets of tasks on various systems.

A Launcher’s state corresponds only to its configuration, not the status of any particular “run” of a Job.

clean(job, output_path=None)

Remove all files created by the Launcher itself

Note that this does not remove any files created by the Job (via its Tasks).

configure()

Interactively configure the Launcher

is_valid_mpi_launch(mpi_launch)

Returns whether a given string is acceptable to launch MPI jobs

load_definition(filename)

Loads configuration from a file

prepare_job(job, output_path=None, exec_path=None)

Prepare to submit a job.

Prepares a launch script and forms the launch command. Returns information about jobs that cannot be launched.

Returns success, info, report, submit_data

If success is false, the job could not be prepared, and info and report give more information. submit_data is a dict which is used by submit_job to launch.

This is to give controlling logic, such as that in Harness, the opportunity to print information about a job before launching it (which may block), and to respond to situations where the Launcher is not able to launch a job.

set_mpi_launch(mpi_launch)

Sets the MPI launch command and check its form

submit_job(submit_data)

Submit a prepared job from data prepared by prepare_job

static write_default_definition(conf_filename_in=None)

Writes a default configuration file

write_definition(filename)

Writes configuration to a file

exception sciath.launcher.SciATHLoadException

Exception for a failed load of configuration file

sciath.launcher.job_complete(job, output_path)

Returns True if a Job has completed with the given output path

sciath.launcher.job_launched(job, output_path)

Returns True if a Job has launched with the given output path

Test

SciATH Test class

class sciath.test.Test(job, name=None)

Test describes a test case in a test suite.

It is simply a collection of data about such a case, not about any particular “run” of it.

Thus, it contains:

• A Job, describing how to execute the required operations
• A name
• An implementation of Verifier, defining how to determine success
• A set of group tags

add_group(group)

Tag the test with a space-free string

verify(output_path=None, exec_path=None)

Return (passing, info, report) relative to an output and execution path

Harness

SciATH Harness class

class sciath.harness.Harness(tests=None)

Harness is the central user-facing class in SciATH.

It manages a set of tests, and thus includes:

• A set of uniquely-named Test objects
• A Launcher
• Tools for running and verifying a test suite
• Tools for managing “sandboxing” (managing directories in which to run tests)

It is the exclusive location within SciATH for

• Printing to stdout
• Information about where to launch Job`s from, passed to included `Launcher

A Harness object’s state is confined to the state of a list of internal _TestRun objects, and the state of the included Launcher.

add_test(test)

Add a Test to be run with the harness

add_tests_from_file(filename)

Read a file to add Tests to be run with the harness

clean()

Remove all output from all Tests, preparing or a re-run

determine_overall_success()

Returns a boolean value to denote overall success of the test suite

execute()

Execute all tests

print_all_tests()

Display information about all tests

report()

Compile results into a report and print to stdout and file

run_from_args()

Perform one or more actions, based on command line options

This essentially defines the “main” function for the typical use of SciATH.

Returns an exit code, with 0 indicating success.

update_expected()

Give each active test the chance to update its reference output

verify()

Updates the status of all test runs

exception sciath.harness.SciATHHarnessInconsistentStateException

Exception for unexpected filesystem state

Verifier

SciATH Verifier class

class sciath.verifier.ComparisonVerifier(test, expected_file, output_file=None, comparison_file=None)

A Verifier which compares an output file against a reference file

execute(output_path=None, exec_path=None)

Relative to a given output path, fetch file(s) and determine success.

Returns (passing, info, report), where passing is a boolean value, info is a short string describing the reason for a pass/fail, and report is a longer list of strings giving further details on failure.

Note that there a Verifier must always be able to determine success or failure.

Accepts an output path (where the Launcher puts the files it generates) and an execution path (where the Job was run from). Either might be required for verification: for instance, to check the error code collected by the Launcher, one needs the output path, and to compare with an output file generated by an executable, one needs the execution path.

update_expected(output_path=None, exec_path=None)

Update reference files from output

Note that if multiple tests refer to the same reference file, a given reference file may be updated several times, and so even in cases where verification is solely based on reference files, updating may not be a guarantee that verification will succeed.

class sciath.verifier.ExitCodeVerifier(test)

Verifier implementation which checks exit codes

execute(output_path, exec_path=None)

Relative to a given output path, fetch file(s) and produce status,report

set_exit_codes_success(exit_codes_success)

Sets a single exit code per Task to interpret as success

exception sciath.verifier.SciATHVerifierMissingFileException

Exception for a missing file required for a Verifier operation

class sciath.verifier.Verifier(test)

Base class for verification of a Test

execute(output_path, exec_path=None)

Relative to a given output path, fetch file(s) and determine success.

Returns (passing, info, report), where passing is a boolean value, info is a short string describing the reason for a pass/fail, and report is a longer list of strings giving further details on failure.

Note that there a Verifier must always be able to determine success or failure.

Accepts an output path (where the Launcher puts the files it generates) and an execution path (where the Job was run from). Either might be required for verification: for instance, to check the error code collected by the Launcher, one needs the output path, and to compare with an output file generated by an executable, one needs the execution path.

SciATH LineVerifier class

class sciath.verifier_line.LineVerifier(test, expected_file, output_file=None, comparison_file=None)

An Verifier which compares sets of lines from expected and output files

This is accomplished by defining a set of “rules”. Each rule contains a regular expression to determine which lines to match, and a function to determine success when comparing the the set of matching lines from the two files.

sciath.verifier_line.compare_float_values_line(line_expected, line_out, abs_tol, rel_tol)

Compares floating point numbers on a pair of lines

sciath.verifier_line.float_pairs_function(match_expected, match_out, abs_tol, rel_tol, max_err_count=100)

Compares all floating point numbers, looking for abs. or rel. tol

sciath.verifier_line.key_and_float_rule(key=None, rel_tol=None, abs_tol=None)

A rule to match lines starting with a key, comparing all numbers as floats

Preceding whitespace is accepted in matching the key. If the key is not provided, all lines are matched.

sciath.verifier_line.string_get_floats(line)

Extracts floating point numbers from a string