Configuration

• All tests are driven by the config.toml file. TOML is a simple configuration language designed to be human-friendly.
• A full schema is defined in config_schema.json

config.toml

output_dir

string: Name of directory in which to place all output files

[build]

optional object: Optional section containing build requirements

[[test]]

array: Array of test objects

[output]

object: Contains template information for logging output.

build

required_files

optional array: Array of {file="str", dest="str", secret=false} objects. Files should be stored in .config, and can be copied anywhere within a project subdirectory as needed.

Note

secret is optional; set to true if you want the files to be removed after the tests have run. This could be useful if students have access to the folders being graded (such as on a networked drive).

commands

optional array: Array of strings. Commands to be executed in order for building the project.

Tests

All tests share some common configuration values:

name

string: Display name of test used when logging results.

type

string: Defines the type of test to run. Can be "junit", "diff", "custom"

JUnit

Note

You can attach point values to invidual JUnit tests. To do so, use an annotation on the test function: @DisplayName("<Name here>points:10"). Points can be any floating point value.

classname

string: Filename of compiled classfile with JUnit test suite. Should be paired with required_files in the build section to copy the classfile into each project directory.

Diff

points

number: All or nothing point value for test.

command

string: command to execute. Output will be captured and compared.

glob_command

boolean: Use bash-style globbing to resolve the executable name.

Note

Useful when executables follow a common naming format but have unique names (CS255P01<LastName>.exe would be matched by CS225P01*.exe).

Arguments can still be used with globbed commands, but only the executable name itself will be resolved.

stdin OR stdinFile

string: Raw text, or name of text file, to feed to STDIN.

stdout OR stdoutFile

string: Raw text, or name of text file, containing correct STDOUT compare against.

stderr OR stderrFile

string: Raw text, or name of text file, containing correct STDERR to compare against.

* Must have at least one of stdout and stderr defined

Custom

command

string: command to execute. Output will be captured.

output

This section consists of python template strings used to format output.

Note

The valid identifiers are $name, $cmd, $retval, $stdout, $stderr, $points, $diffout. Any identifiers that do not pertain to the current test will be replaced with an empty string.

build

string: Used for each build command

junit

string: Used for JUnit tests

diff

string: Used for diff tests

custom

string: Used for custom tests