third_party/exiv2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update the documentation for the test cases

Browse Source
This commit is contained in:
LeoHsiao 2020-08-10 22:16:48 +08:00
parent 15c0366087
commit d231bae4f7
4 changed files with 117 additions and 107 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

160
tests/doc.md → tests/README.md
View File

@ -1,7 +1,27 @@
# TL;DR
- [Introduction](#introduction)
- [Running the test suite](#running-the-test-suite)
- [Writing new tests](#writing-new-tests)
- [Test suite](#test-suite)
- [Configuration](#configuration)
- [INI style](#ini-style)
- [Parameters](#parameters)
- [Test cases](#test-cases)
- [Multiline strings](#multiline-strings)
- [Paths](#paths)
- [Advanced test cases](#advanced-test-cases)
- [Providing standard input to commands](#providing-standard-input-to-commands)
- [Using a different output encoding](#using-a-different-output-encoding)
- [Working with binary output](#working-with-binary-output)
- [Setting and modifying environment variables](#setting-and-modifying-environment-variables)
- [Creating file copies](#creating-file-copies)
- [Customizing the output check](#customizing-the-output-check)
- [Running all commands under valgrind](#running-all-commands-under-valgrind)
- [Manually expanding variables in strings](#manually-expanding-variables-in-strings)
- [Hooks](#hooks)
- [Possible pitfalls](#possible-pitfalls)
- [Bash test cases](#bash-test-cases)
<!-- The TOC is generated by Markdown Preview Enhanced -->
If you just want to write a simple test case, check out the file
`writing_tests.md`.
# Introduction
@ -11,6 +31,95 @@ especially useful for a regression test suite, but can be also used for testing
of new features where unit testing is not feasible, e.g. to test new command
line parameters.
## Running the test suite
The test suite is written for Python 3 and is not compatible with Python 2, thus
it must be run with `python3` and not with `python` (which is usually an alias
for Python 2).
Then navigate to the `tests/` subdirectory and run:
``` shell
python3 runner.py
```
One can supply the script with a directory where the suite should look for the
tests (it will search the directory recursively). If omitted, the runner will
look in the directory where the configuration file is located. It is also
possible to instead pass a file as the parameter, the test suite will then only
run the tests from this file.
The runner script also supports the optional arguments `--config_file` which
allows to provide a different test suite configuration file than the default
`suite.conf`. It also forwards the verbosity setting via the `-v`/`--verbose`
flags to Python's unittest module.
Optionally one can provide the `--debug` flag which will instruct test suite to
print all command invocations and all expected and obtained outputs to the
standard output.
## Writing new tests
The test suite is intended to run a binary and compare its standard output,
standard error and return value against provided values. This is implemented
using Python's [unittest](https://docs.python.org/3/library/unittest.html) module and thus all test files are Python files.
The simplest test has the following structure:
``` python
# -*- coding: utf-8 -*-
import system_tests
class GoodTestName(metaclass=system_tests.CaseMeta):
filename = "$data_path/test_file"
commands = ["$exiv2 $filename", "$exiv2 $filename" + '_2']
stdout = [""] * 2
stderr = ["""$exiv2_exception_msg $filename:
$kerFailedToReadImageData
"""] * 2
retval = [1] * 2
```
The test suite will run the provided commands in `commands` and compare them to
the output in `stdout` and `stderr` and it will compare the return values.
The strings after a `$` are variables either defined in this test's class or are
taken from the suite's configuration file (see `doc.md` for a complete
explanation).
When creating new tests, follow roughly these steps:
1. Choose an appropriate subdirectory where the test belongs. If none fits
create a new one and put an empty `__init__.py` file there.
2. Create a new file with a name matching `test_*.py`. Copy the class definition
from the above example and choose an appropriate class name.
3. Run the test suite via `python3 runner.py` and ensure that your test case is
actually run! Either run the suite with the `-v` option which will output all
test cases that were run or simply add an error and check if errors occur.
## Test suite
The test suite itself uses the builtin `unittest` module of Python to discover
and run the individual test cases. The test cases themselves are implemented in
Python source files, but the required Python knowledge is minimal.
The test suite is configured via one configuration file whose location
automatically sets the root directory of the test suite. The `unittest` module
then recursively searches all sub-directories with a `__init__.py` file for
files of the form `test_*.py`, which it automatically interprets as test cases
(more about these in the next section). Python will automatically interpret each
directory as a module and use this to format the output, e.g. the test case
`regression/crashes/test_bug_15.py` will be interpreted as the module
`regression.crashes.test_bug_15`. Thus one can use the directory structure to
group test cases.
### Configuration
#### INI style
The test suite is configured via `INI` style files using Python's builtin
[ConfigParser](https://docs.python.org/3/library/configparser.html)
module. Such a configuration file looks roughly like this:
@ -55,21 +164,7 @@ is equivalent to this:
some_var:some value with whitespaces before and after
```
The test suite itself uses the builtin `unittest` module of Python to discover
and run the individual test cases. The test cases themselves are implemented in
Python source files, but the required Python knowledge is minimal.
## Test suite
The test suite is configured via one configuration file whose location
automatically sets the root directory of the test suite. The `unittest` module
then recursively searches all sub-directories with a `__init__.py` file for
files of the form `test_*.py`, which it automatically interprets as test cases
(more about these in the next section). Python will automatically interpret each
directory as a module and use this to format the output, e.g. the test case
`regression/crashes/test_bug_15.py` will be interpreted as the module
`regression.crashes.test_bug_15`. Thus one can use the directory structure to
group test cases.
#### Parameters
The test suite's configuration file should have the following form:
@ -161,7 +256,7 @@ Please note that while the `INI` file allows for variables with whitespaces or
variable names in Python.
## Test cases
### Test cases
The test cases are defined in Python source files utilizing the unittest module,
thus every file must also be a valid Python file. Each file defining a test case
@ -692,28 +787,7 @@ class AnInformativeName(metaclass=system_tests.CaseMeta):
for expansion.
## Running the test suite
## Bash test cases
The test suite is written for Python 3 and is not compatible with Python 2, thus
it must be run with `python3` and not with `python` (which is usually an alias
for Python 2).
Then navigate to the `tests/` subdirectory and run:
``` shell
python3 runner.py
```
One can supply the script with a directory where the suite should look for the
tests (it will search the directory recursively). If omitted, the runner will
look in the directory where the configuration file is located. It is also
possible to instead pass a file as the parameter, the test suite will then only
run the tests from this file.
The runner script also supports the optional arguments `--config_file` which
allows to provide a different test suite configuration file than the default
`suite.conf`. It also forwards the verbosity setting via the `-v`/`--verbose`
flags to Python's unittest module.
Optionally one can provide the `--debug` flag which will instruct test suite to
print all command invocations and all expected and obtained outputs to the
standard output.
- Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_test`.
- These Python test scripts are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in [Writing new tests](#writing-new-tests), but can be executed compatibly by `python3 runner.py`.

42
tests/WRITING_TESTS.md
View File

@ -1,42 +0,0 @@
## Writing new tests
The test suite is intended to run a binary and compare its standard output,
standard error and return value against provided values. This is implemented
using Python's `unittest` module and thus all test files are Python files.
The simplest test has the following structure:
``` python
# -*- coding: utf-8 -*-
import system_tests
class GoodTestName(metaclass=system_tests.CaseMeta):
filename = "$data_path/test_file"
commands = ["$exiv2 $filename", "$exiv2 $filename" + '_2']
stdout = [""] * 2
stderr = ["""$exiv2_exception_msg $filename:
$kerFailedToReadImageData
"""] * 2
retval = [1] * 2
```
The test suite will run the provided commands in `commands` and compare them to
the output in `stdout` and `stderr` and it will compare the return values.
The strings after a `$` are variables either defined in this test's class or are
taken from the suite's configuration file (see `doc.md` for a complete
explanation).
When creating new tests, follow roughly these steps:
1. Choose an appropriate subdirectory where the test belongs. If none fits
create a new one and put an empty `__init__.py` file there.
2. Create a new file with a name matching `test_*.py`. Copy the class definition
from the above example and choose an appropriate class name.
3. Run the test suite via `python3 runner.py` and ensure that your test case is
actually run! Either run the suite with the `-v` option which will output all
test cases that were run or simply add an error and check if errors occur.

21
tests/bash_test/README.md
View File

@ -1,21 +0,0 @@
# README
- Plan to convert the bash test scripts `../../test/*.sh` into Python scripts and save them to this directory.
- The test cases in this directory are based on [unittest](https://docs.python.org/3/library/unittest.html) and written in a common format, which is different from the format described in `../writing_tests.md`, but can be executed compatibly.
## Running the test cases
- Use runner.py to collect test cases and execute them:
```sh
cd EXIV2_DIR/tests
python3 runner.py -v unit_test
```
- Also, you can use pytest to execute the test cases for unittest, so that the test report looks better:
```sh
cd EXIV2_DIR/tests
pytest -v unit_test
```
- If no exception occurs during the execution of the test case, it passes. Otherwise it fails.

1
tests/bash_test/utils.py
View File

@ -13,7 +13,6 @@ EXIV2_DIR = os.path.normpath(os.path.join(os.path.abspath(__file__), '../../../'
BIN_DIR = os.path.join(EXIV2_DIR, 'build/bin')
DATA_DIR = os.path.join(EXIV2_DIR, 'test/data')
TEST_DIR = os.path.join(EXIV2_DIR, 'test/tmp')
BIN_SUFFIX = '' # TODO: Determine if the suffix is '.exe'
def log_info(s):