aboutsummaryrefslogtreecommitdiffstats
path: root/NEWS.md
diff options
context:
space:
mode:
Diffstat (limited to 'NEWS.md')
-rw-r--r--NEWS.md622
1 files changed, 622 insertions, 0 deletions
diff --git a/NEWS.md b/NEWS.md
new file mode 100644
index 000000000000..304cfe94695a
--- /dev/null
+++ b/NEWS.md
@@ -0,0 +1,622 @@
+Major changes between releases
+==============================
+
+
+Changes in version 0.14
+-----------------------
+
+**NOT RELEASED YET; STILL UNDER DEVELOPMENT.**
+
+* Explicitly require C++11 language features when compiling Kyua.
+
+
+Changes in version 0.13
+-----------------------
+
+**Released on August 26th, 2016.**
+
+* Fixed execution of test cases as an unprivileged user, at least under
+ NetBSD 7.0. Kyua-level failures were probably a regression introduced
+ in Kyua 0.12, but the underlying may have existed for much longer:
+ test cases might have previously failed for mysterious reasons when
+ running under an unprivileged user.
+
+* Issue #134: Fixed metadata test broken on 32-bit platforms.
+
+* Issue #139: Added per-test case start/end timestamps to all reports.
+
+* Issue #156: Fixed crashes due to the invalid handling of cleanup
+ routine data and triggered by the reuse of PIDs in long-running Kyua
+ instances.
+
+* Issue #159: Fixed TAP parser to ignore case while matching `TODO` and
+ `SKIP` directives, and to also recognize `Skipped`.
+
+* Fixed potential crash due to a race condition in the unprogramming of
+ timers to control test deadlines.
+
+
+Changes in version 0.12
+-----------------------
+
+**Released on November 22nd, 2015.**
+
+This is a huge release and marks a major milestone for Kyua as it finally
+implements a long-standing feature request: the ability to execute test
+cases in parallel. This is a big deal because test cases are rarely
+CPU-bound: running them in parallel yields much faster execution times for
+large test suites, allowing faster iteration of changes during development.
+
+As an example: the FreeBSD test suite as of this date contains 3285 test
+cases. With sequential execution, a full test suite run takes around 12
+minutes to complete, whereas on a 4-core machine with a high level of
+parallelism it takes a little over 1 minute.
+
+Implementing parallel execution required rewriting most of Kyua's core and
+partly explains explains why there has not been a new release for over a
+year. The current implementation is purely subprocess-based, which works
+but has some limitations and has resulted in a core that is really complex
+and difficult to understand. Future versions will investigate the use of
+threads instead for a simplified programming model and additional
+parallelization possibilities.
+
+* Issue #2: Implemented support to execute test cases in parallel when
+ invoking `kyua test`. Parallel execution is *only* enabled when the new
+ `parallelism` configuration variable is set to a value greater than `1`.
+ The default behavior is still to run tests sequentially because some test
+ suites contain test cases with side-effects that might fail when run in
+ parallel. To resolve this, the new metadata property `is_exclusive` can
+ be set to `true` on a test basis to indicate that the test must be run on
+ its own.
+
+* Known regression: Running `kyua debug` on a TAP-based test program does
+ not currently report the output in real time. The output will only be
+ displayed once the test program completes. This is a shortcoming of
+ the new parallel execution engine and will be resolved.
+
+* Removed the external C-based testers code in favor of the new built-in
+ implementations. The new approach feels significantly faster than the
+ previous one.
+
+* Fixed the handling of relative paths in the `fs.*` functions available
+ in `Kyuafile`s. All paths are now resolved relative to the location of
+ the caller `Kyuafile`. `Kyuafile.top` has been updated with these
+ changes and you should update custom copies of this file with the new
+ version.
+
+* Changed temporary directory creation to always grant search
+ permissions on temporary directories. This is to prevent potential
+ problems when running Kyua as root and executing test cases that require
+ dropping privileges (as they may later be unable to use absolute paths
+ that point inside their work directory).
+
+* The cleanup of work directories does not longer attempt to deal with
+ mount points. If a test case mounts a file system and forgets to unmount
+ it, the mount point will be left behind. It is now the responsibility of
+ the test case to clean after itself. The reasons for this change are
+ simplicity and clarity: there are many more things that a test case can
+ do that have side-effects on the system and Kyua cannot protect against
+ them all, so it is better to just have the test undo anything it might
+ have done.
+
+* Improved `kyua report --verbose` to properly handle environment
+ variables with continuation lines in them, and fixed the integration
+ tests for this command to avoid false negatives.
+
+* Changed the configuration file format to accept the definition of
+ unknown variables without declaring them local. The syntax version
+ number remains at 2. This is to allow configuration files for newer Kyua
+ versions to work on older Kyua versions, as there is no reason to forbid
+ this.
+
+* Fixed stacktrace gathering with FreeBSD's ancient version of GDB.
+ GDB 6.1.1 (circa 2004) does not have the `-ex` flag so we need to
+ generate a temporary GDB script and feed it to GDB with `-x` instead.
+
+* Issue #136: Fixed the XML escaping in the JUnit output so that
+ non-printable characters are properly handled when they appear in the
+ process's stdout or stderr.
+
+* Issue #141: Improved reporting of errors triggered by sqlite3. In
+ particular, all error messages are now tagged with their corresponding
+ database filename and, if they are API-level errors, the name of the
+ sqlite3 function that caused them.
+
+* Issue #144: Improved documentation on the support for custom properties
+ in the test metadata.
+
+* Converted the `INSTALL`, `NEWS`, and `README` distribution documents to
+ Markdown for better formatting online.
+
+
+Changes in version 0.11
+-----------------------
+
+**Released on October 23rd, 2014.**
+
+* Added support to print the details of all test cases (metadata and
+ their output) to `report`. This is via a new `--verbose` flag which
+ replaces the previous `--show-context`.
+
+* Added support to specify the amount of physical disk space required
+ by a test case. This is in the form of a new `required_disk_space`
+ metadata property, which can also be provided by ATF test cases as
+ `require.diskspace`.
+
+* Assimilated the contents of all the `kyua-*-tester(1)` and
+ `kyua-*-interface(7)` manual pages into more relevant places. In
+ particular, added more details on test program registration and their
+ metadata to `kyuafile(5)`, and added `kyua-test-isolation(7)`
+ describing the isolation features of the test execution.
+
+* Assimilated the contents of all auxiliary manual pages, including
+ `kyua-build-root(7)`, `kyua-results-files(7)`, `kyua-test-filters(7)`
+ and `kyua-test-isolation(7)`, into the relevant command-specific
+ manual pages. This is for easier discoverability of relevant
+ information when reading how specific Kyua commands work.
+
+* Issue #30: Plumbed through support to query configuration variables
+ from ATF's test case heads. This resolves the confusing situation
+ where test cases could only do this from their body and cleanup
+ routines.
+
+* Issue #49: Extended `report` to support test case filters as
+ command-line arguments. Combined with `--verbose`, this allows
+ inspecting the details of a test case failure after execution.
+
+* Issue #55: Deprecated support for specifying `test_suite` overrides on
+ a test program basis. This idiom should not be used but support for
+ it remains in place.
+
+* Issue #72: Added caching support to the `getcwd(3)` test in configure
+ so that the result can be overriden for cross-compilation purposes.
+
+* Issue #83: Changed manual page headings to include a `kyua` prefix in
+ their name. This prevents some possible confusion when displaying,
+ for example, the `kyua-test` manual page with a plain name of `test`.
+
+* Issue #84: Started passing test-suite configuration variables to plain
+ and TAP test programs via the environment. The name of the
+ environment variables set this way is prefixed by `TEST_ENV_`, so a
+ configuration variable of the form
+ `test_suites.some_name.allow_unsafe_ops=yes` in `kyua.conf` becomes
+ `TEST_ENV_allow_unsafe_ops=YES` in the environment.
+
+* Issues #97 and #116: Fixed the build on Illumos.
+
+* Issue #102: Set `TMPDIR` to the test case's work directory when running
+ the test case. If the test case happens to use the `mktemp(3)` family
+ of functions (due to misunderstandings on how Kyua works or due to
+ the reuse of legacy test code), we don't want it to easily escape the
+ automanaged work directory.
+
+* Issue #103: Started being more liberal in the parsing of TAP test
+ results by treating the number in `ok` and `not ok` lines as optional.
+
+* Issue #105: Started using tmpfs instead of md as a temporary file
+ system for tests in FreeBSD so that we do not leak `md(4)` devices.
+
+* Issue #109: Changed the privilege dropping code to start properly
+ dropping group privileges when `unprivileged_user` is set. Also fixes
+ `testers/run_test:fork_wait__unprivileged_group`.
+
+* Issue #110: Changed `help` to display version information and clarified
+ the purpose of the `about` command in its documentation.
+
+* Issue #111: Fixed crash when defining a test program in a `Kyuafile`
+ that has not yet specified the test suite name.
+
+* Issue #114: Improved the `kyuafile(5)` manual page by clarifying the
+ restrictions of the `include()` directive and by adding abundant
+ examples.
+
+
+Changes in version 0.10
+-----------------------
+
+**Experimental version released on August 14th, 2014.**
+
+* Merged `kyua-cli` and `kyua-testers` into a single `kyua` package.
+
+* Dropped the `kyua-atf-compat` package.
+
+* Issue #100: Do not try to drop privileges to `unprivileged_user` when we
+ are already running as an unprivileged user. Doing so is not possible
+ and thus causes spurious test failures when the current user is not
+ root and the current user and `unprivileged_user` do not match.
+
+* Issue #79: Mention `kyua.conf(5)` in the *See also* section of `kyua(1)`.
+
+* Issue #75: Change the `rewrite__expected_signal__bad_arg` test in
+ `testers/atf_result_test` to use a different signal value. This is to
+ prevent triggering a core dump that made the test fail in some platforms.
+
+
+Changes in kyua-cli version 0.9
+-------------------------------
+
+**Experimental version released on August 8th, 2014.**
+
+Major changes:
+
+The internal architecture of Kyua to record the results of test suite
+runs has completely changed in this release. Kyua no longer stores all
+the different test suite run results as different "actions" within the
+single `store.db` database. Instead, Kyua now generates a separate
+results file inside `~/.kyua/store/` for every test suite run.
+
+Due to the complexity involved in the migration process and the little
+need for it, this is probably going to be the only release where the
+`db-migrate` command is able to convert an old `store.db` file to the
+new scheme.
+
+Changes in more detail:
+
+* Added the `report-junit` command to generate JUnit XML result files.
+ The output has been verified to work within Jenkins.
+
+* Switched to results files specific to their corresponding test suite
+ run. The unified `store.db` file is now gone: `kyua test` creates a
+ new results file for every invocation under `~/.kyua/store/` and the
+ `kyua report*` commands are able to locate the latest file for a
+ corresponding test suite automatically.
+
+* The `db-migrate` command takes an old `store.db` file and generates
+ one results file for every previously-recorded action, later deleting
+ the `store.db` file.
+
+* The `--action` flag has been removed from all commands that accepted
+ it. This has been superseded by the tests results files.
+
+* The `--store` flag that many commands took has been renamed to
+ `--results-file` in line with the semantical changes.
+
+* The `db-exec` command no longer creates an empty database when none
+ is found. This command is now intended to run only over existing
+ files.
+
+
+Changes in kyua-testers version 0.3
+-----------------------------------
+
+**Experimental version released on August 8th, 2014.**
+
+* Made the testers set a "sanitized" value for the `HOME` environment
+ variable where, for example, consecutive and trailing slashes have
+ been cleared. Mac OS X has a tendency to append a trailing slash to
+ the value of `TMPDIR`, which can cause third-party tests to fail if
+ they compare `${HOME}` with `$(pwd)`.
+
+* Issues #85, #86, #90 and #92: Made the TAP parser more complete: mark
+ test cases reported as `TODO` or `SKIP` as passed; handle skip plans;
+ ignore lines that look like `ok` and `not ok` but aren't results; and
+ handle test programs that report a pass but exit with a non-zero code.
+
+
+Changes in kyua-cli version 0.8
+-------------------------------
+
+**Experimental version released on December 7th, 2013.**
+
+* Added support for Lutok 0.4.
+
+* Issue #24: Plug the bootstrap tests back into the test suite. Fixes
+ in `kyua-testers` 0.2 to isolate test cases into their own sessions
+ should allow these to run fine.
+
+* Issue #74: Changed the `kyuafile(5)` parser to automatically discover
+ existing tester interfaces. The various `*_test_program()` functions
+ will now exist (or not) based on tester availability, which simplifies
+ the addition of new testers or the selective installation of them.
+
+
+Changes in kyua-testers version 0.2
+-----------------------------------
+
+**Experimental version released on December 7th, 2013.**
+
+* Issue #74: Added the `kyua-tap-tester`, a new backend to interact with
+ test programs that comply with the Test Anything Protocol.
+
+* Issue #69: Cope with the lack of `AM_PROG_AR` in `configure.ac`, which
+ first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04
+ LTS, which appears stuck in 1.11.1.
+
+* Issue #24: Improve test case isolation by confining the tests to their
+ own session instead of just to their own process group.
+
+
+Changes in kyua-cli version 0.7
+-------------------------------
+
+**Experimental version released on October 18th, 2013.**
+
+* Made failures from testers more resilent. If a tester fails, the
+ corresponding test case will be marked as broken instead of causing
+ kyua to exit.
+
+* Added the `--results-filter` option to the `report-html` command and
+ set its default value to skip passed results from HTML reports. This
+ is to keep these reports more succint and to avoid generating tons of
+ detail files that will be, in general, useless.
+
+* Switched to use Lutok 0.3 to gain compatibility with Lua 5.2.
+
+* Issue #69: Cope with the lack of `AM_PROG_AR` in `configure.ac`, which
+ first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04
+ LTS, which appears stuck in 1.11.1.
+
+
+Changes in kyua-cli version 0.6
+-------------------------------
+
+**Experimental version released on February 22nd, 2013.**
+
+* Issue #36: Changed `kyua help` to not fail when the configuration file
+ is bogus. Help should always work.
+
+* Issue #37: Simplified the `syntax()` calls in configuration and
+ `Kyuafile` files to only specify the requested version instead of also
+ the format name. The format name is implied by the file being loaded, so
+ there is no use in the caller having to specify it. The version number
+ of these file formats has been bumped to 2.
+
+* Issue #39: Added per-test-case metadata values to the HTML reports.
+
+* Issue #40: Rewrote the documentation as manual pages and removed the
+ previous GNU Info document.
+
+* Issue #47: Started using the independent testers in the `kyua-testers`
+ package to run the test cases. Kyua does not implement the logic to
+ invoke test cases any more, which provides for better modularity,
+ extensibility and robustness.
+
+* Issue #57: Added support to specify arbitrary metadata properties for
+ test programs right from the `Kyuafile`. This is to make plain test
+ programs more versatile, by allowing them to specify any of the
+ requirements (allowed architectures, required files, etc.) supported
+ by Kyua.
+
+* Reduced automatic screen line wrapping of messages to the `help`
+ command and the output of tables by `db-exec`. Wrapping any other
+ messages (specially anything going to stderr) was very annoying
+ because it prevented natural copy/pasting of text.
+
+* Increased the granularity of the error codes returned by `kyua(1)` to
+ denote different error conditions. This avoids the overload of `1` to
+ indicate both "expected" errors from specific subcommands and
+ unexpected errors caused by the internals of the code. The manual now
+ correctly explain how the exit codes behave on a command basis.
+
+* Optimized the database schema to make report generation almost
+ instantaneous.
+
+* Bumped the database schema to 2. The database now records the
+ metadata of both test programs and test cases generically, without
+ knowledge of their interface.
+
+* Added the `db-migrate` command to provide a mechanism to upgrade a
+ database with an old schema to the current schema.
+
+* Removed the GDB build-time configuration variable. This is now part
+ of the `kyua-testers` package.
+
+* Issue #31: Rewrote the `Kyuafile` parsing code in C++, which results in
+ a much simpler implementation. As a side-effect, this gets rid of the
+ external Lua files required by `kyua`, which in turn make the tool
+ self-contained.
+
+* Added caching of various configure test results (particularly in those
+ tests that need to execute a test program) so that cross-compilers can
+ predefine the results of the tests without having to run the
+ executables.
+
+
+Changes in kyua-testers version 0.1
+-----------------------------------
+
+**Experimental version released on February 19th, 2013.**
+
+This is the first public release of the `kyua-testers` package.
+
+The goal of this first release is to adopt all the test case execution
+code of `kyua-cli` 0.5 and ship it as a collection of independent tester
+binaries. The `kyua-cli` package will rely on these binaries to run the
+tests, which provides better modularity and simplicity to the
+architecture of Kyua.
+
+The code in this package is all C as opposed to the current C++ codebase
+of `kyua-cli`, which means that the overall build times of Kyua are now
+reduced.
+
+
+Changes in kyua-cli version 0.5
+-------------------------------
+
+**Experimental version released on July 10th, 2012.**
+
+* Issue #15: Added automatic stacktrace gathering of crashing test cases.
+ This relies on GDB and is a best-effort operation.
+
+* Issue #32: Added the `--build-root` option to the debug, list and test
+ commands. This allows executing test programs from a different
+ directory than where the `Kyuafile` scripts live. See the *Build roots*
+ section in the manual for more details.
+
+* Issue #33: Removed the `kyuaify.sh` script. This has been renamed to
+ atf2kyua and moved to the `kyua-atf-compat` module, where it ships as a
+ first-class utility (with a manual page and tests).
+
+* Issue #34: Changed the HTML reports to include the stdout and stderr of
+ every test case.
+
+* Fixed the build when using a "build directory" and a clean source tree
+ from the repository.
+
+
+Changes in kyua-cli version 0.4
+-------------------------------
+
+**Experimental version released on June 6th, 2012.**
+
+* Added the `report-html` command to generate HTML reports of the
+ execution of any recorded action.
+
+* Changed the `--output` flag of the `report` command to only take a
+ path to the target file, not its format. Different formats are better
+ supported by implementing different subcommands, as the options they
+ may receive will vary from format to format.
+
+* Added a `--with-atf` flag to the configure script to control whether
+ the ATF tests get built or not. May be useful for packaging systems
+ that do not have ATF in them yet. Disabling ATF also cuts down the
+ build time of Kyua significantly, but with the obvious drawbacks.
+
+* Grouped `kyua` subcommands by topic both in the output of `help` and
+ in the documentation. In general, the user needs to be aware of
+ commands that rely on a current project and those commands that rely
+ purely on the database to generate reports.
+
+* Made `help` print the descriptions of options and commands properly
+ tabulated.
+
+* Changed most informational messages to automatically wrap on screen
+ boundaries.
+
+* Rewrote the configuration file parsing module for extensibility. This
+ will allow future versions of Kyua to provide additional user-facing
+ options in the configuration file.
+
+ No syntax changes have been made, so existing configuration files
+ (version 1) will continue to be parsed without problems. There is one
+ little exception though: all variables under the top-level
+ `test_suites` tree must be declared as strings.
+
+ Similarly, the `-v` and `--variable` flags to the command line must
+ now carry a `test_suites.` prefix when referencing any variables under
+ such tree.
+
+
+Changes in kyua-cli version 0.3
+-------------------------------
+
+**Experimental version released on February 24th, 2012.**
+
+* Made the `test` command record the results of the executed test
+ cases into a SQLite database. As a side effect, `test` now supports a
+ `--store` option to indicate where the database lives.
+
+* Added the `report` command to generate plain-text reports of the
+ test results stored in the database. The interface of this command is
+ certainly subject to change at this point.
+
+* Added the `db-exec` command to directly interact with the store
+ database.
+
+* Issue #28: Added support for the `require.memory` test case property
+ introduced in ATF 0.15.
+
+* Renamed the user-specific configuration file from `~/.kyuarc` to
+ `~/.kyua/kyua.conf` for consistency with other files stored in the
+ `~/.kyua/` subdirectory.
+
+* Switched to use Lutok instead of our own wrappers over the Lua C
+ library. Lutok is just what used to be our own utils::lua module, but
+ is now distributed separately.
+
+* Removed the `Atffile`s from the source tree. Kyua is stable enough
+ to generate trustworthy reports, and we do not want to give the
+ impression that atf-run / atf-report are still supported.
+
+* Enabled logging to stderr for our own test programs. This makes it
+ slightly easier to debug problems in our own code when we get a
+ failing test.
+
+
+Changes in kyua-cli version 0.2
+-------------------------------
+
+**Experimental version released on August 24th, 2011.**
+
+The biggest change in this release is the ability for Kyua to run test
+programs implemented using different frameworks. What this means is
+that, now, a Kyua test suite can include not only ATF-based test
+programs, but also "legacy" (aka plain) test programs that do not use
+any framework. I.e. if you have tests that are simple programs that
+exit with 0 on success and 1 on failure, you can plug them in into a
+Kyua test suite.
+
+Other than this, there have been several user-visible changes. The most
+important are the addition of the new `config` and `debug` subcommands
+to the `kyua` binary. The former can be used to inspect the runtime
+configuration of Kyua after parsing, and the latter is useful to
+interact with failing tests cases in order to get more data about the
+failure itself.
+
+Without further ado, here comes the itemized list of changes:
+
+* Generalized the run-time engine to support executing test programs
+ that implement different interfaces. Test programs that use the ATF
+ libraries are just a special case of this. (Issue #18.)
+
+* Added support to the engine to run `plain` test programs: i.e. test
+ programs that do not use any framework and report their pass/fail
+ status as an exit code. This is to simplify the integration of legacy
+ test programs into a test suite, and also to demonstrate that the
+ run-time engine is generic enough to support different test
+ interfaces. (Issue #18.)
+
+* Added the `debug` subcommand. This command allows end users to tweak
+ the execution of a specific test case and to poke into the behavior of
+ its execution. At the moment, all this command allows is to view the
+ stdout and stderr of the command in real time (which the `test`
+ command currently completely hides).
+
+* Added the `config` subcommand. This command allows the end user to
+ inspect the current configuration variables after evaluation, without
+ having to read through configuration files. (Issue #11.)
+
+* Removed the `test_suites_var` function from configuration files. This
+ was used to set the value of test-suite-sepecific variables, but it
+ was ugly-looking. It is now possible to use the more natural syntax
+ `test_suites.<test-suite-name>.<variable> = <value>`. (Issue #11.)
+
+* Added a mechanism to disable the loading of configuration files
+ altogether. Needed for testing purposes and for scriptability.
+ Available by passing the `--config=none` flag.
+
+* Enabled detection of unused parameters and variables in the code and
+ fixed all warnings. (Issue #23.)
+
+* Changed the behavior of "developer mode". Compiler warnings are now
+ enabled unconditionally regardless of whether we are in developer mode
+ or not; developer mode is now only used to perform strict warning
+ checks and to enable assertions. Additionally, developer mode is now
+ only automatically enabled when building from the repository, not for
+ formal releases. (Issue #22.)
+
+* Fixed many build and portability problems to Debian sid with GCC 4.6.3
+ and Ubuntu 10.04.1 LTS. (Issues #20, #21, #26.)
+
+
+Changes in kyua-cli version 0.1
+-------------------------------
+
+**Experimental version released on June 23rd, 2011.**
+
+This is the first public release of the `kyua-cli` package.
+
+The scope of this release is to provide functional replacement for the
+`atf-run` utility included in the atf package. At this point, `kyua`
+can reliably run the NetBSD 5.99.53 test suite delivering the same
+results as `atf-run`.
+
+The reporting facilities of this release are quite limited. There is
+no replacement for `atf-report` yet, and there is no easy way of
+debugging failing test programs other than running them by hand. These
+features will mark future milestones and therefore be part of other
+releases.
+
+Be aware that this release has suffered very limited field testing.
+The test suite for `kyua-cli` is quite comprehensive, but some bugs may
+be left in any place.