diff options
542 files changed, 96704 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 000000000000..d7f1a180d6fb --- /dev/null +++ b/.gitignore @@ -0,0 +1,23 @@ +*.a +*.o +*_helpers +*_inttest +*_test +*~ + +.deps +.dirstamp +Doxyfile +Makefile +Makefile.in +aclocal.m4 +api-docs +autom4te.cache +config.h +config.h.in +config.log +config.status +configure +kyua +local-kyua +stamp-h1 diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 000000000000..619dceaf3d2d --- /dev/null +++ b/.travis.yml @@ -0,0 +1,49 @@ +language: cpp +sudo: required + +before_install: + - ./admin/travis-install-deps.sh + +matrix: + include: + - os: linux + dist: xenial + compiler: clang + env: ARCH=amd64 DO=distcheck AS_ROOT=no + - os: linux + dist: xenial + compiler: gcc + env: ARCH=amd64 DO=distcheck AS_ROOT=no + - os: linux + dist: xenial + compiler: clang + env: ARCH=amd64 DO=apidocs + - os: linux + dist: xenial + compiler: clang + env: ARCH=amd64 DO=style + - os: linux + dist: xenial + compiler: clang + env: ARCH=amd64 DO=distcheck AS_ROOT=yes UNPRIVILEGED_USER=no + - os: linux + dist: xenial + compiler: clang + env: ARCH=amd64 DO=distcheck AS_ROOT=yes UNPRIVILEGED_USER=yes + # TODO(ngie): reenable i386; the libraries were not available in the + # Ubuntu Xenial x86_64 docker image. + #- os: linux + # dist: xenial + # compiler: clang + # env: ARCH=i386 DO=distcheck AS_ROOT=no + #- os: linux + # dist: xenial + # compiler: gcc + # env: ARCH=i386 DO=distcheck AS_ROOT=no + +script: + - ./admin/travis-build.sh + +notifications: + email: + - kyua-log@googlegroups.com diff --git a/AUTHORS b/AUTHORS new file mode 100644 index 000000000000..ac0998fb937c --- /dev/null +++ b/AUTHORS @@ -0,0 +1,11 @@ +# This is the official list of Kyua authors for copyright purposes. +# +# This file is distinct from the CONTRIBUTORS files; see the latter for +# an explanation. +# +# Names are sorted alphabetically and should be added to this file as: +# +# * Name <email address> +# * Organization <optional email address> + +* Google Inc. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000000..daa55c308e97 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,173 @@ +Contributing code to Kyua +========================= + +Want to contribute? Great! But first, please take a few minutes to read this +document in full. Doing so upfront will minimize the turnaround time required +to get your changes incorporated. + + +Legal notes +----------- + +* Before we can use your code, you must sign the + [Google Individual Contributor License + Agreement](https://developers.google.com/open-source/cla/individual), + also known as the CLA, which you can easily do online. The CLA is necessary + mainly because you own the copyright to your changes, even after your + contribution becomes part of our codebase, so we need your permission to use + and distribute your code. We also need to be sure of various other + things--for instance that you will tell us if you know that your code + infringes on other people's patents. You do not have to sign the CLA until + after you have submitted your code for review and a member has approved it, + but you must do it before we can put your code into our codebase. + +* Contributions made by corporations are covered by a different agreement than + the one above: the + [Google Software Grant and Corporate Contributor License + Agreement](https://developers.google.com/open-source/cla/corporate). + Please get your company to sign this agreement instead if your contribution is + on their behalf. + +* Unless you have a strong reason not to, please assign copyright of your + changes to Google Inc. and use the 3-clause BSD license text included + throughout the codebase (see [LICENSE](LICENSE)). Keeping the whole project + owned by a single entity is important, particularly to avoid the problem of + having to replicate potentially hundreds of different copyright notes in + documentation materials, etc. + + +Communication +------------- + +* Before you start working on a larger contribution, you should get in touch + with us first through the + [kyua-discuss mailing + list](https://groups.google.com/forum/#!forum/kyua-discuss) + with your idea so that we can help out and possibly guide you. Coordinating + upfront makes it much easier to avoid frustration later on. + +* Subscribe to the + [kyua-log mailing list](https://groups.google.com/forum/#!forum/kyua-log) to + get notifications on new commits, Travis CI results, or changes to bugs. + + +Git workflow +------------ + +* Always work on a non-master branch. + +* Make sure the history of your branch is clean. (Ab)use `git rebase -i master` + to ensure the sequence of commits you want pulled is easy to follow and that + every commit does one (and only one) thing. In particular, commits of the + form `Fix previous` or `Fix build` should never ever exist; merge those fixes + into the relevant commits so that the history is clean at pull time. + +* Always trigger Travis CI builds for your changes (hence why working on a + branch is important). Push your branch to GitHub so that Travis CI picks it + up and performs a build. If you have forked the repository, you may need to + enable Travis CI builds on your end. Wait for a green result. + +* It is OK and expected for you to `git push --force` on **non-master** + branches. This is required if you need to go through the commit/test cycle + more than once for any given branch after you have "fixed-up" commits to + correct problems spotted in earlier builds. + +* Do not send pull requests that subsume other/older pull requests. Each major + change being submitted belongs in a different pull request, which is trivial + to achieve if you use one branch per change as requested in this workflow. + + +Code reviews +------------ + +* All changes will be subject to code reviews pre-merge time. In other words: + all pull requests will be carefully inspected before being accepted and they + will be returned to you with comments if there are issues to be fixed. + +* Be careful of stylistic errors in your code (see below for style guidelines). + Style violations hinder the review process and distract from the actual code. + By keeping your code clean of style issues upfront, you will speed up the + review process and avoid frustration along the way. + +* Whenever you are ready to submit a pull request, review the *combined diff* + you are requesting to be pulled and look for issues. This is the diff that + will be subject to review, not necessarily the individual commits. You can + view this diff in GitHub at the bottom of the `Open a pull request` form that + appears when you click the button to file a pull request, or you can see the + diff by typing `git diff <your-branch> master`. + + +Commit messages +--------------- + +* Follow standard Git commit message guidelines. The first line has a maximum + length of 50 characters, does not terminate in a period, and has to summarize + the whole commit. Then a blank line comes, and then multiple plain-text + paragraphs provide details on the commit if necessary with a maximum length of + 72-75 characters per line. Vim has syntax highlighting for Git commit + messages and will let you know when you go above the maximum line lengths. + +* Use the imperative tense. Say `Add foo-bar` or `Fix baz` instead of `Adding + blah`, `Adds bleh`, or `Added bloh`. + + +Handling bug tracker issues +--------------------------- + +* All changes pushed to `master` should cross-reference one or more issues in + the bug tracker. This is particularly important for bug fixes, but also + applies to major feature improvements. + +* Unless you have a good reason to do otherwise, name your branch `issue-N` + where `N` is the number of the issue being fixed. + +* If the fix to the issue can be done *in a single commit*, terminate the commit + message with `Fixes #N.` where `N` is the number of the issue being fixed and + include a note in `NEWS` about the issue in the same commit. Such fixes can + be merged onto master using fast-forward (the default behavior of `git + merge`). + +* If the fix to the issue requires *more than one commit*, do **not** include + `Fixes #N.` in any of the individual commit messages of the branch nor include + any changes to the `NEWS` file in those commits. These "announcement" changes + belong in the merge commit onto `master`, which is done by `git merge --no-ff + --no-commit your-branch`, followed by an edit of `NEWS`, and terminated with a + `git commit -a` with the proper note on the bug being fixed. + + +Style guide +----------- + +These notes are generic and certainly *non-exhaustive*: + +* Respect formatting of existing files. Note where braces are placed, number of + blank lines between code chunks, how continuation lines are indented, how + docstrings are typed, etc. + +* Indentation is *always* done using spaces, not tabs. The only exception is in + `Makefile`s, where any continuation line within a target must be prefixed by a + *single tab*. + +* [Be mindful of spelling and + grammar.](http://julipedia.meroh.net/2013/06/readability-mind-your-typos-and-grammar.html) + Mistakes of this kind are enough of a reason to return a pull request. + +* Use proper punctuation for all sentences. Always start with a capital letter + and terminate with a period. + +* Respect lexicographical sorting wherever possible. + +* Lines must not be over 80 characters. + +* No trailing whitespace. + +* Two spaces after end-of-sentence periods. + +* Two blank lines between functions. If there are two blank lines among code + blocks, they usually exist for a reason: keep them. + +* In C++ code, prefix all C identifiers (those coming from `extern "C"` + includes) with `::`. + +* Getter functions/methods only need to be documented via `\return`. A + redundant summary is not necessary. diff --git a/CONTRIBUTORS b/CONTRIBUTORS new file mode 100644 index 000000000000..faf726a4fefd --- /dev/null +++ b/CONTRIBUTORS @@ -0,0 +1,20 @@ +# This is the list of people who have agreed to one of the CLAs and can +# contribute patches to the Kyua project. +# +# The AUTHORS file lists the copyright holders; this file lists people. +# For example: Google employees are listed here but not in AUTHORS +# because Google holds the copyright. +# +# See the following links for details on the CLA: +# +# https://developers.google.com/open-source/cla/individual +# https://developers.google.com/open-source/cla/corporate +# +# Names are sorted by last name and should be added as: +# +# * Name <email address> + +* Sergey Bronnikov <sergeyb@openvz.org> +* Enji Cooper <yaneurabeya@gmail.com> +* Julio Merino <jmmv@google.com> +* Craig Rodrigues <rodrigc@crodrigues.org> diff --git a/Doxyfile.in b/Doxyfile.in new file mode 100644 index 000000000000..e28d82f8999a --- /dev/null +++ b/Doxyfile.in @@ -0,0 +1,59 @@ +# Copyright 2010 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +BUILTIN_STL_SUPPORT = YES +ENABLE_PREPROCESSING = YES +EXCLUDE_SYMBOLS = "ATF_TC*" +EXTRACT_ANON_NSPACES = YES +EXTRACT_LOCAL_CLASSES = YES +EXTRACT_PRIVATE = YES +EXTRACT_STATIC = YES +EXPAND_ONLY_PREDEF = YES +EXTENSION_MAPPING = ipp = C++ +FILE_PATTERNS = *.c *.h *.cpp *.hpp *.ipp +GENERATE_LATEX = NO +GENERATE_TAGFILE = @top_builddir@/api-docs/api-docs.tag +HIDE_FRIEND_COMPOUNDS = YES +INPUT = @top_srcdir@ +INPUT_ENCODING = ISO-8859-1 +JAVADOC_AUTOBRIEF = YES +MACRO_EXPANSION = YES +OUTPUT_DIRECTORY = @top_builddir@/api-docs +OUTPUT_LANGUAGE = English +PREDEFINED += "KYUA_DEFS_NORETURN=" +PREDEFINED += "KYUA_DEFS_FORMAT_PRINTF(x, y)=" +PROJECT_NAME = "@PACKAGE_NAME@" +PROJECT_NUMBER = @VERSION@ +QUIET = YES +RECURSIVE = YES +SORT_BY_SCOPE_NAME = YES +SORT_MEMBERS_CTORS_1ST = YES +WARN_IF_DOC_ERROR = YES +WARN_IF_UNDOCUMENTED = YES +WARN_NO_PARAMDOC = YES +WARNINGS = YES diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 000000000000..d3dcab49cb74 --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,268 @@ +Installation instructions +========================= + +Kyua uses the GNU Automake, GNU Autoconf and GNU Libtool utilities as +its build system. These are used only when compiling the application +from the source code package. If you want to install Kyua from a binary +package, you do not need to read this document. + +For the impatient: + + $ ./configure + $ make + $ make check + Gain root privileges + # make install + Drop root privileges + $ make installcheck + +Or alternatively, install as a regular user into your home directory: + + $ ./configure --prefix ~/local + $ make + $ make check + $ make install + $ make installcheck + + +Dependencies +------------ + +To build and use Kyua successfully you need: + +* A standards-compliant C and C++ complier. +* Lutok 0.4. +* pkg-config. +* SQLite 3.6.22. + +To build the Kyua tests, you optionally need: + +* The Automated Testing Framework (ATF), version 0.15 or greater. This + is required if you want to create a distribution file. + +If you are building Kyua from the code on the repository, you will also +need the following tools: + +* GNU Autoconf. +* GNU Automake. +* GNU Libtool. + + +Regenerating the build system +----------------------------- + +This is not necessary if you are building from a formal release +distribution file. + +On the other hand, if you are building Kyua from code extracted from the +repository, you must first regenerate the files used by the build +system. You will also need to do this if you modify `configure.ac`, +`Makefile.am` or any of the other build system files. To do this, simply +run: + + $ autoreconf -i -s + +If ATF is installed in a different prefix than Autoconf, you will also +need to tell autoreconf where the ATF M4 macros are located. Otherwise, +the configure script will be incomplete and will show confusing syntax +errors mentioning, for example, `ATF_CHECK_SH`. To fix this, you have +to run autoreconf in the following manner, replacing `<atf-prefix>` with +the appropriate path: + + $ autoreconf -i -s -I <atf-prefix>/share/aclocal + + +General build procedure +----------------------- + +To build and install the source package, you must follow these steps: + +1. Configure the sources to adapt to your operating system. This is + done using the `configure` script located on the sources' top + directory, and it is usually invoked without arguments unless you + want to change the installation prefix. More details on this + procedure are given on a later section. + +2. Build the sources to generate the binaries and scripts. Simply run + `make` on the sources' top directory after configuring them. No + problems should arise. + +3. Check that the built programs work by running `make check`. You do + not need to be root to do this, but if you are not, some checks will + be skipped. + +4. Install the program by running `make install`. You may need to + become root to issue this step. + +5. Issue any manual installation steps that may be required. These are + described later in their own section. + +6. Check that the installed programs work by running `make + installcheck`. You do not need to be root to do this, but if you are + not, some checks will be skipped. + + +Configuration flags +------------------- + +The most common, standard flags given to `configure` are: + +* `--prefix=directory`: + **Possible values:** Any path. + **Default:** `/usr/local`. + + Specifies where the program (binaries and all associated files) will + be installed. + +* `--sysconfdir=directory`: + **Possible values:** Any path. + **Default:** `/usr/local/etc`. + + Specifies where the installed programs will look for configuration + files. `/kyua` will be appended to the given path unless + `KYUA_CONFSUBDIR` is redefined as explained later on. + +* `--help`: + + Shows information about all available flags and exits immediately, + without running any configuration tasks. + +The following environment variables are specific to Kyua's `configure` +script: + +* `GDB`: + **Possible values:** empty, absolute path to GNU GDB. + **Default:** empty. + + Specifies the path to the GNU GDB binary that Kyua will use to gather a + stack trace of a crashing test program. If empty, the configure script + will try to find a suitable binary for you and, if not found, Kyua will + attempt to do the search at run time. + +* `KYUA_ARCHITECTURE`: + **Possible values:** name of a CPU architecture (e.g. `x86_64`, `powerpc`). + **Default:** autodetected; typically the output of `uname -p`. + + Specifies the name of the CPU architecture on which Kyua will run. + This value is used at run-time to determine tests that are not + applicable to the host system. + +* `KYUA_CONFSUBDIR`: + **Possible values:** empty, a relative path. + **Default:** `kyua`. + + Specifies the subdirectory of the configuration directory (given by + the `--sysconfdir` argument) under which Kyua will search for its + configuration files. + +* `KYUA_CONFIG_FILE_FOR_CHECK`: + **Possible values:** none, an absolute path to an existing file. + **Default:** none. + + Specifies the `kyua.conf` configuration file to use when running any + of the `check`, `installcheck` or `distcheck` targets on this source + tree. This setting is exclusively used to customize the test runs of + Kyua itself and has no effect whatsoever on the built product. + +* `KYUA_MACHINE`: + **Possible values:** name of a machine type (e.g. `amd64`, `macppc`). + **Default:** autodetected; typically the output of `uname -m`. + + Specifies the name of the machine type on which Kyua will run. This + value is used at run-time to determine tests that are not applicable + to the host system. + +* `KYUA_TMPDIR`: + **Possible values:** an absolute path to a temporary directory. + **Default:** `/tmp`. + + Specifies the path that Kyua will use to create temporary directories + in by default. + +The following flags are specific to Kyua's `configure` script: + +* `--enable-developer`: + **Possible values:** `yes`, `no`. + **Default:** `yes` in Git `HEAD` builds; `no` in formal releases. + + Enables several features useful for development, such as the inclusion + of debugging symbols in all objects or the enforcement of compilation + warnings. + + The compiler will be executed with an exhaustive collection of warning + detection features regardless of the value of this flag. However, such + warnings are only fatal when `--enable-developer` is `yes`. + +* `--with-atf`: + **Possible values:** `yes`, `no`, `auto`. + **Default:** `auto`. + + Enables usage of ATF to build (and later install) the tests. + + Setting this to `yes` causes the configure script to look for ATF + unconditionally and abort if not found. Setting this to `auto` lets + configure perform the best decision based on availability of ATF. + Setting this to `no` explicitly disables ATF usage. + + When support for tests is enabled, the build process will generate the + test programs and will later install them into the tests tree. + Running `make check` or `make installcheck` from within the source + directory will cause these tests to be run with Kyua. + +* `--with-doxygen`: + **Possible values:** `yes`, `no`, `auto` or a path. + **Default:** `auto`. + + Enables usage of Doxygen to generate documentation for internal APIs. + This documentation is *not* installed and is only provided to help the + developer of this package. Therefore, enabling or disabling Doxygen + causes absolutely no differences on the files installed by this + package. + + Setting this to `yes` causes the configure script to look for Doxygen + unconditionally and abort if not found. Setting this to `auto` lets + configure perform the best decision based on availability of Doxygen. + Setting this to `no` explicitly disables Doxygen usage. And, lastly, + setting this to a path forces configure to use a specific Doxygen + binary, which must exist. + + +Post-installation steps +----------------------- + +Copy the `Kyuafile.top` file installed in the examples directory to the +root of your tests hierarchy and name it `Kyuafile`. For example: + + # cp /usr/local/share/kyua/examples/Kyuafile.top \ + /usr/local/tests/Kyuafile + +This will allow you to simply go into `/usr/tests` and run the tests +from there. + + +Run the tests! +-------------- + +Lastly, after a successful installation, you should periodically run the +tests from the final location to ensure things remain stable. Do so as +follows: + + $ cd /usr/local/kyua && kyua test + +The following configuration variables are specific to the 'kyua' test +suite and can be given to Kyua with arguments of the form +`-v test_suites.kyua.<variable_name>=<value>`: + +* `run_coredump_tests`: + **Possible values:** `true` or `false`. + **Default:** `true`. + + Avoids running tests that crash subprocesses on purpose to make them + dump core. Such tests are particularly slow on macOS, and it is + sometimes handy to disable them for quicker development iteration. + +If you see any tests fail, do not hesitate to report them in: + + https://github.com/jmmv/kyua/issues/ + +Thank you! diff --git a/Kyuafile b/Kyuafile new file mode 100644 index 000000000000..e986218a45f4 --- /dev/null +++ b/Kyuafile @@ -0,0 +1,18 @@ +syntax(2) + +test_suite("kyua") + +include("bootstrap/Kyuafile") +include("cli/Kyuafile") +if fs.exists("doc/Kyuafile") then + -- The tests for the docs are not installed because they only cover the + -- build-time process of the manual pages. + include("doc/Kyuafile") +end +include("drivers/Kyuafile") +include("engine/Kyuafile") +include("examples/Kyuafile") +include("integration/Kyuafile") +include("model/Kyuafile") +include("store/Kyuafile") +include("utils/Kyuafile") diff --git a/LICENSE b/LICENSE new file mode 100644 index 000000000000..ffb8e3da7d86 --- /dev/null +++ b/LICENSE @@ -0,0 +1,27 @@ +Copyright 2010-2015 The Kyua Authors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +* Neither the name of Google Inc. nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/Makefile.am b/Makefile.am new file mode 100644 index 000000000000..d7f3cd27e73b --- /dev/null +++ b/Makefile.am @@ -0,0 +1,186 @@ +# Copyright 2010 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ACLOCAL_AMFLAGS = -I m4 + +CHECK_BOOTSTRAP_DEPS = +CHECK_KYUA_DEPS = +CHECK_LOCAL = +CLEAN_TARGETS = +DIST_HOOKS = +PHONY_TARGETS = +CLEANFILES = + +EXTRA_DIST = +noinst_DATA = +noinst_LIBRARIES = +noinst_SCRIPTS = + +doc_DATA = AUTHORS CONTRIBUTING.md CONTRIBUTORS LICENSE NEWS.md +noinst_DATA += INSTALL.md README.md +EXTRA_DIST += $(doc_DATA) INSTALL.md README.md + +if WITH_ATF +tests_topdir = $(pkgtestsdir) + +tests_top_DATA = Kyuafile +EXTRA_DIST += $(tests_top_DATA) +endif + +include admin/Makefile.am.inc +include bootstrap/Makefile.am.inc +include cli/Makefile.am.inc +include doc/Makefile.am.inc +include drivers/Makefile.am.inc +include engine/Makefile.am.inc +include examples/Makefile.am.inc +include integration/Makefile.am.inc +include misc/Makefile.am.inc +include model/Makefile.am.inc +include store/Makefile.am.inc +include utils/Makefile.am.inc + +bin_PROGRAMS = kyua +kyua_SOURCES = main.cpp +kyua_CXXFLAGS = $(CLI_CFLAGS) $(ENGINE_CFLAGS) $(UTILS_CFLAGS) +kyua_LDADD = $(CLI_LIBS) $(ENGINE_LIBS) $(UTILS_LIBS) + +CHECK_ENVIRONMENT = KYUA_CONFDIR="/non-existent" \ + KYUA_DOCDIR="$(abs_top_srcdir)" \ + KYUA_EXAMPLESDIR="$(abs_top_srcdir)/examples" \ + KYUA_MISCDIR="$(abs_top_srcdir)/misc" \ + KYUA_STOREDIR="$(abs_top_srcdir)/store" \ + KYUA_STORETESTDATADIR="$(abs_top_srcdir)/store" \ + PATH="$(abs_top_builddir):$${PATH}" +INSTALLCHECK_ENVIRONMENT = KYUA_CONFDIR="/non-existent" \ + PATH="$(prefix)/bin:$${PATH}" + +# Generate local-kyua, a wrapper shell script to run the just-built 'kyua' +# binary by pointing it to the possibly not-yet-installed data files in the +# build tree. +noinst_SCRIPTS += local-kyua +CLEANFILES += local-kyua local-kyua.tmp +local-kyua: Makefile + $(AM_V_GEN)echo '#!/bin/sh' >local-kyua.tmp; \ + echo 'env $(CHECK_ENVIRONMENT) $(TESTS_ENVIRONMENT)' \ + '"$(abs_top_builddir)/kyua" \ + --config='$(KYUA_CONFIG_FILE_FOR_CHECK)' \ + "$${@}"' >>local-kyua.tmp; \ + chmod +x local-kyua.tmp; \ + mv -f local-kyua.tmp local-kyua + +if WITH_ATF +CHECK_LOCAL += dump-ulimits check-kyua +PHONY_TARGETS += check-kyua +check-kyua: $(CHECK_KYUA_DEPS) + @failed=no; \ + ./local-kyua test \ + --kyuafile='$(top_srcdir)/Kyuafile' --build-root='$(top_builddir)' \ + || failed=yes; \ + if [ "$${failed}" = yes ]; then \ + ./local-kyua report --results-file='$(abs_top_srcdir)' \ + --verbose --results-filter=broken,failed; \ + exit 1; \ + fi + +installcheck-local: dump-ulimits installcheck-kyua +PHONY_TARGETS += installcheck-kyua +installcheck-kyua: + @failed=no; \ + cd $(pkgtestsdir) && $(INSTALLCHECK_ENVIRONMENT) $(TESTS_ENVIRONMENT) \ + kyua --config='$(KYUA_CONFIG_FILE_FOR_CHECK)' test \ + || failed=yes; \ + if [ "$${failed}" = yes ]; then \ + cd $(pkgtestsdir) && $(INSTALLCHECK_ENVIRONMENT) \ + $(TESTS_ENVIRONMENT) \ + kyua --config='$(KYUA_CONFIG_FILE_FOR_CHECK)' report \ + --verbose --results-filter=broken,failed; \ + exit 1; \ + fi + +# TODO(jmmv): kyua should probably be recording this information itself as part +# of the execution context, just as we record environment variables. +PHONY_TARGETS += dump-ulimits +dump-ulimits: + @echo "Resource limits:" + @{ \ + ulimit -a | sed -e 's,$$, (soft),'; \ + ulimit -a -H | sed -e 's,$$, (hard),'; \ + } | sort | sed -e 's,^, ,' + @echo +else +DIST_HOOKS += forbid-dist +PHONY_TARGETS += forbid-dist +forbid-dist: + @echo "Sorry; cannot make dist without atf." + @false +endif +check-local: $(CHECK_LOCAL) + +if WITH_DOXYGEN +# Runs doxygen on the source tree and validates the contents of the docstrings. +# We do not do this by default, even if doxygen has been enabled, because this +# step takes a long time. Instead, we just rely on a Travis CI build to catch +# inconsistencies. +PHONY_TARGETS += check-api-docs +check-api-docs: api-docs/api-docs.tag + @$(AWK) -f $(srcdir)/admin/check-api-docs.awk api-docs/doxygen.out + +api-docs/api-docs.tag: $(builddir)/Doxyfile $(SOURCES) + @$(MKDIR_P) api-docs + @rm -f api-docs/doxygen.out api-docs/doxygen.out.tmp + $(AM_V_GEN)$(DOXYGEN) $(builddir)/Doxyfile \ + >api-docs/doxygen.out.tmp 2>&1 && \ + mv api-docs/doxygen.out.tmp api-docs/doxygen.out + +CLEAN_TARGETS += clean-api-docs +clean-api-docs: + rm -rf api-docs +endif + +# Replace Automake's builtin check-news functionality so that we can validate +# the NEWS.md file instead of NEWS. +DIST_HOOKS += check-news +PHONY_TARGETS += check-news +check-news: + @case "$$(sed 15q "$(srcdir)/NEWS.md")" in \ + *"$(VERSION)"*) : ;; \ + *) \ + echo "NEWS.md not updated; not releasing" 1>&2; \ + exit 1 \ + ;; \ + esac + +clean-local: $(CLEAN_TARGETS) +dist-hook: $(DIST_HOOKS) + +PHONY_TARGETS += clean-all +clean-all: + GIT="$(GIT)" $(SH) $(srcdir)/admin/clean-all.sh + +.PHONY: $(PHONY_TARGETS) diff --git a/NEWS.md b/NEWS.md new file mode 100644 index 000000000000..304cfe94695a --- /dev/null +++ b/NEWS.md @@ -0,0 +1,622 @@ +Major changes between releases +============================== + + +Changes in version 0.14 +----------------------- + +**NOT RELEASED YET; STILL UNDER DEVELOPMENT.** + +* Explicitly require C++11 language features when compiling Kyua. + + +Changes in version 0.13 +----------------------- + +**Released on August 26th, 2016.** + +* Fixed execution of test cases as an unprivileged user, at least under + NetBSD 7.0. Kyua-level failures were probably a regression introduced + in Kyua 0.12, but the underlying may have existed for much longer: + test cases might have previously failed for mysterious reasons when + running under an unprivileged user. + +* Issue #134: Fixed metadata test broken on 32-bit platforms. + +* Issue #139: Added per-test case start/end timestamps to all reports. + +* Issue #156: Fixed crashes due to the invalid handling of cleanup + routine data and triggered by the reuse of PIDs in long-running Kyua + instances. + +* Issue #159: Fixed TAP parser to ignore case while matching `TODO` and + `SKIP` directives, and to also recognize `Skipped`. + +* Fixed potential crash due to a race condition in the unprogramming of + timers to control test deadlines. + + +Changes in version 0.12 +----------------------- + +**Released on November 22nd, 2015.** + +This is a huge release and marks a major milestone for Kyua as it finally +implements a long-standing feature request: the ability to execute test +cases in parallel. This is a big deal because test cases are rarely +CPU-bound: running them in parallel yields much faster execution times for +large test suites, allowing faster iteration of changes during development. + +As an example: the FreeBSD test suite as of this date contains 3285 test +cases. With sequential execution, a full test suite run takes around 12 +minutes to complete, whereas on a 4-core machine with a high level of +parallelism it takes a little over 1 minute. + +Implementing parallel execution required rewriting most of Kyua's core and +partly explains explains why there has not been a new release for over a +year. The current implementation is purely subprocess-based, which works +but has some limitations and has resulted in a core that is really complex +and difficult to understand. Future versions will investigate the use of +threads instead for a simplified programming model and additional +parallelization possibilities. + +* Issue #2: Implemented support to execute test cases in parallel when + invoking `kyua test`. Parallel execution is *only* enabled when the new + `parallelism` configuration variable is set to a value greater than `1`. + The default behavior is still to run tests sequentially because some test + suites contain test cases with side-effects that might fail when run in + parallel. To resolve this, the new metadata property `is_exclusive` can + be set to `true` on a test basis to indicate that the test must be run on + its own. + +* Known regression: Running `kyua debug` on a TAP-based test program does + not currently report the output in real time. The output will only be + displayed once the test program completes. This is a shortcoming of + the new parallel execution engine and will be resolved. + +* Removed the external C-based testers code in favor of the new built-in + implementations. The new approach feels significantly faster than the + previous one. + +* Fixed the handling of relative paths in the `fs.*` functions available + in `Kyuafile`s. All paths are now resolved relative to the location of + the caller `Kyuafile`. `Kyuafile.top` has been updated with these + changes and you should update custom copies of this file with the new + version. + +* Changed temporary directory creation to always grant search + permissions on temporary directories. This is to prevent potential + problems when running Kyua as root and executing test cases that require + dropping privileges (as they may later be unable to use absolute paths + that point inside their work directory). + +* The cleanup of work directories does not longer attempt to deal with + mount points. If a test case mounts a file system and forgets to unmount + it, the mount point will be left behind. It is now the responsibility of + the test case to clean after itself. The reasons for this change are + simplicity and clarity: there are many more things that a test case can + do that have side-effects on the system and Kyua cannot protect against + them all, so it is better to just have the test undo anything it might + have done. + +* Improved `kyua report --verbose` to properly handle environment + variables with continuation lines in them, and fixed the integration + tests for this command to avoid false negatives. + +* Changed the configuration file format to accept the definition of + unknown variables without declaring them local. The syntax version + number remains at 2. This is to allow configuration files for newer Kyua + versions to work on older Kyua versions, as there is no reason to forbid + this. + +* Fixed stacktrace gathering with FreeBSD's ancient version of GDB. + GDB 6.1.1 (circa 2004) does not have the `-ex` flag so we need to + generate a temporary GDB script and feed it to GDB with `-x` instead. + +* Issue #136: Fixed the XML escaping in the JUnit output so that + non-printable characters are properly handled when they appear in the + process's stdout or stderr. + +* Issue #141: Improved reporting of errors triggered by sqlite3. In + particular, all error messages are now tagged with their corresponding + database filename and, if they are API-level errors, the name of the + sqlite3 function that caused them. + +* Issue #144: Improved documentation on the support for custom properties + in the test metadata. + +* Converted the `INSTALL`, `NEWS`, and `README` distribution documents to + Markdown for better formatting online. + + +Changes in version 0.11 +----------------------- + +**Released on October 23rd, 2014.** + +* Added support to print the details of all test cases (metadata and + their output) to `report`. This is via a new `--verbose` flag which + replaces the previous `--show-context`. + +* Added support to specify the amount of physical disk space required + by a test case. This is in the form of a new `required_disk_space` + metadata property, which can also be provided by ATF test cases as + `require.diskspace`. + +* Assimilated the contents of all the `kyua-*-tester(1)` and + `kyua-*-interface(7)` manual pages into more relevant places. In + particular, added more details on test program registration and their + metadata to `kyuafile(5)`, and added `kyua-test-isolation(7)` + describing the isolation features of the test execution. + +* Assimilated the contents of all auxiliary manual pages, including + `kyua-build-root(7)`, `kyua-results-files(7)`, `kyua-test-filters(7)` + and `kyua-test-isolation(7)`, into the relevant command-specific + manual pages. This is for easier discoverability of relevant + information when reading how specific Kyua commands work. + +* Issue #30: Plumbed through support to query configuration variables + from ATF's test case heads. This resolves the confusing situation + where test cases could only do this from their body and cleanup + routines. + +* Issue #49: Extended `report` to support test case filters as + command-line arguments. Combined with `--verbose`, this allows + inspecting the details of a test case failure after execution. + +* Issue #55: Deprecated support for specifying `test_suite` overrides on + a test program basis. This idiom should not be used but support for + it remains in place. + +* Issue #72: Added caching support to the `getcwd(3)` test in configure + so that the result can be overriden for cross-compilation purposes. + +* Issue #83: Changed manual page headings to include a `kyua` prefix in + their name. This prevents some possible confusion when displaying, + for example, the `kyua-test` manual page with a plain name of `test`. + +* Issue #84: Started passing test-suite configuration variables to plain + and TAP test programs via the environment. The name of the + environment variables set this way is prefixed by `TEST_ENV_`, so a + configuration variable of the form + `test_suites.some_name.allow_unsafe_ops=yes` in `kyua.conf` becomes + `TEST_ENV_allow_unsafe_ops=YES` in the environment. + +* Issues #97 and #116: Fixed the build on Illumos. + +* Issue #102: Set `TMPDIR` to the test case's work directory when running + the test case. If the test case happens to use the `mktemp(3)` family + of functions (due to misunderstandings on how Kyua works or due to + the reuse of legacy test code), we don't want it to easily escape the + automanaged work directory. + +* Issue #103: Started being more liberal in the parsing of TAP test + results by treating the number in `ok` and `not ok` lines as optional. + +* Issue #105: Started using tmpfs instead of md as a temporary file + system for tests in FreeBSD so that we do not leak `md(4)` devices. + +* Issue #109: Changed the privilege dropping code to start properly + dropping group privileges when `unprivileged_user` is set. Also fixes + `testers/run_test:fork_wait__unprivileged_group`. + +* Issue #110: Changed `help` to display version information and clarified + the purpose of the `about` command in its documentation. + +* Issue #111: Fixed crash when defining a test program in a `Kyuafile` + that has not yet specified the test suite name. + +* Issue #114: Improved the `kyuafile(5)` manual page by clarifying the + restrictions of the `include()` directive and by adding abundant + examples. + + +Changes in version 0.10 +----------------------- + +**Experimental version released on August 14th, 2014.** + +* Merged `kyua-cli` and `kyua-testers` into a single `kyua` package. + +* Dropped the `kyua-atf-compat` package. + +* Issue #100: Do not try to drop privileges to `unprivileged_user` when we + are already running as an unprivileged user. Doing so is not possible + and thus causes spurious test failures when the current user is not + root and the current user and `unprivileged_user` do not match. + +* Issue #79: Mention `kyua.conf(5)` in the *See also* section of `kyua(1)`. + +* Issue #75: Change the `rewrite__expected_signal__bad_arg` test in + `testers/atf_result_test` to use a different signal value. This is to + prevent triggering a core dump that made the test fail in some platforms. + + +Changes in kyua-cli version 0.9 +------------------------------- + +**Experimental version released on August 8th, 2014.** + +Major changes: + +The internal architecture of Kyua to record the results of test suite +runs has completely changed in this release. Kyua no longer stores all +the different test suite run results as different "actions" within the +single `store.db` database. Instead, Kyua now generates a separate +results file inside `~/.kyua/store/` for every test suite run. + +Due to the complexity involved in the migration process and the little +need for it, this is probably going to be the only release where the +`db-migrate` command is able to convert an old `store.db` file to the +new scheme. + +Changes in more detail: + +* Added the `report-junit` command to generate JUnit XML result files. + The output has been verified to work within Jenkins. + +* Switched to results files specific to their corresponding test suite + run. The unified `store.db` file is now gone: `kyua test` creates a + new results file for every invocation under `~/.kyua/store/` and the + `kyua report*` commands are able to locate the latest file for a + corresponding test suite automatically. + +* The `db-migrate` command takes an old `store.db` file and generates + one results file for every previously-recorded action, later deleting + the `store.db` file. + +* The `--action` flag has been removed from all commands that accepted + it. This has been superseded by the tests results files. + +* The `--store` flag that many commands took has been renamed to + `--results-file` in line with the semantical changes. + +* The `db-exec` command no longer creates an empty database when none + is found. This command is now intended to run only over existing + files. + + +Changes in kyua-testers version 0.3 +----------------------------------- + +**Experimental version released on August 8th, 2014.** + +* Made the testers set a "sanitized" value for the `HOME` environment + variable where, for example, consecutive and trailing slashes have + been cleared. Mac OS X has a tendency to append a trailing slash to + the value of `TMPDIR`, which can cause third-party tests to fail if + they compare `${HOME}` with `$(pwd)`. + +* Issues #85, #86, #90 and #92: Made the TAP parser more complete: mark + test cases reported as `TODO` or `SKIP` as passed; handle skip plans; + ignore lines that look like `ok` and `not ok` but aren't results; and + handle test programs that report a pass but exit with a non-zero code. + + +Changes in kyua-cli version 0.8 +------------------------------- + +**Experimental version released on December 7th, 2013.** + +* Added support for Lutok 0.4. + +* Issue #24: Plug the bootstrap tests back into the test suite. Fixes + in `kyua-testers` 0.2 to isolate test cases into their own sessions + should allow these to run fine. + +* Issue #74: Changed the `kyuafile(5)` parser to automatically discover + existing tester interfaces. The various `*_test_program()` functions + will now exist (or not) based on tester availability, which simplifies + the addition of new testers or the selective installation of them. + + +Changes in kyua-testers version 0.2 +----------------------------------- + +**Experimental version released on December 7th, 2013.** + +* Issue #74: Added the `kyua-tap-tester`, a new backend to interact with + test programs that comply with the Test Anything Protocol. + +* Issue #69: Cope with the lack of `AM_PROG_AR` in `configure.ac`, which + first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04 + LTS, which appears stuck in 1.11.1. + +* Issue #24: Improve test case isolation by confining the tests to their + own session instead of just to their own process group. + + +Changes in kyua-cli version 0.7 +------------------------------- + +**Experimental version released on October 18th, 2013.** + +* Made failures from testers more resilent. If a tester fails, the + corresponding test case will be marked as broken instead of causing + kyua to exit. + +* Added the `--results-filter` option to the `report-html` command and + set its default value to skip passed results from HTML reports. This + is to keep these reports more succint and to avoid generating tons of + detail files that will be, in general, useless. + +* Switched to use Lutok 0.3 to gain compatibility with Lua 5.2. + +* Issue #69: Cope with the lack of `AM_PROG_AR` in `configure.ac`, which + first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04 + LTS, which appears stuck in 1.11.1. + + +Changes in kyua-cli version 0.6 +------------------------------- + +**Experimental version released on February 22nd, 2013.** + +* Issue #36: Changed `kyua help` to not fail when the configuration file + is bogus. Help should always work. + +* Issue #37: Simplified the `syntax()` calls in configuration and + `Kyuafile` files to only specify the requested version instead of also + the format name. The format name is implied by the file being loaded, so + there is no use in the caller having to specify it. The version number + of these file formats has been bumped to 2. + +* Issue #39: Added per-test-case metadata values to the HTML reports. + +* Issue #40: Rewrote the documentation as manual pages and removed the + previous GNU Info document. + +* Issue #47: Started using the independent testers in the `kyua-testers` + package to run the test cases. Kyua does not implement the logic to + invoke test cases any more, which provides for better modularity, + extensibility and robustness. + +* Issue #57: Added support to specify arbitrary metadata properties for + test programs right from the `Kyuafile`. This is to make plain test + programs more versatile, by allowing them to specify any of the + requirements (allowed architectures, required files, etc.) supported + by Kyua. + +* Reduced automatic screen line wrapping of messages to the `help` + command and the output of tables by `db-exec`. Wrapping any other + messages (specially anything going to stderr) was very annoying + because it prevented natural copy/pasting of text. + +* Increased the granularity of the error codes returned by `kyua(1)` to + denote different error conditions. This avoids the overload of `1` to + indicate both "expected" errors from specific subcommands and + unexpected errors caused by the internals of the code. The manual now + correctly explain how the exit codes behave on a command basis. + +* Optimized the database schema to make report generation almost + instantaneous. + +* Bumped the database schema to 2. The database now records the + metadata of both test programs and test cases generically, without + knowledge of their interface. + +* Added the `db-migrate` command to provide a mechanism to upgrade a + database with an old schema to the current schema. + +* Removed the GDB build-time configuration variable. This is now part + of the `kyua-testers` package. + +* Issue #31: Rewrote the `Kyuafile` parsing code in C++, which results in + a much simpler implementation. As a side-effect, this gets rid of the + external Lua files required by `kyua`, which in turn make the tool + self-contained. + +* Added caching of various configure test results (particularly in those + tests that need to execute a test program) so that cross-compilers can + predefine the results of the tests without having to run the + executables. + + +Changes in kyua-testers version 0.1 +----------------------------------- + +**Experimental version released on February 19th, 2013.** + +This is the first public release of the `kyua-testers` package. + +The goal of this first release is to adopt all the test case execution +code of `kyua-cli` 0.5 and ship it as a collection of independent tester +binaries. The `kyua-cli` package will rely on these binaries to run the +tests, which provides better modularity and simplicity to the +architecture of Kyua. + +The code in this package is all C as opposed to the current C++ codebase +of `kyua-cli`, which means that the overall build times of Kyua are now +reduced. + + +Changes in kyua-cli version 0.5 +------------------------------- + +**Experimental version released on July 10th, 2012.** + +* Issue #15: Added automatic stacktrace gathering of crashing test cases. + This relies on GDB and is a best-effort operation. + +* Issue #32: Added the `--build-root` option to the debug, list and test + commands. This allows executing test programs from a different + directory than where the `Kyuafile` scripts live. See the *Build roots* + section in the manual for more details. + +* Issue #33: Removed the `kyuaify.sh` script. This has been renamed to + atf2kyua and moved to the `kyua-atf-compat` module, where it ships as a + first-class utility (with a manual page and tests). + +* Issue #34: Changed the HTML reports to include the stdout and stderr of + every test case. + +* Fixed the build when using a "build directory" and a clean source tree + from the repository. + + +Changes in kyua-cli version 0.4 +------------------------------- + +**Experimental version released on June 6th, 2012.** + +* Added the `report-html` command to generate HTML reports of the + execution of any recorded action. + +* Changed the `--output` flag of the `report` command to only take a + path to the target file, not its format. Different formats are better + supported by implementing different subcommands, as the options they + may receive will vary from format to format. + +* Added a `--with-atf` flag to the configure script to control whether + the ATF tests get built or not. May be useful for packaging systems + that do not have ATF in them yet. Disabling ATF also cuts down the + build time of Kyua significantly, but with the obvious drawbacks. + +* Grouped `kyua` subcommands by topic both in the output of `help` and + in the documentation. In general, the user needs to be aware of + commands that rely on a current project and those commands that rely + purely on the database to generate reports. + +* Made `help` print the descriptions of options and commands properly + tabulated. + +* Changed most informational messages to automatically wrap on screen + boundaries. + +* Rewrote the configuration file parsing module for extensibility. This + will allow future versions of Kyua to provide additional user-facing + options in the configuration file. + + No syntax changes have been made, so existing configuration files + (version 1) will continue to be parsed without problems. There is one + little exception though: all variables under the top-level + `test_suites` tree must be declared as strings. + + Similarly, the `-v` and `--variable` flags to the command line must + now carry a `test_suites.` prefix when referencing any variables under + such tree. + + +Changes in kyua-cli version 0.3 +------------------------------- + +**Experimental version released on February 24th, 2012.** + +* Made the `test` command record the results of the executed test + cases into a SQLite database. As a side effect, `test` now supports a + `--store` option to indicate where the database lives. + +* Added the `report` command to generate plain-text reports of the + test results stored in the database. The interface of this command is + certainly subject to change at this point. + +* Added the `db-exec` command to directly interact with the store + database. + +* Issue #28: Added support for the `require.memory` test case property + introduced in ATF 0.15. + +* Renamed the user-specific configuration file from `~/.kyuarc` to + `~/.kyua/kyua.conf` for consistency with other files stored in the + `~/.kyua/` subdirectory. + +* Switched to use Lutok instead of our own wrappers over the Lua C + library. Lutok is just what used to be our own utils::lua module, but + is now distributed separately. + +* Removed the `Atffile`s from the source tree. Kyua is stable enough + to generate trustworthy reports, and we do not want to give the + impression that atf-run / atf-report are still supported. + +* Enabled logging to stderr for our own test programs. This makes it + slightly easier to debug problems in our own code when we get a + failing test. + + +Changes in kyua-cli version 0.2 +------------------------------- + +**Experimental version released on August 24th, 2011.** + +The biggest change in this release is the ability for Kyua to run test +programs implemented using different frameworks. What this means is +that, now, a Kyua test suite can include not only ATF-based test +programs, but also "legacy" (aka plain) test programs that do not use +any framework. I.e. if you have tests that are simple programs that +exit with 0 on success and 1 on failure, you can plug them in into a +Kyua test suite. + +Other than this, there have been several user-visible changes. The most +important are the addition of the new `config` and `debug` subcommands +to the `kyua` binary. The former can be used to inspect the runtime +configuration of Kyua after parsing, and the latter is useful to +interact with failing tests cases in order to get more data about the +failure itself. + +Without further ado, here comes the itemized list of changes: + +* Generalized the run-time engine to support executing test programs + that implement different interfaces. Test programs that use the ATF + libraries are just a special case of this. (Issue #18.) + +* Added support to the engine to run `plain` test programs: i.e. test + programs that do not use any framework and report their pass/fail + status as an exit code. This is to simplify the integration of legacy + test programs into a test suite, and also to demonstrate that the + run-time engine is generic enough to support different test + interfaces. (Issue #18.) + +* Added the `debug` subcommand. This command allows end users to tweak + the execution of a specific test case and to poke into the behavior of + its execution. At the moment, all this command allows is to view the + stdout and stderr of the command in real time (which the `test` + command currently completely hides). + +* Added the `config` subcommand. This command allows the end user to + inspect the current configuration variables after evaluation, without + having to read through configuration files. (Issue #11.) + +* Removed the `test_suites_var` function from configuration files. This + was used to set the value of test-suite-sepecific variables, but it + was ugly-looking. It is now possible to use the more natural syntax + `test_suites.<test-suite-name>.<variable> = <value>`. (Issue #11.) + +* Added a mechanism to disable the loading of configuration files + altogether. Needed for testing purposes and for scriptability. + Available by passing the `--config=none` flag. + +* Enabled detection of unused parameters and variables in the code and + fixed all warnings. (Issue #23.) + +* Changed the behavior of "developer mode". Compiler warnings are now + enabled unconditionally regardless of whether we are in developer mode + or not; developer mode is now only used to perform strict warning + checks and to enable assertions. Additionally, developer mode is now + only automatically enabled when building from the repository, not for + formal releases. (Issue #22.) + +* Fixed many build and portability problems to Debian sid with GCC 4.6.3 + and Ubuntu 10.04.1 LTS. (Issues #20, #21, #26.) + + +Changes in kyua-cli version 0.1 +------------------------------- + +**Experimental version released on June 23rd, 2011.** + +This is the first public release of the `kyua-cli` package. + +The scope of this release is to provide functional replacement for the +`atf-run` utility included in the atf package. At this point, `kyua` +can reliably run the NetBSD 5.99.53 test suite delivering the same +results as `atf-run`. + +The reporting facilities of this release are quite limited. There is +no replacement for `atf-report` yet, and there is no easy way of +debugging failing test programs other than running them by hand. These +features will mark future milestones and therefore be part of other +releases. + +Be aware that this release has suffered very limited field testing. +The test suite for `kyua-cli` is quite comprehensive, but some bugs may +be left in any place. diff --git a/README.md b/README.md new file mode 100644 index 000000000000..eb34c0fd4550 --- /dev/null +++ b/README.md @@ -0,0 +1,84 @@ +Welcome to the Kyua project! +============================ + +Kyua is a **testing framework** for infrastructure software, originally +designed to equip BSD-based operating systems with a test suite. This +means that Kyua is lightweight and simple, and that Kyua integrates well +with various build systems and continuous integration frameworks. + +Kyua features an **expressive test suite definition language**, a **safe +runtime engine** for test suites and a **powerful report generation +engine**. + +Kyua is for **both developers *and* users**, from the developer applying a +simple fix to a library to the system administrator deploying a new release +on a production machine. + +Kyua is **able to execute test programs written with a plethora of testing +libraries and languages**. The library of choice is +[ATF](https://github.com/jmmv/atf/), for which Kyua was originally +designed, but simple, framework-less test programs and TAP-compliant test +programs can also be executed through Kyua. + +Kyua is licensed under a **[liberal BSD 3-clause license](LICENSE)**. +This is not an official Google product. + +[Read more about Kyua in the About wiki page.](../../wiki/About) + + +Download +-------- + +The latest version of Kyua is 0.13 and was released on August 26th, 2016. + +Download: [kyua-0.13](../../releases/tag/kyua-0.13). + +See the [release notes](NEWS.md) for information about the changes in this +and all previous releases. + + +Installation +------------ + +You are encouraged to install binary packages for your operating system +wherever available: + +* Fedora 20 and above: install the `kyua-cli` package with `yum install + kyua-cli`. + +* FreeBSD 10.0 and above: install the `kyua` package with `pkg install kyua`. + +* NetBSD with pkgsrc: install the `pkgsrc/devel/kyua` package. + +* OpenBSD with packages: install the `kyua` package with `pkg_add kyua`. + +* OS X (with Homebrew): install the `kyua` package with `brew install kyua`. + +Should you want to build and install Kyua from the source tree provided +here, follow the instructions in the +[INSTALL.md file](INSTALL.md). + +You should also install the ATF libraries to assist in the development of +test programs. To that end, see the +[ATF project page](https://github.com/jmmv/atf/). + + +Contributing +------------ + +Want to contribute? Great! But please first read the guidelines provided +in [CONTRIBUTING.md](CONTRIBUTING.md). + +If you are curious about who made this project possible, you can check out +the [list of copyright holders](AUTHORS) and the [list of +individuals](CONTRIBUTORS). + + +Support +------- + +Please use the [kyua-discuss mailing +list](https://groups.google.com/forum/#!forum/kyua-discuss) for any support +inquiries. + +*Homepage:* https://github.com/jmmv/kyua/ diff --git a/admin/.gitignore b/admin/.gitignore new file mode 100644 index 000000000000..1b34cbb4e096 --- /dev/null +++ b/admin/.gitignore @@ -0,0 +1,6 @@ +ar-lib +compile +depcomp +install-sh +mdate-sh +missing diff --git a/admin/Makefile.am.inc b/admin/Makefile.am.inc new file mode 100644 index 000000000000..7d02b0e611c3 --- /dev/null +++ b/admin/Makefile.am.inc @@ -0,0 +1,41 @@ +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +PHONY_TARGETS += check-style +check-style: + @$(srcdir)/admin/check-style.sh \ + -b "$(abs_top_builddir)" \ + -s "$(abs_top_srcdir)" \ + -t "$(PACKAGE_TARNAME)" + +EXTRA_DIST += admin/check-style-common.awk \ + admin/check-style-cpp.awk \ + admin/check-style-make.awk \ + admin/check-style-man.awk \ + admin/check-style-shell.awk \ + admin/check-style.sh diff --git a/admin/build-bintray-dist.sh b/admin/build-bintray-dist.sh new file mode 100755 index 000000000000..99cd439892c5 --- /dev/null +++ b/admin/build-bintray-dist.sh @@ -0,0 +1,131 @@ +#! /bin/sh +# Copyright 2017 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# \file admin/build-bintray-dist.sh +# Builds a full Kyua installation under /usr/local for Ubuntu. +# +# This script is used to create the bintray distribution packages in lieu +# of real Debian packages for Kyua. The result of this script is a +# tarball that provides the contents of /usr/local for Kyua. + +set -e -x + +err() { + echo "${@}" 1>&2 + exit 1 +} + +install_deps() { + sudo apt-get update -qq + + local pkgsuffix= + local packages= + packages="${packages} autoconf" + packages="${packages} automake" + packages="${packages} clang" + packages="${packages} g++" + packages="${packages} gdb" + packages="${packages} git" + packages="${packages} libtool" + packages="${packages} make" + if [ "${ARCH?}" = i386 ]; then + pkgsuffix=:i386 + packages="${packages} gcc-multilib" + packages="${packages} g++-multilib" + fi + packages="${packages} liblua5.2-0${pkgsuffix}" + packages="${packages} liblua5.2-dev${pkgsuffix}" + packages="${packages} libsqlite3-0${pkgsuffix}" + packages="${packages} libsqlite3-dev${pkgsuffix}" + packages="${packages} pkg-config${pkgsuffix}" + packages="${packages} sqlite3" + sudo apt-get install -y ${packages} +} + +install_from_github() { + local name="${1}"; shift + local release="${1}"; shift + + local distname="${name}-${release}" + + local baseurl="https://github.com/jmmv/${name}" + wget --no-check-certificate \ + "${baseurl}/releases/download/${distname}/${distname}.tar.gz" + tar -xzvf "${distname}.tar.gz" + + local archflags= + [ "${ARCH?}" != i386 ] || archflags=-m32 + + cd "${distname}" + ./configure \ + --disable-developer \ + --without-atf \ + --without-doxygen \ + CC="${CC?}" \ + CFLAGS="${archflags}" \ + CPPFLAGS="-I/usr/local/include" \ + CXX="${CXX?}" \ + CXXFLAGS="${archflags}" \ + LDFLAGS="-L/usr/local/lib -Wl,-R/usr/local/lib" \ + PKG_CONFIG_PATH="/usr/local/lib/pkgconfig" + make + sudo make install + cd - + + rm -rf "${distname}" "${distname}.tar.gz" +} + +main() { + [ "${ARCH+set}" = set ] || err "ARCH must be set in the environment" + [ "${CC+set}" = set ] || err "CC must be set in the environment" + [ "${CXX+set}" = set ] || err "CXX must be set in the environment" + + [ ! -f /root/local.tgz ] || err "/root/local.tgz already exists" + tar -czf /root/local.tgz /usr/local + restore() { + rm -rf /usr/local + tar -xz -C / -f /root/local.tgz + rm /root/local.tgz + } + trap restore EXIT + rm -rf /usr/local + mkdir /usr/local + + install_deps + install_from_github atf 0.21 + install_from_github lutok 0.4 + install_from_github kyua 0.13 + + local version="$(lsb_release -rs | cut -d . -f 1-2 | tr . -)" + local name="$(date +%Y%m%d)-usr-local-kyua" + name="${name}-ubuntu-${version}-${ARCH?}-${CC?}.tar.gz" + tar -czf "${name}" /usr/local +} + +main "${@}" diff --git a/admin/check-api-docs.awk b/admin/check-api-docs.awk new file mode 100644 index 000000000000..358e3d54c177 --- /dev/null +++ b/admin/check-api-docs.awk @@ -0,0 +1,72 @@ +#! /bin/sh +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +BEGIN { + failed = 0 +} + +# Skip empty lines. +/^$/ {next} + +# Skip lines that do not directly reference a file. +/^[^\/]/ {next} + +# Ignore known problems. As far as I can tell, all the cases listed here are +# well-documented in the code but Doxygen fails, for some reason or another, to +# properly locate the docstrings. +/engine\/kyuafile\.cpp.*no matching class member/ {next} +/engine\/scheduler\.hpp.*Member setup\(void\).*friend/ {next} +/engine\/scheduler\.hpp.*Member wait_any\(void\)/ {next} +/utils\/optional\.ipp.*no matching file member/ {next} +/utils\/optional\.hpp.*Member make_optional\(const T &\)/ {next} +/utils\/config\/nodes\.hpp.*Member set_lua\(lutok::state &, const int\)/ {next} +/utils\/config\/nodes\.hpp.*Member push_lua\(lutok::state &\)/ {next} +/utils\/config\/nodes\.hpp.*Member set_string\(const std::string &\)/ {next} +/utils\/config\/nodes\.hpp.*Member to_string\(void\)/ {next} +/utils\/config\/nodes\.hpp.*Member is_set\(void\)/ {next} +/utils\/process\/executor\.hpp.*Member spawn\(Hook.*\)/ {next} +/utils\/process\/executor\.hpp.*Member spawn_followup\(Hook.*\)/ {next} +/utils\/process\/executor\.hpp.*Member setup\(void\).*friend/ {next} +/utils\/signals\/timer\.hpp.*Member detail::invoke_do_fired.*friend/ {next} +/utils\/stacktrace_test\.cpp.*no matching class member/ {next} + +# Dump any other problems and account for the failure. +{ + failed = 1 + print +} + +END { + if (failed) { + print "ERROR: Unexpected docstring problems encountered" + exit 1 + } else { + exit 0 + } +} diff --git a/admin/check-style-common.awk b/admin/check-style-common.awk new file mode 100644 index 000000000000..39516d00d4e5 --- /dev/null +++ b/admin/check-style-common.awk @@ -0,0 +1,79 @@ +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +function warn(msg) { + print FILENAME "[" FNR "]: " msg > "/dev/stderr" + error = 1 +} + +BEGIN { + skip = 0 + error = 0 +} + +/CHECK_STYLE_DISABLE/ { + skip = 1 + next +} + +/CHECK_STYLE_ENABLE/ { + skip = 0 + next +} + +/CHECK_STYLE_(ENABLE|DISABLE)/ { + next +} + +{ + if (skip) + next + + if (length > 80 && NF > 1) + warn("Line too long to fit on screen") +} + +/^ *\t+/ { + if (! match(FILENAME, "Makefile")) + warn("Tab character used for indentation"); +} + +/[ \t]+$/ { + warn("Trailing spaces or tabs"); +} + +/^#![^ ]/ { + warn("Missing space after #!"); +} + +END { + if (skip) + warn("Missing CHECK_STYLE_ENABLE"); + if (error) + exit 1 +} diff --git a/admin/check-style-cpp.awk b/admin/check-style-cpp.awk new file mode 100644 index 000000000000..126789ca9262 --- /dev/null +++ b/admin/check-style-cpp.awk @@ -0,0 +1,87 @@ +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +function warn(msg) { + print FILENAME "[" FNR "]: " msg > "/dev/stderr" + error = 1 +} + +BEGIN { + skip = 0 + error = 0 +} + +/CHECK_STYLE_DISABLE/ { + skip = 1 + next +} + +/CHECK_STYLE_ENABLE/ { + skip = 0 + next +} + +/CHECK_STYLE_(ENABLE|DISABLE)/ { + next +} + +{ + if (skip) + next +} + +/#ifdef/ { + warn("Undesired usage of #ifdef; use #if defined()") +} + +/#ifndef/ { + warn("Undesired usage of #ifndef; use #if !defined()") +} + +/assert[ \t]*\(/ { + warn("Use the macros in sanity.hpp instead of assert"); +} + +/#.*include.*assert/ { + warn("Do not include assert.h nor cassert"); +} + +/std::endl/ { + warn("Use \\n instead of std::endl"); +} + +/\/\*/ && ! /\*\// { + warn("Do not use multi-line C-style comments"); +} + +END { + if (skip) + warn("Missing CHECK_STYLE_ENABLE"); + if (error) + exit 1 +} diff --git a/admin/check-style-make.awk b/admin/check-style-make.awk new file mode 100644 index 000000000000..9a6c532e7131 --- /dev/null +++ b/admin/check-style-make.awk @@ -0,0 +1,71 @@ +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +function warn(msg) { + print FILENAME "[" FNR "]: " msg > "/dev/stderr" + error = 1 +} + +BEGIN { + skip = 0 + error = 0 +} + +/CHECK_STYLE_DISABLE/ { + skip = 1 + next +} + +/CHECK_STYLE_ENABLE/ { + skip = 0 + next +} + +/CHECK_STYLE_(ENABLE|DISABLE)/ { + next +} + +{ + if (skip) + next +} + +/^\t *\t/ { + warn("Continuation lines must use a single tab"); +} + +/mkdir.*-p/ { + warn("Use $(MKDIR_P) instead of mkdir -p"); +} + +END { + if (skip) + warn("Missing CHECK_STYLE_ENABLE"); + if (error) + exit 1 +} diff --git a/admin/check-style-man.awk b/admin/check-style-man.awk new file mode 100644 index 000000000000..5c4a2c261b96 --- /dev/null +++ b/admin/check-style-man.awk @@ -0,0 +1,71 @@ +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +function warn(msg) { + print FILENAME "[" FNR "]: " msg > "/dev/stderr" + error = 1 +} + +BEGIN { + skip = 0 + error = 0 +} + +/CHECK_STYLE_DISABLE|^\.Bd/ { + skip = 1 + next +} + +/CHECK_STYLE_ENABLE|^\.Ed/ { + skip = 0 + next +} + +/CHECK_STYLE_(ENABLE|DISABLE)/ { + next +} + +/^\.\\"/ { + next +} + +{ + if (skip) + next +} + +/\.\.|e\.g\.|i\.e\./ { + next +} + +END { + if (skip) + warn("Missing CHECK_STYLE_ENABLE"); + if (error) + exit 1 +} diff --git a/admin/check-style-shell.awk b/admin/check-style-shell.awk new file mode 100644 index 000000000000..43d3472cb45b --- /dev/null +++ b/admin/check-style-shell.awk @@ -0,0 +1,95 @@ +# Copyright 2015 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +function warn(msg) { + print FILENAME "[" FNR "]: " msg > "/dev/stderr" + error = 1 +} + +BEGIN { + skip = 0 + error = 0 +} + +/CHECK_STYLE_DISABLE/ { + skip = 1 + next +} + +/CHECK_STYLE_ENABLE/ { + skip = 0 + next +} + +/CHECK_STYLE_(ENABLE|DISABLE)/ { + next +} + +{ + if (skip) + next +} + +/^[ \t]*#/ { + next +} + +/[$ \t]+_[a-zA-Z0-9]+=/ { + warn("Variable should not start with an underline") +} + +/[^\\]\$[^0-9!'"$?@#*{}(|\/,]+/ { + warn("Missing braces around variable name") +} + +/=(""|'')/ { + warn("Assignment to the empty string does not need quotes"); +} + +/basename[ \t]+/ { + warn("Use parameter expansion instead of basename"); +} + +/if[ \t]+(test|![ \t]+test)/ { + warn("Use [ instead of test"); +} + +/[ \t]+(test|\[).*==/ { + warn("test(1)'s == operator is not portable"); +} + +/if.*;[ \t]*fi$/ { + warn("Avoid using a single-line if conditional"); +} + +END { + if (skip) + warn("Missing CHECK_STYLE_ENABLE"); + if (error) + exit 1 +} diff --git a/admin/check-style.sh b/admin/check-style.sh new file mode 100755 index 000000000000..696f9247a74a --- /dev/null +++ b/admin/check-style.sh @@ -0,0 +1,170 @@ +#! /bin/sh +# Copyright 2011 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# \file admin/check-style.sh +# +# Sanity checks the coding style of all source files in the project tree. + +ProgName="${0##*/}" + + +# Prints an error message and exits. +# +# \param ... Parts of the error message; concatenated using a space as the +# separator. +err() { + echo "${ProgName}:" "${@}" 1>&2 + exit 1 +} + + +# Locates all source files within the project directory. +# +# We require the project to have been configured in a directory that is separate +# from the source tree. This is to allow us to easily filter out build +# artifacts from our search. +# +# \param srcdir Absolute path to the source directory. +# \param builddir Absolute path to the build directory. +# \param tarname Basename of the project's tar file, to skip possible distfile +# directories. +find_sources() { + local srcdir="${1}"; shift + local builddir="${1}"; shift + local tarname="${1}"; shift + + ( + cd "${srcdir}" + find . -type f -a \ + \! -path "*/.git/*" \ + \! -path "*/.deps/*" \ + \! -path "*/autom4te.cache/*" \ + \! -path "*/${tarname}-[0-9]*/*" \ + \! -path "*/${builddir##*/}/*" \ + \! -name "Makefile.in" \ + \! -name "aclocal.m4" \ + \! -name "config.h.in" \ + \! -name "configure" \ + \! -name "testsuite" + ) +} + + +# Prints the style rules applicable to a given file. +# +# \param file Path to the source file. +guess_rules() { + local file="${1}"; shift + + case "${file}" in + */ax_cxx_compile_stdcxx.m4) ;; + */ltmain.sh) ;; + *Makefile*) echo common make ;; + *.[0-9]) echo common man ;; + *.cpp|*.hpp) echo common cpp ;; + *.sh) echo common shell ;; + *) echo common ;; + esac +} + + +# Validates a given file against the rules that apply to it. +# +# \param srcdir Absolute path to the source directory. +# \param file Name of the file to validate relative to srcdir. +# +# \return 0 if the file is valid; 1 otherwise, in which case the style +# violations are printed to the output. +check_file() { + local srcdir="${1}"; shift + local file="${1}"; shift + + local err=0 + for rule in $(guess_rules "${file}"); do + awk -f "${srcdir}/admin/check-style-${rule}.awk" \ + "${srcdir}/${file}" || err=1 + done + + return ${err} +} + + +# Entry point. +main() { + local builddir=. + local srcdir=. + local tarname=UNKNOWN + + local arg + while getopts :b:s:t: arg; do + case "${arg}" in + b) + builddir="${OPTARG}" + ;; + + s) + srcdir="${OPTARG}" + ;; + + t) + tarname="${OPTARG}" + ;; + + \?) + err "Unknown option -${OPTARG}" + ;; + esac + done + shift $(expr ${OPTIND} - 1) + + srcdir="$(cd "${srcdir}" && pwd -P)" + builddir="$(cd "${builddir}" && pwd -P)" + [ "${srcdir}" != "${builddir}" ] || \ + err "srcdir and builddir cannot match; reconfigure the package" \ + "in a separate directory" + + local sources + if [ ${#} -gt 0 ]; then + sources="${@}" + else + sources="$(find_sources "${srcdir}" "${builddir}" "${tarname}")" + fi + + local ok=0 + for file in ${sources}; do + local file="$(echo ${file} | sed -e "s,\\./,,")" + + check_file "${srcdir}" "${file}" || ok=1 + done + + return "${ok}" +} + + +main "${@}" diff --git a/admin/clean-all.sh b/admin/clean-all.sh new file mode 100755 index 000000000000..bc02f1e811f4 --- /dev/null +++ b/admin/clean-all.sh @@ -0,0 +1,90 @@ +#! /bin/sh +# Copyright 2010 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Prog_Name=${0##*/} + +if [ ! -f ./main.cpp ]; then + echo "${Prog_Name}: must be run from the source top directory" 1>&2 + exit 1 +fi + +if [ ! -f configure ]; then + echo "${Prog_Name}: configure not found; nothing to clean?" 1>&2 + exit 1 +fi + +[ -f Makefile ] || ./configure +make distclean + +# Top-level directory. +rm -f Makefile.in +rm -f aclocal.m4 +rm -rf autom4te.cache +rm -f config.h.in +rm -f configure +rm -f mkinstalldirs +rm -f kyua-*.tar.gz + +# admin directory. +rm -f admin/ar-lib +rm -f admin/compile +rm -f admin/config.guess +rm -f admin/config.sub +rm -f admin/depcomp +rm -f admin/install-sh +rm -f admin/ltmain.sh +rm -f admin/mdate-sh +rm -f admin/missing + +# bootstrap directory. +rm -f bootstrap/package.m4 +rm -f bootstrap/testsuite + +# doc directory. +rm -f doc/*.info +rm -f doc/stamp-vti +rm -f doc/version.texi + +# m4 directory. +rm -f m4/libtool.m4 +rm -f m4/lt*.m4 + +# Files and directories spread all around the tree. +find . -name '#*' | xargs rm -rf +find . -name '*~' | xargs rm -rf +find . -name .deps | xargs rm -rf +find . -name .gdb_history | xargs rm -rf +find . -name .libs | xargs rm -rf +find . -name .tmp | xargs rm -rf + +# Show remaining files. +if [ -n "${GIT}" ]; then + echo ">>> untracked and ignored files" + "${GIT}" status --porcelain --ignored | grep -E '^(\?\?|!!)' || true +fi diff --git a/admin/travis-build.sh b/admin/travis-build.sh new file mode 100755 index 000000000000..e69f271c13f1 --- /dev/null +++ b/admin/travis-build.sh @@ -0,0 +1,98 @@ +#! /bin/sh +# Copyright 2014 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +set -e -x + +run_autoreconf() { + if [ -d /usr/local/share/aclocal ]; then + autoreconf -isv -I/usr/local/share/aclocal + else + autoreconf -isv + fi +} + +do_apidocs() { + run_autoreconf || return 1 + ./configure --with-doxygen || return 1 + make check-api-docs +} + +do_distcheck() { + run_autoreconf || return 1 + ./configure || return 1 + + sudo sysctl -w "kernel.core_pattern=core.%p" + + local archflags= + [ "${ARCH?}" != i386 ] || archflags=-m32 + + cat >kyua.conf <<EOF +syntax(2) + +-- We do not know how many CPUs the test machine has. However, parallelizing +-- the execution of our tests to _any_ degree speeds up the time it takes to +-- complete a test run because many of our tests are blocking. +parallelism = 4 +EOF + [ "${UNPRIVILEGED_USER:-no}" = no ] || \ + echo "unprivileged_user = 'travis'" >>kyua.conf + + local f= + f="${f} CFLAGS='${archflags}'" + f="${f} CPPFLAGS='-I/usr/local/include'" + f="${f} CXXFLAGS='${archflags}'" + f="${f} LDFLAGS='-L/usr/local/lib -Wl,-R/usr/local/lib'" + f="${f} PKG_CONFIG_PATH='/usr/local/lib/pkgconfig'" + f="${f} KYUA_CONFIG_FILE_FOR_CHECK=$(pwd)/kyua.conf" + if [ "${AS_ROOT:-no}" = yes ]; then + sudo -H PATH="${PATH}" make distcheck DISTCHECK_CONFIGURE_FLAGS="${f}" + else + make distcheck DISTCHECK_CONFIGURE_FLAGS="${f}" + fi +} + +do_style() { + run_autoreconf || return 1 + mkdir build + cd build + ../configure || return 1 + make check-style +} + +main() { + if [ -z "${DO}" ]; then + echo "DO must be defined" 1>&2 + exit 1 + fi + for step in ${DO}; do + "do_${DO}" || exit 1 + done +} + +main "${@}" diff --git a/admin/travis-install-deps.sh b/admin/travis-install-deps.sh new file mode 100755 index 000000000000..9341c43895b1 --- /dev/null +++ b/admin/travis-install-deps.sh @@ -0,0 +1,83 @@ +#! /bin/sh +# Copyright 2014 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +set -e -x + +install_deps() { + local pkgsuffix= + local packages= + if [ "${ARCH?}" = i386 ]; then + pkgsuffix=:i386 + packages="${packages} gcc-multilib" + packages="${packages} g++-multilib" + sudo dpkg --add-architecture i386 + fi + packages="${packages} gdb" + packages="${packages} liblua5.2-0${pkgsuffix}" + packages="${packages} liblua5.2-dev${pkgsuffix}" + packages="${packages} libsqlite3-0${pkgsuffix}" + packages="${packages} libsqlite3-dev${pkgsuffix}" + packages="${packages} pkg-config${pkgsuffix}" + packages="${packages} sqlite3" + sudo apt-get update -qq + sudo apt-get install -y ${packages} +} + +install_kyua() { + local name="20190321-usr-local-kyua-ubuntu-16-04-${ARCH?}-${CC?}.tar.gz" + wget -O "${name}" "http://dl.bintray.com/ngie-eign/kyua/${name}" || return 1 + sudo tar -xzvp -C / -f "${name}" + rm -f "${name}" +} + +do_apidocs() { + sudo apt-get install -y doxygen +} + +do_distcheck() { + : +} + +do_style() { + : +} + +main() { + if [ -z "${DO}" ]; then + echo "DO must be defined" 1>&2 + exit 1 + fi + install_deps + install_kyua + for step in ${DO}; do + "do_${DO}" || exit 1 + done +} + +main "${@}" diff --git a/bootstrap/.gitignore b/bootstrap/.gitignore new file mode 100644 index 000000000000..effaef8e6b4a --- /dev/null +++ b/bootstrap/.gitignore @@ -0,0 +1,4 @@ +atconfig +package.m4 +testsuite +testsuite.log diff --git a/bootstrap/Kyuafile b/bootstrap/Kyuafile new file mode 100644 index 000000000000..0f161b2d66eb --- /dev/null +++ b/bootstrap/Kyuafile @@ -0,0 +1,5 @@ +syntax(2) + +test_suite("kyua") + +plain_test_program{name="testsuite"} diff --git a/bootstrap/Makefile.am.inc b/bootstrap/Makefile.am.inc new file mode 100644 index 000000000000..0dbf26002ce9 --- /dev/null +++ b/bootstrap/Makefile.am.inc @@ -0,0 +1,90 @@ +# Copyright 2010 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +if WITH_ATF +tests_bootstrapdir = $(pkgtestsdir)/bootstrap + +tests_bootstrap_DATA = bootstrap/Kyuafile +EXTRA_DIST += $(tests_bootstrap_DATA) + +DISTCLEANFILES = bootstrap/atconfig \ + bootstrap/testsuite.lineno \ + bootstrap/testsuite.log + +distclean-local: distclean-testsuite +distclean-testsuite: + -rm -rf bootstrap/testsuite.dir + +EXTRA_DIST += bootstrap/Kyuafile \ + bootstrap/testsuite \ + bootstrap/package.m4 \ + bootstrap/testsuite.at + +tests_bootstrap_PROGRAMS = bootstrap/atf_helpers +bootstrap_atf_helpers_SOURCES = bootstrap/atf_helpers.cpp +bootstrap_atf_helpers_CXXFLAGS = $(ATF_CXX_CFLAGS) +bootstrap_atf_helpers_LDADD = $(ATF_CXX_LIBS) + +tests_bootstrap_PROGRAMS += bootstrap/plain_helpers +bootstrap_plain_helpers_SOURCES = bootstrap/plain_helpers.cpp +bootstrap_plain_helpers_CXXFLAGS = $(UTILS_CFLAGS) + +tests_bootstrap_SCRIPTS = bootstrap/testsuite +@target_srcdir@bootstrap/package.m4: $(top_srcdir)/configure.ac + $(AM_V_GEN){ \ + echo '# Signature of the current package.'; \ + echo 'm4_define(AT_PACKAGE_NAME, @PACKAGE_NAME@)'; \ + echo 'm4_define(AT_PACKAGE_TARNAME, @PACKAGE_TARNAME@)'; \ + echo 'm4_define(AT_PACKAGE_VERSION, @PACKAGE_VERSION@)'; \ + echo 'm4_define(AT_PACKAGE_STRING, @PACKAGE_STRING@)'; \ + echo 'm4_define(AT_PACKAGE_BUGREPORT, @PACKAGE_BUGREPORT@)'; \ + } >$(srcdir)/bootstrap/package.m4 + +@target_srcdir@bootstrap/testsuite: $(srcdir)/bootstrap/testsuite.at \ + @target_srcdir@bootstrap/package.m4 + $(AM_V_GEN)autom4te --language=Autotest -I $(srcdir) \ + -I $(srcdir)/bootstrap \ + $(srcdir)/bootstrap/testsuite.at -o $@.tmp; \ + mv $@.tmp $@ + +CHECK_LOCAL += check-bootstrap +PHONY_TARGETS += check-bootstrap +check-bootstrap: @target_srcdir@bootstrap/testsuite $(check_PROGRAMS) \ + $(CHECK_BOOTSTRAP_DEPS) + cd bootstrap && $(CHECK_ENVIRONMENT) $(TESTS_ENVIRONMENT) \ + ./testsuite + +if !TARGET_SRCDIR_EMPTY +CHECK_BOOTSTRAP_DEPS += copy-bootstrap-testsuite +CHECK_KYUA_DEPS += copy-bootstrap-testsuite +PHONY_TARGETS += copy-bootstrap-testsuite +copy-bootstrap-testsuite: + cp -f @target_srcdir@bootstrap/testsuite bootstrap/testsuite +CLEANFILES += bootstrap/testsuite +endif +endif diff --git a/bootstrap/atf_helpers.cpp b/bootstrap/atf_helpers.cpp new file mode 100644 index 000000000000..6a31b4ced994 --- /dev/null +++ b/bootstrap/atf_helpers.cpp @@ -0,0 +1,71 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include <cstdlib> +#include <string> + +#include <atf-c++.hpp> + + +ATF_TEST_CASE_WITHOUT_HEAD(fails); +ATF_TEST_CASE_BODY(fails) +{ + fail("Failed on purpose"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(passes); +ATF_TEST_CASE_BODY(passes) +{ +} + + +ATF_TEST_CASE_WITHOUT_HEAD(skips); +ATF_TEST_CASE_BODY(skips) +{ + skip("Skipped on purpose"); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + std::string enabled; + + const char* tests = std::getenv("TESTS"); + if (tests == NULL) + enabled = "fails passes skips"; + else + enabled = tests; + + if (enabled.find("fails") != std::string::npos) + ATF_ADD_TEST_CASE(tcs, fails); + if (enabled.find("passes") != std::string::npos) + ATF_ADD_TEST_CASE(tcs, passes); + if (enabled.find("skips") != std::string::npos) + ATF_ADD_TEST_CASE(tcs, skips); +} diff --git a/bootstrap/plain_helpers.cpp b/bootstrap/plain_helpers.cpp new file mode 100644 index 000000000000..7de629a99d4d --- /dev/null +++ b/bootstrap/plain_helpers.cpp @@ -0,0 +1,141 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include <cstdlib> +#include <cstring> +#include <iostream> + +#include "utils/defs.hpp" +#include "utils/test_utils.ipp" + + +namespace { + + +/// Prints a fake but valid test case list and then aborts. +/// +/// \param argv The original arguments of the program. +/// +/// \return Nothing because this dies before returning. +static int +helper_abort_test_cases_list(int /* argc */, char** argv) +{ + for (const char* const* arg = argv; *arg != NULL; arg++) { + if (std::strcmp(*arg, "-l") == 0) { + std::cout << "Content-Type: application/X-atf-tp; " + "version=\"1\"\n\n"; + std::cout << "ident: foo\n"; + } + } + utils::abort_without_coredump(); +} + + +/// Just returns without printing anything as the test case list. +/// +/// \return Always 0, as required for test programs. +static int +helper_empty_test_cases_list(int /* argc */, char** /* argv */) +{ + return EXIT_SUCCESS; +} + + +/// Prints a correctly-formatted test case list but empty. +/// +/// \param argv The original arguments of the program. +/// +/// \return Always 0, as required for test programs. +static int +helper_zero_test_cases(int /* argc */, char** argv) +{ + for (const char* const* arg = argv; *arg != NULL; arg++) { + if (std::strcmp(*arg, "-l") == 0) + std::cout << "Content-Type: application/X-atf-tp; " + "version=\"1\"\n\n"; + } + return EXIT_SUCCESS; +} + + +/// Mapping of the name of a helper to its implementation. +struct helper { + /// The name of the helper, as will be provided by the user on the CLI. + const char* name; + + /// A pointer to the function implementing the helper. + int (*hook)(int, char**); +}; + + +/// NULL-terminated table mapping helper names to their implementations. +static const helper helpers[] = { + { "abort_test_cases_list", helper_abort_test_cases_list, }, + { "empty_test_cases_list", helper_empty_test_cases_list, }, + { "zero_test_cases", helper_zero_test_cases, }, + { NULL, NULL, }, +}; + + +} // anonymous namespace + + +/// Entry point to the ATF-less helpers. +/// +/// The caller must select a helper to execute by defining the HELPER +/// environment variable to the name of the desired helper. Think of this main +/// method as a subprogram dispatcher, to avoid having many individual helper +/// binaries. +/// +/// \todo Maybe we should really have individual helper binaries. It would +/// avoid a significant amount of complexity here and in the tests, at the +/// expense of some extra files and extra build logic. +/// +/// \param argc The user argument count; delegated to the helper. +/// \param argv The user arguments; delegated to the helper. +/// +/// \return The exit code of the helper, which depends on the requested helper. +int +main(int argc, char** argv) +{ + const char* command = std::getenv("HELPER"); + if (command == NULL) { + std::cerr << "Usage error: HELPER must be set to a helper name\n"; + std::exit(EXIT_FAILURE); + } + + const struct helper* iter = helpers; + for (; iter->name != NULL && std::strcmp(iter->name, command) != 0; iter++) + ; + if (iter->name == NULL) { + std::cerr << "Usage error: unknown command " << command << "\n"; + std::exit(EXIT_FAILURE); + } + + return iter->hook(argc, argv); +} diff --git a/bootstrap/testsuite.at b/bootstrap/testsuite.at new file mode 100644 index 000000000000..10200a67a5ca --- /dev/null +++ b/bootstrap/testsuite.at @@ -0,0 +1,200 @@ +dnl Copyright 2010 The Kyua Authors. +dnl All rights reserved. +dnl +dnl Redistribution and use in source and binary forms, with or without +dnl modification, are permitted provided that the following conditions are +dnl met: +dnl +dnl * Redistributions of source code must retain the above copyright +dnl notice, this list of conditions and the following disclaimer. +dnl * Redistributions in binary form must reproduce the above copyright +dnl notice, this list of conditions and the following disclaimer in the +dnl documentation and/or other materials provided with the distribution. +dnl * Neither the name of Google Inc. nor the names of its contributors +dnl may be used to endorse or promote products derived from this software +dnl without specific prior written permission. +dnl +dnl THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +dnl "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +dnl LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +dnl A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +dnl OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +dnl SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +dnl LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +dnl DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +dnl THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +dnl (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +dnl OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +AT_INIT([bootstrapping tests]) + + +m4_define([GUESS_TOPDIR], { + old=$(pwd) + cd "$(dirname ${as_myself})" + # We need to locate a build product, not a source file, because the + # test suite may be run outside of the source tree (think distcheck). + while test $(pwd) != '/' -a ! -e bootstrap/plain_helpers; do + cd .. + done + topdir=$(pwd) + cd ${old} + echo ${topdir} +}) + + +m4_define([CREATE_ATF_HELPERS], [ + AT_DATA([Kyuafile], [ +syntax(2) +test_suite("bootstrap") +atf_test_program{name="atf_helpers"} +]) + ln -s $(GUESS_TOPDIR)/bootstrap/atf_helpers atf_helpers +]) +m4_define([RUN_ATF_HELPERS], + [HOME=$(pwd) TESTS="$1" kyua --config=none \ + test --results-file=bootstrap.db $2]) + + +m4_define([CREATE_PLAIN_HELPERS], [ + AT_DATA([Kyuafile], [ +syntax(2) +test_suite("bootstrap") +atf_test_program{name="plain_helpers"} +]) + ln -s $(GUESS_TOPDIR)/bootstrap/plain_helpers plain_helpers +]) +m4_define([RUN_PLAIN_HELPER], + [HOME=$(pwd) HELPER="$1" kyua --config=none \ + test --results-file=bootstrap.db]) + + +AT_SETUP([test program crashes in test list]) +AT_TESTED([kyua]) + +CREATE_PLAIN_HELPERS +AT_CHECK([RUN_PLAIN_HELPER([abort_test_cases_list])], [1], [stdout], []) +re='plain_helpers:__test_cases_list__.*broken.*Test program received signal' +AT_CHECK([grep "${re}" stdout], [0], [ignore], []) + +AT_CLEANUP + + +AT_SETUP([test program prints an empty test list]) +AT_TESTED([kyua]) + +CREATE_PLAIN_HELPERS +AT_CHECK([RUN_PLAIN_HELPER([empty_test_cases_list])], [1], [stdout], []) +re="plain_helpers:__test_cases_list__.*broken.*Invalid header.*got ''" +AT_CHECK([grep "${re}" stdout], [0], [ignore], []) + +AT_CLEANUP + + +AT_SETUP([test program with zero test cases]) +AT_TESTED([kyua]) + +CREATE_PLAIN_HELPERS +AT_CHECK([RUN_PLAIN_HELPER([zero_test_cases])], [1], [stdout], []) +re='plain_helpers:__test_cases_list__.*broken.*No test cases' +AT_CHECK([grep "${re}" stdout], [0], [ignore], []) + +AT_CLEANUP + + +AT_SETUP([run test case that passes]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([RUN_ATF_HELPERS([passes])], [0], [stdout], []) +AT_CHECK([grep "atf_helpers:fails" stdout], [1], [], []) +AT_CHECK([grep "atf_helpers:passes.*passed" stdout], [0], [ignore], []) +AT_CHECK([grep "atf_helpers:skips" stdout], [1], [], []) + +AT_CLEANUP + + +AT_SETUP([run test case that fails]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([RUN_ATF_HELPERS([fails])], [1], [stdout], []) +AT_CHECK([grep "atf_helpers:fails.*failed.*Failed on purpose" stdout], + [0], [ignore], []) +AT_CHECK([grep "atf_helpers:passes" stdout], [1], [], []) +AT_CHECK([grep "atf_helpers:skips" stdout], [1], [], []) + +AT_CLEANUP + + +AT_SETUP([run test case that skips]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([RUN_ATF_HELPERS([skips])], [0], [stdout], []) +AT_CHECK([grep "atf_helpers:fails" stdout], [1], [], []) +AT_CHECK([grep "atf_helpers:passes" stdout], [1], [], []) +AT_CHECK([grep "atf_helpers:skips.*skipped.*Skipped on purpose" stdout], + [0], [ignore], []) + +AT_CLEANUP + + +AT_SETUP([run two test cases, success]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([RUN_ATF_HELPERS([passes skips])], [0], [stdout], []) +AT_CHECK([grep "atf_helpers:fails" stdout], [1], [], []) +AT_CHECK([grep "atf_helpers:passes.*passed" stdout], [0], [ignore], []) +AT_CHECK([grep "atf_helpers:skips.*skipped.*Skipped on purpose" stdout], + [0], [ignore], []) + +AT_CLEANUP + + +AT_SETUP([run two test cases, failure]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([RUN_ATF_HELPERS([fails passes])], [1], [stdout], []) +AT_CHECK([grep "atf_helpers:fails.*failure.*Failed on purpose" stdout], + [1], [], []) +AT_CHECK([grep "atf_helpers:passes.*passed" stdout], [0], [ignore], []) +AT_CHECK([grep "atf_helpers:skips" stdout], [1], [], []) + +AT_CLEANUP + + +AT_SETUP([run mixed test cases]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([RUN_ATF_HELPERS([fails passes skips])], [1], [stdout], []) +AT_CHECK([grep "atf_helpers:fails.*failure.*Failed on purpose" stdout], + [1], [], []) +AT_CHECK([grep "atf_helpers:passes.*passed" stdout], [0], [ignore], []) +AT_CHECK([grep "atf_helpers:skips.*skipped.*Skipped on purpose" stdout], + [0], [ignore], []) + +AT_CLEANUP + + +AT_SETUP([run tests from build directories]) +AT_TESTED([kyua]) + +CREATE_ATF_HELPERS +AT_CHECK([mkdir src], [0], [], []) +AT_CHECK([mv Kyuafile src], [0], [], []) +AT_CHECK([mkdir obj], [0], [], []) +AT_CHECK([mv atf_helpers obj], [0], [], []) +AT_CHECK([RUN_ATF_HELPERS([fails passes skips], + [--kyuafile=src/Kyuafile --build-root=obj])], + [1], [stdout], []) +AT_CHECK([grep "atf_helpers:fails.*failure.*Failed on purpose" stdout], + [1], [], []) +AT_CHECK([grep "atf_helpers:passes.*passed" stdout], [0], [ignore], []) +AT_CHECK([grep "atf_helpers:skips.*skipped.*Skipped on purpose" stdout], + [0], [ignore], []) + +AT_CLEANUP diff --git a/cli/Kyuafile b/cli/Kyuafile new file mode 100644 index 000000000000..f5b797d760c3 --- /dev/null +++ b/cli/Kyuafile @@ -0,0 +1,14 @@ +syntax(2) + +test_suite("kyua") + +atf_test_program{name="cmd_about_test"} +atf_test_program{name="cmd_config_test"} +atf_test_program{name="cmd_db_exec_test"} +atf_test_program{name="cmd_debug_test"} +atf_test_program{name="cmd_help_test"} +atf_test_program{name="cmd_list_test"} +atf_test_program{name="cmd_test_test"} +atf_test_program{name="common_test"} +atf_test_program{name="config_test"} +atf_test_program{name="main_test"} diff --git a/cli/Makefile.am.inc b/cli/Makefile.am.inc new file mode 100644 index 000000000000..27872088a1b7 --- /dev/null +++ b/cli/Makefile.am.inc @@ -0,0 +1,123 @@ +# Copyright 2010 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +CLI_CFLAGS = $(DRIVERS_CFLAGS) +CLI_LIBS = libcli.a $(DRIVERS_LIBS) + +noinst_LIBRARIES += libcli.a +libcli_a_SOURCES = cli/cmd_about.cpp +libcli_a_SOURCES += cli/cmd_about.hpp +libcli_a_SOURCES += cli/cmd_config.cpp +libcli_a_SOURCES += cli/cmd_config.hpp +libcli_a_SOURCES += cli/cmd_db_exec.cpp +libcli_a_SOURCES += cli/cmd_db_exec.hpp +libcli_a_SOURCES += cli/cmd_db_migrate.cpp +libcli_a_SOURCES += cli/cmd_db_migrate.hpp +libcli_a_SOURCES += cli/cmd_debug.cpp +libcli_a_SOURCES += cli/cmd_debug.hpp +libcli_a_SOURCES += cli/cmd_help.cpp +libcli_a_SOURCES += cli/cmd_help.hpp +libcli_a_SOURCES += cli/cmd_list.cpp +libcli_a_SOURCES += cli/cmd_list.hpp +libcli_a_SOURCES += cli/cmd_report.cpp +libcli_a_SOURCES += cli/cmd_report.hpp +libcli_a_SOURCES += cli/cmd_report_html.cpp +libcli_a_SOURCES += cli/cmd_report_html.hpp +libcli_a_SOURCES += cli/cmd_report_junit.cpp +libcli_a_SOURCES += cli/cmd_report_junit.hpp +libcli_a_SOURCES += cli/cmd_test.cpp +libcli_a_SOURCES += cli/cmd_test.hpp +libcli_a_SOURCES += cli/common.cpp +libcli_a_SOURCES += cli/common.hpp +libcli_a_SOURCES += cli/common.ipp +libcli_a_SOURCES += cli/config.cpp +libcli_a_SOURCES += cli/config.hpp +libcli_a_SOURCES += cli/main.cpp +libcli_a_SOURCES += cli/main.hpp +libcli_a_CPPFLAGS = -DKYUA_CONFDIR="\"$(kyua_confdir)\"" +libcli_a_CPPFLAGS += -DKYUA_DOCDIR="\"$(docdir)\"" +libcli_a_CPPFLAGS += -DKYUA_MISCDIR="\"$(miscdir)\"" +libcli_a_CPPFLAGS += $(DRIVERS_CFLAGS) +libcli_a_LIBADD = libutils.a + +if WITH_ATF +tests_clidir = $(pkgtestsdir)/cli + +tests_cli_DATA = cli/Kyuafile +EXTRA_DIST += $(tests_cli_DATA) + +tests_cli_PROGRAMS = cli/cmd_about_test +cli_cmd_about_test_SOURCES = cli/cmd_about_test.cpp +cli_cmd_about_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_about_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/cmd_config_test +cli_cmd_config_test_SOURCES = cli/cmd_config_test.cpp +cli_cmd_config_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_config_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/cmd_db_exec_test +cli_cmd_db_exec_test_SOURCES = cli/cmd_db_exec_test.cpp +cli_cmd_db_exec_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_db_exec_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/cmd_debug_test +cli_cmd_debug_test_SOURCES = cli/cmd_debug_test.cpp +cli_cmd_debug_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_debug_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/cmd_help_test +cli_cmd_help_test_SOURCES = cli/cmd_help_test.cpp +cli_cmd_help_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_help_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/cmd_list_test +cli_cmd_list_test_SOURCES = cli/cmd_list_test.cpp +cli_cmd_list_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_list_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/cmd_test_test +cli_cmd_test_test_SOURCES = cli/cmd_test_test.cpp +cli_cmd_test_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_cmd_test_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/common_test +cli_common_test_SOURCES = cli/common_test.cpp +cli_common_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_common_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/config_test +cli_config_test_SOURCES = cli/config_test.cpp +cli_config_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_config_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) + +tests_cli_PROGRAMS += cli/main_test +cli_main_test_SOURCES = cli/main_test.cpp +cli_main_test_CXXFLAGS = $(CLI_CFLAGS) $(ATF_CXX_CFLAGS) +cli_main_test_LDADD = $(CLI_LIBS) $(ATF_CXX_LIBS) +endif diff --git a/cli/cmd_about.cpp b/cli/cmd_about.cpp new file mode 100644 index 000000000000..f2b3f99e0ada --- /dev/null +++ b/cli/cmd_about.cpp @@ -0,0 +1,160 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_about.hpp" + +#include <cstdlib> +#include <fstream> +#include <utility> +#include <vector> + +#include "cli/common.ipp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/defs.hpp" +#include "utils/env.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/path.hpp" +#include "utils/sanity.hpp" +#include "utils/text/regex.hpp" + +#if defined(HAVE_CONFIG_H) +# include "config.h" +#endif + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; +namespace text = utils::text; + +using cli::cmd_about; + + +namespace { + + +/// Print the contents of a document. +/// +/// If the file cannot be opened for whatever reason, an error message is +/// printed to the output of the program instead of the contents of the file. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param file The file to print. +/// \param filter_re Regular expression to match the lines to print. If empty, +/// no filtering is applied. +/// +/// \return True if the file was printed, false otherwise. +static bool +cat_file(cmdline::ui* ui, const fs::path& file, + const std::string& filter_re = "") +{ + std::ifstream input(file.c_str()); + if (!input) { + ui->err(F("Failed to open %s") % file); + return false; + } + + std::string line; + if (filter_re.empty()) { + while (std::getline(input, line).good()) { + ui->out(line); + } + } else { + const text::regex filter = text::regex::compile(filter_re, 0); + while (std::getline(input, line).good()) { + if (filter.match(line)) { + ui->out(line); + } + } + } + input.close(); + return true; +} + + +} // anonymous namespace + + +/// Default constructor for cmd_about. +cmd_about::cmd_about(void) : cli_command( + "about", "[authors|license|version]", 0, 1, + "Shows detailed authors and contributors; license; and version information") +{ +} + + +/// Entry point for the "about" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 if everything is OK, 1 if any of the necessary documents cannot be +/// opened. +int +cmd_about::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + const fs::path docdir(utils::getenv_with_default( + "KYUA_DOCDIR", KYUA_DOCDIR)); + + bool success = true; + + static const char* list_re = "^\\* "; + + if (cmdline.arguments().empty()) { + ui->out(PACKAGE " (" PACKAGE_NAME ") " PACKAGE_VERSION); + ui->out(""); + ui->out("License terms:"); + ui->out(""); + success &= cat_file(ui, docdir / "LICENSE"); + ui->out(""); + ui->out("Brought to you by:"); + ui->out(""); + success &= cat_file(ui, docdir / "AUTHORS", list_re); + ui->out(""); + success &= cat_file(ui, docdir / "CONTRIBUTORS", list_re); + ui->out(""); + ui->out(F("Homepage: %s") % PACKAGE_URL); + } else { + const std::string& topic = cmdline.arguments()[0]; + + if (topic == "authors") { + success &= cat_file(ui, docdir / "AUTHORS", list_re); + success &= cat_file(ui, docdir / "CONTRIBUTORS", list_re); + } else if (topic == "license") { + success &= cat_file(ui, docdir / "LICENSE"); + } else if (topic == "version") { + write_version_header(ui); + } else { + throw cmdline::usage_error(F("Invalid about topic '%s'") % topic); + } + } + + return success ? EXIT_SUCCESS : EXIT_FAILURE; +} diff --git a/cli/cmd_about.hpp b/cli/cmd_about.hpp new file mode 100644 index 000000000000..2d1ed57a498b --- /dev/null +++ b/cli/cmd_about.hpp @@ -0,0 +1,57 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_about.hpp +/// Provides the cmd_about class. + +#if !defined(CLI_CMD_ABOUT_HPP) +#define CLI_CMD_ABOUT_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "about" subcommand. +class cmd_about : public cli_command +{ + /// Path to the directory containing the distribution documents. + const std::string _docdir; + +public: + cmd_about(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_ABOUT_HPP) diff --git a/cli/cmd_about_test.cpp b/cli/cmd_about_test.cpp new file mode 100644 index 000000000000..da75db3b3871 --- /dev/null +++ b/cli/cmd_about_test.cpp @@ -0,0 +1,306 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_about.hpp" + +#if defined(HAVE_CONFIG_H) +# include "config.h" +#endif + +#include <cstdlib> + +#include <atf-c++.hpp> + +#include "cli/common.ipp" +#include "engine/config.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/config/tree.ipp" +#include "utils/env.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" + +namespace cmdline = utils::cmdline; +namespace fs = utils::fs; + +using cli::cmd_about; + + +ATF_TEST_CASE_WITHOUT_HEAD(all_topics__ok); +ATF_TEST_CASE_BODY(all_topics__ok) +{ + cmdline::args_vector args; + args.push_back("about"); + + fs::mkdir(fs::path("fake-docs"), 0755); + atf::utils::create_file("fake-docs/AUTHORS", + "Content of AUTHORS\n" + "* First author\n" + " * garbage\n" + "* Second author\n"); + atf::utils::create_file("fake-docs/CONTRIBUTORS", + "Content of CONTRIBUTORS\n" + "* First contributor\n" + " * garbage\n" + "* Second contributor\n"); + atf::utils::create_file("fake-docs/LICENSE", "Content of LICENSE\n"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + + ATF_REQUIRE(atf::utils::grep_string(PACKAGE_NAME, ui.out_log()[0])); + ATF_REQUIRE(atf::utils::grep_string(PACKAGE_VERSION, ui.out_log()[0])); + + ATF_REQUIRE(!atf::utils::grep_collection("Content of AUTHORS", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* First author", ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("garbage", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* Second author", ui.out_log())); + + ATF_REQUIRE(!atf::utils::grep_collection("Content of CONTRIBUTORS", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* First contributor", + ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("garbage", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* Second contributor", + ui.out_log())); + + ATF_REQUIRE(atf::utils::grep_collection("Content of LICENSE", + ui.out_log())); + + ATF_REQUIRE(atf::utils::grep_collection("Homepage", ui.out_log())); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(all_topics__missing_docs); +ATF_TEST_CASE_BODY(all_topics__missing_docs) +{ + cmdline::args_vector args; + args.push_back("about"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_FAILURE, cmd.main(&ui, args, engine::default_config())); + + ATF_REQUIRE(atf::utils::grep_string(PACKAGE_NAME, ui.out_log()[0])); + ATF_REQUIRE(atf::utils::grep_string(PACKAGE_VERSION, ui.out_log()[0])); + + ATF_REQUIRE(atf::utils::grep_collection("Homepage", ui.out_log())); + + ATF_REQUIRE(atf::utils::grep_collection("Failed to open.*AUTHORS", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Failed to open.*CONTRIBUTORS", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Failed to open.*LICENSE", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(topic_authors__ok); +ATF_TEST_CASE_BODY(topic_authors__ok) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("authors"); + + fs::mkdir(fs::path("fake-docs"), 0755); + atf::utils::create_file("fake-docs/AUTHORS", + "Content of AUTHORS\n" + "* First author\n" + " * garbage\n" + "* Second author\n"); + atf::utils::create_file("fake-docs/CONTRIBUTORS", + "Content of CONTRIBUTORS\n" + "* First contributor\n" + " * garbage\n" + "* Second contributor\n"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(!atf::utils::grep_string(PACKAGE_NAME, ui.out_log()[0])); + + ATF_REQUIRE(!atf::utils::grep_collection("Content of AUTHORS", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* First author", ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("garbage", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* Second author", ui.out_log())); + + ATF_REQUIRE(!atf::utils::grep_collection("Content of CONTRIBUTORS", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* First contributor", + ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("garbage", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("\\* Second contributor", + ui.out_log())); + + ATF_REQUIRE(!atf::utils::grep_collection("LICENSE", ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("Homepage", ui.out_log())); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(topic_authors__missing_doc); +ATF_TEST_CASE_BODY(topic_authors__missing_doc) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("authors"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_FAILURE, cmd.main(&ui, args, engine::default_config())); + + ATF_REQUIRE_EQ(0, ui.out_log().size()); + + ATF_REQUIRE(atf::utils::grep_collection("Failed to open.*AUTHORS", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Failed to open.*CONTRIBUTORS", + ui.err_log())); + ATF_REQUIRE(!atf::utils::grep_collection("Failed to open.*LICENSE", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(topic_license__ok); +ATF_TEST_CASE_BODY(topic_license__ok) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("license"); + + fs::mkdir(fs::path("fake-docs"), 0755); + atf::utils::create_file("fake-docs/LICENSE", "Content of LICENSE\n"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(!atf::utils::grep_string(PACKAGE_NAME, ui.out_log()[0])); + ATF_REQUIRE(!atf::utils::grep_collection("AUTHORS", ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("CONTRIBUTORS", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("Content of LICENSE", + ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection("Homepage", ui.out_log())); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(topic_license__missing_doc); +ATF_TEST_CASE_BODY(topic_license__missing_doc) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("license"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_FAILURE, cmd.main(&ui, args, engine::default_config())); + + ATF_REQUIRE_EQ(0, ui.out_log().size()); + + ATF_REQUIRE(!atf::utils::grep_collection("Failed to open.*AUTHORS", + ui.err_log())); + ATF_REQUIRE(!atf::utils::grep_collection("Failed to open.*CONTRIBUTORS", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Failed to open.*LICENSE", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(topic_version__ok); +ATF_TEST_CASE_BODY(topic_version__ok) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("version"); + + utils::setenv("KYUA_DOCDIR", "fake-docs"); + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE_EQ(1, ui.out_log().size()); + ATF_REQUIRE(atf::utils::grep_string(PACKAGE_NAME, ui.out_log()[0])); + ATF_REQUIRE(atf::utils::grep_string(PACKAGE_VERSION, ui.out_log()[0])); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(invalid_args); +ATF_TEST_CASE_BODY(invalid_args) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("first"); + args.push_back("second"); + + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_THROW_RE(cmdline::usage_error, "Too many arguments", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(invalid_topic); +ATF_TEST_CASE_BODY(invalid_topic) +{ + cmdline::args_vector args; + args.push_back("about"); + args.push_back("foo"); + + cmd_about cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_THROW_RE(cmdline::usage_error, "Invalid about topic 'foo'", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, all_topics__ok); + ATF_ADD_TEST_CASE(tcs, all_topics__missing_docs); + ATF_ADD_TEST_CASE(tcs, topic_authors__ok); + ATF_ADD_TEST_CASE(tcs, topic_authors__missing_doc); + ATF_ADD_TEST_CASE(tcs, topic_license__ok); + ATF_ADD_TEST_CASE(tcs, topic_license__missing_doc); + ATF_ADD_TEST_CASE(tcs, topic_version__ok); + ATF_ADD_TEST_CASE(tcs, invalid_args); + ATF_ADD_TEST_CASE(tcs, invalid_topic); +} diff --git a/cli/cmd_config.cpp b/cli/cmd_config.cpp new file mode 100644 index 000000000000..947449aacc2d --- /dev/null +++ b/cli/cmd_config.cpp @@ -0,0 +1,122 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_config.hpp" + +#include <cstdlib> + +#include "cli/common.ipp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/config/tree.ipp" +#include "utils/format/macros.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; + +using cli::cmd_config; + + +namespace { + + +/// Prints all configuration variables. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param properties The key/value map representing all the configuration +/// variables. +/// +/// \return 0 for success. +static int +print_all(cmdline::ui* ui, const config::properties_map& properties) +{ + for (config::properties_map::const_iterator iter = properties.begin(); + iter != properties.end(); iter++) + ui->out(F("%s = %s") % (*iter).first % (*iter).second); + return EXIT_SUCCESS; +} + + +/// Prints the configuration variables that the user requests. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param properties The key/value map representing all the configuration +/// variables. +/// \param filters The names of the configuration variables to print. +/// +/// \return 0 if all specified filters are valid; 1 otherwise. +static int +print_some(cmdline::ui* ui, const config::properties_map& properties, + const cmdline::args_vector& filters) +{ + bool ok = true; + + for (cmdline::args_vector::const_iterator iter = filters.begin(); + iter != filters.end(); iter++) { + const config::properties_map::const_iterator match = + properties.find(*iter); + if (match == properties.end()) { + cmdline::print_warning(ui, F("'%s' is not defined.") % *iter); + ok = false; + } else + ui->out(F("%s = %s") % (*match).first % (*match).second); + } + + return ok ? EXIT_SUCCESS : EXIT_FAILURE; +} + + +} // anonymous namespace + + +/// Default constructor for cmd_config. +cmd_config::cmd_config(void) : cli_command( + "config", "[variable1 .. variableN]", 0, -1, + "Inspects the values of configuration variables") +{ +} + + +/// Entry point for the "config" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// \param user_config The runtime configuration of the program. +/// +/// \return 0 if everything is OK, 1 if any of the necessary documents cannot be +/// opened. +int +cmd_config::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& user_config) +{ + const config::properties_map properties = user_config.all_properties(); + if (cmdline.arguments().empty()) + return print_all(ui, properties); + else + return print_some(ui, properties, cmdline.arguments()); +} diff --git a/cli/cmd_config.hpp b/cli/cmd_config.hpp new file mode 100644 index 000000000000..42f5abd90c28 --- /dev/null +++ b/cli/cmd_config.hpp @@ -0,0 +1,54 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_config.hpp +/// Provides the cmd_config class. + +#if !defined(CLI_CMD_CONFIG_HPP) +#define CLI_CMD_CONFIG_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "config" subcommand. +class cmd_config : public cli_command +{ +public: + cmd_config(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_CONFIG_HPP) diff --git a/cli/cmd_config_test.cpp b/cli/cmd_config_test.cpp new file mode 100644 index 000000000000..f084f99bb90a --- /dev/null +++ b/cli/cmd_config_test.cpp @@ -0,0 +1,144 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_config.hpp" + +#include <cstdlib> + +#include <atf-c++.hpp> + +#include "cli/common.ipp" +#include "engine/config.hpp" +#include "utils/cmdline/globals.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/config/tree.ipp" +#include "utils/optional.ipp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; + +using cli::cmd_config; +using utils::none; + + +namespace { + + +/// Instantiates a fake user configuration for testing purposes. +/// +/// The user configuration is populated with a collection of test-suite +/// properties and some hardcoded values for the generic configuration options. +/// +/// \return A new user configuration object. +static config::tree +fake_config(void) +{ + config::tree user_config = engine::default_config(); + user_config.set_string("architecture", "the-architecture"); + user_config.set_string("parallelism", "128"); + user_config.set_string("platform", "the-platform"); + //user_config.set_string("unprivileged_user", ""); + user_config.set_string("test_suites.foo.bar", "first"); + user_config.set_string("test_suites.foo.baz", "second"); + return user_config; +} + + +} // anonymous namespace + + +ATF_TEST_CASE_WITHOUT_HEAD(all); +ATF_TEST_CASE_BODY(all) +{ + cmdline::args_vector args; + args.push_back("config"); + + cmd_config cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, fake_config())); + + ATF_REQUIRE_EQ(5, ui.out_log().size()); + ATF_REQUIRE_EQ("architecture = the-architecture", ui.out_log()[0]); + ATF_REQUIRE_EQ("parallelism = 128", ui.out_log()[1]); + ATF_REQUIRE_EQ("platform = the-platform", ui.out_log()[2]); + ATF_REQUIRE_EQ("test_suites.foo.bar = first", ui.out_log()[3]); + ATF_REQUIRE_EQ("test_suites.foo.baz = second", ui.out_log()[4]); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(some__ok); +ATF_TEST_CASE_BODY(some__ok) +{ + cmdline::args_vector args; + args.push_back("config"); + args.push_back("platform"); + args.push_back("test_suites.foo.baz"); + + cmd_config cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, fake_config())); + + ATF_REQUIRE_EQ(2, ui.out_log().size()); + ATF_REQUIRE_EQ("platform = the-platform", ui.out_log()[0]); + ATF_REQUIRE_EQ("test_suites.foo.baz = second", ui.out_log()[1]); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(some__fail); +ATF_TEST_CASE_BODY(some__fail) +{ + cmdline::args_vector args; + args.push_back("config"); + args.push_back("platform"); + args.push_back("unknown"); + args.push_back("test_suites.foo.baz"); + + cmdline::init("progname"); + + cmd_config cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_FAILURE, cmd.main(&ui, args, fake_config())); + + ATF_REQUIRE_EQ(2, ui.out_log().size()); + ATF_REQUIRE_EQ("platform = the-platform", ui.out_log()[0]); + ATF_REQUIRE_EQ("test_suites.foo.baz = second", ui.out_log()[1]); + ATF_REQUIRE_EQ(1, ui.err_log().size()); + ATF_REQUIRE(atf::utils::grep_string("unknown.*not defined", + ui.err_log()[0])); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, all); + ATF_ADD_TEST_CASE(tcs, some__ok); + ATF_ADD_TEST_CASE(tcs, some__fail); +} diff --git a/cli/cmd_db_exec.cpp b/cli/cmd_db_exec.cpp new file mode 100644 index 000000000000..54304e6643de --- /dev/null +++ b/cli/cmd_db_exec.cpp @@ -0,0 +1,200 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_db_exec.hpp" + +#include <algorithm> +#include <cstdlib> +#include <iterator> +#include <sstream> +#include <string> + +#include "cli/common.ipp" +#include "store/exceptions.hpp" +#include "store/layout.hpp" +#include "store/read_backend.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/defs.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/path.hpp" +#include "utils/sanity.hpp" +#include "utils/sqlite/database.hpp" +#include "utils/sqlite/exceptions.hpp" +#include "utils/sqlite/statement.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; +namespace layout = store::layout; +namespace sqlite = utils::sqlite; + +using cli::cmd_db_exec; + + +namespace { + + +/// Concatenates a vector into a string using ' ' as a separator. +/// +/// \param args The objects to join. This cannot be empty. +/// +/// \return The concatenation of all the objects in the set. +static std::string +flatten_args(const cmdline::args_vector& args) +{ + std::ostringstream output; + std::copy(args.begin(), args.end(), + std::ostream_iterator< std::string >(output, " ")); + + std::string result = output.str(); + result.erase(result.end() - 1); + return result; +} + + +} // anonymous namespace + + +/// Formats a particular cell of a statement result. +/// +/// \param stmt The statement whose cell to format. +/// \param index The index of the cell to format. +/// +/// \return A textual representation of the cell. +std::string +cli::format_cell(sqlite::statement& stmt, const int index) +{ + switch (stmt.column_type(index)) { + case sqlite::type_blob: { + const sqlite::blob blob = stmt.column_blob(index); + return F("BLOB of %s bytes") % blob.size; + } + + case sqlite::type_float: + return F("%s") % stmt.column_double(index); + + case sqlite::type_integer: + return F("%s") % stmt.column_int64(index); + + case sqlite::type_null: + return "NULL"; + + case sqlite::type_text: + return stmt.column_text(index); + } + + UNREACHABLE; +} + + +/// Formats the column names of a statement for output as CSV. +/// +/// \param stmt The statement whose columns to format. +/// +/// \return A comma-separated list of column names. +std::string +cli::format_headers(sqlite::statement& stmt) +{ + std::string output; + int i = 0; + for (; i < stmt.column_count() - 1; ++i) + output += stmt.column_name(i) + ','; + output += stmt.column_name(i); + return output; +} + + +/// Formats a row of a statement for output as CSV. +/// +/// \param stmt The statement whose current row to format. +/// +/// \return A comma-separated list of values. +std::string +cli::format_row(sqlite::statement& stmt) +{ + std::string output; + int i = 0; + for (; i < stmt.column_count() - 1; ++i) + output += cli::format_cell(stmt, i) + ','; + output += cli::format_cell(stmt, i); + return output; +} + + +/// Default constructor for cmd_db_exec. +cmd_db_exec::cmd_db_exec(void) : cli_command( + "db-exec", "sql_statement", 1, -1, + "Executes an arbitrary SQL statement in a results file and prints " + "the resulting table") +{ + add_option(results_file_open_option); + add_option(cmdline::bool_option("no-headers", "Do not show headers in the " + "output table")); +} + + +/// Entry point for the "db-exec" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 if everything is OK, 1 if the statement is invalid or if there is +/// any other problem. +int +cmd_db_exec::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + try { + const fs::path results_file = layout::find_results( + results_file_open(cmdline)); + + // TODO(jmmv): Shouldn't be using store::detail here... + sqlite::database db = store::detail::open_and_setup( + results_file, sqlite::open_readwrite); + sqlite::statement stmt = db.create_statement( + flatten_args(cmdline.arguments())); + + if (stmt.step()) { + if (!cmdline.has_option("no-headers")) + ui->out(cli::format_headers(stmt)); + do + ui->out(cli::format_row(stmt)); + while (stmt.step()); + } + + return EXIT_SUCCESS; + } catch (const sqlite::error& e) { + cmdline::print_error(ui, F("SQLite error: %s.") % e.what()); + return EXIT_FAILURE; + } catch (const store::error& e) { + cmdline::print_error(ui, F("%s.") % e.what()); + return EXIT_FAILURE; + } +} diff --git a/cli/cmd_db_exec.hpp b/cli/cmd_db_exec.hpp new file mode 100644 index 000000000000..18aa16108553 --- /dev/null +++ b/cli/cmd_db_exec.hpp @@ -0,0 +1,61 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_db_exec.hpp +/// Provides the cmd_db_exec class. + +#if !defined(CLI_CMD_DB_EXEC_HPP) +#define CLI_CMD_DB_EXEC_HPP + +#include <string> + +#include "cli/common.hpp" +#include "utils/sqlite/statement_fwd.hpp" + +namespace cli { + + +std::string format_cell(utils::sqlite::statement&, const int); +std::string format_headers(utils::sqlite::statement&); +std::string format_row(utils::sqlite::statement&); + + +/// Implementation of the "db-exec" subcommand. +class cmd_db_exec : public cli_command +{ +public: + cmd_db_exec(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + +#endif // !defined(CLI_CMD_DB_EXEC_HPP) diff --git a/cli/cmd_db_exec_test.cpp b/cli/cmd_db_exec_test.cpp new file mode 100644 index 000000000000..1bf6b2e074a9 --- /dev/null +++ b/cli/cmd_db_exec_test.cpp @@ -0,0 +1,165 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_db_exec.hpp" + +#include <cstring> + +#include <atf-c++.hpp> + +#include "utils/format/macros.hpp" +#include "utils/sqlite/database.hpp" +#include "utils/sqlite/statement.ipp" + +namespace sqlite = utils::sqlite; + + +namespace { + + +/// Performs a test for the cli::format_cell() function. +/// +/// \tparam Cell The type of the value to insert into the test column. +/// \param column_type The SQL type of the test column. +/// \param value The value to insert into the test column. +/// \param exp_value The expected return value of cli::format_cell(). +template< class Cell > +static void +do_format_cell_test(const std::string column_type, + const Cell& value, const std::string& exp_value) +{ + sqlite::database db = sqlite::database::in_memory(); + + sqlite::statement create = db.create_statement( + F("CREATE TABLE test (column %s)") % column_type); + create.step_without_results(); + + sqlite::statement insert = db.create_statement( + "INSERT INTO test (column) VALUES (:column)"); + insert.bind(":column", value); + insert.step_without_results(); + + sqlite::statement query = db.create_statement("SELECT * FROM test"); + ATF_REQUIRE(query.step()); + ATF_REQUIRE_EQ(exp_value, cli::format_cell(query, 0)); + ATF_REQUIRE(!query.step()); +} + + +} // anonymous namespace + + +ATF_TEST_CASE_WITHOUT_HEAD(format_cell__blob); +ATF_TEST_CASE_BODY(format_cell__blob) +{ + const char* contents = "Some random contents"; + do_format_cell_test( + "BLOB", sqlite::blob(contents, std::strlen(contents)), + F("BLOB of %s bytes") % strlen(contents)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_cell__float); +ATF_TEST_CASE_BODY(format_cell__float) +{ + do_format_cell_test("FLOAT", 3.5, "3.5"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_cell__integer); +ATF_TEST_CASE_BODY(format_cell__integer) +{ + do_format_cell_test("INTEGER", 123456, "123456"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_cell__null); +ATF_TEST_CASE_BODY(format_cell__null) +{ + do_format_cell_test("TEXT", sqlite::null(), "NULL"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_cell__text); +ATF_TEST_CASE_BODY(format_cell__text) +{ + do_format_cell_test("TEXT", "Hello, world", "Hello, world"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_headers); +ATF_TEST_CASE_BODY(format_headers) +{ + sqlite::database db = sqlite::database::in_memory(); + + sqlite::statement create = db.create_statement( + "CREATE TABLE test (c1 TEXT, c2 TEXT, c3 TEXT)"); + create.step_without_results(); + + sqlite::statement query = db.create_statement( + "SELECT c1, c2, c3 AS c3bis FROM test"); + ATF_REQUIRE_EQ("c1,c2,c3bis", cli::format_headers(query)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_row); +ATF_TEST_CASE_BODY(format_row) +{ + sqlite::database db = sqlite::database::in_memory(); + + sqlite::statement create = db.create_statement( + "CREATE TABLE test (c1 TEXT, c2 BLOB)"); + create.step_without_results(); + + const char* memory = "BLOB contents"; + sqlite::statement insert = db.create_statement( + "INSERT INTO test VALUES (:v1, :v2)"); + insert.bind(":v1", "A string"); + insert.bind(":v2", sqlite::blob(memory, std::strlen(memory))); + insert.step_without_results(); + + sqlite::statement query = db.create_statement("SELECT * FROM test"); + query.step(); + ATF_REQUIRE_EQ( + (F("A string,BLOB of %s bytes") % std::strlen(memory)).str(), + cli::format_row(query)); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, format_cell__blob); + ATF_ADD_TEST_CASE(tcs, format_cell__float); + ATF_ADD_TEST_CASE(tcs, format_cell__integer); + ATF_ADD_TEST_CASE(tcs, format_cell__null); + ATF_ADD_TEST_CASE(tcs, format_cell__text); + + ATF_ADD_TEST_CASE(tcs, format_headers); + + ATF_ADD_TEST_CASE(tcs, format_row); +} diff --git a/cli/cmd_db_migrate.cpp b/cli/cmd_db_migrate.cpp new file mode 100644 index 000000000000..c6076c6afa4d --- /dev/null +++ b/cli/cmd_db_migrate.cpp @@ -0,0 +1,82 @@ +// Copyright 2013 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_db_migrate.hpp" + +#include <cstdlib> + +#include "cli/common.ipp" +#include "store/exceptions.hpp" +#include "store/layout.hpp" +#include "store/migrate.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/ui.hpp" +#include "utils/defs.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/path.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; +namespace layout = store::layout; + +using cli::cmd_db_migrate; + + +/// Default constructor for cmd_db_migrate. +cmd_db_migrate::cmd_db_migrate(void) : cli_command( + "db-migrate", "", 0, 0, + "Upgrades the schema of an existing results file to the currently " + "implemented version. A backup of the results file is created, but " + "this operation is not reversible") +{ + add_option(results_file_open_option); +} + + +/// Entry point for the "db-migrate" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 if everything is OK, 1 if the statement is invalid or if there is +/// any other problem. +int +cmd_db_migrate::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + try { + const fs::path results_file = layout::find_results( + results_file_open(cmdline)); + store::migrate_schema(results_file); + return EXIT_SUCCESS; + } catch (const store::error& e) { + cmdline::print_error(ui, F("Migration failed: %s.") % e.what()); + return EXIT_FAILURE; + } +} diff --git a/cli/cmd_db_migrate.hpp b/cli/cmd_db_migrate.hpp new file mode 100644 index 000000000000..ebbe2b8a4ba4 --- /dev/null +++ b/cli/cmd_db_migrate.hpp @@ -0,0 +1,54 @@ +// Copyright 2013 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_db_migrate.hpp +/// Provides the cmd_db_migrate class. + +#if !defined(CLI_CMD_DB_MIGRATE_HPP) +#define CLI_CMD_DB_MIGRATE_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "db-migrate" subcommand. +class cmd_db_migrate : public cli_command +{ +public: + cmd_db_migrate(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_DB_MIGRATE_HPP) diff --git a/cli/cmd_debug.cpp b/cli/cmd_debug.cpp new file mode 100644 index 000000000000..b7a29b7ab804 --- /dev/null +++ b/cli/cmd_debug.cpp @@ -0,0 +1,94 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_debug.hpp" + +#include <cstdlib> + +#include "cli/common.ipp" +#include "drivers/debug_test.hpp" +#include "engine/filters.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/format/macros.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; + +using cli::cmd_debug; + + +/// Default constructor for cmd_debug. +cmd_debug::cmd_debug(void) : cli_command( + "debug", "test_case", 1, 1, + "Executes a single test case providing facilities for debugging") +{ + add_option(build_root_option); + add_option(kyuafile_option); + + add_option(cmdline::path_option( + "stdout", "Where to direct the standard output of the test case", + "path", "/dev/stdout")); + + add_option(cmdline::path_option( + "stderr", "Where to direct the standard error of the test case", + "path", "/dev/stderr")); +} + + +/// Entry point for the "debug" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// \param user_config The runtime debuguration of the program. +/// +/// \return 0 if everything is OK, 1 if any of the necessary documents cannot be +/// opened. +int +cmd_debug::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& user_config) +{ + const std::string& test_case_name = cmdline.arguments()[0]; + if (test_case_name.find(':') == std::string::npos) + throw cmdline::usage_error(F("'%s' is not a test case identifier " + "(missing ':'?)") % test_case_name); + const engine::test_filter filter = engine::test_filter::parse( + test_case_name); + + const drivers::debug_test::result result = drivers::debug_test::drive( + kyuafile_path(cmdline), build_root_path(cmdline), filter, user_config, + cmdline.get_option< cmdline::path_option >("stdout"), + cmdline.get_option< cmdline::path_option >("stderr")); + + ui->out(F("%s -> %s") % cli::format_test_case_id(result.test_case) % + cli::format_result(result.test_result)); + + return result.test_result.good() ? EXIT_SUCCESS : EXIT_FAILURE; +} diff --git a/cli/cmd_debug.hpp b/cli/cmd_debug.hpp new file mode 100644 index 000000000000..2d9e8dee1797 --- /dev/null +++ b/cli/cmd_debug.hpp @@ -0,0 +1,54 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_debug.hpp +/// Provides the cmd_debug class. + +#if !defined(CLI_CMD_DEBUG_HPP) +#define CLI_CMD_DEBUG_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "debug" subcommand. +class cmd_debug : public cli_command +{ +public: + cmd_debug(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_DEBUG_HPP) diff --git a/cli/cmd_debug_test.cpp b/cli/cmd_debug_test.cpp new file mode 100644 index 000000000000..28137e028962 --- /dev/null +++ b/cli/cmd_debug_test.cpp @@ -0,0 +1,82 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_debug.hpp" + +#include <stdexcept> + +#include <atf-c++.hpp> + +#include "cli/common.ipp" +#include "engine/config.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/config/tree.ipp" + +namespace cmdline = utils::cmdline; + + +ATF_TEST_CASE_WITHOUT_HEAD(invalid_filter); +ATF_TEST_CASE_BODY(invalid_filter) +{ + cmdline::args_vector args; + args.push_back("debug"); + args.push_back("incorrect:"); + + cli::cmd_debug cmd; + cmdline::ui_mock ui; + // TODO(jmmv): This error should really be cmdline::usage_error. + ATF_REQUIRE_THROW_RE(std::runtime_error, "Test case.*'incorrect:'.*empty", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(filter_without_test_case); +ATF_TEST_CASE_BODY(filter_without_test_case) +{ + cmdline::args_vector args; + args.push_back("debug"); + args.push_back("program"); + + cli::cmd_debug cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_THROW_RE(cmdline::error, "'program'.*not a test case", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, invalid_filter); + ATF_ADD_TEST_CASE(tcs, filter_without_test_case); +} diff --git a/cli/cmd_help.cpp b/cli/cmd_help.cpp new file mode 100644 index 000000000000..9ebe6f50c852 --- /dev/null +++ b/cli/cmd_help.cpp @@ -0,0 +1,250 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_help.hpp" + +#include <algorithm> +#include <cstdlib> + +#include "cli/common.ipp" +#include "utils/cmdline/commands_map.ipp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/globals.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui.hpp" +#include "utils/defs.hpp" +#include "utils/format/macros.hpp" +#include "utils/sanity.hpp" +#include "utils/text/table.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace text = utils::text; + +using cli::cmd_help; + + +namespace { + + +/// Creates a table with the help of a set of options. +/// +/// \param options The set of options to describe. May be empty. +/// +/// \return A 2-column wide table with the description of the options. +static text::table +options_help(const cmdline::options_vector& options) +{ + text::table table(2); + + for (cmdline::options_vector::const_iterator iter = options.begin(); + iter != options.end(); iter++) { + const cmdline::base_option* option = *iter; + + std::string description = option->description(); + if (option->needs_arg() && option->has_default_value()) + description += F(" (default: %s)") % option->default_value(); + + text::table_row row; + + if (option->has_short_name()) + row.push_back(F("%s, %s") % option->format_short_name() % + option->format_long_name()); + else + row.push_back(F("%s") % option->format_long_name()); + row.push_back(F("%s.") % description); + + table.add_row(row); + } + + return table; +} + + +/// Prints the summary of commands and generic options. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param options The set of program-wide options for which to print help. +/// \param commands The set of commands for which to print help. +static void +general_help(cmdline::ui* ui, const cmdline::options_vector* options, + const cmdline::commands_map< cli::cli_command >* commands) +{ + PRE(!commands->empty()); + + cli::write_version_header(ui); + ui->out(""); + ui->out_tag_wrap( + "Usage: ", + F("%s [general_options] command [command_options] [args]") % + cmdline::progname(), false); + + const text::table options_table = options_help(*options); + text::widths_vector::value_type first_width = + options_table.column_width(0); + + std::map< std::string, text::table > command_tables; + + for (cmdline::commands_map< cli::cli_command >::const_iterator + iter = commands->begin(); iter != commands->end(); iter++) { + const std::string& category = (*iter).first; + const std::set< std::string >& command_names = (*iter).second; + + command_tables.insert(std::map< std::string, text::table >::value_type( + category, text::table(2))); + text::table& table = command_tables.find(category)->second; + + for (std::set< std::string >::const_iterator i2 = command_names.begin(); + i2 != command_names.end(); i2++) { + const cli::cli_command* command = commands->find(*i2); + text::table_row row; + row.push_back(command->name()); + row.push_back(F("%s.") % command->short_description()); + table.add_row(row); + } + + if (table.column_width(0) > first_width) + first_width = table.column_width(0); + } + + text::table_formatter formatter; + formatter.set_column_width(0, first_width); + formatter.set_column_width(1, text::table_formatter::width_refill); + formatter.set_separator(" "); + + if (!options_table.empty()) { + ui->out_wrap(""); + ui->out_wrap("Available general options:"); + ui->out_table(options_table, formatter, " "); + } + + // Iterate using the same loop as above to preserve ordering. + for (cmdline::commands_map< cli::cli_command >::const_iterator + iter = commands->begin(); iter != commands->end(); iter++) { + const std::string& category = (*iter).first; + ui->out_wrap(""); + ui->out_wrap(F("%s commands:") % + (category.empty() ? "Generic" : category)); + ui->out_table(command_tables.find(category)->second, formatter, " "); + } + + ui->out_wrap(""); + ui->out_wrap("See kyua(1) for more details."); +} + + +/// Prints help for a particular subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param general_options The options that apply to all commands. +/// \param command Pointer to the command to describe. +static void +subcommand_help(cmdline::ui* ui, + const utils::cmdline::options_vector* general_options, + const cli::cli_command* command) +{ + cli::write_version_header(ui); + ui->out(""); + ui->out_tag_wrap( + "Usage: ", F("%s [general_options] %s%s%s") % + cmdline::progname() % command->name() % + (command->options().empty() ? "" : " [command_options]") % + (command->arg_list().empty() ? "" : (" " + command->arg_list())), + false); + ui->out_wrap(""); + ui->out_wrap(F("%s.") % command->short_description()); + + const text::table general_table = options_help(*general_options); + const text::table command_table = options_help(command->options()); + + const text::widths_vector::value_type first_width = + std::max(general_table.column_width(0), command_table.column_width(0)); + text::table_formatter formatter; + formatter.set_column_width(0, first_width); + formatter.set_column_width(1, text::table_formatter::width_refill); + formatter.set_separator(" "); + + if (!general_table.empty()) { + ui->out_wrap(""); + ui->out_wrap("Available general options:"); + ui->out_table(general_table, formatter, " "); + } + + if (!command_table.empty()) { + ui->out_wrap(""); + ui->out_wrap("Available command options:"); + ui->out_table(command_table, formatter, " "); + } + + ui->out_wrap(""); + ui->out_wrap(F("See kyua-%s(1) for more details.") % command->name()); +} + + +} // anonymous namespace + + +/// Default constructor for cmd_help. +/// +/// \param options_ The set of program-wide options for which to provide help. +/// \param commands_ The set of commands for which to provide help. +cmd_help::cmd_help(const cmdline::options_vector* options_, + const cmdline::commands_map< cli_command >* commands_) : + cli_command("help", "[subcommand]", 0, 1, "Shows usage information"), + _options(options_), + _commands(commands_) +{ +} + + +/// Entry point for the "help" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 to indicate success. +int +cmd_help::run(utils::cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + if (cmdline.arguments().empty()) { + general_help(ui, _options, _commands); + } else { + INV(cmdline.arguments().size() == 1); + const std::string& cmdname = cmdline.arguments()[0]; + const cli::cli_command* command = _commands->find(cmdname); + if (command == NULL) + throw cmdline::usage_error(F("The command %s does not exist") % + cmdname); + else + subcommand_help(ui, _options, command); + } + + return EXIT_SUCCESS; +} diff --git a/cli/cmd_help.hpp b/cli/cmd_help.hpp new file mode 100644 index 000000000000..5f3b19db901d --- /dev/null +++ b/cli/cmd_help.hpp @@ -0,0 +1,62 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_help.hpp +/// Provides the cmd_help class. + +#if !defined(CLI_CMD_HELP_HPP) +#define CLI_CMD_HELP_HPP + +#include "cli/common.hpp" +#include "utils/cmdline/commands_map_fwd.hpp" + +namespace cli { + + +/// Implementation of the "help" subcommand. +class cmd_help : public cli_command +{ + /// The set of program-wide options for which to provide help. + const utils::cmdline::options_vector* _options; + + /// The set of commands for which to provide help. + const utils::cmdline::commands_map< cli_command >* _commands; + +public: + cmd_help(const utils::cmdline::options_vector*, + const utils::cmdline::commands_map< cli_command >*); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_HELP_HPP) diff --git a/cli/cmd_help_test.cpp b/cli/cmd_help_test.cpp new file mode 100644 index 000000000000..d292090be451 --- /dev/null +++ b/cli/cmd_help_test.cpp @@ -0,0 +1,347 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_help.hpp" + +#include <algorithm> +#include <cstdlib> +#include <iterator> + +#include <atf-c++.hpp> + +#include "cli/common.ipp" +#include "engine/config.hpp" +#include "utils/cmdline/commands_map.ipp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/globals.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/config/tree.ipp" +#include "utils/defs.hpp" +#include "utils/sanity.hpp" + +#if defined(HAVE_CONFIG_H) +# include "config.h" +#endif + +namespace cmdline = utils::cmdline; +namespace config = utils::config; + +using cli::cmd_help; + + +namespace { + + +/// Mock command with a simple definition (no options, no arguments). +/// +/// Attempting to run this command will result in a crash. It is only provided +/// to validate the generation of interactive help. +class cmd_mock_simple : public cli::cli_command { +public: + /// Constructs a new mock command. + /// + /// \param name_ The name of the command to create. + cmd_mock_simple(const char* name_) : cli::cli_command( + name_, "", 0, 0, "Simple command") + { + } + + /// Runs the mock command. + /// + /// \return Nothing because this function is never called. + int + run(cmdline::ui* /* ui */, + const cmdline::parsed_cmdline& /* cmdline */, + const config::tree& /* user_config */) + { + UNREACHABLE; + } +}; + + +/// Mock command with a complex definition (some options, some arguments). +/// +/// Attempting to run this command will result in a crash. It is only provided +/// to validate the generation of interactive help. +class cmd_mock_complex : public cli::cli_command { +public: + /// Constructs a new mock command. + /// + /// \param name_ The name of the command to create. + cmd_mock_complex(const char* name_) : cli::cli_command( + name_, "[arg1 .. argN]", 0, 2, "Complex command") + { + add_option(cmdline::bool_option("flag_a", "Flag A")); + add_option(cmdline::bool_option('b', "flag_b", "Flag B")); + add_option(cmdline::string_option('c', "flag_c", "Flag C", "c_arg")); + add_option(cmdline::string_option("flag_d", "Flag D", "d_arg", "foo")); + } + + /// Runs the mock command. + /// + /// \return Nothing because this function is never called. + int + run(cmdline::ui* /* ui */, + const cmdline::parsed_cmdline& /* cmdline */, + const config::tree& /* user_config */) + { + UNREACHABLE; + } +}; + + +/// Initializes the cmdline library and generates the set of test commands. +/// +/// \param [out] commands A mapping that is updated to contain the commands to +/// use for testing. +static void +setup(cmdline::commands_map< cli::cli_command >& commands) +{ + cmdline::init("progname"); + + commands.insert(new cmd_mock_simple("mock_simple")); + commands.insert(new cmd_mock_complex("mock_complex")); + + commands.insert(new cmd_mock_simple("mock_simple_2"), "First"); + commands.insert(new cmd_mock_complex("mock_complex_2"), "First"); + + commands.insert(new cmd_mock_simple("mock_simple_3"), "Second"); +} + + +/// Performs a test on the global help (not that of a subcommand). +/// +/// \param general_options The genral options supported by the tool, if any. +/// \param expected_options Expected lines of help output documenting the +/// options in general_options. +/// \param ui The cmdline::mock_ui object to which to write the output. +static void +global_test(const cmdline::options_vector& general_options, + const std::vector< std::string >& expected_options, + cmdline::ui_mock& ui) +{ + cmdline::commands_map< cli::cli_command > mock_commands; + setup(mock_commands); + + cmdline::args_vector args; + args.push_back("help"); + + cmd_help cmd(&general_options, &mock_commands); + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + + std::vector< std::string > expected; + + expected.push_back(PACKAGE " (" PACKAGE_NAME ") " PACKAGE_VERSION); + expected.push_back(""); + expected.push_back("Usage: progname [general_options] command " + "[command_options] [args]"); + if (!general_options.empty()) { + expected.push_back(""); + expected.push_back("Available general options:"); + std::copy(expected_options.begin(), expected_options.end(), + std::back_inserter(expected)); + } + expected.push_back(""); + expected.push_back("Generic commands:"); + expected.push_back(" mock_complex Complex command."); + expected.push_back(" mock_simple Simple command."); + expected.push_back(""); + expected.push_back("First commands:"); + expected.push_back(" mock_complex_2 Complex command."); + expected.push_back(" mock_simple_2 Simple command."); + expected.push_back(""); + expected.push_back("Second commands:"); + expected.push_back(" mock_simple_3 Simple command."); + expected.push_back(""); + expected.push_back("See kyua(1) for more details."); + + ATF_REQUIRE(expected == ui.out_log()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +} // anonymous namespace + + +ATF_TEST_CASE_WITHOUT_HEAD(global__no_options); +ATF_TEST_CASE_BODY(global__no_options) +{ + cmdline::ui_mock ui; + + cmdline::options_vector general_options; + + global_test(general_options, std::vector< std::string >(), ui); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(global__some_options); +ATF_TEST_CASE_BODY(global__some_options) +{ + cmdline::ui_mock ui; + + cmdline::options_vector general_options; + const cmdline::bool_option flag_a("flag_a", "Flag A"); + general_options.push_back(&flag_a); + const cmdline::string_option flag_c('c', "lc", "Flag C", "X"); + general_options.push_back(&flag_c); + + std::vector< std::string > expected; + expected.push_back(" --flag_a Flag A."); + expected.push_back(" -c X, --lc=X Flag C."); + + global_test(general_options, expected, ui); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(subcommand__simple); +ATF_TEST_CASE_BODY(subcommand__simple) +{ + cmdline::options_vector general_options; + + cmdline::commands_map< cli::cli_command > mock_commands; + setup(mock_commands); + + cmdline::args_vector args; + args.push_back("help"); + args.push_back("mock_simple"); + + cmd_help cmd(&general_options, &mock_commands); + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(atf::utils::grep_collection( + "^kyua.*" PACKAGE_VERSION, ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection( + "^Usage: progname \\[general_options\\] mock_simple$", ui.out_log())); + ATF_REQUIRE(!atf::utils::grep_collection( + "Available.*options", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection( + "^See kyua-mock_simple\\(1\\) for more details.", ui.out_log())); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(subcommand__complex); +ATF_TEST_CASE_BODY(subcommand__complex) +{ + cmdline::options_vector general_options; + const cmdline::bool_option global_a("global_a", "Global A"); + general_options.push_back(&global_a); + const cmdline::string_option global_c('c', "global_c", "Global C", + "c_global"); + general_options.push_back(&global_c); + + cmdline::commands_map< cli::cli_command > mock_commands; + setup(mock_commands); + + cmdline::args_vector args; + args.push_back("help"); + args.push_back("mock_complex"); + + cmd_help cmd(&general_options, &mock_commands); + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_SUCCESS, cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(atf::utils::grep_collection( + "^kyua.*" PACKAGE_VERSION, ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection( + "^Usage: progname \\[general_options\\] mock_complex " + "\\[command_options\\] \\[arg1 .. argN\\]$", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("Available general options", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("--global_a", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("--global_c=c_global", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("Available command options", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("--flag_a *Flag A", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection("-b.*--flag_b *Flag B", + ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection( + "-c c_arg.*--flag_c=c_arg *Flag C", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection( + "--flag_d=d_arg *Flag D.*default.*foo", ui.out_log())); + ATF_REQUIRE(atf::utils::grep_collection( + "^See kyua-mock_complex\\(1\\) for more details.", ui.out_log())); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(subcommand__unknown); +ATF_TEST_CASE_BODY(subcommand__unknown) +{ + cmdline::options_vector general_options; + + cmdline::commands_map< cli::cli_command > mock_commands; + setup(mock_commands); + + cmdline::args_vector args; + args.push_back("help"); + args.push_back("foobar"); + + cmd_help cmd(&general_options, &mock_commands); + cmdline::ui_mock ui; + ATF_REQUIRE_THROW_RE(cmdline::usage_error, "command foobar.*not exist", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(invalid_args); +ATF_TEST_CASE_BODY(invalid_args) +{ + cmdline::options_vector general_options; + + cmdline::commands_map< cli::cli_command > mock_commands; + setup(mock_commands); + + cmdline::args_vector args; + args.push_back("help"); + args.push_back("mock_simple"); + args.push_back("mock_complex"); + + cmd_help cmd(&general_options, &mock_commands); + cmdline::ui_mock ui; + ATF_REQUIRE_THROW_RE(cmdline::usage_error, "Too many arguments", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, global__no_options); + ATF_ADD_TEST_CASE(tcs, global__some_options); + ATF_ADD_TEST_CASE(tcs, subcommand__simple); + ATF_ADD_TEST_CASE(tcs, subcommand__complex); + ATF_ADD_TEST_CASE(tcs, subcommand__unknown); + ATF_ADD_TEST_CASE(tcs, invalid_args); +} diff --git a/cli/cmd_list.cpp b/cli/cmd_list.cpp new file mode 100644 index 000000000000..ed0e4980fc47 --- /dev/null +++ b/cli/cmd_list.cpp @@ -0,0 +1,161 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_list.hpp" + +#include <cstdlib> +#include <utility> +#include <vector> + +#include "cli/common.ipp" +#include "drivers/list_tests.hpp" +#include "engine/filters.hpp" +#include "model/metadata.hpp" +#include "model/test_case.hpp" +#include "model/test_program.hpp" +#include "model/types.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/defs.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/path.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; + + +namespace { + + +/// Hooks for list_tests to print test cases as they come. +class progress_hooks : public drivers::list_tests::base_hooks { + /// The ui object to which to print the test cases. + cmdline::ui* _ui; + + /// Whether to print test case details or just their names. + bool _verbose; + +public: + /// Initializes the hooks. + /// + /// \param ui_ The ui object to which to print the test cases. + /// \param verbose_ Whether to print test case details or just their names. + progress_hooks(cmdline::ui* ui_, const bool verbose_) : + _ui(ui_), + _verbose(verbose_) + { + } + + /// Reports a test case as soon as it is found. + /// + /// \param test_program The test program containing the test case. + /// \param test_case_name The name of the located test case. + void + got_test_case(const model::test_program& test_program, + const std::string& test_case_name) + { + cli::detail::list_test_case(_ui, _verbose, test_program, + test_case_name); + } +}; + + +} // anonymous namespace + + +/// Lists a single test case. +/// +/// \param [out] ui Object to interact with the I/O of the program. +/// \param verbose Whether to be verbose or not. +/// \param test_program The test program containing the test case to print. +/// \param test_case_name The name of the test case to print. +void +cli::detail::list_test_case(cmdline::ui* ui, const bool verbose, + const model::test_program& test_program, + const std::string& test_case_name) +{ + const model::test_case& test_case = test_program.find(test_case_name); + + const std::string id = format_test_case_id(test_program, test_case_name); + if (!verbose) { + ui->out(id); + } else { + ui->out(F("%s (%s)") % id % test_program.test_suite_name()); + + // TODO(jmmv): Running these for every test case is probably not the + // fastest thing to do. + const model::metadata default_md = model::metadata_builder().build(); + const model::properties_map default_props = default_md.to_properties(); + + const model::metadata& test_md = test_case.get_metadata(); + const model::properties_map test_props = test_md.to_properties(); + + for (model::properties_map::const_iterator iter = test_props.begin(); + iter != test_props.end(); iter++) { + const model::properties_map::const_iterator default_iter = + default_props.find((*iter).first); + if (default_iter == default_props.end() || + (*iter).second != (*default_iter).second) + ui->out(F(" %s = %s") % (*iter).first % (*iter).second); + } + } +} + + +/// Default constructor for cmd_list. +cli::cmd_list::cmd_list(void) : + cli_command("list", "[test-program ...]", 0, -1, + "Lists test cases and their meta-data") +{ + add_option(build_root_option); + add_option(kyuafile_option); + add_option(cmdline::bool_option('v', "verbose", "Show properties")); +} + + +/// Entry point for the "list" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// \param user_config The runtime configuration of the program. +/// +/// \return 0 to indicate success. +int +cli::cmd_list::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& user_config) +{ + progress_hooks hooks(ui, cmdline.has_option("verbose")); + const drivers::list_tests::result result = drivers::list_tests::drive( + kyuafile_path(cmdline), build_root_path(cmdline), + parse_filters(cmdline.arguments()), user_config, hooks); + + return report_unused_filters(result.unused_filters, ui) ? + EXIT_FAILURE : EXIT_SUCCESS; +} diff --git a/cli/cmd_list.hpp b/cli/cmd_list.hpp new file mode 100644 index 000000000000..cbdc084a6e16 --- /dev/null +++ b/cli/cmd_list.hpp @@ -0,0 +1,65 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_list.hpp +/// Provides the cmd_list class. + +#if !defined(CLI_CMD_LIST_HPP) +#define CLI_CMD_LIST_HPP + +#include <string> + +#include "cli/common.hpp" +#include "model/test_program_fwd.hpp" +#include "utils/fs/path_fwd.hpp" + +namespace cli { + + +namespace detail { + +void list_test_case(utils::cmdline::ui*, const bool, const model::test_program&, + const std::string&); + +} // namespace detail + + +/// Implementation of the "list" subcommand. +class cmd_list : public cli_command +{ +public: + cmd_list(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + +#endif // !defined(CLI_CMD_LIST_HPP) diff --git a/cli/cmd_list_test.cpp b/cli/cmd_list_test.cpp new file mode 100644 index 000000000000..19078abd7d48 --- /dev/null +++ b/cli/cmd_list_test.cpp @@ -0,0 +1,112 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_list.hpp" + +#include <atf-c++.hpp> + +#include "model/metadata.hpp" +#include "model/test_program.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/fs/path.hpp" + +namespace cmdline = utils::cmdline; +namespace fs = utils::fs; + + +ATF_TEST_CASE_WITHOUT_HEAD(list_test_case__no_verbose); +ATF_TEST_CASE_BODY(list_test_case__no_verbose) +{ + const model::metadata md = model::metadata_builder() + .set_description("This should not be shown") + .build(); + const model::test_program test_program = model::test_program_builder( + "mock", fs::path("the/test-program"), fs::path("root"), "suite") + .add_test_case("abc", md) + .set_metadata(md) + .build(); + + cmdline::ui_mock ui; + cli::detail::list_test_case(&ui, false, test_program, "abc"); + ATF_REQUIRE_EQ(1, ui.out_log().size()); + ATF_REQUIRE_EQ("the/test-program:abc", ui.out_log()[0]); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(list_test_case__verbose__no_properties); +ATF_TEST_CASE_BODY(list_test_case__verbose__no_properties) +{ + const model::test_program test_program = model::test_program_builder( + "mock", fs::path("hello/world"), fs::path("root"), "the-suite") + .add_test_case("my_name") + .build(); + + cmdline::ui_mock ui; + cli::detail::list_test_case(&ui, true, test_program, "my_name"); + ATF_REQUIRE_EQ(1, ui.out_log().size()); + ATF_REQUIRE_EQ("hello/world:my_name (the-suite)", ui.out_log()[0]); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(list_test_case__verbose__some_properties); +ATF_TEST_CASE_BODY(list_test_case__verbose__some_properties) +{ + const model::metadata md = model::metadata_builder() + .add_custom("my-property", "value") + .set_description("Some description") + .set_has_cleanup(true) + .build(); + const model::test_program test_program = model::test_program_builder( + "mock", fs::path("hello/world"), fs::path("root"), "the-suite") + .add_test_case("my_name", md) + .set_metadata(md) + .build(); + + cmdline::ui_mock ui; + cli::detail::list_test_case(&ui, true, test_program, "my_name"); + ATF_REQUIRE_EQ(4, ui.out_log().size()); + ATF_REQUIRE_EQ("hello/world:my_name (the-suite)", ui.out_log()[0]); + ATF_REQUIRE_EQ(" custom.my-property = value", ui.out_log()[1]); + ATF_REQUIRE_EQ(" description = Some description", ui.out_log()[2]); + ATF_REQUIRE_EQ(" has_cleanup = true", ui.out_log()[3]); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, list_test_case__no_verbose); + ATF_ADD_TEST_CASE(tcs, list_test_case__verbose__no_properties); + ATF_ADD_TEST_CASE(tcs, list_test_case__verbose__some_properties); + + // Tests for cmd_list::run are located in integration/cmd_list_test. +} diff --git a/cli/cmd_report.cpp b/cli/cmd_report.cpp new file mode 100644 index 000000000000..27827e893de7 --- /dev/null +++ b/cli/cmd_report.cpp @@ -0,0 +1,421 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_report.hpp" + +#include <algorithm> +#include <cstddef> +#include <cstdlib> +#include <map> +#include <ostream> +#include <string> +#include <vector> + +#include "cli/common.ipp" +#include "drivers/scan_results.hpp" +#include "model/context.hpp" +#include "model/metadata.hpp" +#include "model/test_case.hpp" +#include "model/test_program.hpp" +#include "model/test_result.hpp" +#include "model/types.hpp" +#include "store/layout.hpp" +#include "store/read_transaction.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/datetime.hpp" +#include "utils/defs.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/path.hpp" +#include "utils/optional.ipp" +#include "utils/sanity.hpp" +#include "utils/stream.hpp" +#include "utils/text/operations.ipp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace datetime = utils::datetime; +namespace fs = utils::fs; +namespace layout = store::layout; +namespace text = utils::text; + +using cli::cmd_report; +using utils::optional; + + +namespace { + + +/// Generates a plain-text report intended to be printed to the console. +class report_console_hooks : public drivers::scan_results::base_hooks { + /// Stream to which to write the report. + std::ostream& _output; + + /// Whether to include details in the report or not. + const bool _verbose; + + /// Collection of result types to include in the report. + const cli::result_types& _results_filters; + + /// Path to the results file being read. + const fs::path& _results_file; + + /// The start time of the first test. + optional< utils::datetime::timestamp > _start_time; + + /// The end time of the last test. + optional< utils::datetime::timestamp > _end_time; + + /// The total run time of the tests. Note that we cannot subtract _end_time + /// from _start_time to compute this due to parallel execution. + utils::datetime::delta _runtime; + + /// Representation of a single result. + struct result_data { + /// The relative path to the test program. + utils::fs::path binary_path; + + /// The name of the test case. + std::string test_case_name; + + /// The result of the test case. + model::test_result result; + + /// The duration of the test case execution. + utils::datetime::delta duration; + + /// Constructs a new results data. + /// + /// \param binary_path_ The relative path to the test program. + /// \param test_case_name_ The name of the test case. + /// \param result_ The result of the test case. + /// \param duration_ The duration of the test case execution. + result_data(const utils::fs::path& binary_path_, + const std::string& test_case_name_, + const model::test_result& result_, + const utils::datetime::delta& duration_) : + binary_path(binary_path_), test_case_name(test_case_name_), + result(result_), duration(duration_) + { + } + }; + + /// Results received, broken down by their type. + /// + /// Note that this may not include all results, as keeping the whole list in + /// memory may be too much. + std::map< model::test_result_type, std::vector< result_data > > _results; + + /// Pretty-prints the value of an environment variable. + /// + /// \param indent Prefix for the lines to print. Continuation lines + /// use this indentation twice. + /// \param name Name of the variable. + /// \param value Value of the variable. Can have newlines. + void + print_env_var(const char* indent, const std::string& name, + const std::string& value) + { + const std::vector< std::string > lines = text::split(value, '\n'); + if (lines.size() == 0) { + _output << F("%s%s=\n") % indent % name;; + } else { + _output << F("%s%s=%s\n") % indent % name % lines[0]; + for (std::vector< std::string >::size_type i = 1; + i < lines.size(); ++i) { + _output << F("%s%s%s\n") % indent % indent % lines[i]; + } + } + } + + /// Prints the execution context to the output. + /// + /// \param context The context to dump. + void + print_context(const model::context& context) + { + _output << "===> Execution context\n"; + + _output << F("Current directory: %s\n") % context.cwd(); + const std::map< std::string, std::string >& env = context.env(); + if (env.empty()) + _output << "No environment variables recorded\n"; + else { + _output << "Environment variables:\n"; + for (std::map< std::string, std::string >::const_iterator + iter = env.begin(); iter != env.end(); iter++) { + print_env_var(" ", (*iter).first, (*iter).second); + } + } + } + + /// Dumps a detailed view of the test case. + /// + /// \param result_iter Results iterator pointing at the test case to be + /// dumped. + void + print_test_case_and_result(const store::results_iterator& result_iter) + { + const model::test_case& test_case = + result_iter.test_program()->find(result_iter.test_case_name()); + const model::properties_map props = + test_case.get_metadata().to_properties(); + + _output << F("===> %s:%s\n") % + result_iter.test_program()->relative_path() % + result_iter.test_case_name(); + _output << F("Result: %s\n") % + cli::format_result(result_iter.result()); + _output << F("Start time: %s\n") % + result_iter.start_time().to_iso8601_in_utc(); + _output << F("End time: %s\n") % + result_iter.end_time().to_iso8601_in_utc(); + _output << F("Duration: %s\n") % + cli::format_delta(result_iter.end_time() - + result_iter.start_time()); + + _output << "\n"; + _output << "Metadata:\n"; + for (model::properties_map::const_iterator iter = props.begin(); + iter != props.end(); ++iter) { + if ((*iter).second.empty()) { + _output << F(" %s is empty\n") % (*iter).first; + } else { + _output << F(" %s = %s\n") % (*iter).first % (*iter).second; + } + } + + const std::string stdout_contents = result_iter.stdout_contents(); + if (!stdout_contents.empty()) { + _output << "\n" + << "Standard output:\n" + << stdout_contents; + } + + const std::string stderr_contents = result_iter.stderr_contents(); + if (!stderr_contents.empty()) { + _output << "\n" + << "Standard error:\n" + << stderr_contents; + } + } + + /// Counts how many results of a given type have been received. + /// + /// \param type Test result type to count results for. + /// + /// \return The number of test results with \p type. + std::size_t + count_results(const model::test_result_type type) + { + const std::map< model::test_result_type, + std::vector< result_data > >::const_iterator iter = + _results.find(type); + if (iter == _results.end()) + return 0; + else + return (*iter).second.size(); + } + + /// Prints a set of results. + /// + /// \param type Test result type to print results for. + /// \param title Title used when printing results. + void + print_results(const model::test_result_type type, + const char* title) + { + const std::map< model::test_result_type, + std::vector< result_data > >::const_iterator iter2 = + _results.find(type); + if (iter2 == _results.end()) + return; + const std::vector< result_data >& all = (*iter2).second; + + _output << F("===> %s\n") % title; + for (std::vector< result_data >::const_iterator iter = all.begin(); + iter != all.end(); iter++) { + _output << F("%s:%s -> %s [%s]\n") % (*iter).binary_path % + (*iter).test_case_name % + cli::format_result((*iter).result) % + cli::format_delta((*iter).duration); + } + } + +public: + /// Constructor for the hooks. + /// + /// \param [out] output_ Stream to which to write the report. + /// \param verbose_ Whether to include details in the output or not. + /// \param results_filters_ The result types to include in the report. + /// Cannot be empty. + /// \param results_file_ Path to the results file being read. + report_console_hooks(std::ostream& output_, const bool verbose_, + const cli::result_types& results_filters_, + const fs::path& results_file_) : + _output(output_), + _verbose(verbose_), + _results_filters(results_filters_), + _results_file(results_file_) + { + PRE(!results_filters_.empty()); + } + + /// Callback executed when the context is loaded. + /// + /// \param context The context loaded from the database. + void + got_context(const model::context& context) + { + if (_verbose) + print_context(context); + } + + /// Callback executed when a test results is found. + /// + /// \param iter Container for the test result's data. + void + got_result(store::results_iterator& iter) + { + if (!_start_time || _start_time.get() > iter.start_time()) + _start_time = iter.start_time(); + if (!_end_time || _end_time.get() < iter.end_time()) + _end_time = iter.end_time(); + + const datetime::delta duration = iter.end_time() - iter.start_time(); + + _runtime += duration; + const model::test_result result = iter.result(); + _results[result.type()].push_back( + result_data(iter.test_program()->relative_path(), + iter.test_case_name(), iter.result(), duration)); + + if (_verbose) { + // TODO(jmmv): _results_filters is a list and is small enough for + // std::find to not be an expensive operation here (probably). But + // we should be using a std::set instead. + if (std::find(_results_filters.begin(), _results_filters.end(), + iter.result().type()) != _results_filters.end()) { + print_test_case_and_result(iter); + } + } + } + + /// Prints the tests summary. + void + end(const drivers::scan_results::result& /* r */) + { + typedef std::map< model::test_result_type, const char* > types_map; + + types_map titles; + titles[model::test_result_broken] = "Broken tests"; + titles[model::test_result_expected_failure] = "Expected failures"; + titles[model::test_result_failed] = "Failed tests"; + titles[model::test_result_passed] = "Passed tests"; + titles[model::test_result_skipped] = "Skipped tests"; + + for (cli::result_types::const_iterator iter = _results_filters.begin(); + iter != _results_filters.end(); ++iter) { + const types_map::const_iterator match = titles.find(*iter); + INV_MSG(match != titles.end(), "Conditional does not match user " + "input validation in parse_types()"); + print_results((*match).first, (*match).second); + } + + const std::size_t broken = count_results(model::test_result_broken); + const std::size_t failed = count_results(model::test_result_failed); + const std::size_t passed = count_results(model::test_result_passed); + const std::size_t skipped = count_results(model::test_result_skipped); + const std::size_t xfail = count_results( + model::test_result_expected_failure); + const std::size_t total = broken + failed + passed + skipped + xfail; + + _output << "===> Summary\n"; + _output << F("Results read from %s\n") % _results_file; + _output << F("Test cases: %s total, %s skipped, %s expected failures, " + "%s broken, %s failed\n") % + total % skipped % xfail % broken % failed; + if (_verbose && _start_time) { + INV(_end_time); + _output << F("Start time: %s\n") % + _start_time.get().to_iso8601_in_utc(); + _output << F("End time: %s\n") % + _end_time.get().to_iso8601_in_utc(); + } + _output << F("Total time: %s\n") % cli::format_delta(_runtime); + } +}; + + +} // anonymous namespace + + +/// Default constructor for cmd_report. +cmd_report::cmd_report(void) : cli_command( + "report", "", 0, -1, + "Generates a report with the results of a test suite run") +{ + add_option(results_file_open_option); + add_option(cmdline::bool_option( + "verbose", "Include the execution context and the details of each test " + "case in the report")); + add_option(cmdline::path_option("output", "Path to the output file", "path", + "/dev/stdout")); + add_option(results_filter_option); +} + + +/// Entry point for the "report" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 if everything is OK, 1 if the statement is invalid or if there is +/// any other problem. +int +cmd_report::run(cmdline::ui* ui, + const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + std::auto_ptr< std::ostream > output = utils::open_ostream( + cmdline.get_option< cmdline::path_option >("output")); + + const fs::path results_file = layout::find_results( + results_file_open(cmdline)); + + const result_types types = get_result_types(cmdline); + report_console_hooks hooks(*output.get(), cmdline.has_option("verbose"), + types, results_file); + const drivers::scan_results::result result = drivers::scan_results::drive( + results_file, parse_filters(cmdline.arguments()), hooks); + + return report_unused_filters(result.unused_filters, ui) ? + EXIT_FAILURE : EXIT_SUCCESS; +} diff --git a/cli/cmd_report.hpp b/cli/cmd_report.hpp new file mode 100644 index 000000000000..3d73c592ed9b --- /dev/null +++ b/cli/cmd_report.hpp @@ -0,0 +1,54 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_report.hpp +/// Provides the cmd_report class. + +#if !defined(CLI_CMD_REPORT_HPP) +#define CLI_CMD_REPORT_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "report" subcommand. +class cmd_report : public cli_command +{ +public: + cmd_report(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_REPORT_HPP) diff --git a/cli/cmd_report_html.cpp b/cli/cmd_report_html.cpp new file mode 100644 index 000000000000..b2133a8de047 --- /dev/null +++ b/cli/cmd_report_html.cpp @@ -0,0 +1,474 @@ +// Copyright 2012 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_report_html.hpp" + +#include <algorithm> +#include <cerrno> +#include <cstdlib> +#include <set> +#include <stdexcept> + +#include "cli/common.ipp" +#include "drivers/scan_results.hpp" +#include "engine/filters.hpp" +#include "model/context.hpp" +#include "model/metadata.hpp" +#include "model/test_case.hpp" +#include "model/test_program.hpp" +#include "model/test_result.hpp" +#include "store/layout.hpp" +#include "store/read_transaction.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/datetime.hpp" +#include "utils/env.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/exceptions.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" +#include "utils/optional.ipp" +#include "utils/text/templates.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace datetime = utils::datetime; +namespace fs = utils::fs; +namespace layout = store::layout; +namespace text = utils::text; + +using utils::optional; + + +namespace { + + +/// Creates the report's top directory and fails if it exists. +/// +/// \param directory The directory to create. +/// \param force Whether to wipe an existing directory or not. +/// +/// \throw std::runtime_error If the directory already exists; this is a user +/// error that the user must correct. +/// \throw fs::error If the directory creation fails for any other reason. +static void +create_top_directory(const fs::path& directory, const bool force) +{ + if (force) { + if (fs::exists(directory)) + fs::rm_r(directory); + } + + try { + fs::mkdir(directory, 0755); + } catch (const fs::system_error& e) { + if (e.original_errno() == EEXIST) + throw std::runtime_error(F("Output directory '%s' already exists; " + "maybe use --force?") % + directory); + else + throw e; + } +} + + +/// Generates a flat unique filename for a given test case. +/// +/// \param test_program The test program for which to genereate the name. +/// \param test_case_name The test case name. +/// +/// \return A filename unique within a directory with a trailing HTML extension. +static std::string +test_case_filename(const model::test_program& test_program, + const std::string& test_case_name) +{ + static const char* special_characters = "/:"; + + std::string name = cli::format_test_case_id(test_program, test_case_name); + std::string::size_type pos = name.find_first_of(special_characters); + while (pos != std::string::npos) { + name.replace(pos, 1, "_"); + pos = name.find_first_of(special_characters, pos + 1); + } + return name + ".html"; +} + + +/// Adds a string to string map to the templates. +/// +/// \param [in,out] templates The templates to add the map to. +/// \param props The map to add to the templates. +/// \param key_vector Name of the template vector that holds the keys. +/// \param value_vector Name of the template vector that holds the values. +static void +add_map(text::templates_def& templates, const config::properties_map& props, + const std::string& key_vector, const std::string& value_vector) +{ + templates.add_vector(key_vector); + templates.add_vector(value_vector); + + for (config::properties_map::const_iterator iter = props.begin(); + iter != props.end(); ++iter) { + templates.add_to_vector(key_vector, (*iter).first); + templates.add_to_vector(value_vector, (*iter).second); + } +} + + +/// Generates an HTML report. +class html_hooks : public drivers::scan_results::base_hooks { + /// User interface object where to report progress. + cmdline::ui* _ui; + + /// The top directory in which to create the HTML files. + fs::path _directory; + + /// Collection of result types to include in the report. + const cli::result_types& _results_filters; + + /// The start time of the first test. + optional< utils::datetime::timestamp > _start_time; + + /// The end time of the last test. + optional< utils::datetime::timestamp > _end_time; + + /// The total run time of the tests. Note that we cannot subtract _end_time + /// from _start_time to compute this due to parallel execution. + utils::datetime::delta _runtime; + + /// Templates accumulator to generate the index.html file. + text::templates_def _summary_templates; + + /// Mapping of result types to the amount of tests with such result. + std::map< model::test_result_type, std::size_t > _types_count; + + /// Generates a common set of templates for all of our files. + /// + /// \return A new templates object with common parameters. + static text::templates_def + common_templates(void) + { + text::templates_def templates; + templates.add_variable("css", "report.css"); + return templates; + } + + /// Adds a test case result to the summary. + /// + /// \param test_program The test program with the test case to be added. + /// \param test_case_name Name of the test case. + /// \param result The result of the test case. + /// \param has_detail If true, the result of the test case has not been + /// filtered and therefore there exists a separate file for the test + /// with all of its information. + void + add_to_summary(const model::test_program& test_program, + const std::string& test_case_name, + const model::test_result& result, + const bool has_detail) + { + ++_types_count[result.type()]; + + if (!has_detail) + return; + + std::string test_cases_vector; + std::string test_cases_file_vector; + switch (result.type()) { + case model::test_result_broken: + test_cases_vector = "broken_test_cases"; + test_cases_file_vector = "broken_test_cases_file"; + break; + + case model::test_result_expected_failure: + test_cases_vector = "xfail_test_cases"; + test_cases_file_vector = "xfail_test_cases_file"; + break; + + case model::test_result_failed: + test_cases_vector = "failed_test_cases"; + test_cases_file_vector = "failed_test_cases_file"; + break; + + case model::test_result_passed: + test_cases_vector = "passed_test_cases"; + test_cases_file_vector = "passed_test_cases_file"; + break; + + case model::test_result_skipped: + test_cases_vector = "skipped_test_cases"; + test_cases_file_vector = "skipped_test_cases_file"; + break; + } + INV(!test_cases_vector.empty()); + INV(!test_cases_file_vector.empty()); + + _summary_templates.add_to_vector( + test_cases_vector, + cli::format_test_case_id(test_program, test_case_name)); + _summary_templates.add_to_vector( + test_cases_file_vector, + test_case_filename(test_program, test_case_name)); + } + + /// Instantiate a template to generate an HTML file in the output directory. + /// + /// \param templates The templates to use. + /// \param template_name The name of the template. This is automatically + /// searched for in the installed directory, so do not provide a path. + /// \param output_name The name of the output file. This is a basename to + /// be created within the output directory. + /// + /// \throw text::error If there is any problem applying the templates. + void + generate(const text::templates_def& templates, + const std::string& template_name, + const std::string& output_name) const + { + const fs::path miscdir(utils::getenv_with_default( + "KYUA_MISCDIR", KYUA_MISCDIR)); + const fs::path template_file = miscdir / template_name; + const fs::path output_path(_directory / output_name); + + _ui->out(F("Generating %s") % output_path); + text::instantiate(templates, template_file, output_path); + } + + /// Gets the number of tests with a given result type. + /// + /// \param type The type to be queried. + /// + /// \return The number of tests of the given type, or 0 if none have yet + /// been registered by add_to_summary(). + std::size_t + get_count(const model::test_result_type type) const + { + const std::map< model::test_result_type, std::size_t >::const_iterator + iter = _types_count.find(type); + if (iter == _types_count.end()) + return 0; + else + return (*iter).second; + } + +public: + /// Constructor for the hooks. + /// + /// \param ui_ User interface object where to report progress. + /// \param directory_ The directory in which to create the HTML files. + /// \param results_filters_ The result types to include in the report. + /// Cannot be empty. + html_hooks(cmdline::ui* ui_, const fs::path& directory_, + const cli::result_types& results_filters_) : + _ui(ui_), + _directory(directory_), + _results_filters(results_filters_), + _summary_templates(common_templates()) + { + PRE(!results_filters_.empty()); + + // Keep in sync with add_to_summary(). + _summary_templates.add_vector("broken_test_cases"); + _summary_templates.add_vector("broken_test_cases_file"); + _summary_templates.add_vector("xfail_test_cases"); + _summary_templates.add_vector("xfail_test_cases_file"); + _summary_templates.add_vector("failed_test_cases"); + _summary_templates.add_vector("failed_test_cases_file"); + _summary_templates.add_vector("passed_test_cases"); + _summary_templates.add_vector("passed_test_cases_file"); + _summary_templates.add_vector("skipped_test_cases"); + _summary_templates.add_vector("skipped_test_cases_file"); + } + + /// Callback executed when the context is loaded. + /// + /// \param context The context loaded from the database. + void + got_context(const model::context& context) + { + text::templates_def templates = common_templates(); + templates.add_variable("cwd", context.cwd().str()); + add_map(templates, context.env(), "env_var", "env_var_value"); + generate(templates, "context.html", "context.html"); + } + + /// Callback executed when a test results is found. + /// + /// \param iter Container for the test result's data. + void + got_result(store::results_iterator& iter) + { + const model::test_program_ptr test_program = iter.test_program(); + const std::string& test_case_name = iter.test_case_name(); + const model::test_result result = iter.result(); + + if (std::find(_results_filters.begin(), _results_filters.end(), + result.type()) == _results_filters.end()) { + add_to_summary(*test_program, test_case_name, result, false); + return; + } + + add_to_summary(*test_program, test_case_name, result, true); + + if (!_start_time || _start_time.get() > iter.start_time()) + _start_time = iter.start_time(); + if (!_end_time || _end_time.get() < iter.end_time()) + _end_time = iter.end_time(); + + const datetime::delta duration = iter.end_time() - iter.start_time(); + + _runtime += duration; + + text::templates_def templates = common_templates(); + templates.add_variable("test_case", + cli::format_test_case_id(*test_program, + test_case_name)); + templates.add_variable("test_program", + test_program->absolute_path().str()); + templates.add_variable("result", cli::format_result(result)); + templates.add_variable("start_time", + iter.start_time().to_iso8601_in_utc()); + templates.add_variable("end_time", + iter.end_time().to_iso8601_in_utc()); + templates.add_variable("duration", cli::format_delta(duration)); + + const model::test_case& test_case = test_program->find(test_case_name); + add_map(templates, test_case.get_metadata().to_properties(), + "metadata_var", "metadata_value"); + + { + const std::string stdout_text = iter.stdout_contents(); + if (!stdout_text.empty()) + templates.add_variable("stdout", stdout_text); + } + { + const std::string stderr_text = iter.stderr_contents(); + if (!stderr_text.empty()) + templates.add_variable("stderr", stderr_text); + } + + generate(templates, "test_result.html", + test_case_filename(*test_program, test_case_name)); + } + + /// Writes the index.html file in the output directory. + /// + /// This should only be called once all the processing has been done; + /// i.e. when the scan_results driver returns. + void + write_summary(void) + { + const std::size_t n_passed = get_count(model::test_result_passed); + const std::size_t n_failed = get_count(model::test_result_failed); + const std::size_t n_skipped = get_count(model::test_result_skipped); + const std::size_t n_xfail = get_count( + model::test_result_expected_failure); + const std::size_t n_broken = get_count(model::test_result_broken); + + const std::size_t n_bad = n_broken + n_failed; + + if (_start_time) { + INV(_end_time); + _summary_templates.add_variable( + "start_time", _start_time.get().to_iso8601_in_utc()); + _summary_templates.add_variable( + "end_time", _end_time.get().to_iso8601_in_utc()); + } else { + _summary_templates.add_variable("start_time", "No tests run"); + _summary_templates.add_variable("end_time", "No tests run"); + } + _summary_templates.add_variable("duration", + cli::format_delta(_runtime)); + _summary_templates.add_variable("passed_tests_count", + F("%s") % n_passed); + _summary_templates.add_variable("failed_tests_count", + F("%s") % n_failed); + _summary_templates.add_variable("skipped_tests_count", + F("%s") % n_skipped); + _summary_templates.add_variable("xfail_tests_count", + F("%s") % n_xfail); + _summary_templates.add_variable("broken_tests_count", + F("%s") % n_broken); + _summary_templates.add_variable("bad_tests_count", F("%s") % n_bad); + + generate(text::templates_def(), "report.css", "report.css"); + generate(_summary_templates, "index.html", "index.html"); + } +}; + + +} // anonymous namespace + + +/// Default constructor for cmd_report_html. +cli::cmd_report_html::cmd_report_html(void) : cli_command( + "report-html", "", 0, 0, + "Generates an HTML report with the result of a test suite run") +{ + add_option(results_file_open_option); + add_option(cmdline::bool_option( + "force", "Wipe the output directory before generating the new report; " + "use care")); + add_option(cmdline::path_option( + "output", "The directory in which to store the HTML files", + "path", "html")); + add_option(cmdline::list_option( + "results-filter", "Comma-separated list of result types to include in " + "the report", "types", "skipped,xfail,broken,failed")); +} + + +/// Entry point for the "report-html" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 if everything is OK, 1 if the statement is invalid or if there is +/// any other problem. +int +cli::cmd_report_html::run(cmdline::ui* ui, + const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + const result_types types = get_result_types(cmdline); + + const fs::path results_file = layout::find_results( + results_file_open(cmdline)); + + const fs::path directory = + cmdline.get_option< cmdline::path_option >("output"); + create_top_directory(directory, cmdline.has_option("force")); + html_hooks hooks(ui, directory, types); + drivers::scan_results::drive(results_file, + std::set< engine::test_filter >(), + hooks); + hooks.write_summary(); + + return EXIT_SUCCESS; +} diff --git a/cli/cmd_report_html.hpp b/cli/cmd_report_html.hpp new file mode 100644 index 000000000000..fadc138293ad --- /dev/null +++ b/cli/cmd_report_html.hpp @@ -0,0 +1,55 @@ +// Copyright 2012 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_report_html.hpp +/// Provides the cmd_report_html class. + +#if !defined(CLI_CMD_REPORT_HTML_HPP) +#define CLI_CMD_REPORT_HTML_HPP + +#include "cli/common.hpp" +#include "utils/cmdline/ui_fwd.hpp" + +namespace cli { + + +/// Implementation of the "report-html" subcommand. +class cmd_report_html : public cli_command +{ +public: + cmd_report_html(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_REPORT_HTML_HPP) diff --git a/cli/cmd_report_junit.cpp b/cli/cmd_report_junit.cpp new file mode 100644 index 000000000000..c4846c8795f2 --- /dev/null +++ b/cli/cmd_report_junit.cpp @@ -0,0 +1,89 @@ +// Copyright 2014 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_report_junit.hpp" + +#include <cstddef> +#include <cstdlib> +#include <set> + +#include "cli/common.ipp" +#include "drivers/report_junit.hpp" +#include "drivers/scan_results.hpp" +#include "engine/filters.hpp" +#include "store/layout.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/defs.hpp" +#include "utils/optional.ipp" +#include "utils/stream.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; +namespace layout = store::layout; + +using cli::cmd_report_junit; +using utils::optional; + + +/// Default constructor for cmd_report. +cmd_report_junit::cmd_report_junit(void) : cli_command( + "report-junit", "", 0, 0, + "Generates a JUnit report with the result of a test suite run") +{ + add_option(results_file_open_option); + add_option(cmdline::path_option("output", "Path to the output file", "path", + "/dev/stdout")); +} + + +/// Entry point for the "report" subcommand. +/// +/// \param cmdline Representation of the command line to the subcommand. +/// +/// \return 0 if everything is OK, 1 if the statement is invalid or if there is +/// any other problem. +int +cmd_report_junit::run(cmdline::ui* /* ui */, + const cmdline::parsed_cmdline& cmdline, + const config::tree& /* user_config */) +{ + const fs::path results_file = layout::find_results( + results_file_open(cmdline)); + + std::auto_ptr< std::ostream > output = utils::open_ostream( + cmdline.get_option< cmdline::path_option >("output")); + + drivers::report_junit_hooks hooks(*output.get()); + drivers::scan_results::drive(results_file, + std::set< engine::test_filter >(), + hooks); + + return EXIT_SUCCESS; +} diff --git a/cli/cmd_report_junit.hpp b/cli/cmd_report_junit.hpp new file mode 100644 index 000000000000..1dc0bb731645 --- /dev/null +++ b/cli/cmd_report_junit.hpp @@ -0,0 +1,54 @@ +// Copyright 2014 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_report_junit.hpp +/// Provides the cmd_report_junit class. + +#if !defined(CLI_CMD_REPORT_JUNIT_HPP) +#define CLI_CMD_REPORT_JUNIT_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "report-junit" subcommand. +class cmd_report_junit : public cli_command +{ +public: + cmd_report_junit(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_REPORT_JUNIT_HPP) diff --git a/cli/cmd_test.cpp b/cli/cmd_test.cpp new file mode 100644 index 000000000000..cfaeec9b74cc --- /dev/null +++ b/cli/cmd_test.cpp @@ -0,0 +1,186 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_test.hpp" + +#include <cstdlib> + +#include "cli/common.ipp" +#include "drivers/run_tests.hpp" +#include "model/test_program.hpp" +#include "model/test_result.hpp" +#include "store/layout.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/config/tree.ipp" +#include "utils/datetime.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/path.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace datetime = utils::datetime; +namespace fs = utils::fs; +namespace layout = store::layout; + +using cli::cmd_test; + + +namespace { + + +/// Hooks to print a progress report of the execution of the tests. +class print_hooks : public drivers::run_tests::base_hooks { + /// Object to interact with the I/O of the program. + cmdline::ui* _ui; + + /// Whether the tests are executed in parallel or not. + bool _parallel; + +public: + /// The amount of positive test results found so far. + unsigned long good_count; + + /// The amount of negative test results found so far. + unsigned long bad_count; + + /// Constructor for the hooks. + /// + /// \param ui_ Object to interact with the I/O of the program. + /// \param parallel_ True if we are executing more than one test at once. + print_hooks(cmdline::ui* ui_, const bool parallel_) : + _ui(ui_), + _parallel(parallel_), + good_count(0), + bad_count(0) + { + } + + /// Called when the processing of a test case begins. + /// + /// \param test_program The test program containing the test case. + /// \param test_case_name The name of the test case being executed. + virtual void + got_test_case(const model::test_program& test_program, + const std::string& test_case_name) + { + if (!_parallel) { + _ui->out(F("%s -> ") % + cli::format_test_case_id(test_program, test_case_name), + false); + } + } + + /// Called when a result of a test case becomes available. + /// + /// \param test_program The test program containing the test case. + /// \param test_case_name The name of the test case being executed. + /// \param result The result of the execution of the test case. + /// \param duration The time it took to run the test. + virtual void + got_result(const model::test_program& test_program, + const std::string& test_case_name, + const model::test_result& result, + const datetime::delta& duration) + { + if (_parallel) { + _ui->out(F("%s -> ") % + cli::format_test_case_id(test_program, test_case_name), + false); + } + _ui->out(F("%s [%s]") % cli::format_result(result) % + cli::format_delta(duration)); + if (result.good()) + good_count++; + else + bad_count++; + } +}; + + +} // anonymous namespace + + +/// Default constructor for cmd_test. +cmd_test::cmd_test(void) : cli_command( + "test", "[test-program ...]", 0, -1, "Run tests") +{ + add_option(build_root_option); + add_option(kyuafile_option); + add_option(results_file_create_option); +} + + +/// Entry point for the "test" subcommand. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param cmdline Representation of the command line to the subcommand. +/// \param user_config The runtime configuration of the program. +/// +/// \return 0 if all tests passed, 1 otherwise. +int +cmd_test::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, + const config::tree& user_config) +{ + const layout::results_id_file_pair results = layout::new_db( + results_file_create(cmdline), kyuafile_path(cmdline).branch_path()); + + const bool parallel = (user_config.lookup< config::positive_int_node >( + "parallelism") > 1); + + print_hooks hooks(ui, parallel); + const drivers::run_tests::result result = drivers::run_tests::drive( + kyuafile_path(cmdline), build_root_path(cmdline), results.second, + parse_filters(cmdline.arguments()), user_config, hooks); + + int exit_code; + if (hooks.good_count > 0 || hooks.bad_count > 0) { + ui->out(""); + if (!results.first.empty()) { + ui->out(F("Results file id is %s") % results.first); + } + ui->out(F("Results saved to %s") % results.second); + ui->out(""); + + ui->out(F("%s/%s passed (%s failed)") % hooks.good_count % + (hooks.good_count + hooks.bad_count) % hooks.bad_count); + + exit_code = (hooks.bad_count == 0 ? EXIT_SUCCESS : EXIT_FAILURE); + } else { + // TODO(jmmv): Delete created empty file; it's useless! + if (!results.first.empty()) { + ui->out(F("Results file id is %s") % results.first); + } + ui->out(F("Results saved to %s") % results.second); + exit_code = EXIT_SUCCESS; + } + + return report_unused_filters(result.unused_filters, ui) ? + EXIT_FAILURE : exit_code; +} diff --git a/cli/cmd_test.hpp b/cli/cmd_test.hpp new file mode 100644 index 000000000000..22d8422cb293 --- /dev/null +++ b/cli/cmd_test.hpp @@ -0,0 +1,54 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/cmd_test.hpp +/// Provides the cmd_test class. + +#if !defined(CLI_CMD_TEST_HPP) +#define CLI_CMD_TEST_HPP + +#include "cli/common.hpp" + +namespace cli { + + +/// Implementation of the "test" subcommand. +class cmd_test : public cli_command +{ +public: + cmd_test(void); + + int run(utils::cmdline::ui*, const utils::cmdline::parsed_cmdline&, + const utils::config::tree&); +}; + + +} // namespace cli + + +#endif // !defined(CLI_CMD_TEST_HPP) diff --git a/cli/cmd_test_test.cpp b/cli/cmd_test_test.cpp new file mode 100644 index 000000000000..fb623323dd86 --- /dev/null +++ b/cli/cmd_test_test.cpp @@ -0,0 +1,63 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/cmd_test.hpp" + +#include <atf-c++.hpp> + +#include "cli/common.ipp" +#include "engine/config.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/config/tree.ipp" + +namespace cmdline = utils::cmdline; + + +ATF_TEST_CASE_WITHOUT_HEAD(invalid_filter); +ATF_TEST_CASE_BODY(invalid_filter) +{ + cmdline::args_vector args; + args.push_back("test"); + args.push_back("correct"); + args.push_back("incorrect:"); + + cli::cmd_test cmd; + cmdline::ui_mock ui; + ATF_REQUIRE_THROW_RE(cmdline::error, "Test case.*'incorrect:'.*empty", + cmd.main(&ui, args, engine::default_config())); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, invalid_filter); +} diff --git a/cli/common.cpp b/cli/common.cpp new file mode 100644 index 000000000000..dbb7f12f18e0 --- /dev/null +++ b/cli/common.cpp @@ -0,0 +1,411 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/common.hpp" + +#include <algorithm> +#include <fstream> +#include <iostream> +#include <stdexcept> + +#include "engine/filters.hpp" +#include "model/test_program.hpp" +#include "model/test_result.hpp" +#include "store/layout.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/datetime.hpp" +#include "utils/env.hpp" +#include "utils/format/macros.hpp" +#include "utils/logging/macros.hpp" +#include "utils/fs/exceptions.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" +#include "utils/optional.ipp" +#include "utils/sanity.hpp" + +#if defined(HAVE_CONFIG_H) +# include "config.h" +#endif + +namespace cmdline = utils::cmdline; +namespace datetime = utils::datetime; +namespace fs = utils::fs; +namespace layout = store::layout; + +using utils::none; +using utils::optional; + + +/// Standard definition of the option to specify the build root. +const cmdline::path_option cli::build_root_option( + "build-root", + "Path to the built test programs, if different from the location of the " + "Kyuafile scripts", + "path"); + + +/// Standard definition of the option to specify a Kyuafile. +const cmdline::path_option cli::kyuafile_option( + 'k', "kyuafile", + "Path to the test suite definition", + "file", "Kyuafile"); + + +/// Standard definition of the option to specify filters on test results. +const cmdline::list_option cli::results_filter_option( + "results-filter", "Comma-separated list of result types to include in " + "the report", "types", "skipped,xfail,broken,failed"); + + +/// Standard definition of the option to specify the results file. +/// +/// TODO(jmmv): Should support a git-like syntax to go back in time, like +/// --results-file=LATEST^N where N indicates how many runs to go back to. +const cmdline::string_option cli::results_file_create_option( + 'r', "results-file", + "Path to the results file to create; if left to the default value, the " + "name of the file is automatically computed for the current test suite", + "file", layout::results_auto_create_name); + + +/// Standard definition of the option to specify the results file. +/// +/// TODO(jmmv): Should support a git-like syntax to go back in time, like +/// --results-file=LATEST^N where N indicates how many runs to go back to. +const cmdline::string_option cli::results_file_open_option( + 'r', "results-file", + "Path to the results file to open or the identifier of the current test " + "suite or a previous results file for automatic lookup; if left to the " + "default value, uses the current directory as the test suite name", + "file", layout::results_auto_open_name); + + +namespace { + + +/// Gets the path to the historical database if it exists. +/// +/// TODO(jmmv): This function should go away. It only exists as a temporary +/// transitional path to force the use of the stale ~/.kyua/store.db if it +/// exists. +/// +/// \return A path if the file is found; none otherwise. +static optional< fs::path > +get_historical_db(void) +{ + optional< fs::path > home = utils::get_home(); + if (home) { + const fs::path old_db = home.get() / ".kyua/store.db"; + if (fs::exists(old_db)) { + if (old_db.is_absolute()) + return utils::make_optional(old_db); + else + return utils::make_optional(old_db.to_absolute()); + } else { + return none; + } + } else { + return none; + } +} + + +/// Converts a set of result type names to identifiers. +/// +/// \param names The collection of names to process; may be empty. +/// +/// \return The result type identifiers corresponding to the input names. +/// +/// \throw std::runtime_error If any name in the input names is invalid. +static cli::result_types +parse_types(const std::vector< std::string >& names) +{ + typedef std::map< std::string, model::test_result_type > types_map; + types_map valid_types; + valid_types["broken"] = model::test_result_broken; + valid_types["failed"] = model::test_result_failed; + valid_types["passed"] = model::test_result_passed; + valid_types["skipped"] = model::test_result_skipped; + valid_types["xfail"] = model::test_result_expected_failure; + + cli::result_types types; + for (std::vector< std::string >::const_iterator iter = names.begin(); + iter != names.end(); ++iter) { + const types_map::const_iterator match = valid_types.find(*iter); + if (match == valid_types.end()) + throw std::runtime_error(F("Unknown result type '%s'") % *iter); + else + types.push_back((*match).second); + } + return types; +} + + +} // anonymous namespace + + +/// Gets the path to the build root, if any. +/// +/// This is just syntactic sugar to simplify quierying the 'build_root_option'. +/// +/// \param cmdline The parsed command line. +/// +/// \return The path to the build root, if specified; none otherwise. +optional< fs::path > +cli::build_root_path(const cmdline::parsed_cmdline& cmdline) +{ + optional< fs::path > build_root; + if (cmdline.has_option(build_root_option.long_name())) + build_root = cmdline.get_option< cmdline::path_option >( + build_root_option.long_name()); + return build_root; +} + + +/// Gets the path to the Kyuafile to be loaded. +/// +/// This is just syntactic sugar to simplify quierying the 'kyuafile_option'. +/// +/// \param cmdline The parsed command line. +/// +/// \return The path to the Kyuafile to be loaded. +fs::path +cli::kyuafile_path(const cmdline::parsed_cmdline& cmdline) +{ + return cmdline.get_option< cmdline::path_option >( + kyuafile_option.long_name()); +} + + +/// Gets the value of the results-file flag for the creation of a new file. +/// +/// \param cmdline The parsed command line from which to extract any possible +/// override for the location of the database via the --results-file flag. +/// +/// \return The path to the database to be used. +/// +/// \throw cmdline::error If the value passed to the flag is invalid. +std::string +cli::results_file_create(const cmdline::parsed_cmdline& cmdline) +{ + std::string results_file = cmdline.get_option< cmdline::string_option >( + results_file_create_option.long_name()); + if (results_file == results_file_create_option.default_value()) { + const optional< fs::path > historical_db = get_historical_db(); + if (historical_db) + results_file = historical_db.get().str(); + } else { + try { + (void)fs::path(results_file); + } catch (const fs::error& e) { + throw cmdline::usage_error(F("Invalid value passed to --%s") % + results_file_create_option.long_name()); + } + } + return results_file; +} + + +/// Gets the value of the results-file flag for the lookup of the file. +/// +/// \param cmdline The parsed command line from which to extract any possible +/// override for the location of the database via the --results-file flag. +/// +/// \return The path to the database to be used. +/// +/// \throw cmdline::error If the value passed to the flag is invalid. +std::string +cli::results_file_open(const cmdline::parsed_cmdline& cmdline) +{ + std::string results_file = cmdline.get_option< cmdline::string_option >( + results_file_open_option.long_name()); + if (results_file == results_file_open_option.default_value()) { + const optional< fs::path > historical_db = get_historical_db(); + if (historical_db) + results_file = historical_db.get().str(); + } else { + try { + (void)fs::path(results_file); + } catch (const fs::error& e) { + throw cmdline::usage_error(F("Invalid value passed to --%s") % + results_file_open_option.long_name()); + } + } + return results_file; +} + + +/// Gets the filters for the result types. +/// +/// \param cmdline The parsed command line. +/// +/// \return A collection of result types to be used for filtering. +/// +/// \throw std::runtime_error If any of the user-provided filters is invalid. +cli::result_types +cli::get_result_types(const utils::cmdline::parsed_cmdline& cmdline) +{ + result_types types = parse_types( + cmdline.get_option< cmdline::list_option >("results-filter")); + if (types.empty()) { + types.push_back(model::test_result_passed); + types.push_back(model::test_result_skipped); + types.push_back(model::test_result_expected_failure); + types.push_back(model::test_result_broken); + types.push_back(model::test_result_failed); + } + return types; +} + + +/// Parses a set of command-line arguments to construct test filters. +/// +/// \param args The command-line arguments representing test filters. +/// +/// \return A set of test filters. +/// +/// \throw cmdline:error If any of the arguments is invalid, or if they +/// represent a non-disjoint collection of filters. +std::set< engine::test_filter > +cli::parse_filters(const cmdline::args_vector& args) +{ + std::set< engine::test_filter > filters; + + try { + for (cmdline::args_vector::const_iterator iter = args.begin(); + iter != args.end(); iter++) { + const engine::test_filter filter(engine::test_filter::parse(*iter)); + if (filters.find(filter) != filters.end()) + throw cmdline::error(F("Duplicate filter '%s'") % filter.str()); + filters.insert(filter); + } + check_disjoint_filters(filters); + } catch (const std::runtime_error& e) { + throw cmdline::error(e.what()); + } + + return filters; +} + + +/// Reports the filters that have not matched any tests as errors. +/// +/// \param unused The collection of unused filters to report. +/// \param ui The user interface object through which errors are to be reported. +/// +/// \return True if there are any unused filters. The caller should report this +/// as an error to the user by means of a non-successful exit code. +bool +cli::report_unused_filters(const std::set< engine::test_filter >& unused, + cmdline::ui* ui) +{ + for (std::set< engine::test_filter >::const_iterator iter = unused.begin(); + iter != unused.end(); iter++) { + cmdline::print_warning(ui, F("No test cases matched by the filter " + "'%s'.") % (*iter).str()); + } + + return !unused.empty(); +} + + +/// Formats a time delta for user presentation. +/// +/// \param delta The time delta to format. +/// +/// \return A user-friendly representation of the time delta. +std::string +cli::format_delta(const datetime::delta& delta) +{ + return F("%.3ss") % (delta.seconds + (delta.useconds / 1000000.0)); +} + + +/// Formats a test case result for user presentation. +/// +/// \param result The result to format. +/// +/// \return A user-friendly representation of the result. +std::string +cli::format_result(const model::test_result& result) +{ + std::string text; + + switch (result.type()) { + case model::test_result_broken: text = "broken"; break; + case model::test_result_expected_failure: text = "expected_failure"; break; + case model::test_result_failed: text = "failed"; break; + case model::test_result_passed: text = "passed"; break; + case model::test_result_skipped: text = "skipped"; break; + } + INV(!text.empty()); + + if (!result.reason().empty()) + text += ": " + result.reason(); + + return text; +} + + +/// Formats the identifier of a test case for user presentation. +/// +/// \param test_program The test program containing the test case. +/// \param test_case_name The name of the test case. +/// +/// \return A string representing the test case uniquely within a test suite. +std::string +cli::format_test_case_id(const model::test_program& test_program, + const std::string& test_case_name) +{ + return F("%s:%s") % test_program.relative_path() % test_case_name; +} + + +/// Formats a filter using the same syntax of a test case. +/// +/// \param test_filter The filter to format. +/// +/// \return A string representing the test filter. +std::string +cli::format_test_case_id(const engine::test_filter& test_filter) +{ + return F("%s:%s") % test_filter.test_program % test_filter.test_case; +} + + +/// Prints the version header information to the interface output. +/// +/// \param ui Interface to which to write the version details. +void +cli::write_version_header(utils::cmdline::ui* ui) +{ + ui->out(PACKAGE " (" PACKAGE_NAME ") " PACKAGE_VERSION); +} diff --git a/cli/common.hpp b/cli/common.hpp new file mode 100644 index 000000000000..15a7e9fa3344 --- /dev/null +++ b/cli/common.hpp @@ -0,0 +1,104 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/common.hpp +/// Utility functions to implement CLI subcommands. + +#if !defined(CLI_COMMON_HPP) +#define CLI_COMMON_HPP + +#include <memory> +#include <set> +#include <vector> + +#include "engine/filters_fwd.hpp" +#include "model/test_program_fwd.hpp" +#include "model/test_result.hpp" +#include "utils/cmdline/base_command.hpp" +#include "utils/cmdline/options_fwd.hpp" +#include "utils/cmdline/parser_fwd.hpp" +#include "utils/cmdline/ui_fwd.hpp" +#include "utils/config/tree_fwd.hpp" +#include "utils/datetime_fwd.hpp" +#include "utils/fs/path_fwd.hpp" +#include "utils/optional_fwd.hpp" + +namespace cli { + + +extern const utils::cmdline::path_option build_root_option; +extern const utils::cmdline::path_option kyuafile_option; +extern const utils::cmdline::string_option results_file_create_option; +extern const utils::cmdline::string_option results_file_open_option; +extern const utils::cmdline::list_option results_filter_option; +extern const utils::cmdline::property_option variable_option; + + +/// Base type for commands defined in the cli module. +/// +/// All commands in Kyua receive a configuration object as their runtime +/// data parameter because the configuration file applies to all the +/// commands. +typedef utils::cmdline::base_command< utils::config::tree > cli_command; + + +/// Scoped, strictly owned pointer to a cli_command. +typedef std::auto_ptr< cli_command > cli_command_ptr; + + +/// Collection of result types. +/// +/// This is a vector rather than a set because we want to respect the order in +/// which the user provided the types. +typedef std::vector< model::test_result_type > result_types; + + +utils::optional< utils::fs::path > build_root_path( + const utils::cmdline::parsed_cmdline&); +utils::fs::path kyuafile_path(const utils::cmdline::parsed_cmdline&); +std::string results_file_create(const utils::cmdline::parsed_cmdline&); +std::string results_file_open(const utils::cmdline::parsed_cmdline&); +result_types get_result_types(const utils::cmdline::parsed_cmdline&); + +std::set< engine::test_filter > parse_filters( + const utils::cmdline::args_vector&); +bool report_unused_filters(const std::set< engine::test_filter >&, + utils::cmdline::ui*); + +std::string format_delta(const utils::datetime::delta&); +std::string format_result(const model::test_result&); +std::string format_test_case_id(const model::test_program&, const std::string&); +std::string format_test_case_id(const engine::test_filter&); + + +void write_version_header(utils::cmdline::ui*); + + +} // namespace cli + +#endif // !defined(CLI_COMMON_HPP) diff --git a/cli/common.ipp b/cli/common.ipp new file mode 100644 index 000000000000..c0de4e44ccc1 --- /dev/null +++ b/cli/common.ipp @@ -0,0 +1,30 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/common.hpp" +#include "utils/cmdline/base_command.ipp" diff --git a/cli/common_test.cpp b/cli/common_test.cpp new file mode 100644 index 000000000000..05bb187ace22 --- /dev/null +++ b/cli/common_test.cpp @@ -0,0 +1,488 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/common.hpp" + +#include <fstream> + +#include <atf-c++.hpp> + +#include "engine/exceptions.hpp" +#include "engine/filters.hpp" +#include "model/metadata.hpp" +#include "model/test_program.hpp" +#include "model/test_result.hpp" +#include "store/layout.hpp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/globals.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/datetime.hpp" +#include "utils/env.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/exceptions.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" +#include "utils/optional.ipp" +#include "utils/sanity.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace datetime = utils::datetime; +namespace fs = utils::fs; +namespace layout = store::layout; + +using utils::optional; + + +namespace { + + +/// Syntactic sugar to instantiate engine::test_filter objects. +/// +/// \param test_program Test program. +/// \param test_case Test case. +/// +/// \return A \code test_filter \endcode object, based on \p test_program and +/// \p test_case. +inline engine::test_filter +mkfilter(const char* test_program, const char* test_case) +{ + return engine::test_filter(fs::path(test_program), test_case); +} + + +} // anonymous namespace + + +ATF_TEST_CASE_WITHOUT_HEAD(build_root_path__default); +ATF_TEST_CASE_BODY(build_root_path__default) +{ + std::map< std::string, std::vector< std::string > > options; + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE(!cli::build_root_path(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(build_root_path__explicit); +ATF_TEST_CASE_BODY(build_root_path__explicit) +{ + std::map< std::string, std::vector< std::string > > options; + options["build-root"].push_back("/my//path"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE(cli::build_root_path(mock_cmdline)); + ATF_REQUIRE_EQ("/my/path", cli::build_root_path(mock_cmdline).get().str()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(kyuafile_path__default); +ATF_TEST_CASE_BODY(kyuafile_path__default) +{ + std::map< std::string, std::vector< std::string > > options; + options["kyuafile"].push_back(cli::kyuafile_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_EQ(cli::kyuafile_option.default_value(), + cli::kyuafile_path(mock_cmdline).str()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(kyuafile_path__explicit); +ATF_TEST_CASE_BODY(kyuafile_path__explicit) +{ + std::map< std::string, std::vector< std::string > > options; + options["kyuafile"].push_back("/my//path"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_EQ("/my/path", cli::kyuafile_path(mock_cmdline).str()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(result_types__default); +ATF_TEST_CASE_BODY(result_types__default) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-filter"].push_back( + cli::results_filter_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + cli::result_types exp_types; + exp_types.push_back(model::test_result_skipped); + exp_types.push_back(model::test_result_expected_failure); + exp_types.push_back(model::test_result_broken); + exp_types.push_back(model::test_result_failed); + ATF_REQUIRE(exp_types == cli::get_result_types(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(result_types__empty); +ATF_TEST_CASE_BODY(result_types__empty) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-filter"].push_back(""); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + cli::result_types exp_types; + exp_types.push_back(model::test_result_passed); + exp_types.push_back(model::test_result_skipped); + exp_types.push_back(model::test_result_expected_failure); + exp_types.push_back(model::test_result_broken); + exp_types.push_back(model::test_result_failed); + ATF_REQUIRE(exp_types == cli::get_result_types(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(result_types__explicit__all); +ATF_TEST_CASE_BODY(result_types__explicit__all) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-filter"].push_back("passed,skipped,xfail,broken,failed"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + cli::result_types exp_types; + exp_types.push_back(model::test_result_passed); + exp_types.push_back(model::test_result_skipped); + exp_types.push_back(model::test_result_expected_failure); + exp_types.push_back(model::test_result_broken); + exp_types.push_back(model::test_result_failed); + ATF_REQUIRE(exp_types == cli::get_result_types(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(result_types__explicit__some); +ATF_TEST_CASE_BODY(result_types__explicit__some) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-filter"].push_back("skipped,broken"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + cli::result_types exp_types; + exp_types.push_back(model::test_result_skipped); + exp_types.push_back(model::test_result_broken); + ATF_REQUIRE(exp_types == cli::get_result_types(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(result_types__explicit__invalid); +ATF_TEST_CASE_BODY(result_types__explicit__invalid) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-filter"].push_back("skipped,foo,broken"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_THROW_RE(std::runtime_error, "Unknown result type 'foo'", + cli::get_result_types(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(results_file_create__default__new); +ATF_TEST_CASE_BODY(results_file_create__default__new) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-file"].push_back( + cli::results_file_create_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const fs::path home("homedir"); + utils::setenv("HOME", home.str()); + + ATF_REQUIRE_EQ(cli::results_file_create_option.default_value(), + cli::results_file_create(mock_cmdline)); + ATF_REQUIRE(!fs::exists(home / ".kyua")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(results_file_create__default__historical); +ATF_TEST_CASE_BODY(results_file_create__default__historical) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-file"].push_back( + cli::results_file_create_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const fs::path home("homedir"); + utils::setenv("HOME", home.str()); + fs::mkdir_p(fs::path("homedir/.kyua"), 0755); + atf::utils::create_file("homedir/.kyua/store.db", "fake store"); + + ATF_REQUIRE_EQ(fs::path("homedir/.kyua/store.db").to_absolute(), + fs::path(cli::results_file_create(mock_cmdline))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(results_file_create__explicit); +ATF_TEST_CASE_BODY(results_file_create__explicit) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-file"].push_back("/my//path/f.db"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_EQ("/my//path/f.db", + cli::results_file_create(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(results_file_open__default__latest); +ATF_TEST_CASE_BODY(results_file_open__default__latest) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-file"].push_back( + cli::results_file_open_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const fs::path home("homedir"); + utils::setenv("HOME", home.str()); + + ATF_REQUIRE_EQ(cli::results_file_open_option.default_value(), + cli::results_file_open(mock_cmdline)); + ATF_REQUIRE(!fs::exists(home / ".kyua")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(results_file_open__default__historical); +ATF_TEST_CASE_BODY(results_file_open__default__historical) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-file"].push_back( + cli::results_file_open_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const fs::path home("homedir"); + utils::setenv("HOME", home.str()); + fs::mkdir_p(fs::path("homedir/.kyua"), 0755); + atf::utils::create_file("homedir/.kyua/store.db", "fake store"); + + ATF_REQUIRE_EQ(fs::path("homedir/.kyua/store.db").to_absolute(), + fs::path(cli::results_file_open(mock_cmdline))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(results_file_open__explicit); +ATF_TEST_CASE_BODY(results_file_open__explicit) +{ + std::map< std::string, std::vector< std::string > > options; + options["results-file"].push_back("/my//path/f.db"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_EQ("/my//path/f.db", cli::results_file_open(mock_cmdline)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(parse_filters__none); +ATF_TEST_CASE_BODY(parse_filters__none) +{ + const cmdline::args_vector args; + const std::set< engine::test_filter > filters = cli::parse_filters(args); + ATF_REQUIRE(filters.empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(parse_filters__ok); +ATF_TEST_CASE_BODY(parse_filters__ok) +{ + cmdline::args_vector args; + args.push_back("foo"); + args.push_back("bar/baz"); + args.push_back("other:abc"); + args.push_back("other:bcd"); + const std::set< engine::test_filter > filters = cli::parse_filters(args); + + std::set< engine::test_filter > exp_filters; + exp_filters.insert(mkfilter("foo", "")); + exp_filters.insert(mkfilter("bar/baz", "")); + exp_filters.insert(mkfilter("other", "abc")); + exp_filters.insert(mkfilter("other", "bcd")); + + ATF_REQUIRE(exp_filters == filters); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(parse_filters__duplicate); +ATF_TEST_CASE_BODY(parse_filters__duplicate) +{ + cmdline::args_vector args; + args.push_back("foo/bar//baz"); + args.push_back("hello/world:yes"); + args.push_back("foo//bar/baz"); + ATF_REQUIRE_THROW_RE(cmdline::error, "Duplicate.*'foo/bar/baz'", + cli::parse_filters(args)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(parse_filters__nondisjoint); +ATF_TEST_CASE_BODY(parse_filters__nondisjoint) +{ + cmdline::args_vector args; + args.push_back("foo/bar"); + args.push_back("hello/world:yes"); + args.push_back("foo/bar:baz"); + ATF_REQUIRE_THROW_RE(cmdline::error, "'foo/bar'.*'foo/bar:baz'.*disjoint", + cli::parse_filters(args)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(report_unused_filters__none); +ATF_TEST_CASE_BODY(report_unused_filters__none) +{ + std::set< engine::test_filter > unused; + + cmdline::ui_mock ui; + ATF_REQUIRE(!cli::report_unused_filters(unused, &ui)); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(report_unused_filters__some); +ATF_TEST_CASE_BODY(report_unused_filters__some) +{ + std::set< engine::test_filter > unused; + unused.insert(mkfilter("a/b", "")); + unused.insert(mkfilter("hey/d", "yes")); + + cmdline::ui_mock ui; + cmdline::init("progname"); + ATF_REQUIRE(cli::report_unused_filters(unused, &ui)); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE_EQ(2, ui.err_log().size()); + ATF_REQUIRE( atf::utils::grep_collection("No.*matched.*'a/b'", + ui.err_log())); + ATF_REQUIRE( atf::utils::grep_collection("No.*matched.*'hey/d:yes'", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_delta); +ATF_TEST_CASE_BODY(format_delta) +{ + ATF_REQUIRE_EQ("0.000s", cli::format_delta(datetime::delta())); + ATF_REQUIRE_EQ("0.012s", cli::format_delta(datetime::delta(0, 12300))); + ATF_REQUIRE_EQ("0.999s", cli::format_delta(datetime::delta(0, 999000))); + ATF_REQUIRE_EQ("51.321s", cli::format_delta(datetime::delta(51, 321000))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_result__no_reason); +ATF_TEST_CASE_BODY(format_result__no_reason) +{ + ATF_REQUIRE_EQ("passed", cli::format_result( + model::test_result(model::test_result_passed))); + ATF_REQUIRE_EQ("failed", cli::format_result( + model::test_result(model::test_result_failed))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_result__with_reason); +ATF_TEST_CASE_BODY(format_result__with_reason) +{ + ATF_REQUIRE_EQ("broken: Something", cli::format_result( + model::test_result(model::test_result_broken, "Something"))); + ATF_REQUIRE_EQ("expected_failure: A B C", cli::format_result( + model::test_result(model::test_result_expected_failure, "A B C"))); + ATF_REQUIRE_EQ("failed: More text", cli::format_result( + model::test_result(model::test_result_failed, "More text"))); + ATF_REQUIRE_EQ("skipped: Bye", cli::format_result( + model::test_result(model::test_result_skipped, "Bye"))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_test_case_id__test_case); +ATF_TEST_CASE_BODY(format_test_case_id__test_case) +{ + const model::test_program test_program = model::test_program_builder( + "mock", fs::path("foo/bar/baz"), fs::path("unused-root"), + "unused-suite-name") + .add_test_case("abc") + .build(); + ATF_REQUIRE_EQ("foo/bar/baz:abc", + cli::format_test_case_id(test_program, "abc")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(format_test_case_id__test_filter); +ATF_TEST_CASE_BODY(format_test_case_id__test_filter) +{ + const engine::test_filter filter(fs::path("foo/bar"), "baz"); + ATF_REQUIRE_EQ("foo/bar:baz", cli::format_test_case_id(filter)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(write_version_header); +ATF_TEST_CASE_BODY(write_version_header) +{ + cmdline::ui_mock ui; + cli::write_version_header(&ui); + ATF_REQUIRE_EQ(1, ui.out_log().size()); + ATF_REQUIRE_MATCH("^kyua .*[0-9]+\\.[0-9]+$", ui.out_log()[0]); + ATF_REQUIRE(ui.err_log().empty()); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, build_root_path__default); + ATF_ADD_TEST_CASE(tcs, build_root_path__explicit); + + ATF_ADD_TEST_CASE(tcs, kyuafile_path__default); + ATF_ADD_TEST_CASE(tcs, kyuafile_path__explicit); + + ATF_ADD_TEST_CASE(tcs, result_types__default); + ATF_ADD_TEST_CASE(tcs, result_types__empty); + ATF_ADD_TEST_CASE(tcs, result_types__explicit__all); + ATF_ADD_TEST_CASE(tcs, result_types__explicit__some); + ATF_ADD_TEST_CASE(tcs, result_types__explicit__invalid); + + ATF_ADD_TEST_CASE(tcs, results_file_create__default__new); + ATF_ADD_TEST_CASE(tcs, results_file_create__default__historical); + ATF_ADD_TEST_CASE(tcs, results_file_create__explicit); + + ATF_ADD_TEST_CASE(tcs, results_file_open__default__latest); + ATF_ADD_TEST_CASE(tcs, results_file_open__default__historical); + ATF_ADD_TEST_CASE(tcs, results_file_open__explicit); + + ATF_ADD_TEST_CASE(tcs, parse_filters__none); + ATF_ADD_TEST_CASE(tcs, parse_filters__ok); + ATF_ADD_TEST_CASE(tcs, parse_filters__duplicate); + ATF_ADD_TEST_CASE(tcs, parse_filters__nondisjoint); + + ATF_ADD_TEST_CASE(tcs, report_unused_filters__none); + ATF_ADD_TEST_CASE(tcs, report_unused_filters__some); + + ATF_ADD_TEST_CASE(tcs, format_delta); + + ATF_ADD_TEST_CASE(tcs, format_result__no_reason); + ATF_ADD_TEST_CASE(tcs, format_result__with_reason); + + ATF_ADD_TEST_CASE(tcs, format_test_case_id__test_case); + ATF_ADD_TEST_CASE(tcs, format_test_case_id__test_filter); + + ATF_ADD_TEST_CASE(tcs, write_version_header); +} diff --git a/cli/config.cpp b/cli/config.cpp new file mode 100644 index 000000000000..0049103706bf --- /dev/null +++ b/cli/config.cpp @@ -0,0 +1,223 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/config.hpp" + +#include "cli/common.hpp" +#include "engine/config.hpp" +#include "engine/exceptions.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/config/tree.ipp" +#include "utils/format/macros.hpp" +#include "utils/fs/exceptions.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" +#include "utils/env.hpp" +#include "utils/logging/macros.hpp" +#include "utils/optional.ipp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; + +using utils::optional; + + +namespace { + + +/// Basename of the configuration file. +static const char* config_basename = "kyua.conf"; + + +/// Magic string to disable loading of configuration files. +static const char* none_config = "none"; + + +/// Textual description of the default configuration files. +/// +/// This is just an auxiliary string required to define the option below, which +/// requires a pointer to a static C string. +/// +/// \todo If the user overrides the KYUA_CONFDIR environment variable, we don't +/// reflect this fact here. We don't want to query the variable during program +/// initialization due to the side-effects it may have. Therefore, fixing this +/// is tricky as it may require a whole rethink of this module. +static const std::string config_lookup_names = + (fs::path("~/.kyua") / config_basename).str() + " or " + + (fs::path(KYUA_CONFDIR) / config_basename).str(); + + +/// Loads the configuration file for this session, if any. +/// +/// This is a helper function that does not apply user-specified overrides. See +/// the documentation for cli::load_config() for more details. +/// +/// \param cmdline The parsed command line. +/// +/// \return The loaded configuration file, or the configuration defaults if the +/// loading is disabled. +/// +/// \throw engine::error If the parsing of the configuration file fails. +/// TODO(jmmv): I'm not sure if this is the raised exception. And even if +/// it is, we should make it more accurate. +config::tree +load_config_file(const cmdline::parsed_cmdline& cmdline) +{ + // TODO(jmmv): We should really be able to use cmdline.has_option here to + // detect whether the option was provided or not instead of checking against + // the default value. + const fs::path filename = cmdline.get_option< cmdline::path_option >( + cli::config_option.long_name()); + if (filename.str() == none_config) { + LD("Configuration loading disabled; using defaults"); + return engine::default_config(); + } else if (filename.str() != cli::config_option.default_value()) + return engine::load_config(filename); + + const optional< fs::path > home = utils::get_home(); + if (home) { + const fs::path path = home.get() / ".kyua" / config_basename; + try { + if (fs::exists(path)) + return engine::load_config(path); + } catch (const fs::error& e) { + // Fall through. If we fail to load the user-specific configuration + // file because it cannot be openend, we try to load the system-wide + // one. + LW(F("Failed to load user-specific configuration file '%s': %s") % + path % e.what()); + } + } + + const fs::path confdir(utils::getenv_with_default( + "KYUA_CONFDIR", KYUA_CONFDIR)); + + const fs::path path = confdir / config_basename; + if (fs::exists(path)) { + return engine::load_config(path); + } else { + return engine::default_config(); + } +} + + +/// Loads the configuration file for this session, if any. +/// +/// This is a helper function for cli::load_config() that attempts to load the +/// configuration unconditionally. +/// +/// \param cmdline The parsed command line. +/// +/// \return The loaded configuration file data. +/// +/// \throw engine::error If the parsing of the configuration file fails. +static config::tree +load_required_config(const cmdline::parsed_cmdline& cmdline) +{ + config::tree user_config = load_config_file(cmdline); + + if (cmdline.has_option(cli::variable_option.long_name())) { + typedef std::pair< std::string, std::string > override_pair; + + const std::vector< override_pair >& overrides = + cmdline.get_multi_option< cmdline::property_option >( + cli::variable_option.long_name()); + + for (std::vector< override_pair >::const_iterator + iter = overrides.begin(); iter != overrides.end(); iter++) { + try { + user_config.set_string((*iter).first, (*iter).second); + } catch (const config::error& e) { + // TODO(jmmv): Raising this type from here is obviously the + // wrong thing to do. + throw engine::error(e.what()); + } + } + } + + return user_config; +} + + +} // anonymous namespace + + +/// Standard definition of the option to specify a configuration file. +/// +/// You must use load_config() to load a configuration file while honoring the +/// value of this flag. +const cmdline::path_option cli::config_option( + 'c', "config", + (std::string("Path to the configuration file; '") + none_config + + "' to disable loading").c_str(), + "file", config_lookup_names.c_str()); + + +/// Standard definition of the option to specify a configuration variable. +const cmdline::property_option cli::variable_option( + 'v', "variable", + "Overrides a particular configuration variable", + "K=V"); + + +/// Loads the configuration file for this session, if any. +/// +/// The algorithm implemented here is as follows: +/// 1) If ~/.kyua/kyua.conf exists, load it. +/// 2) Otherwise, if sysconfdir/kyua.conf exists, load it. +/// 3) Otherwise, use the built-in settings. +/// 4) Lastly, apply any user-provided overrides. +/// +/// \param cmdline The parsed command line. +/// \param required Whether the loading of the configuration file must succeed. +/// Some commands should run regardless, and therefore we need to set this +/// to false for those commands. +/// +/// \return The loaded configuration file data. If required was set to false, +/// this might be the default configuration data if the requested file could not +/// be properly loaded. +/// +/// \throw engine::error If the parsing of the configuration file fails. +config::tree +cli::load_config(const cmdline::parsed_cmdline& cmdline, + const bool required) +{ + try { + return load_required_config(cmdline); + } catch (const engine::error& e) { + if (required) { + throw; + } else { + LW(F("Ignoring failure to load configuration because the requested " + "command should not fail: %s") % e.what()); + return engine::default_config(); + } + } +} diff --git a/cli/config.hpp b/cli/config.hpp new file mode 100644 index 000000000000..d948208ee5d0 --- /dev/null +++ b/cli/config.hpp @@ -0,0 +1,55 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/config.hpp +/// Utility functions to load configuration files. +/// +/// \todo All this should probably just be merged into the main module +/// as nothing else should have access to this. + +#if !defined(CLI_CONFIG_HPP) +#define CLI_CONFIG_HPP + +#include "utils/cmdline/options_fwd.hpp" +#include "utils/cmdline/parser_fwd.hpp" +#include "utils/config/tree_fwd.hpp" + +namespace cli { + + +extern const utils::cmdline::path_option config_option; +extern const utils::cmdline::property_option variable_option; + + +utils::config::tree load_config(const utils::cmdline::parsed_cmdline&, + const bool); + + +} // namespace cli + +#endif // !defined(CLI_CONFIG_HPP) diff --git a/cli/config_test.cpp b/cli/config_test.cpp new file mode 100644 index 000000000000..7a20c2941d8c --- /dev/null +++ b/cli/config_test.cpp @@ -0,0 +1,351 @@ +// Copyright 2011 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/config.hpp" + +#include <atf-c++.hpp> + +#include "engine/config.hpp" +#include "engine/exceptions.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/config/tree.ipp" +#include "utils/env.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; + + +namespace { + + +/// Creates a configuration file for testing purposes. +/// +/// To ensure that the loaded file is the one created by this function, use +/// validate_mock_config(). +/// +/// \param name The name of the configuration file to create. +/// \param cookie The magic value to set in the configuration file, or NULL if a +/// broken configuration file is desired. +static void +create_mock_config(const char* name, const char* cookie) +{ + if (cookie != NULL) { + atf::utils::create_file( + name, + F("syntax(2)\n" + "test_suites.suite.magic_value = '%s'\n") % cookie); + } else { + atf::utils::create_file(name, "syntax(200)\n"); + } +} + + +/// Creates an invalid system configuration. +/// +/// \param cookie The magic value to set in the configuration file, or NULL if a +/// broken configuration file is desired. +static void +mock_system_config(const char* cookie) +{ + fs::mkdir(fs::path("system-dir"), 0755); + utils::setenv("KYUA_CONFDIR", (fs::current_path() / "system-dir").str()); + create_mock_config("system-dir/kyua.conf", cookie); +} + + +/// Creates an invalid user configuration. +/// +/// \param cookie The magic value to set in the configuration file, or NULL if a +/// broken configuration file is desired. +static void +mock_user_config(const char* cookie) +{ + fs::mkdir(fs::path("user-dir"), 0755); + fs::mkdir(fs::path("user-dir/.kyua"), 0755); + utils::setenv("HOME", (fs::current_path() / "user-dir").str()); + create_mock_config("user-dir/.kyua/kyua.conf", cookie); +} + + +/// Ensures that a loaded configuration was created with create_mock_config(). +/// +/// \param user_config The configuration to validate. +/// \param cookie The magic value to expect in the configuration file. +static void +validate_mock_config(const config::tree& user_config, const char* cookie) +{ + const config::properties_map& properties = user_config.all_properties( + "test_suites.suite", true); + const config::properties_map::const_iterator iter = + properties.find("magic_value"); + ATF_REQUIRE(iter != properties.end()); + ATF_REQUIRE_EQ(cookie, (*iter).second); +} + + +/// Ensures that two configuration trees are equal. +/// +/// \param exp_tree The expected configuration tree. +/// \param actual_tree The configuration tree being validated against exp_tree. +static void +require_eq(const config::tree& exp_tree, const config::tree& actual_tree) +{ + ATF_REQUIRE(exp_tree.all_properties() == actual_tree.all_properties()); +} + + +} // anonymous namespace + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__none); +ATF_TEST_CASE_BODY(load_config__none) +{ + utils::setenv("KYUA_CONFDIR", "/the/system/does/not/exist"); + utils::setenv("HOME", "/the/user/does/not/exist"); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + require_eq(engine::default_config(), + cli::load_config(mock_cmdline, true)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__explicit__ok); +ATF_TEST_CASE_BODY(load_config__explicit__ok) +{ + mock_system_config(NULL); + mock_user_config(NULL); + + create_mock_config("test-file", "hello"); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back("test-file"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const config::tree user_config = cli::load_config(mock_cmdline, true); + validate_mock_config(user_config, "hello"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__explicit__disable); +ATF_TEST_CASE_BODY(load_config__explicit__disable) +{ + mock_system_config(NULL); + mock_user_config(NULL); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back("none"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + require_eq(engine::default_config(), + cli::load_config(mock_cmdline, true)); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__explicit__fail); +ATF_TEST_CASE_BODY(load_config__explicit__fail) +{ + mock_system_config("ok1"); + mock_user_config("ok2"); + + create_mock_config("test-file", NULL); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back("test-file"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_THROW_RE(engine::error, "200", + cli::load_config(mock_cmdline, true)); + + const config::tree config = cli::load_config(mock_cmdline, false); + require_eq(engine::default_config(), config); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__user__ok); +ATF_TEST_CASE_BODY(load_config__user__ok) +{ + mock_system_config(NULL); + mock_user_config("I am the user config"); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const config::tree user_config = cli::load_config(mock_cmdline, true); + validate_mock_config(user_config, "I am the user config"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__user__fail); +ATF_TEST_CASE_BODY(load_config__user__fail) +{ + mock_system_config("valid"); + mock_user_config(NULL); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_THROW_RE(engine::error, "200", + cli::load_config(mock_cmdline, true)); + + const config::tree config = cli::load_config(mock_cmdline, false); + require_eq(engine::default_config(), config); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__user__bad_home); +ATF_TEST_CASE_BODY(load_config__user__bad_home) +{ + mock_system_config("Fallback system config"); + utils::setenv("HOME", ""); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const config::tree user_config = cli::load_config(mock_cmdline, true); + validate_mock_config(user_config, "Fallback system config"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__system__ok); +ATF_TEST_CASE_BODY(load_config__system__ok) +{ + mock_system_config("I am the system config"); + utils::setenv("HOME", "/the/user/does/not/exist"); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const config::tree user_config = cli::load_config(mock_cmdline, true); + validate_mock_config(user_config, "I am the system config"); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__system__fail); +ATF_TEST_CASE_BODY(load_config__system__fail) +{ + mock_system_config(NULL); + utils::setenv("HOME", "/the/user/does/not/exist"); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_THROW_RE(engine::error, "200", + cli::load_config(mock_cmdline, true)); + + const config::tree config = cli::load_config(mock_cmdline, false); + require_eq(engine::default_config(), config); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__overrides__no); +ATF_TEST_CASE_BODY(load_config__overrides__no) +{ + utils::setenv("KYUA_CONFDIR", fs::current_path().str()); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + options["variable"].push_back("architecture=1"); + options["variable"].push_back("platform=2"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const config::tree user_config = cli::load_config(mock_cmdline, true); + ATF_REQUIRE_EQ("1", + user_config.lookup< config::string_node >("architecture")); + ATF_REQUIRE_EQ("2", + user_config.lookup< config::string_node >("platform")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__overrides__yes); +ATF_TEST_CASE_BODY(load_config__overrides__yes) +{ + atf::utils::create_file( + "config", + "syntax(2)\n" + "architecture = 'do not see me'\n" + "platform = 'see me'\n"); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back("config"); + options["variable"].push_back("architecture=overriden"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + const config::tree user_config = cli::load_config(mock_cmdline, true); + ATF_REQUIRE_EQ("overriden", + user_config.lookup< config::string_node >("architecture")); + ATF_REQUIRE_EQ("see me", + user_config.lookup< config::string_node >("platform")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(load_config__overrides__fail); +ATF_TEST_CASE_BODY(load_config__overrides__fail) +{ + utils::setenv("KYUA_CONFDIR", fs::current_path().str()); + + std::map< std::string, std::vector< std::string > > options; + options["config"].push_back(cli::config_option.default_value()); + options["variable"].push_back(".a=d"); + const cmdline::parsed_cmdline mock_cmdline(options, cmdline::args_vector()); + + ATF_REQUIRE_THROW_RE(engine::error, "Empty component in key.*'\\.a'", + cli::load_config(mock_cmdline, true)); + + const config::tree config = cli::load_config(mock_cmdline, false); + require_eq(engine::default_config(), config); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, load_config__none); + ATF_ADD_TEST_CASE(tcs, load_config__explicit__ok); + ATF_ADD_TEST_CASE(tcs, load_config__explicit__disable); + ATF_ADD_TEST_CASE(tcs, load_config__explicit__fail); + ATF_ADD_TEST_CASE(tcs, load_config__user__ok); + ATF_ADD_TEST_CASE(tcs, load_config__user__fail); + ATF_ADD_TEST_CASE(tcs, load_config__user__bad_home); + ATF_ADD_TEST_CASE(tcs, load_config__system__ok); + ATF_ADD_TEST_CASE(tcs, load_config__system__fail); + ATF_ADD_TEST_CASE(tcs, load_config__overrides__no); + ATF_ADD_TEST_CASE(tcs, load_config__overrides__yes); + ATF_ADD_TEST_CASE(tcs, load_config__overrides__fail); +} diff --git a/cli/main.cpp b/cli/main.cpp new file mode 100644 index 000000000000..531c252b0a75 --- /dev/null +++ b/cli/main.cpp @@ -0,0 +1,356 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/main.hpp" + +#if defined(HAVE_CONFIG_H) +# include "config.h" +#endif + +extern "C" { +#include <signal.h> +#include <unistd.h> +} + +#include <cstdlib> +#include <iostream> +#include <string> +#include <utility> + +#include "cli/cmd_about.hpp" +#include "cli/cmd_config.hpp" +#include "cli/cmd_db_exec.hpp" +#include "cli/cmd_db_migrate.hpp" +#include "cli/cmd_debug.hpp" +#include "cli/cmd_help.hpp" +#include "cli/cmd_list.hpp" +#include "cli/cmd_report.hpp" +#include "cli/cmd_report_html.hpp" +#include "cli/cmd_report_junit.hpp" +#include "cli/cmd_test.hpp" +#include "cli/common.ipp" +#include "cli/config.hpp" +#include "engine/atf.hpp" +#include "engine/plain.hpp" +#include "engine/scheduler.hpp" +#include "engine/tap.hpp" +#include "store/exceptions.hpp" +#include "utils/cmdline/commands_map.ipp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/globals.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.ipp" +#include "utils/cmdline/ui.hpp" +#include "utils/config/tree.ipp" +#include "utils/env.hpp" +#include "utils/format/macros.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" +#include "utils/logging/macros.hpp" +#include "utils/logging/operations.hpp" +#include "utils/optional.ipp" +#include "utils/sanity.hpp" +#include "utils/signals/exceptions.hpp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace fs = utils::fs; +namespace logging = utils::logging; +namespace signals = utils::signals; +namespace scheduler = engine::scheduler; + +using utils::none; +using utils::optional; + + +namespace { + + +/// Registers all valid scheduler interfaces. +/// +/// This is part of Kyua's setup but it is a bit strange to find it here. I am +/// not sure what a better location would be though, so for now this is good +/// enough. +static void +register_scheduler_interfaces(void) +{ + scheduler::register_interface( + "atf", std::shared_ptr< scheduler::interface >( + new engine::atf_interface())); + scheduler::register_interface( + "plain", std::shared_ptr< scheduler::interface >( + new engine::plain_interface())); + scheduler::register_interface( + "tap", std::shared_ptr< scheduler::interface >( + new engine::tap_interface())); +} + + +/// Executes the given subcommand with proper usage_error reporting. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param command The subcommand to execute. +/// \param args The part of the command line passed to the subcommand. The +/// first item of this collection must match the command name. +/// \param user_config The runtime configuration to pass to the subcommand. +/// +/// \return The exit code of the command. Typically 0 on success, some other +/// integer otherwise. +/// +/// \throw cmdline::usage_error If the user input to the subcommand is invalid. +/// This error does not encode the command name within it, so this function +/// extends the message in the error to specify which subcommand was +/// affected. +/// \throw std::exception This propagates any uncaught exception. Such +/// exceptions are bugs, but we let them propagate so that the runtime will +/// abort and dump core. +static int +run_subcommand(cmdline::ui* ui, cli::cli_command* command, + const cmdline::args_vector& args, + const config::tree& user_config) +{ + try { + PRE(command->name() == args[0]); + return command->main(ui, args, user_config); + } catch (const cmdline::usage_error& e) { + throw std::pair< std::string, cmdline::usage_error >( + command->name(), e); + } +} + + +/// Exception-safe version of main. +/// +/// This function provides the real meat of the entry point of the program. It +/// is allowed to throw some known exceptions which are parsed by the caller. +/// Doing so keeps this function simpler and allow tests to actually validate +/// that the errors reported are accurate. +/// +/// \return The exit code of the program. Should be EXIT_SUCCESS on success and +/// EXIT_FAILURE on failure. The caller extends this to additional integers for +/// errors reported through exceptions. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param argc The number of arguments passed on the command line. +/// \param argv NULL-terminated array containing the command line arguments. +/// \param mock_command An extra command provided for testing purposes; should +/// just be NULL other than for tests. +/// +/// \throw cmdline::usage_error If the user ran the program with invalid +/// arguments. +/// \throw std::exception This propagates any uncaught exception. Such +/// exceptions are bugs, but we let them propagate so that the runtime will +/// abort and dump core. +static int +safe_main(cmdline::ui* ui, int argc, const char* const argv[], + cli::cli_command_ptr mock_command) +{ + cmdline::options_vector options; + options.push_back(&cli::config_option); + options.push_back(&cli::variable_option); + const cmdline::string_option loglevel_option( + "loglevel", "Level of the messages to log", "level", "info"); + options.push_back(&loglevel_option); + const cmdline::path_option logfile_option( + "logfile", "Path to the log file", "file", + cli::detail::default_log_name().c_str()); + options.push_back(&logfile_option); + + cmdline::commands_map< cli::cli_command > commands; + + commands.insert(new cli::cmd_about()); + commands.insert(new cli::cmd_config()); + commands.insert(new cli::cmd_db_exec()); + commands.insert(new cli::cmd_db_migrate()); + commands.insert(new cli::cmd_help(&options, &commands)); + + commands.insert(new cli::cmd_debug(), "Workspace"); + commands.insert(new cli::cmd_list(), "Workspace"); + commands.insert(new cli::cmd_test(), "Workspace"); + + commands.insert(new cli::cmd_report(), "Reporting"); + commands.insert(new cli::cmd_report_html(), "Reporting"); + commands.insert(new cli::cmd_report_junit(), "Reporting"); + + if (mock_command.get() != NULL) + commands.insert(mock_command); + + const cmdline::parsed_cmdline cmdline = cmdline::parse(argc, argv, options); + + const fs::path logfile(cmdline.get_option< cmdline::path_option >( + "logfile")); + fs::mkdir_p(logfile.branch_path(), 0755); + LD(F("Log file is %s") % logfile); + utils::install_crash_handlers(logfile.str()); + try { + logging::set_persistency(cmdline.get_option< cmdline::string_option >( + "loglevel"), logfile); + } catch (const std::range_error& e) { + throw cmdline::usage_error(e.what()); + } + + if (cmdline.arguments().empty()) + throw cmdline::usage_error("No command provided"); + const std::string cmdname = cmdline.arguments()[0]; + + const config::tree user_config = cli::load_config(cmdline, + cmdname != "help"); + + cli::cli_command* command = commands.find(cmdname); + if (command == NULL) + throw cmdline::usage_error(F("Unknown command '%s'") % cmdname); + register_scheduler_interfaces(); + return run_subcommand(ui, command, cmdline.arguments(), user_config); +} + + +} // anonymous namespace + + +/// Gets the name of the default log file. +/// +/// \return The path to the log file. +fs::path +cli::detail::default_log_name(void) +{ + // Update doc/troubleshooting.texi if you change this algorithm. + const optional< std::string > home(utils::getenv("HOME")); + if (home) { + return logging::generate_log_name(fs::path(home.get()) / ".kyua" / + "logs", cmdline::progname()); + } else { + const optional< std::string > tmpdir(utils::getenv("TMPDIR")); + if (tmpdir) { + return logging::generate_log_name(fs::path(tmpdir.get()), + cmdline::progname()); + } else { + return logging::generate_log_name(fs::path("/tmp"), + cmdline::progname()); + } + } +} + + +/// Testable entry point, with catch-all exception handlers. +/// +/// This entry point does not perform any initialization of global state; it is +/// provided to allow unit-testing of the utility's entry point. +/// +/// \param ui Object to interact with the I/O of the program. +/// \param argc The number of arguments passed on the command line. +/// \param argv NULL-terminated array containing the command line arguments. +/// \param mock_command An extra command provided for testing purposes; should +/// just be NULL other than for tests. +/// +/// \return 0 on success, some other integer on error. +/// +/// \throw std::exception This propagates any uncaught exception. Such +/// exceptions are bugs, but we let them propagate so that the runtime will +/// abort and dump core. +int +cli::main(cmdline::ui* ui, const int argc, const char* const* const argv, + cli_command_ptr mock_command) +{ + try { + const int exit_code = safe_main(ui, argc, argv, mock_command); + + // Codes above 1 are reserved to report conditions captured as + // exceptions below. + INV(exit_code == EXIT_SUCCESS || exit_code == EXIT_FAILURE); + + return exit_code; + } catch (const signals::interrupted_error& e) { + cmdline::print_error(ui, F("%s.") % e.what()); + // Re-deliver the interruption signal to self so that we terminate with + // the right status. At this point we should NOT have any custom signal + // handlers in place. + ::kill(getpid(), e.signo()); + LD("Interrupt signal re-delivery did not terminate program"); + // If we reach this, something went wrong because we did not exit as + // intended. Return an internal error instead. (Would be nicer to + // abort in principle, but it wouldn't be a nice experience if it ever + // happened.) + return 2; + } catch (const std::pair< std::string, cmdline::usage_error >& e) { + const std::string message = F("Usage error for command %s: %s.") % + e.first % e.second.what(); + LE(message); + ui->err(message); + ui->err(F("Type '%s help %s' for usage information.") % + cmdline::progname() % e.first); + return 3; + } catch (const cmdline::usage_error& e) { + const std::string message = F("Usage error: %s.") % e.what(); + LE(message); + ui->err(message); + ui->err(F("Type '%s help' for usage information.") % + cmdline::progname()); + return 3; + } catch (const store::old_schema_error& e) { + const std::string message = F("The database has schema version %s, " + "which is too old; please use db-migrate " + "to upgrade it.") % e.old_version(); + cmdline::print_error(ui, message); + return 2; + } catch (const std::runtime_error& e) { + cmdline::print_error(ui, F("%s.") % e.what()); + return 2; + } +} + + +/// Delegate for ::main(). +/// +/// This function is supposed to be called directly from the top-level ::main() +/// function. It takes care of initializing internal libraries and then calls +/// main(ui, argc, argv). +/// +/// \pre This function can only be called once. +/// +/// \throw std::exception This propagates any uncaught exception. Such +/// exceptions are bugs, but we let them propagate so that the runtime will +/// abort and dump core. +int +cli::main(const int argc, const char* const* const argv) +{ + logging::set_inmemory(); + + LI(F("%s %s") % PACKAGE % VERSION); + + std::string plain_args; + for (const char* const* arg = argv; *arg != NULL; arg++) + plain_args += F(" %s") % *arg; + LI(F("Command line:%s") % plain_args); + + cmdline::init(argv[0]); + cmdline::ui ui; + + const int exit_code = main(&ui, argc, argv); + LI(F("Clean exit with code %s") % exit_code); + return exit_code; +} diff --git a/cli/main.hpp b/cli/main.hpp new file mode 100644 index 000000000000..00e53c5a4ab2 --- /dev/null +++ b/cli/main.hpp @@ -0,0 +1,61 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file cli/main.hpp +/// Entry point for the program. +/// +/// These entry points are separate from the top-level ::main() function to +/// allow unit-testing of the main code. + +#if !defined(CLI_MAIN_HPP) +#define CLI_MAIN_HPP + +#include "cli/common.hpp" +#include "utils/cmdline/ui_fwd.hpp" +#include "utils/fs/path_fwd.hpp" + +namespace cli { + + +namespace detail { + + +utils::fs::path default_log_name(void); + + +} // namespace detail + + +int main(utils::cmdline::ui*, const int, const char* const* const, + cli_command_ptr = cli_command_ptr()); +int main(const int, const char* const* const); + + +} // namespace cli + +#endif // !defined(CLI_MAIN_HPP) diff --git a/cli/main_test.cpp b/cli/main_test.cpp new file mode 100644 index 000000000000..70d167ff6963 --- /dev/null +++ b/cli/main_test.cpp @@ -0,0 +1,489 @@ +// Copyright 2010 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "cli/main.hpp" + +extern "C" { +#include <signal.h> +} + +#include <cstdlib> + +#include <atf-c++.hpp> + +#include "utils/cmdline/base_command.ipp" +#include "utils/cmdline/exceptions.hpp" +#include "utils/cmdline/globals.hpp" +#include "utils/cmdline/options.hpp" +#include "utils/cmdline/parser.hpp" +#include "utils/cmdline/ui_mock.hpp" +#include "utils/datetime.hpp" +#include "utils/defs.hpp" +#include "utils/env.hpp" +#include "utils/fs/operations.hpp" +#include "utils/fs/path.hpp" +#include "utils/logging/macros.hpp" +#include "utils/logging/operations.hpp" +#include "utils/process/child.ipp" +#include "utils/process/status.hpp" +#include "utils/test_utils.ipp" + +namespace cmdline = utils::cmdline; +namespace config = utils::config; +namespace datetime = utils::datetime; +namespace fs = utils::fs; +namespace logging = utils::logging; +namespace process = utils::process; + + +namespace { + + +/// Fake command implementation that crashes during its execution. +class cmd_mock_crash : public cli::cli_command { +public: + /// Constructs a new mock command. + /// + /// All command parameters are set to irrelevant values. + cmd_mock_crash(void) : + cli::cli_command("mock_error", "", 0, 0, "Mock command that crashes") + { + } + + /// Runs the mock command. + /// + /// \return Nothing because this function always aborts. + int + run(cmdline::ui* /* ui */, + const cmdline::parsed_cmdline& /* cmdline */, + const config::tree& /* user_config */) + { + utils::abort_without_coredump(); + } +}; + + +/// Fake command implementation that throws an exception during its execution. +class cmd_mock_error : public cli::cli_command { + /// Whether the command raises an exception captured by the parent or not. + /// + /// If this is true, the command will raise a std::runtime_error exception + /// or a subclass of it. The main program is in charge of capturing these + /// and reporting them appropriately. If false, this raises another + /// exception that does not inherit from std::runtime_error. + bool _unhandled; + +public: + /// Constructs a new mock command. + /// + /// \param unhandled If true, make run raise an exception not catched by the + /// main program. + cmd_mock_error(const bool unhandled) : + cli::cli_command("mock_error", "", 0, 0, + "Mock command that raises an error"), + _unhandled(unhandled) + { + } + + /// Runs the mock command. + /// + /// \return Nothing because this function always aborts. + /// + /// \throw std::logic_error If _unhandled is true. + /// \throw std::runtime_error If _unhandled is false. + int + run(cmdline::ui* /* ui */, + const cmdline::parsed_cmdline& /* cmdline */, + const config::tree& /* user_config */) + { + if (_unhandled) + throw std::logic_error("This is unhandled"); + else + throw std::runtime_error("Runtime error"); + } +}; + + +/// Fake command implementation that prints messages during its execution. +class cmd_mock_write : public cli::cli_command { +public: + /// Constructs a new mock command. + /// + /// All command parameters are set to irrelevant values. + cmd_mock_write(void) : cli::cli_command( + "mock_write", "", 0, 0, "Mock command that prints output") + { + } + + /// Runs the mock command. + /// + /// \param ui Object to interact with the I/O of the program. + /// + /// \return Nothing because this function always aborts. + int + run(cmdline::ui* ui, + const cmdline::parsed_cmdline& /* cmdline */, + const config::tree& /* user_config */) + { + ui->out("stdout message from subcommand"); + ui->err("stderr message from subcommand"); + return EXIT_FAILURE; + } +}; + + +} // anonymous namespace + + +ATF_TEST_CASE_WITHOUT_HEAD(detail__default_log_name__home); +ATF_TEST_CASE_BODY(detail__default_log_name__home) +{ + datetime::set_mock_now(2011, 2, 21, 21, 10, 30, 0); + cmdline::init("progname1"); + + utils::setenv("HOME", "/home//fake"); + utils::setenv("TMPDIR", "/do/not/use/this"); + ATF_REQUIRE_EQ( + fs::path("/home/fake/.kyua/logs/progname1.20110221-211030.log"), + cli::detail::default_log_name()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(detail__default_log_name__tmpdir); +ATF_TEST_CASE_BODY(detail__default_log_name__tmpdir) +{ + datetime::set_mock_now(2011, 2, 21, 21, 10, 50, 987); + cmdline::init("progname2"); + + utils::unsetenv("HOME"); + utils::setenv("TMPDIR", "/a/b//c"); + ATF_REQUIRE_EQ(fs::path("/a/b/c/progname2.20110221-211050.log"), + cli::detail::default_log_name()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(detail__default_log_name__hardcoded); +ATF_TEST_CASE_BODY(detail__default_log_name__hardcoded) +{ + datetime::set_mock_now(2011, 2, 21, 21, 15, 00, 123456); + cmdline::init("progname3"); + + utils::unsetenv("HOME"); + utils::unsetenv("TMPDIR"); + ATF_REQUIRE_EQ(fs::path("/tmp/progname3.20110221-211500.log"), + cli::detail::default_log_name()); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__no_args); +ATF_TEST_CASE_BODY(main__no_args) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 1; + const char* const argv[] = {"progname", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(atf::utils::grep_collection("Usage error: No command provided", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Type.*progname help", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__unknown_command); +ATF_TEST_CASE_BODY(main__unknown_command) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "foo", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(atf::utils::grep_collection("Usage error: Unknown command.*foo", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Type.*progname help", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__logfile__default); +ATF_TEST_CASE_BODY(main__logfile__default) +{ + logging::set_inmemory(); + datetime::set_mock_now(2011, 2, 21, 21, 30, 00, 0); + cmdline::init("progname"); + + const int argc = 1; + const char* const argv[] = {"progname", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE(!fs::exists(fs::path( + ".kyua/logs/progname.20110221-213000.log"))); + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(fs::exists(fs::path( + ".kyua/logs/progname.20110221-213000.log"))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__logfile__override); +ATF_TEST_CASE_BODY(main__logfile__override) +{ + logging::set_inmemory(); + datetime::set_mock_now(2011, 2, 21, 21, 30, 00, 321); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "--logfile=test.log", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE(!fs::exists(fs::path("test.log"))); + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(!fs::exists(fs::path( + ".kyua/logs/progname.20110221-213000.log"))); + ATF_REQUIRE(fs::exists(fs::path("test.log"))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__loglevel__default); +ATF_TEST_CASE_BODY(main__loglevel__default) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "--logfile=test.log", NULL}; + + LD("Mock debug message"); + LE("Mock error message"); + LI("Mock info message"); + LW("Mock warning message"); + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(!atf::utils::grep_file("Mock debug message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock error message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock info message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock warning message", "test.log")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__loglevel__higher); +ATF_TEST_CASE_BODY(main__loglevel__higher) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 3; + const char* const argv[] = {"progname", "--logfile=test.log", + "--loglevel=debug", NULL}; + + LD("Mock debug message"); + LE("Mock error message"); + LI("Mock info message"); + LW("Mock warning message"); + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(atf::utils::grep_file("Mock debug message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock error message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock info message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock warning message", "test.log")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__loglevel__lower); +ATF_TEST_CASE_BODY(main__loglevel__lower) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 3; + const char* const argv[] = {"progname", "--logfile=test.log", + "--loglevel=warning", NULL}; + + LD("Mock debug message"); + LE("Mock error message"); + LI("Mock info message"); + LW("Mock warning message"); + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(!atf::utils::grep_file("Mock debug message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock error message", "test.log")); + ATF_REQUIRE(!atf::utils::grep_file("Mock info message", "test.log")); + ATF_REQUIRE(atf::utils::grep_file("Mock warning message", "test.log")); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__loglevel__error); +ATF_TEST_CASE_BODY(main__loglevel__error) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 3; + const char* const argv[] = {"progname", "--logfile=test.log", + "--loglevel=i-am-invalid", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, cli::main(&ui, argc, argv)); + ATF_REQUIRE(atf::utils::grep_collection("Usage error.*i-am-invalid", + ui.err_log())); + ATF_REQUIRE(!fs::exists(fs::path("test.log"))); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__subcommand__ok); +ATF_TEST_CASE_BODY(main__subcommand__ok) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "mock_write", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(EXIT_FAILURE, + cli::main(&ui, argc, argv, + cli::cli_command_ptr(new cmd_mock_write()))); + ATF_REQUIRE_EQ(1, ui.out_log().size()); + ATF_REQUIRE_EQ("stdout message from subcommand", ui.out_log()[0]); + ATF_REQUIRE_EQ(1, ui.err_log().size()); + ATF_REQUIRE_EQ("stderr message from subcommand", ui.err_log()[0]); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__subcommand__invalid_args); +ATF_TEST_CASE_BODY(main__subcommand__invalid_args) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 3; + const char* const argv[] = {"progname", "mock_write", "bar", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(3, + cli::main(&ui, argc, argv, + cli::cli_command_ptr(new cmd_mock_write()))); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(atf::utils::grep_collection( + "Usage error for command mock_write: Too many arguments.", + ui.err_log())); + ATF_REQUIRE(atf::utils::grep_collection("Type.*progname help", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__subcommand__runtime_error); +ATF_TEST_CASE_BODY(main__subcommand__runtime_error) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "mock_error", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_EQ(2, cli::main(&ui, argc, argv, + cli::cli_command_ptr(new cmd_mock_error(false)))); + ATF_REQUIRE(ui.out_log().empty()); + ATF_REQUIRE(atf::utils::grep_collection("progname: E: Runtime error.", + ui.err_log())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__subcommand__unhandled_exception); +ATF_TEST_CASE_BODY(main__subcommand__unhandled_exception) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "mock_error", NULL}; + + cmdline::ui_mock ui; + ATF_REQUIRE_THROW(std::logic_error, cli::main(&ui, argc, argv, + cli::cli_command_ptr(new cmd_mock_error(true)))); +} + + +static void +do_subcommand_crash(void) +{ + logging::set_inmemory(); + cmdline::init("progname"); + + const int argc = 2; + const char* const argv[] = {"progname", "mock_error", NULL}; + + cmdline::ui_mock ui; + cli::main(&ui, argc, argv, + cli::cli_command_ptr(new cmd_mock_crash())); +} + + +ATF_TEST_CASE_WITHOUT_HEAD(main__subcommand__crash); +ATF_TEST_CASE_BODY(main__subcommand__crash) +{ + const process::status status = process::child::fork_files( + do_subcommand_crash, fs::path("stdout.txt"), + fs::path("stderr.txt"))->wait(); + ATF_REQUIRE(status.signaled()); + ATF_REQUIRE_EQ(SIGABRT, status.termsig()); + ATF_REQUIRE(atf::utils::grep_file("Fatal signal", "stderr.txt")); +} + + +ATF_INIT_TEST_CASES(tcs) +{ + ATF_ADD_TEST_CASE(tcs, detail__default_log_name__home); + ATF_ADD_TEST_CASE(tcs, detail__default_log_name__tmpdir); + ATF_ADD_TEST_CASE(tcs, detail__default_log_name__hardcoded); + + ATF_ADD_TEST_CASE(tcs, main__no_args); + ATF_ADD_TEST_CASE(tcs, main__unknown_command); + ATF_ADD_TEST_CASE(tcs, main__logfile__default); + ATF_ADD_TEST_CASE(tcs, main__logfile__override); + ATF_ADD_TEST_CASE(tcs, main__loglevel__default); + ATF_ADD_TEST_CASE(tcs, main__loglevel__higher); + ATF_ADD_TEST_CASE(tcs, main__loglevel__lower); + ATF_ADD_TEST_CASE(tcs, main__loglevel__error); + ATF_ADD_TEST_CASE(tcs, main__subcommand__ok); + ATF_ADD_TEST_CASE(tcs, main__subcommand__invalid_args); + ATF_ADD_TEST_CASE(tcs, main__subcommand__runtime_error); + ATF_ADD_TEST_CASE(tcs, main__subcommand__unhandled_exception); + ATF_ADD_TEST_CASE(tcs, main__subcommand__crash); +} diff --git a/configure.ac b/configure.ac new file mode 100644 index 000000000000..a0df977c5226 --- /dev/null +++ b/configure.ac @@ -0,0 +1,173 @@ +dnl Copyright 2010 The Kyua Authors. +dnl All rights reserved. +dnl +dnl Redistribution and use in source and binary forms, with or without +dnl modification, are permitted provided that the following conditions are +dnl met: +dnl +dnl * Redistributions of source code must retain the above copyright +dnl notice, this list of conditions and the following disclaimer. +dnl * Redistributions in binary form must reproduce the above copyright +dnl notice, this list of conditions and the following disclaimer in the +dnl documentation and/or other materials provided with the distribution. +dnl * Neither the name of Google Inc. nor the names of its contributors +dnl may be used to endorse or promote products derived from this software +dnl without specific prior written permission. +dnl +dnl THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +dnl "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +dnl LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +dnl A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +dnl OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +dnl SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +dnl LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +dnl DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +dnl THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +dnl (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +dnl OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +AC_INIT([Kyua], [0.14], [kyua-discuss@googlegroups.com], [kyua], + [https://github.com/jmmv/kyua/]) +AC_PREREQ([2.65]) + + +AC_COPYRIGHT([Copyright 2010 The Kyua Authors.]) +AC_CONFIG_AUX_DIR([admin]) +AC_CONFIG_FILES([Doxyfile Makefile utils/defs.hpp]) +AC_CONFIG_HEADERS([config.h]) +AC_CONFIG_MACRO_DIR([m4]) +AC_CONFIG_SRCDIR([main.cpp]) +AC_CONFIG_TESTDIR([bootstrap]) + + +AM_INIT_AUTOMAKE([1.9 foreign subdir-objects -Wall]) + + +AC_LANG([C++]) +AC_PROG_CXX +AX_CXX_COMPILE_STDCXX([11], [noext], [mandatory]) +m4_ifdef([AM_PROG_AR], [AM_PROG_AR]) +KYUA_DEVELOPER_MODE([C++]) +KYUA_ATTRIBUTE_NORETURN +KYUA_ATTRIBUTE_PURE +KYUA_ATTRIBUTE_UNUSED +KYUA_FS_MODULE +KYUA_GETOPT +KYUA_LAST_SIGNO +KYUA_MEMORY +AC_CHECK_FUNCS([putenv setenv unsetenv]) +AC_CHECK_HEADERS([termios.h]) + + +AC_PROG_RANLIB + + +m4_ifndef([PKG_CHECK_MODULES], + [m4_fatal([Cannot find pkg.m4; see the INSTALL document for help])]) + +m4_ifndef([ATF_CHECK_CXX], + [m4_fatal([Cannot find atf-c++.m4; see the INSTALL document for help])]) +ATF_CHECK_CXX([>= 0.17]) +m4_ifndef([ATF_CHECK_SH], + [m4_fatal([Cannot find atf-sh.m4; see the INSTALL document for help])]) +ATF_CHECK_SH([>= 0.15]) +m4_ifndef([ATF_ARG_WITH], + [m4_fatal([Cannot find atf-common.m4; see the INSTALL document for help])]) +ATF_ARG_WITH + +PKG_CHECK_MODULES([LUTOK], [lutok >= 0.4], + [], + AC_MSG_ERROR([lutok (0.4 or newer) is required])) +PKG_CHECK_MODULES([SQLITE3], [sqlite3 >= 3.6.22], + [], + AC_MSG_ERROR([sqlite3 (3.6.22 or newer) is required])) +KYUA_DOXYGEN +AC_PATH_PROG([GDB], [gdb]) +test -n "${GDB}" || GDB=gdb +AC_PATH_PROG([GIT], [git]) + + +KYUA_UNAME_ARCHITECTURE +KYUA_UNAME_PLATFORM + + +AC_ARG_VAR([KYUA_CONFSUBDIR], + [Subdirectory of sysconfdir under which to look for files]) +if test x"${KYUA_CONFSUBDIR-unset}" = x"unset"; then + KYUA_CONFSUBDIR=kyua +else + case ${KYUA_CONFSUBDIR} in + /*) + AC_MSG_ERROR([KYUA_CONFSUBDIR must hold a relative path]) + ;; + *) + ;; + esac +fi +if test x"${KYUA_CONFSUBDIR}" = x""; then + AC_SUBST(kyua_confdir, \${sysconfdir}) +else + AC_SUBST(kyua_confdir, \${sysconfdir}/${KYUA_CONFSUBDIR}) +fi + + +dnl Allow the caller of 'make check', 'make installcheck' and 'make distcheck' +dnl on the Kyua source tree to override the configuration file passed to our +dnl own test runs. This is for the development of Kyua only and the value of +dnl this setting has no effect on the built product in any way. If we go +dnl through great extents in validating the value of this setting, it is to +dnl minimize the chance of false test run negatives later on. +AC_ARG_VAR([KYUA_CONFIG_FILE_FOR_CHECK], + [kyua.conf file to use at 'make (|dist|install)check' time]) +case "${KYUA_CONFIG_FILE_FOR_CHECK-none}" in +none) + KYUA_CONFIG_FILE_FOR_CHECK=none + ;; +/*) + if test -f "${KYUA_CONFIG_FILE_FOR_CHECK}"; then + : # All good! + else + AC_MSG_ERROR([KYUA_CONFIG_FILE_FOR_CHECK file does not exist]) + fi + ;; +*) + AC_MSG_ERROR([KYUA_CONFIG_FILE_FOR_CHECK must hold an absolute path]) + ;; +esac + + +AC_ARG_VAR([KYUA_TMPDIR], + [Path to the directory in which to place work directories]) +case "${KYUA_TMPDIR:-unset}" in + unset) + KYUA_TMPDIR=/tmp + ;; + /*) + ;; + *) + AC_MSG_ERROR([KYUA_TMPDIR must be an absolute path]) + ;; +esac + + +AC_SUBST(examplesdir, \${pkgdatadir}/examples) +AC_SUBST(luadir, \${pkgdatadir}/lua) +AC_SUBST(miscdir, \${pkgdatadir}/misc) +AC_SUBST(pkgtestsdir, \${testsdir}/kyua) +AC_SUBST(storedir, \${pkgdatadir}/store) +AC_SUBST(testsdir, \${exec_prefix}/tests) + + +dnl BSD make(1) doesn't deal with targets specified as './foo' well: they +dnl need to be specified as 'foo'. The following hack is to workaround this +dnl issue. +if test "${srcdir}" = .; then + target_srcdir= +else + target_srcdir="${srcdir}/" +fi +AM_CONDITIONAL(TARGET_SRCDIR_EMPTY, [test -z "${target_srcdir}"]) +AC_SUBST([target_srcdir]) + + +AC_OUTPUT diff --git a/doc/.gitignore b/doc/.gitignore new file mode 100644 index 000000000000..ecaaf27b9262 --- /dev/null +++ b/doc/.gitignore @@ -0,0 +1,14 @@ +kyua-about.1 +kyua-config.1 +kyua-db-exec.1 +kyua-db-migrate.1 +kyua-debug.1 +kyua-help.1 +kyua-list.1 +kyua-report-html.1 +kyua-report-junit.1 +kyua-report.1 +kyua-test.1 +kyua.1 +kyua.conf.5 +kyuafile.5 diff --git a/doc/Kyuafile b/doc/Kyuafile new file mode 100644 index 000000000000..c538f5b2a531 --- /dev/null +++ b/doc/Kyuafile @@ -0,0 +1,5 @@ +syntax(2) + +test_suite("kyua") + +atf_test_program{name="manbuild_test"} diff --git a/doc/Makefile.am.inc b/doc/Makefile.am.inc new file mode 100644 index 000000000000..638191218bcc --- /dev/null +++ b/doc/Makefile.am.inc @@ -0,0 +1,152 @@ +# Copyright 2011 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +BUILD_MANPAGE = \ + $(MKDIR_P) doc; \ + $(SHELL) $(srcdir)/doc/manbuild.sh \ + -v "CONFDIR=$(kyua_confdir)" \ + -v "DOCDIR=$(docdir)" \ + -v "EGDIR=$(examplesdir)" \ + -v "MISCDIR=$(miscdir)" \ + -v "PACKAGE=$(PACKAGE_TARNAME)" \ + -v "STOREDIR=$(storedir)" \ + -v "TESTSDIR=$(testsdir)" \ + -v "VERSION=$(PACKAGE_VERSION)" \ + "$(srcdir)/doc/$${name}.in" "doc/$${name}" + +DIST_MAN_DEPS = doc/manbuild.sh \ + doc/build-root.mdoc \ + doc/results-file-flag-read.mdoc \ + doc/results-file-flag-write.mdoc \ + doc/results-files.mdoc \ + doc/results-files-report-example.mdoc \ + doc/test-filters.mdoc \ + doc/test-isolation.mdoc +MAN_DEPS = $(DIST_MAN_DEPS) Makefile +EXTRA_DIST += $(DIST_MAN_DEPS) + +man_MANS = doc/kyua-about.1 +CLEANFILES += doc/kyua-about.1 +EXTRA_DIST += doc/kyua-about.1.in +doc/kyua-about.1: $(srcdir)/doc/kyua-about.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-about.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-config.1 +CLEANFILES += doc/kyua-config.1 +EXTRA_DIST += doc/kyua-config.1.in +doc/kyua-config.1: $(srcdir)/doc/kyua-config.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-config.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-db-exec.1 +CLEANFILES += doc/kyua-db-exec.1 +EXTRA_DIST += doc/kyua-db-exec.1.in +doc/kyua-db-exec.1: $(srcdir)/doc/kyua-db-exec.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-db-exec.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-db-migrate.1 +CLEANFILES += doc/kyua-db-migrate.1 +EXTRA_DIST += doc/kyua-db-migrate.1.in +doc/kyua-db-migrate.1: $(srcdir)/doc/kyua-db-migrate.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-db-migrate.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-debug.1 +CLEANFILES += doc/kyua-debug.1 +EXTRA_DIST += doc/kyua-debug.1.in +doc/kyua-debug.1: $(srcdir)/doc/kyua-debug.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-debug.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-help.1 +CLEANFILES += doc/kyua-help.1 +EXTRA_DIST += doc/kyua-help.1.in +doc/kyua-help.1: $(srcdir)/doc/kyua-help.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-help.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-list.1 +CLEANFILES += doc/kyua-list.1 +EXTRA_DIST += doc/kyua-list.1.in +doc/kyua-list.1: $(srcdir)/doc/kyua-list.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-list.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-report-html.1 +CLEANFILES += doc/kyua-report-html.1 +EXTRA_DIST += doc/kyua-report-html.1.in +doc/kyua-report-html.1: $(srcdir)/doc/kyua-report-html.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-report-html.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-report-junit.1 +CLEANFILES += doc/kyua-report-junit.1 +EXTRA_DIST += doc/kyua-report-junit.1.in +doc/kyua-report-junit.1: $(srcdir)/doc/kyua-report-junit.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-report-junit.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-report.1 +CLEANFILES += doc/kyua-report.1 +EXTRA_DIST += doc/kyua-report.1.in +doc/kyua-report.1: $(srcdir)/doc/kyua-report.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-report.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua-test.1 +CLEANFILES += doc/kyua-test.1 +EXTRA_DIST += doc/kyua-test.1.in +doc/kyua-test.1: $(srcdir)/doc/kyua-test.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua-test.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua.1 +CLEANFILES += doc/kyua.1 +EXTRA_DIST += doc/kyua.1.in +doc/kyua.1: $(srcdir)/doc/kyua.1.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua.1; $(BUILD_MANPAGE) + +man_MANS += doc/kyua.conf.5 +CLEANFILES += doc/kyua.conf.5 +EXTRA_DIST += doc/kyua.conf.5.in +doc/kyua.conf.5: $(srcdir)/doc/kyua.conf.5.in $(MAN_DEPS) + $(AM_V_GEN)name=kyua.conf.5; $(BUILD_MANPAGE) + +man_MANS += doc/kyuafile.5 +CLEANFILES += doc/kyuafile.5 +EXTRA_DIST += doc/kyuafile.5.in +doc/kyuafile.5: $(srcdir)/doc/kyuafile.5.in $(MAN_DEPS) + $(AM_V_GEN)name=kyuafile.5; $(BUILD_MANPAGE) + +if WITH_ATF +EXTRA_DIST += doc/Kyuafile + +noinst_SCRIPTS += doc/manbuild_test +CLEANFILES += doc/manbuild_test +EXTRA_DIST += doc/manbuild_test.sh +doc/manbuild_test: $(srcdir)/doc/manbuild_test.sh Makefile + $(AM_V_GEN)$(MKDIR_P) doc; \ + echo "#! $(ATF_SH)" >doc/manbuild_test.tmp; \ + echo "# AUTOMATICALLY GENERATED FROM Makefile" \ + >>doc/manbuild_test.tmp; \ + sed -e 's,__MANBUILD__,$(abs_srcdir)/doc/manbuild.sh,g' \ + <$(srcdir)/doc/manbuild_test.sh >>doc/manbuild_test.tmp; \ + mv doc/manbuild_test.tmp doc/manbuild_test; \ + chmod +x doc/manbuild_test +endif diff --git a/doc/build-root.mdoc b/doc/build-root.mdoc new file mode 100644 index 000000000000..2fb008246f41 --- /dev/null +++ b/doc/build-root.mdoc @@ -0,0 +1,104 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Em Build directories +(or object directories, target directories, product directories, etc.) is +the concept that allows a developer to keep the source tree clean from +build products by asking the build system to place such build products +under a separate subtree. +.Pp +Most build systems today support build directories. +For example, the GNU Automake/Autoconf build system exposes such concept when +invoked as follows: +.Bd -literal -offset indent +$ cd my-project-1.0 +$ mkdir build +$ cd build +$ ../configure +$ make +.Ed +.Pp +Under such invocation, all the results of the build are left in the +.Pa my-project-1.0/build/ +subdirectory while maintaining the contents of +.Pa my-project-1.0/ +intact. +.Pp +Because build directories are an integral part of most build systems, and +because they are a tool that developers use frequently, +.Nm +supports build directories too. +This manifests in the form of +.Nm +being able to run tests from build directories while reading the (often +immutable) test suite definition from the source tree. +.Pp +One important property of build directories is that they follow (or need to +follow) the exact same layout as the source tree. +For example, consider the following directory listings: +.Bd -literal -offset indent +src/Kyuafile +src/bin/ls/ +src/bin/ls/Kyuafile +src/bin/ls/ls.c +src/bin/ls/ls_test.c +src/sbin/su/ +src/sbin/su/Kyuafile +src/sbin/su/su.c +src/sbin/su/su_test.c + +obj/bin/ls/ +obj/bin/ls/ls* +obj/bin/ls/ls_test* +obj/sbin/su/ +obj/sbin/su/su* +obj/sbin/su/su_test* +.Ed +.Pp +Note how the directory layout within +.Pa src/ +matches that of +.Pa obj/ . +The +.Pa src/ +directory contains only source files and the definition of the test suite +(the Kyuafiles), while the +.Pa obj/ +directory contains only the binaries generated during a build. +.Pp +All commands that deal with the workspace support the +.Fl -build-root Ar path +option. +When this option is provided, the directory specified by the +option is considered to be the root of the build directory. +For example, considering our previous fake tree layout, we could invoke +.Nm +as any of the following: +.Bd -literal -offset indent +$ kyua __COMMAND__ --kyuafile=src/Kyuafile --build-root=obj +$ cd src && kyua __COMMAND__ --build-root=../obj +.Ed diff --git a/doc/kyua-about.1.in b/doc/kyua-about.1.in new file mode 100644 index 000000000000..1ea134810e65 --- /dev/null +++ b/doc/kyua-about.1.in @@ -0,0 +1,95 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd May 20, 2015 +.Dt KYUA-ABOUT 1 +.Os +.Sh NAME +.Nm "kyua about" +.Nd Shows detailed authors, license, and version information +.Sh SYNOPSIS +.Nm +.Op Ar authors | license | version +.Sh DESCRIPTION +The +.Sq about +command provides generic information about the +.Xr kyua 1 +tool. +In the default synopsis form (no arguments), the information printed +includes: +.Bl -enum +.It +The name of the package, which is +.Sq __PACKAGE__ . +.It +The version number, which is +.Sq __VERSION__ . +.It +License information. +.It +Authors information. +.It +A link to the project web site. +.El +.Pp +You can customize the information printed by this command by specifying +the desired topic as the single argument to the command. +This can be one of: +.Bl -tag -width authorsXX +.It Ar authors +Displays the list of authors and contributors only. +.It Ar license +Displays the license information and the list of copyrights. +.It Ar version +Displays the package name and the version number in a format that is +compatible with the output of GNU tools that support a +.Fl -version +flag. +Use this whenever you have to query the version number of the package. +.El +.Sh FILES +The following files are read by the +.Nm +command: +.Bl -tag -width XX +.It Pa __DOCDIR__/AUTHORS +List of authors (aka copyright holders). +.It Pa __DOCDIR__/CONTRIBUTORS +List of contributors (aka individuals that have contributed to the project). +.It Pa __DOCDIR__/LICENSE +License information. +.El +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/doc/kyua-config.1.in b/doc/kyua-config.1.in new file mode 100644 index 000000000000..9c13ce06505e --- /dev/null +++ b/doc/kyua-config.1.in @@ -0,0 +1,59 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd September 9, 2012 +.Dt KYUA-CONFIG 1 +.Os +.Sh NAME +.Nm "kyua config" +.Nd Inspects the values of the loaded configuration +.Sh SYNOPSIS +.Nm +.Op Ar variable1 .. variableN +.Sh DESCRIPTION +The +.Nm +command provides a way to list all defined configuration variables and +their current values. +.Pp +This command is intended to help you in resolving the values of the +configuration variables without having to scan over configuration files. +.Pp +In the default synopsis form (no arguments), the command prints all +configuration variables. +If any arguments are provided, the command will only print the +requested variables. +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if any of the specified configuration +variables does not exist. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/doc/kyua-db-exec.1.in b/doc/kyua-db-exec.1.in new file mode 100644 index 000000000000..04f34c7b54e7 --- /dev/null +++ b/doc/kyua-db-exec.1.in @@ -0,0 +1,80 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-DB-EXEC 1 +.Os +.Sh NAME +.Nm "kyua db-exec" +.Nd Executes a SQL statement in a results file +.Sh SYNOPSIS +.Nm +.Op Fl -no-headers +.Op Fl -results-file Ar file +.Ar statement +.Sh DESCRIPTION +The +.Nm +command provides a way to execute an arbitrary SQL statement within the +database. +This command is mostly intended to aid in debugging, but can also be used to +extract information from the database when the current interfaces do not +provide the desired functionality. +.Pp +The input database must exist. +It makes no sense to use +.Nm +on a nonexistent or empty database. +.Pp +The +.Nm +command takes one or more arguments, all of which are concatenated to form +a single SQL statement. +Once the statement is executed, +.Nm +prints the resulting table on the screen, if any. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -no-headers +Avoids printing the headers of the table in the output of the command. +.It Fl -results-file Ar path , Fl s Ar path +__include__ results-file-flag-read.mdoc +.El +.Ss Results files +__include__ results-files.mdoc +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if the SQL statement is invalid or fails +to run. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-test 1 diff --git a/doc/kyua-db-migrate.1.in b/doc/kyua-db-migrate.1.in new file mode 100644 index 000000000000..67e46de46fec --- /dev/null +++ b/doc/kyua-db-migrate.1.in @@ -0,0 +1,63 @@ +.\" Copyright 2013 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-DB-MIGRATE 1 +.Os +.Sh NAME +.Nm "kyua db-migrate" +.Nd Upgrades the schema of an existing results file +.Sh SYNOPSIS +.Nm +.Op Fl -results-file Ar file +.Sh DESCRIPTION +The +.Nm +command migrates the schema of an existing database to the latest +version implemented in +.Xr kyua 1 . +.Pp +This operation is not reversible. +However, a backup of the database is created in the same directory where the +database lives. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -results-file Ar path , Fl s Ar path +__include__ results-file-flag-read.mdoc +.El +.Ss Results files +__include__ results-files.mdoc +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if the migration fails. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/doc/kyua-debug.1.in b/doc/kyua-debug.1.in new file mode 100644 index 000000000000..9e962a465421 --- /dev/null +++ b/doc/kyua-debug.1.in @@ -0,0 +1,145 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-DEBUG 1 +.Os +.Sh NAME +.Nm "kyua debug" +.Nd Executes a single test case with facilities for debugging +.Sh SYNOPSIS +.Nm +.Op Fl -build-root Ar path +.Op Fl -kyuafile Ar file +.Op Fl -stdout Ar path +.Op Fl -stderr Ar path +.Ar test_case +.Sh DESCRIPTION +The +.Nm +command provides a mechanism to execute a single test case bypassing some +of the Kyua infrastructure and allowing the user to poke into the execution +behavior of the test. +.Pp +The test case to run is selected by providing a test filter, described below in +.Sx Test filters , +that matches a single test case. +The test case is executed and its result is printed as the last line of the +output of the tool. +.Pp +The test executed by +.Nm +is run under a controlled environment as described in +.Sx Test isolation . +.Pp +At the moment, the +.Nm +command allows the following aspects of a test case execution to be +tweaked: +.Bl -bullet +.It +Redirection of the test case's stdout and stderr to the console (the +default) or to arbitrary files. +See the +.Fl -stdout +and +.Fl -stderr +options below. +.El +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -build-root Ar path +Specifies the build root in which to find the test programs referenced +by the Kyuafile, if different from the Kyuafile's directory. +See +.Sx Build directories +below for more information. +.It Fl -kyuafile Ar file , Fl k Ar file +Specifies the Kyuafile to process. +Defaults to +.Pa Kyuafile +file in the current directory. +.It Fl -stderr Ar path +Specifies the file to which to send the standard error of the test +program's body. +The default is +.Pa /dev/stderr , +which is a special character device that redirects the output to +standard error on the console. +.It Fl -stdout Ar path +Specifies the file to which to send the standard output of the test +program's body. +The default is +.Pa /dev/stdout , +which is a special character device that redirects the output to +standard output on the console. +.El +.Pp +For example, consider the following Kyua session: +.Bd -literal -offset indent +$ kyua test +kernel/fs:mkdir -> passed +kernel/fs:rmdir -> failed: Invalid argument + +1/2 passed (1 failed) +.Ed +.Pp +At this point, we do not have a lot of information regarding the +failure of the +.Sq kernel/fs:rmdir +test. +We can run this test through the +.Nm +command to inspect its output a bit closer, hoping that the test case is +kind enough to log its progress: +.Bd -literal -offset indent +$ kyua debug kernel/fs:rmdir +Trying rmdir('foo') +Trying rmdir(NULL) +kernel/fs:rmdir -> failed: Invalid argument +.Ed +.Pp +Luckily, the offending test case was printing status lines as it +progressed, so we could see the last attempted call and we can know match +the failure message to the problem. +.Ss Build directories +__include__ build-root.mdoc COMMAND=debug +.Ss Test filters +__include__ test-filters.mdoc +.Ss Test isolation +__include__ test-isolation.mdoc +.Sh EXIT STATUS +The +.Nm +command returns 0 if the test case passes or 1 if the test case fails. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyuafile 5 diff --git a/doc/kyua-help.1.in b/doc/kyua-help.1.in new file mode 100644 index 000000000000..2c4f2bc3859e --- /dev/null +++ b/doc/kyua-help.1.in @@ -0,0 +1,64 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd September 9, 2012 +.Dt KYUA-HELP 1 +.Os +.Sh NAME +.Nm "kyua help" +.Nd Shows usage information +.Sh SYNOPSIS +.Nm +.Op Ar command +.Sh DESCRIPTION +The +.Nm +command provides interactive help on all supported commands and options. +If, for some reason, you happen to spot a discrepancy in the output of this +command and this document, the command is the authoritative source of +information. +.Pp +If no arguments are provided, the command prints the list of common options +and the list of supported subcommands. +.Pp +If the +.Ar command +argument is provided to, this single argument is the name of a valid +subcommand. +In that case, +.Nm +prints a textual description of the command, the list of common options and +the list of subcommand-specific options. +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/doc/kyua-list.1.in b/doc/kyua-list.1.in new file mode 100644 index 000000000000..5774354d9236 --- /dev/null +++ b/doc/kyua-list.1.in @@ -0,0 +1,90 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-LIST 1 +.Os +.Sh NAME +.Nm "kyua list" +.Nd Lists test cases and their metadata +.Sh SYNOPSIS +.Nm +.Op Fl -build-root Ar path +.Op Fl -kyuafile Ar file +.Op Fl -verbose +.Ar test_case1 Op Ar .. test_caseN +.Sh DESCRIPTION +The +.Nm +command scans all the test programs and test cases in a test suite (as +defined by a +.Xr kyuafile 5 ) +and prints a list of all their names, optionally accompanied by any metadata +properties they have. +.Pp +The optional arguments to +.Nm +are used to select which test programs or test cases to run. +These are filters and are described below in +.Sx Test filters . +.Pp +This command must be run within a test suite or a test suite must be +provided with the +.Fl -kyuafile +flag. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -build-root Ar path +Specifies the build root in which to find the test programs referenced +by the Kyuafile, if different from the Kyuafile's directory. +See +.Sx Build directories +below for more information. +.It Fl -kyuafile Ar path , Fl k Ar path +Specifies the Kyuafile to process. +Defaults to a +.Pa Kyuafile +file in the current directory. +.It Fl -verbose , Fl v +Prints metadata properties for every test case. +.El +.Ss Build directories +__include__ build-root.mdoc COMMAND=list +.Ss Test filters +__include__ test-filters.mdoc +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if any of the given test case filters +does not match any test case. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyuafile 5 diff --git a/doc/kyua-report-html.1.in b/doc/kyua-report-html.1.in new file mode 100644 index 000000000000..1f9f55b69a3f --- /dev/null +++ b/doc/kyua-report-html.1.in @@ -0,0 +1,103 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-REPORT-HTML 1 +.Os +.Sh NAME +.Nm "kyua report-html" +.Nd Generates an HTML report with the results of a test suite run +.Sh SYNOPSIS +.Nm +.Op Fl -force +.Op Fl -output Ar path +.Op Fl -results-file Ar file +.Op Fl -results-filter Ar types +.Sh DESCRIPTION +The +.Nm +command provides a simple mechanism to generate HTML reports of the +execution of a test suite. +The command processes a results file and then populates a directory with +multiple HTML and supporting files to describe the results recorded in that +results file. +.Pp +The HTML output is static and self-contained, so it can easily be served by +any simple web server. +The command expects the target directory to not exist, because it would +overwrite any contents if not careful. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -force +Forces the deletion of the output directory if it exists. +Use care, as this effectively means a +.Sq rm -rf . +.It Fl -output Ar directory +Specifies the target directory into which to generate the HTML files. +The directory must not exist unless the +.Fl -force +option is provided. +The default is +.Pa ./html . +.It Fl -results-file Ar path , Fl s Ar path +__include__ results-file-flag-read.mdoc +.It Fl -results-filter Ar types +Comma-separated list of the test result types to include in the report. +The ordering of the values is respected so that you can determine how you +want the list of tests to be shown. +.Pp +The valid values are: +.Sq broken , +.Sq failed , +.Sq passed , +.Sq skipped +and +.Sq xfail . +If the parameter supplied to the option is empty, filtering is suppressed +and all result types are shown in the report. +.Pp +The default value for this flag includes all the test results except the +passed tests. +Showing the passed tests by default clutters the report with too much +information, so only abnormal conditions are included. +.El +.Ss Results files +__include__ results-files.mdoc +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +__include__ results-files-report-example.mdoc REPORT_COMMAND=report-html +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report 1 , +.Xr kyua-report-junit 1 diff --git a/doc/kyua-report-junit.1.in b/doc/kyua-report-junit.1.in new file mode 100644 index 000000000000..f1ad3a2e7f29 --- /dev/null +++ b/doc/kyua-report-junit.1.in @@ -0,0 +1,87 @@ +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-REPORT-JUNIT 1 +.Os +.Sh NAME +.Nm "kyua report-junit" +.Nd Generates a JUnit report with the results of a test suite run +.Sh SYNOPSIS +.Nm +.Op Fl -output Ar path +.Op Fl -results-file Ar file +.Sh DESCRIPTION +The +.Nm +command provides a simple mechanism to generate JUnit reports of the +execution of a test suite. +The command processes a results file and then generates a single XML file +that complies with the JUnit XSchema. +.Pp +The JUnit output is static and self-contained, so it can easily be plugged +into any continuous integration system, like Jenkins. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -output Ar directory +Specifies the file into which to store the JUnit report. +.It Fl -results-file Ar path , Fl s Ar path +__include__ results-file-flag-read.mdoc +.El +.Ss Caveats +Because of limitations in the JUnit XML schema, not all the data collected by +Kyua can be properly represented in JUnit reports. +However, because test data are extremely useful for debugging purposes, the +.Nm +command shovels these data into the JUnit output. +In particular: +.Bl -bullet +.It +The test case metadata values are prepended to the test case's standard error +output. +.It +Test cases that report expected failures as their results are recorded as +passed. +The fact that they failed as expected is recorded in the test case's standard +error output along with the corresponding reason. +.El +.Ss Results files +__include__ results-files.mdoc +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +__include__ results-files-report-example.mdoc REPORT_COMMAND=report-junit +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report 1 , +.Xr kyua-report-html 1 diff --git a/doc/kyua-report.1.in b/doc/kyua-report.1.in new file mode 100644 index 000000000000..8e2485f9c4ac --- /dev/null +++ b/doc/kyua-report.1.in @@ -0,0 +1,118 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-REPORT 1 +.Os +.Sh NAME +.Nm "kyua report" +.Nd Generates reports with the results of a test suite run +.Sh SYNOPSIS +.Nm +.Op Fl -output Ar path +.Op Fl -results-file Ar file +.Op Fl -results-filter Ar types +.Op Fl -verbose +.Op Ar test_filter1 .. test_filterN +.Sh DESCRIPTION +The +.Nm +command parses a results file and generates a user-friendly, plaintext +report for user consumption on the terminal. +By default, these reports only display a summary of the execution of the full +test suite to highlight where problems may lie. +.Pp +The output of +.Nm +can be customized to display full details on all executed test cases. +Additionally, the optional arguments to +.Nm +are used to select which test programs or test cases to display. +These are filters and are described below in +.Sx Test filters . +.Pp +Reports generated by +.Nm +are +.Em not intended to be machine-parseable . +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -output Ar path +Specifies the path to which the report should be written to. +The special values +.Pa /dev/stdout +and +.Pa /dev/stderr +can be used to specify the standard output and the standard error, +respectively. +.It Fl -results-file Ar path , Fl s Ar path +__include__ results-file-flag-read.mdoc +.It Fl -results-filter Ar types +Comma-separated list of the test result types to include in the report. +The ordering of the values is respected so that you can determine how you +want the list of tests to be shown. +.Pp +The valid values are: +.Sq broken , +.Sq failed , +.Sq passed , +.Sq skipped +and +.Sq xfail . +If the parameter supplied to the option is empty, filtering is suppressed +and all result types are shown in the report. +.Pp +The default value for this flag includes all the test results except the +passed tests. +Showing the passed tests by default clutters the report with too much +information, so only abnormal conditions are included. +.It Fl -verbose +Prints a detailed report of the execution. +In addition to all the information printed by default, verbose reports +include the runtime context of the test suite run, the metadata of each +test case, and the verbatim output of the test cases. +.El +.Ss Results files +__include__ results-files.mdoc +.Ss Test filters +__include__ test-filters.mdoc +.Sh EXIT STATUS +The +.Nm +command returns 0 if no filters were specified or if all filters match one +or more test cases. +If any filter fails to match any test case, the command returns 1. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +__include__ results-files-report-example.mdoc REPORT_COMMAND=report +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report-html 1 , +.Xr kyua-report-junit 1 diff --git a/doc/kyua-test.1.in b/doc/kyua-test.1.in new file mode 100644 index 000000000000..8cd5f34ae6af --- /dev/null +++ b/doc/kyua-test.1.in @@ -0,0 +1,102 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-TEST 1 +.Os +.Sh NAME +.Nm "kyua test" +.Nd Runs tests +.Sh SYNOPSIS +.Nm +.Op Fl -build-root Ar path +.Op Fl -kyuafile Ar file +.Op Fl -results-file Ar file +.Op Ar test_filter1 .. test_filterN +.Sh DESCRIPTION +The +.Nm +command loads a test suite definition from a +.Xr kyuafile 5 , +runs the tests defined in it, and records the results into a new results +file. +By default, all tests in the test suite are executed but the optional +arguments to +.Nm +can be used to select which test programs or test cases to run. +These are filters and are described below in +.Sx Test filters . +.Pp +Every test executed by +.Nm +is run under a controlled environment as described in +.Sx Test isolation . +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -build-root Ar path +Specifies the build root in which to find the test programs referenced by +the Kyuafile, if different from the Kyuafile's directory. +See +.Sx Build directories +below for more information. +.It Fl -kyuafile Ar path , Fl k Ar path +Specifies the Kyuafile to process. +Defaults to a +.Pa Kyuafile +file in the current directory. +.It Fl -results-file Ar path , Fl s Ar path +__include__ results-file-flag-write.mdoc +.El +.Pp +You can later inspect the results of the test run in more detail by using +.Xr kyua-report 1 +or you can execute a single test case with debugging functionality by using +.Xr kyua-debug 1 . +.Ss Build directories +__include__ build-root.mdoc COMMAND=test +.Ss Results files +__include__ results-files.mdoc +.Ss Test filters +__include__ test-filters.mdoc +.Ss Test isolation +__include__ test-isolation.mdoc +.Sh EXIT STATUS +The +.Nm +command returns 0 if all executed test cases pass or 1 if any of the +executed test cases fails or if any of the given test case filters does not +match any test case. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +__include__ results-files-report-example.mdoc REPORT_COMMAND=report +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report 1 , +.Xr kyuafile 5 diff --git a/doc/kyua.1.in b/doc/kyua.1.in new file mode 100644 index 000000000000..2fca5eb09f9f --- /dev/null +++ b/doc/kyua.1.in @@ -0,0 +1,400 @@ +.\" Copyright 2011 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd May 12, 2015 +.Dt KYUA 1 +.Os +.Sh NAME +.Nm kyua +.Nd Testing framework for infrastructure software +.Sh SYNOPSIS +.Nm +.Op Fl -config Ar file +.Op Fl -logfile Ar file +.Op Fl -loglevel Ar level +.Op Fl -variable Ar name=value +.Ar command +.Op Ar command_options +.Op Ar command_arguments +.Sh DESCRIPTION +.Em If you are here looking for details on how to run the test suite in +.Pa /usr/tests +.Em ( or +.Pa __TESTSDIR__ ) , +.Em please start by reading the +.Xr tests 7 +.Em manual page that should be supplied by your system . +.Pp +Kyua is a testing framework for infrastructure software, originally +designed to equip BSD-based operating systems with a test suite. +This means that Kyua is lightweight and simple, and that Kyua integrates well +with various build systems and continuous integration frameworks. +.Pp +Kyua features an expressive test suite definition language, a safe +runtime engine for test suites and a powerful report generation engine. +.Pp +Kyua is for both developers and users, from the developer applying a +simple fix to a library to the system administrator deploying a new +release on a production machine. +.Pp +Kyua is able to execute test programs written with a plethora of testing +libraries and languages. +The test program library of choice is ATF, which +.Nm Ns 's +design originated from. +However, framework-less test programs and TAP-compliant test programs can also +be executed through +.Nm +.Ss Overview +As can be observed in the synopsis, the interface of +.Nm +implements a common subcommand-based interface. +The arguments to the tool specify, in this order: a set of common options +that all the commands accept, a required +.Ar command +name that specifies what +.Nm +should do, and +a set of possibly-optional +.Ar command_options +and +.Ar command_arguments +that are specific to the chosen command. +.Pp +The following options are recognized by all the commands. +Keep in mind that these must always be specified before the command name. +.Bl -tag -width XX +.It Fl -config Ar path , Fl c Ar path +Specifies the configuration file to process, which must be in the format +described in +.Xr kyua.conf 5 . +The special value +.Sq none +explicitly disables the loading of any configuration file. +.Pp +Defaults to +.Pa ~/.kyua/kyua.conf +if it exists, otherwise to +.Pa __CONFDIR__/kyua.conf +if it exists, +or else to +.Sq none . +.It Fl -logfile Ar path +Specifies the location of the file to which +.Nm +will log run time events useful for postmortem debugging. +.Pp +The default depends on different environment variables as described in +.Sx Logging , +but typically the file will be stored within the user's home directory. +.It Fl -loglevel Ar level +Specifies the maximum logging level to record in the log file. +See +.Sx Logging +for more details. +.Pp +The default is +.Sq info . +.It Fl -variable Ar name=value , Fl v Ar name=value +Sets the +.Ar name +configuration variable to +.Ar value . +The values set through this option have preference over the values set in the +configuration file. +.Pp +The specified variable can either be a builtin variable or a test-suite +specific variable. +See +.Xr kyua.conf 5 +for more details. +.El +.Pp +The following commands are generic and do not have any relation to the execution +of tests or the inspection of their results: +.Bl -tag -width reportXjunitXX -offset indent +.It Ar about +Shows general program information. +See +.Xr kyua-about 1 . +.It Ar config +Inspects the values of the configuration variables. +See +.Xr kyua-config 1 . +.It Ar db-exec +Executes an arbitrary SQL statement on a results file and prints the +resulting table. +See +.Xr kyua-db-exec 1 . +.It Ar help +Shows usage information. +See +.Xr kyua-help 1 . +.El +.Pp +The following commands are used to generate reports based on the data previously +recorded in a results file: +.Bl -tag -width reportXjunitXX -offset indent +.It Ar report +Generates a plaintext report. +Combined with its +.Fl -verbose +flag and the ability to only display specific test cases, this command can also +be used to debug test failures post-facto on the console. +See +.Xr kyua-report 1 . +.It Ar report-html +Generates an HTML report. +See +.Xr kyua-report-html 1 . +.It Ar report-junit +Generates a JUnit report. +See +.Xr kyua-report-junit 1 . +.El +.Pp +The following commands are used to interact with a test suite: +.Bl -tag -width reportXjunitXX -offset indent +.It Ar debug +Executes a single test case in a controlled environment for debugging purposes. +See +.Xr kyua-debug 1 . +.It Ar list +Lists test cases defined in a test suite by a +.Xr kyuafile 5 +and, optionally, displays their metadata. +See +.Xr kyua-list 1 . +.It Ar test +Runs tests defined in a test suite by a +.Xr kyuafile 5 . +See +.Xr kyua-test 1 . +.El +.Ss Logging +.Nm +has a logging facility that collects all kinds of events at run time. +These events are always logged to a file so that the log is available when +it is most needed: right after a non-reproducible problem happens. +The only way to disable logging is by sending the log to +.Pa /dev/null . +.Pp +The location of the log file can be manually specified with the +.Fl -logfile +option, which applies to all commands. +If no file is explicitly specified, the location of the log files is chosen in +this order: +.Bl -enum -offset indent +.It +.Pa ${HOME}/.kyua/logs/ +if +.Va HOME +is defined. +.It +.Pa ${TMPDIR}/ +if +.Va TMPDIR +is defined. +.It +.Pa /tmp/ . +.El +.Pp +And the default naming scheme of the log files is: +.Sq <progname>.<timestamp>.log . +.Pp +The messages stored in the log file have a level (or severity) attached to +them. +These are: +.Bl -tag -width warningXX -offset indent +.It error +Fatal error messages. +The program generally terminates after these, either in a clean manner or by +crashing. +.It warning +Non-fatal error messages. +These generally report a condition that must be addressed but the application +can continue to run. +.It info +Informational messages. +These tell the user what the program was doing at a general level of +operation. +.It debug +Detailed informational messages. +These are often useful when debugging problems in the application, as they +contain lots of internal details. +.El +.Pp +The default log level is +.Sq info +unless explicitly overridden with +.Fl -loglevel . +.Pp +The log file is a plain text file containing one line per log record. +The format of each line is as follows: +.Bd -literal -offset indent +timestamp entry_type pid file:line: message +.Ed +.Pp +.Ar entry_type +can be one of: +.Sq E +for an error, +.Sq W +for a warning, +.Sq I +for an informational message and +.Sq D +for a debug message. +.Ss Bug reporting +If you think you have encountered a bug in +.Nm , +please take the time to let the developers know about it. +This will ensure that the bug is addressed and potentially fixed in the next +Kyua release. +.Pp +The first step in reporting a bug is to check if there already is a similar +bug in the database. +You can check what issues are currently in the database by going to: +.Bd -literal -offset indent +https://github.com/jmmv/kyua/issues/ +.Ed +.Pp +If there is no existing issue that describes an issue similar to the +one you are experiencing, you can open a new one by visiting: +.Bd -literal -offset indent +https://github.com/jmmv/kyua/issues/new/ +.Ed +.Pp +When doing so, please include as much detail as possible. +Among other things, explain what operating system and platform you are running +.Nm +on, what were you trying to do, what exact messages you saw on the screen, +how did you expect the program to behave, and any other details that you +may find relevant. +.Pp +Also, please include a copy of the log file corresponding to the problem +you are experiencing. +Unless you have changed the location of the log files, you can most likely +find them in +.Pa ~/.kyua/logs/ . +If the problem is reproducible, it is good idea to regenerate the log file +with an increased log level so as to provide more information. +For example: +.Bd -literal -offset indent +$ kyua --logfile=problem.log --loglevel=debug \\ + [rest of the command line] +.Ed +.Sh ENVIRONMENT +The following variables are recognized and can be freely tuned by the end user: +.Bl -tag -width COLUMNSXX +.It Va COLUMNS +The width of the screen, in number of characters. +.Nm +uses this to wrap long lines. +If not present, the width of the screen is determined from the terminal +stdout is connected to, and, if the guessing fails, this defaults to infinity. +.It Va HOME +Path to the user's home directory. +.Nm +uses this location to determine paths to configuration files and default log +files. +.It Va TMPDIR +Path to the system-wide temporary directory. +.Nm +uses this location to place the work directory of test cases, among other +things. +.Pp +The default value of this variable depends on the operating system. +In general, it is +.Pa /tmp . +.El +.Pp +The following variables are also recognized, but you should not need to set them +during normal operation. +They are only provided to override the value of built-in values, which is useful +when testing +.Nm +itself: +.Bl -tag -width KYUAXCONFDIRXX +.It Va KYUA_CONFDIR +Path to the system-wide configuration files for +.Nm . +.Pp +Defaults to +.Pa __CONFDIR__ . +.It Va KYUA_DOCDIR +Path to the location of installed documentation. +.Pp +Defaults to +.Pa __DOCDIR__ . +.It Va KYUA_MISCDIR +Path to the location of the installed miscellaneous scripts and data +files provided by +.Nm . +.Pp +Defaults to +.Pa __MISCDIR__ . +.It Va KYUA_STOREDIR +Path to the location of the installed store support files; e.g., the +directory containing the SQL database schema. +.Pp +Defaults to +.Pa __STOREDIR__ . +.El +.Sh FILES +.Bl -tag -width XXXX +.It Pa ~/.kyua/store/ +Default location for the results files. +.It Pa ~/.kyua/kyua.conf +User-specific configuration file. +.It Pa ~/.kyua/logs/ +Default location for the collected log files. +.It Pa __CONFDIR__/kyua.conf +System-wide configuration file. +.El +.Sh EXIT STATUS +.Nm +returns 0 on success, 1 on a controlled error condition in the given +subcommand, 2 on a general unexpected error and 3 on a usage error. +.Pp +The documentation of the subcommands in the corresponding manual pages only +details the difference between a successful exit (0) and the detection of a +controlled error (1). +Even though when those manual pages do not describe any other exit statuses, +codes above 1 can be returned. +.Sh SEE ALSO +.Xr kyua.conf 5 , +.Xr kyuafile 5 , +.Xr atf 7 , +.Xr tests 7 +.Sh AUTHORS +For more details on the people that made +.Nm +possible and the license terms, run: +.Bd -literal -offset indent +$ kyua about +.Ed diff --git a/doc/kyua.conf.5.in b/doc/kyua.conf.5.in new file mode 100644 index 000000000000..05a9499b48c4 --- /dev/null +++ b/doc/kyua.conf.5.in @@ -0,0 +1,141 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd February 20, 2015 +.Dt KYUA.CONF 5 +.Os +.Sh NAME +.Nm kyua.conf +.Nd Configuration file for the kyua tool +.Sh SYNOPSIS +.Fn syntax "int version" +.Pp +Variables: +.Va architecture , +.Va platform , +.Va test_suites , +.Va unprivileged_user . +.Sh DESCRIPTION +The configuration of Kyua is a simple collection of key/value pairs called +configuration variables. +There are configuration variables that have a special meaning to the runtime +engine implemented by +.Xr kyua 1 , +and there are variables that only have meaning in the context of particular +test suites. +.Pp +Configuration files are Lua scripts. +In their most basic form, their whole purpose is to assign values to +variables, but the user has the freedom to implement any logic he desires +to compute such values. +.Ss File versioning +Every +.Nm +file starts with a call to +.Fn syntax "int version" . +This call determines the specific schema used by the file so that future +backwards-incompatible modifications to the file can be introduced. +.Pp +Any new +.Nm +file should set +.Fa version +to +.Sq 2 . +.Ss Runtime configuration variables +The following variables are internally recognized by +.Xr kyua 1 : +.Bl -tag -width XX -offset indent +.It Va architecture +Name of the system architecture (aka processor type). +.It Va parallelism +Maximum number of test cases to execute concurrently. +.It Va platform +Name of the system platform (aka machine type). +.It Va unprivileged_user +Name or UID of the unprivileged user. +.Pp +If set, the given user must exist in the system and his privileges will be +used to run test cases that need regular privileges when +.Xr kyua 1 +is executed as root. +.El +.Ss Test-suite configuration variables +Each test suite is able to recognize arbitrary configuration variables, and +their type and meaning is specific to the test suite. +Because the existence and naming of these variables depends on every test +suite, this manual page cannot detail them; please refer to the documentation +of the test suite you are working with for more details on this topic. +.Pp +Test-suite specific configuration variables are defined inside the +.Va test_suites +dictionary. +The general syntax is: +.Bd -literal -offset indent +test_suites.<test_suite_name>.<variable_name> = <value> +.Ed +.Pp +where +.Va test_suite_name +is the name of the test suite, +.Va variable_name +is the name of the variable to set, and +.Va value +is a value. +The value can be a string, an integer or a boolean. +.Sh FILES +.Bl -tag -width XX +.It __EGDIR__/kyua.conf +Sample configuration file. +.El +.Sh EXAMPLES +The following +.Nm +shows a simple configuration file that overrides a bunch of the built-in +.Xr kyua 1 +configuration variables: +.Bd -literal -offset indent +syntax(2) + +architecture = 'x86_64' +platform = 'amd64' +.Ed +.Pp +The following is a more complex example that introduces the definition of +per-test suite configuration variables: +.Bd -literal -offset indent +syntax(2) + +-- Assign built-in variables. +unprivileged_user = '_tests' + +-- Assign test-suite variables. All of these must be strings. +test_suites.NetBSD.file_systems = 'ffs ext2fs' +test_suites.X11.graphics_driver = 'vesa' +.Ed +.Sh SEE ALSO +.Xr kyua 1 diff --git a/doc/kyuafile.5.in b/doc/kyuafile.5.in new file mode 100644 index 000000000000..06cb2dbc42a8 --- /dev/null +++ b/doc/kyuafile.5.in @@ -0,0 +1,407 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd July 3, 2015 +.Dt KYUAFILE 5 +.Os +.Sh NAME +.Nm Kyuafile +.Nd Test suite description files +.Sh SYNOPSIS +.Fn atf_test_program "string name" "[string metadata]" +.Fn current_kyuafile +.Fn fs.basename "string path" +.Fn fs.dirname "string path" +.Fn fs.exists "string path" +.Fn fs.files "string path" +.Fn fs.is_absolute "string path" +.Fn fs.join "string path" "string path" +.Fn include "string path" +.Fn plain_test_program "string name" "[string metadata]" +.Fn syntax "int version" +.Fn tap_test_program "string name" "[string metadata]" +.Fn test_suite "string name" +.Sh DESCRIPTION +A test suite is a collection of test programs and is represented by a +hierarchical layout of test binaries on the file system. +Any subtree of the file system can represent a test suite, provided that it +includes one or more +.Nm Ns s , +which are the test suite definition files. +.Pp +A +.Nm +is a Lua script whose purpose is to describe the structure of the test +suite it belongs to. +To do so, the script has access to a collection of special functions provided +by +.Xr kyua 1 +as described in +.Sx Helper functions . +.Ss File versioning +Every +.Nm +file starts with a call to +.Fn syntax "int version" . +This call determines the specific schema used by the file so that future +backwards-incompatible modifications to the file can be introduced. +.Pp +Any new +.Nm +file should set +.Fa version +to +.Sq 2 . +.Ss Test suite definition +If the +.Nm +registers any test programs, +the +.Nm +must define the name of the test suite the test programs belong to by using the +.Fn test_suite +function at the very beginning of the file. +.Pp +The test suite name provided in the +.Fn test_suite +call tells +.Xr kyua 1 +which set of configuration variables from +.Xr kyua.conf 5 +to pass to the test programs at run time. +.Ss Test program registration +A +.Nm +can register test programs by means of a variety of +.Fn *_test_program +functions, all of which take the name of a test program and a set of +optional metadata properties that describe such test program. +.Pp +The test programs to be registered must live in the current directory; in +other words, the various +.Fn *_test_program +calls cannot reference test programs in other directories. +The rationale for this is to force all +.Nm +files to be self-contained, and to simplify their internal representation. +.Pp +.Em ATF test programs +are those that use the +.Xr atf 7 +libraries. +They can be registered with the +.Fn atf_test_program +table constructor. +This function takes the +.Fa name +of the test program and a collection of optional metadata settings for all +the test cases in the test program. +Any metadata properties defined by the test cases themselves override the +metadata values defined here. +.Pp +.Em Plain test programs +are those that return 0 on success and non-0 on failure; in general, most test +programs (even those that use fancy unit-testing libraries) behave this way and +thus also qualify as plain test programs. +They can be registered with the +.Fn plain_test_program +table constructor. +This function takes the +.Fa name +of the test program, an optional +.Fa test_suite +name that overrides the global test suite name, and a collection of optional +metadata settings for the test program. +.Pp +.Em TAP test programs +are those that implement the Test Anything Protocol. +They can be registered with the +.Fn tap_test_program +table constructor. +This function takes the +.Fa name +of the test program and a collection of optional metadata settings for the +test program. +.Pp +The following metadata properties can be passed to any test program definition: +.Bl -tag -width XX -offset indent +.It Va allowed_architectures +Whitespace-separated list of machine architecture names allowed by the test. +If empty or not defined, the test is allowed to run on any machine +architecture. +.It Va allowed_platforms +Whitespace-separated list of machine platform names allowed by the test. +If empty or not defined, the test is allowed to run on any machine +platform. +.It Va custom.NAME +Custom variable defined by the test where +.Sq NAME +denotes the name of the variable. +These variables are useful to tag your tests with information specific to +your project. +The values of such variables are propagated all the way from the tests to the +results files and later to any generated reports. +.Pp +Note that if the name happens to have dashes or any other special characters +in it, you will have to use a special Lua syntax to define the property. +Refer to the +.Sx EXAMPLES +section below for clarification. +.It Va description +Textual description of the test. +.It Va is_exclusive +If true, indicates that this test program cannot be executed along any other +programs at the same time. +Test programs that affect global system state, such as those that modify the +value of a +.Xr sysctl 8 +setting, must set themselves as exclusive to prevent failures due to race +conditions. +Defaults to false. +.It Va required_configs +Whitespace-separated list of configuration variables that the test requires +to be defined before it can run. +.It Va required_disk_space +Amount of available disk space that the test needs to run successfully. +.It Va required_files +Whitespace-separated list of paths that the test requires to exist before +it can run. +.It Va required_memory +Amount of physical memory that the test needs to run successfully. +.It Va required_programs +Whitespace-separated list of basenames or absolute paths pointing to executable +binaries that the test requires to exist before it can run. +.It Va required_user +If empty, the test has no restrictions on the calling user for it to run. +If set to +.Sq unprivileged , +the test needs to not run as root. +If set to +.Sq root , +the test must run as root. +.It Va timeout +Amount of seconds that the test is allowed to execute before being killed. +.El +.Ss Recursion +To reference test programs in another subdirectory, a different +.Nm +must be created in that directory and it must be included into the original +.Nm +by means of the +.Fn include +function. +.Pp +.Fn include +may only be called with a relative path and with at most one directory +component. +This is by design: Kyua uses the file system structure as the layout of the +test suite definition. +Therefore, each subdirectory in a test suite must include its own +.Nm +and each +.Nm +can only descend into the +.Nm Ns s +of immediate subdirectories. +.Pp +If you need to source a +.Nm +located in disjoint parts of your file system namespace, you will have to +create a +.Sq shadow tree +using symbolic links and possibly helper +.Nm Ns s +to plug the various subdirectories together. +See the +.Sx EXAMPLES +section below for details. +.Pp +Note that each file is processed in its own Lua environment: there is no +mechanism to pass state from one file to the other. +The reason for this is that there is no such thing as a +.Dq top-level +.Nm +in a test suite: the user has to be able to run the test suite from any +directory in a given hierarchy, and this execution must not depend on files +that live in parent directories. +.Ss Top-level Kyuafile +Every system has a top directory into which test suites get installed. +The default is +.Pa __TESTSDIR__ . +Within this directory live test suites, each of which is in an independent +subdirectory. +Each subdirectory can be provided separately by independent third-party +packages. +.Pp +Kyua allows running all the installed test suites at once in order to +provide comprehensive cross-component reports. +In order to do this, there is a special file in the top directory that knows +how to inspect the subdirectories in search for other Kyuafiles and include +them. +.Pp +The +.Sx FILES +section includes more details on where this file lives. +.Ss Helper functions +The +.Sq base , +.Sq string , +and +.Sq table +Lua modules are fully available in the context of a +.Nm . +.Pp +The following extra functions are provided by Kyua: +.Bl -tag -width XX -offset indent +.It Ft string Fn current_kyuafile +Returns the absolute path to the current +.Nm . +.It Ft string Fn fs.basename "string path" +Returns the last component of the given path. +.It Ft string Fn fs.dirname "string path" +Returns the given path without its last component or a dot if the path has +a single component. +.It Ft bool Fn fs.exists "string path" +Checks if the given path exists. +If the path is not absolute, it is relative to the directory containing the +.Nm +in which the call to this function occurs. +.It Ft iterator Fn fs.files "string path" +Opens a directory for scanning of its entries. +The returned iterator yields an entry on each call, and the entry is simply +the filename. +If the path is not absolute, it is relative to the directory containing the +.Nm +in which the call to this function occurs. +.It Ft is_absolute Fn fs.is_absolute "string path" +Returns true if the given path is absolute; false otherwise. +.It Ft join Fn fs.join "string path" "string path" +Concatenates the two paths. +The second path cannot be absolute. +.El +.Sh FILES +.Bl -tag -width XX +.It Pa __TESTSDIR__/Kyuafile . +Top-level +.Nm +for the current system. +.It Pa __EGDIR__/Kyuafile.top . +Sample file to serve as a top-level +.Nm . +.El +.Sh EXAMPLES +The following +.Nm +is the simplest you can define. +It provides a test suite definition and registers a couple of different test +programs using different interfaces: +.Bd -literal -offset indent +syntax(2) + +test_suite('first') + +atf_test_program{name='integration_test'} +plain_test_program{name='legacy_test'} +.Ed +.Pp +The following example is a bit more elaborate. +It introduces some metadata properties to the test program definitions and +recurses into a couple of subdirectories: +.Bd -literal -offset indent +syntax(2) + +test_suite('second') + +plain_test_program{name='legacy_test', + allowed_architectures='amd64 i386', + required_files='/bin/ls', + timeout=30} + +tap_test_program{name='privileged_test', + required_user='root'} + +include('module-1/Kyuafile') +include('module-2/Kyuafile') +.Ed +.Pp +The syntax to define custom properties may be not obvious if their names +have any characters that make the property name not be a valid Lua identifier. +Dashes are just one example. +To set such properties, do something like this: +.Bd -literal -offset indent +syntax(2) + +test_suite('FreeBSD') + +plain_test_program{name='the_test', + ['custom.FreeBSD-Bug-Id']='category/12345'} +.Ed +.Ss Connecting disjoint test suites +Now suppose you had various test suites on your file system and you would +like to connect them together so that they could be executed and treated as +a single unit. +The test suites we would like to connect live under +.Pa /usr/tests , +.Pa /usr/local/tests +and +.Pa ~/local/tests . +.Pp +We cannot create a +.Nm +that references these because the +.Fn include +directive does not support absolute paths. +Instead, what we can do is create a shadow tree using symbolic links: +.Bd -literal -offset indent +$ mkdir ~/everything +$ ln -s /usr/tests ~/everything/system-tests +$ ln -s /usr/local/tests ~/everything/local-tests +$ ln -s ~/local/tests ~/everything/home-tests +.Ed +.Pp +And then we create an +.Pa ~/everything/Kyuafile +file to drive the execution of the integrated test suite: +.Bd -literal -offset indent +syntax(2) + +test_suite('test-all-the-things') + +include('system-tests/Kyuafile') +include('local-tests/Kyuafile') +include('home-tests/Kyuafile') +.Ed +.Pp +Or, simply, you could reuse the sample top-level +.Nm +to avoid having to manually craft the list of directories into which to +recurse: +.Bd -literal -offset indent +$ cp __EGDIR__/Kyuafile.top ~/everything/Kyuafile +.Ed +.Sh SEE ALSO +.Xr kyua 1 diff --git a/doc/manbuild.sh b/doc/manbuild.sh new file mode 100755 index 000000000000..e01239909183 --- /dev/null +++ b/doc/manbuild.sh @@ -0,0 +1,171 @@ +#! /bin/sh +# Copyright 2014 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# \file doc/manbuild.sh +# Generates a manual page from a source file. +# +# Input files can have __VAR__-style patterns in them that are replaced +# with the values provided by the caller via the -v VAR=VALUE flag. +# +# Input files can also include other files using the __include__ directive, +# which takes a relative path to the file to include plus an optional +# collection of additional variables to replace in the included file. + + +# Name of the running program for error reporting purposes. +Prog_Name="${0##*/}" + + +# Prints an error message and exits. +# +# Args: +# ...: The error message to print. Multiple arguments are joined with a +# single space separator. +err() { + echo "${Prog_Name}: ${*}" 1>&2 + exit 1 +} + + +# Invokes sed(1) translating input variables to expressions. +# +# Args: +# ...: List of var=value pairs to replace. +# +# Returns: +# True if the operation succeeds; false otherwise. +sed_with_vars() { + local vars="${*}" + + set -- + for pair in ${vars}; do + local var="$(echo "${pair}" | cut -d = -f 1)" + local value="$(echo "${pair}" | cut -d = -f 2-)" + set -- "${@}" -e"s&__${var}__&${value}&g" + done + + if [ "${#}" -gt 0 ]; then + sed "${@}" + else + cat + fi +} + + +# Generates the manual page reading from stdin and dumping to stdout. +# +# Args: +# include_dir: Path to the directory containing the include files. +# ...: List of var=value pairs to replace in the manpage. +# +# Returns: +# True if the generation succeeds; false otherwise. +generate() { + local include_dir="${1}"; shift + + while :; do + local read_ok=yes + local oldifs="${IFS}" + IFS= + read -r line || read_ok=no + IFS="${oldifs}" + [ "${read_ok}" = yes ] || break + + case "${line}" in + __include__*) + local file="$(echo "${line}" | cut -d ' ' -f 2)" + local extra_vars="$(echo "${line}" | cut -d ' ' -f 3-)" + # If we fail to output the included file, just leave the line as + # is. validate_file() will later error out. + [ -f "${include_dir}/${file}" ] || echo "${line}" + generate <"${include_dir}/${file}" "${include_dir}" \ + "${@}" ${extra_vars} || echo "${line}" + ;; + + *) + echo "${line}" + ;; + esac + done | sed_with_vars "${@}" +} + + +# Validates that the manual page has been properly generated. +# +# In particular, this checks if any directives or common replacement patterns +# have been left in place. +# +# Returns: +# True if the manual page is valid; false otherwise. +validate_file() { + local filename="${1}" + + if grep '__[A-Za-z0-9]*__' "${filename}" >/dev/null; then + return 1 + else + return 0 + fi +} + + +# Program entry point. +main() { + local vars= + + while getopts :v: arg; do + case "${arg}" in + v) + vars="${vars} ${OPTARG}" + ;; + + \?) + err "Unknown option -${OPTARG}" + ;; + esac + done + shift $((${OPTIND} - 1)) + + [ ${#} -eq 2 ] || err "Must provide input and output names as arguments" + local input="${1}"; shift + local output="${1}"; shift + + trap "rm -f '${output}.tmp'" EXIT HUP INT TERM + generate "$(dirname "${input}")" ${vars} \ + <"${input}" >"${output}.tmp" \ + || err "Failed to generate ${output}" + if validate_file "${output}.tmp"; then + : + else + err "Failed to generate ${output}; some patterns were left unreplaced" + fi + mv "${output}.tmp" "${output}" +} + + +main "${@}" diff --git a/doc/manbuild_test.sh b/doc/manbuild_test.sh new file mode 100755 index 000000000000..87234324e829 --- /dev/null +++ b/doc/manbuild_test.sh @@ -0,0 +1,235 @@ +# Copyright 2014 The Kyua Authors. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of Google Inc. nor the names of its contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +# Absolute path to the uninstalled script. +MANBUILD="__MANBUILD__" + + +atf_test_case empty +empty_body() { + touch input + atf_check "${MANBUILD}" input output + atf_check cat output +} + + +atf_test_case no_replacements +no_replacements_body() { + cat >input <<EOF +This is a manpage. + +With more than one line. +EOF + atf_check "${MANBUILD}" input output + atf_check -o file:input cat output +} + + +atf_test_case one_replacement +one_replacement_body() { + cat >input <<EOF +This is a manpage. +Where __FOO__ gets replaced. +And nothing more. +EOF + atf_check "${MANBUILD}" -v FOO=this input output + cat >expout <<EOF +This is a manpage. +Where this gets replaced. +And nothing more. +EOF + atf_check -o file:expout cat output +} + + +atf_test_case some_replacements +some_replacements_body() { + cat >input <<EOF +This is a manpage. +Where __FOO__ gets __BAR__. +And nothing more. +EOF + atf_check "${MANBUILD}" -v FOO=this -v BAR=replaced input output + cat >expout <<EOF +This is a manpage. +Where this gets replaced. +And nothing more. +EOF + atf_check -o file:expout cat output +} + + +atf_test_case preserve_tricky_lines +preserve_tricky_lines_body() { + cat >input <<EOF +Begin + This line is intended. +This other \\ + continues later. +\*(LtAnd this has strange characters\*(Gt +End +EOF + atf_check "${MANBUILD}" input output + cat >expout <<EOF +Begin + This line is intended. +This other \\ + continues later. +\*(LtAnd this has strange characters\*(Gt +End +EOF + atf_check -o file:expout cat output +} + + +atf_test_case includes_ok +includes_ok_body() { + mkdir doc doc/subdir + cat >doc/input <<EOF +This is a manpage. +__include__ subdir/chunk +There is more... +__include__ chunk +And done! +EOF + cat >doc/subdir/chunk <<EOF +This is the first inclusion +and worked __OK__. +EOF + cat >doc/chunk <<EOF +This is the second inclusion. +EOF + atf_check "${MANBUILD}" -v OK=ok doc/input output + cat >expout <<EOF +This is a manpage. +This is the first inclusion +and worked ok. +There is more... +This is the second inclusion. +And done! +EOF + atf_check -o file:expout cat output +} + + +atf_test_case includes_parameterized +includes_parameterized_body() { + cat >input <<EOF +__include__ chunk value=first +__include__ chunk value=second +EOF + cat >chunk <<EOF +This is a chunk with value: __value__. +EOF + atf_check "${MANBUILD}" input output + cat >expout <<EOF +This is a chunk with value: first. +This is a chunk with value: second. +EOF + atf_check -o file:expout cat output +} + + +atf_test_case includes_fail +includes_fail_body() { + cat >input <<EOF +This is a manpage. +__include__ missing +EOF + atf_check -s exit:1 -o ignore \ + -e match:"manbuild.sh: Failed to generate output.*left unreplaced" \ + "${MANBUILD}" input output + [ ! -f output ] || atf_fail "Output file was generated but it should" \ + "not have been" +} + + +atf_test_case generate_fail +generate_fail_body() { + touch input + atf_check -s exit:1 -o ignore \ + -e match:"manbuild.sh: Failed to generate output" \ + "${MANBUILD}" -v 'malformed&name=value' input output + [ ! -f output ] || atf_fail "Output file was generated but it should" \ + "not have been" +} + + +atf_test_case validate_fail +validate_fail_body() { + cat >input <<EOF +This is a manpage. +Where __FOO__ gets replaced. +But where __BAR__ doesn't. +EOF + atf_check -s exit:1 -o ignore \ + -e match:"manbuild.sh: Failed to generate output.*left unreplaced" \ + "${MANBUILD}" -v FOO=this input output + [ ! -f output ] || atf_fail "Output file was generated but it should" \ + "not have been" +} + + +atf_test_case bad_args +bad_args_body() { + atf_check -s exit:1 \ + -e match:'manbuild.sh: Must provide input and output names' \ + "${MANBUILD}" + + atf_check -s exit:1 \ + -e match:'manbuild.sh: Must provide input and output names' \ + "${MANBUILD}" foo + + atf_check -s exit:1 \ + -e match:'manbuild.sh: Must provide input and output names' \ + "${MANBUILD}" foo bar baz +} + + +atf_test_case bad_option +bad_option_body() { + atf_check -s exit:1 -e match:'manbuild.sh: Unknown option -Z' \ + "${MANBUILD}" -Z +} + + +atf_init_test_cases() { + atf_add_test_case empty + atf_add_test_case no_replacements + atf_add_test_case one_replacement + atf_add_test_case some_replacements + atf_add_test_case preserve_tricky_lines + atf_add_test_case includes_ok + atf_add_test_case includes_parameterized + atf_add_test_case includes_fail + atf_add_test_case generate_fail + atf_add_test_case validate_fail + atf_add_test_case bad_args + atf_add_test_case bad_option +} diff --git a/doc/results-file-flag-read.mdoc b/doc/results-file-flag-read.mdoc new file mode 100644 index 000000000000..a0a24cfe0eec --- /dev/null +++ b/doc/results-file-flag-read.mdoc @@ -0,0 +1,53 @@ +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to operate on. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically load the latest results file from the current test suite. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq LATEST +Requests the load of the latest results file available for the test suite rooted +at the current directory. +.It Directory +Requests the load of the latest results file available for the test suite rooted +at the given directory. +.It Test suite name +Requests the load of the latest results file available for the given test suite. +.It Results identifier +Requests the load of a specific results file. +.It Explicit file name (aka everything else) +Load the specified results file. +.El +.Pp +See +.Sx Results files +for more details. diff --git a/doc/results-file-flag-write.mdoc b/doc/results-file-flag-write.mdoc new file mode 100644 index 000000000000..5960560a0665 --- /dev/null +++ b/doc/results-file-flag-write.mdoc @@ -0,0 +1,46 @@ +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to create. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically generate a new results file for the test run. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq NEW +Requests the automatic generation of a new results filename based on the test +suite being run and the current time. +.It Explicit filename (aka everything else) +Store the results file where indicated. +.El +.Pp +See +.Sx Results files +for more details. diff --git a/doc/results-files-report-example.mdoc b/doc/results-files-report-example.mdoc new file mode 100644 index 000000000000..74ef99f16da4 --- /dev/null +++ b/doc/results-files-report-example.mdoc @@ -0,0 +1,32 @@ +.Ss Workflow with results files +If one runs the following command twice in a row: +.Bd -literal -offset indent +kyua test -k /usr/tests/Kyuafile +.Ed +.Pp +the two executions will generate two different files with names like: +.Bd -literal -offset indent +~/.kyua/store/results.usr_tests.20140731-150500-196784.db +~/.kyua/store/results.usr_tests.20140731-151730-997451.db +.Ed +.Pp +Taking advantage of the default naming scheme, the following commands would all +generate a report for the results of the +.Em latest +execution of the test suite: +.Bd -literal -offset indent +cd /usr/tests && kyua __REPORT_COMMAND__ +cd /usr/tests && kyua __REPORT_COMMAND__ --results-file=LATEST +kyua __REPORT_COMMAND__ --results-file=/usr/tests +kyua __REPORT_COMMAND__ --results-file=usr_tests +kyua __REPORT_COMMAND__ --results-file=usr_tests.20140731-151730-997451 +.Ed +.Pp +But it is also possible to explicitly load data for older runs or from +explicitly-named files: +.Bd -literal -offset indent +kyua __REPORT_COMMAND__ \\ + --results-file=usr_tests.20140731-150500-196784 +kyua __REPORT_COMMAND__ \\ + --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db +.Ed diff --git a/doc/results-files.mdoc b/doc/results-files.mdoc new file mode 100644 index 000000000000..3d93a7b16943 --- /dev/null +++ b/doc/results-files.mdoc @@ -0,0 +1,68 @@ +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa __STOREDIR__/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. diff --git a/doc/test-filters.mdoc b/doc/test-filters.mdoc new file mode 100644 index 000000000000..d2d335999b9e --- /dev/null +++ b/doc/test-filters.mdoc @@ -0,0 +1,40 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +A +.Em test filter +is a string that is used to match test cases or test programs in a test suite. +Filters have the following form: +.Bd -literal -offset indent +test_program_name[:test_case_name] +.Ed +.Pp +Where +.Sq test_program_name +is the name of a test program or a subdirectory in the test suite, and +.Sq test_case_name +is the name of a test case. diff --git a/doc/test-isolation.mdoc b/doc/test-isolation.mdoc new file mode 100644 index 000000000000..1072edb41105 --- /dev/null +++ b/doc/test-isolation.mdoc @@ -0,0 +1,112 @@ +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +The test programs and test cases run by +.Nm +are all executed in a deterministic environment. +This known, clean environment serves to make the test execution as +reproducible as possible and also to prevent clashes between tests that may, +for example, create auxiliary files with overlapping names. +.Pp +For plain test programs and for TAP test programs, the whole test program +is run under a single instance of the environment described in this page. +For ATF test programs (see +.Xr atf 7 ) , +each individual test case +.Em and +test cleanup routine are executed in separate environments. +.Bl -tag -width XX +.It Process space +Each test is executed in an independent processes. +Corollary: the test can do whatever it wants to the current process (such +as modify global variables) without having to undo such changes. +.It Session and process group +The test is executed in its own session and its own process group. +There is no controlling terminal attached to the session. +.Pp +Should the test spawn any children, the children should maintain the same +session and process group. +Modifying any of these settings pr |