aboutsummaryrefslogtreecommitdiffstats
path: root/doc/kyua.1.in
blob: 2fca5eb09f9f7d8bf26c47e706d84c87cb6f2405 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
.\" Copyright 2011 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\"   notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\"   notice, this list of conditions and the following disclaimer in the
.\"   documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\"   may be used to endorse or promote products derived from this software
.\"   without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd May 12, 2015
.Dt KYUA 1
.Os
.Sh NAME
.Nm kyua
.Nd Testing framework for infrastructure software
.Sh SYNOPSIS
.Nm
.Op Fl -config Ar file
.Op Fl -logfile Ar file
.Op Fl -loglevel Ar level
.Op Fl -variable Ar name=value
.Ar command
.Op Ar command_options
.Op Ar command_arguments
.Sh DESCRIPTION
.Em If you are here looking for details on how to run the test suite in
.Pa /usr/tests
.Em ( or
.Pa __TESTSDIR__ ) ,
.Em please start by reading the
.Xr tests 7
.Em manual page that should be supplied by your system .
.Pp
Kyua is a testing framework for infrastructure software, originally
designed to equip BSD-based operating systems with a test suite.
This means that Kyua is lightweight and simple, and that Kyua integrates well
with various build systems and continuous integration frameworks.
.Pp
Kyua features an expressive test suite definition language, a safe
runtime engine for test suites and a powerful report generation engine.
.Pp
Kyua is for both developers and users, from the developer applying a
simple fix to a library to the system administrator deploying a new
release on a production machine.
.Pp
Kyua is able to execute test programs written with a plethora of testing
libraries and languages.
The test program library of choice is ATF, which
.Nm Ns 's
design originated from.
However, framework-less test programs and TAP-compliant test programs can also
be executed through
.Nm
.Ss Overview
As can be observed in the synopsis, the interface of
.Nm
implements a common subcommand-based interface.
The arguments to the tool specify, in this order: a set of common options
that all the commands accept, a required
.Ar command
name that specifies what
.Nm
should do, and
a set of possibly-optional
.Ar command_options
and
.Ar command_arguments
that are specific to the chosen command.
.Pp
The following options are recognized by all the commands.
Keep in mind that these must always be specified before the command name.
.Bl -tag -width XX
.It Fl -config Ar path , Fl c Ar path
Specifies the configuration file to process, which must be in the format
described in
.Xr kyua.conf 5 .
The special value
.Sq none
explicitly disables the loading of any configuration file.
.Pp
Defaults to
.Pa ~/.kyua/kyua.conf
if it exists, otherwise to
.Pa __CONFDIR__/kyua.conf
if it exists,
or else to
.Sq none .
.It Fl -logfile Ar path
Specifies the location of the file to which
.Nm
will log run time events useful for postmortem debugging.
.Pp
The default depends on different environment variables as described in
.Sx Logging ,
but typically the file will be stored within the user's home directory.
.It Fl -loglevel Ar level
Specifies the maximum logging level to record in the log file.
See
.Sx Logging
for more details.
.Pp
The default is
.Sq info .
.It Fl -variable Ar name=value , Fl v Ar name=value
Sets the
.Ar name
configuration variable to
.Ar value .
The values set through this option have preference over the values set in the
configuration file.
.Pp
The specified variable can either be a builtin variable or a test-suite
specific variable.
See
.Xr kyua.conf 5
for more details.
.El
.Pp
The following commands are generic and do not have any relation to the execution
of tests or the inspection of their results:
.Bl -tag -width reportXjunitXX -offset indent
.It Ar about
Shows general program information.
See
.Xr kyua-about 1 .
.It Ar config
Inspects the values of the configuration variables.
See
.Xr kyua-config 1 .
.It Ar db-exec
Executes an arbitrary SQL statement on a results file and prints the
resulting table.
See
.Xr kyua-db-exec 1 .
.It Ar help
Shows usage information.
See
.Xr kyua-help 1 .
.El
.Pp
The following commands are used to generate reports based on the data previously
recorded in a results file:
.Bl -tag -width reportXjunitXX -offset indent
.It Ar report
Generates a plaintext report.
Combined with its
.Fl -verbose
flag and the ability to only display specific test cases, this command can also
be used to debug test failures post-facto on the console.
See
.Xr kyua-report 1 .
.It Ar report-html
Generates an HTML report.
See
.Xr kyua-report-html 1 .
.It Ar report-junit
Generates a JUnit report.
See
.Xr kyua-report-junit 1 .
.El
.Pp
The following commands are used to interact with a test suite:
.Bl -tag -width reportXjunitXX -offset indent
.It Ar debug
Executes a single test case in a controlled environment for debugging purposes.
See
.Xr kyua-debug 1 .
.It Ar list
Lists test cases defined in a test suite by a
.Xr kyuafile 5
and, optionally, displays their metadata.
See
.Xr kyua-list 1 .
.It Ar test
Runs tests defined in a test suite by a
.Xr kyuafile 5 .
See
.Xr kyua-test 1 .
.El
.Ss Logging
.Nm
has a logging facility that collects all kinds of events at run time.
These events are always logged to a file so that the log is available when
it is most needed: right after a non-reproducible problem happens.
The only way to disable logging is by sending the log to
.Pa /dev/null .
.Pp
The location of the log file can be manually specified with the
.Fl -logfile
option, which applies to all commands.
If no file is explicitly specified, the location of the log files is chosen in
this order:
.Bl -enum -offset indent
.It
.Pa ${HOME}/.kyua/logs/
if
.Va HOME
is defined.
.It
.Pa ${TMPDIR}/
if
.Va TMPDIR
is defined.
.It
.Pa /tmp/ .
.El
.Pp
And the default naming scheme of the log files is:
.Sq <progname>.<timestamp>.log .
.Pp
The messages stored in the log file have a level (or severity) attached to
them.
These are:
.Bl -tag -width warningXX -offset indent
.It error
Fatal error messages.
The program generally terminates after these, either in a clean manner or by
crashing.
.It warning
Non-fatal error messages.
These generally report a condition that must be addressed but the application
can continue to run.
.It info
Informational messages.
These tell the user what the program was doing at a general level of
operation.
.It debug
Detailed informational messages.
These are often useful when debugging problems in the application, as they
contain lots of internal details.
.El
.Pp
The default log level is
.Sq info
unless explicitly overridden with
.Fl -loglevel .
.Pp
The log file is a plain text file containing one line per log record.
The format of each line is as follows:
.Bd -literal -offset indent
timestamp entry_type pid file:line: message
.Ed
.Pp
.Ar entry_type
can be one of:
.Sq E
for an error,
.Sq W
for a warning,
.Sq I
for an informational message and
.Sq D
for a debug message.
.Ss Bug reporting
If you think you have encountered a bug in
.Nm ,
please take the time to let the developers know about it.
This will ensure that the bug is addressed and potentially fixed in the next
Kyua release.
.Pp
The first step in reporting a bug is to check if there already is a similar
bug in the database.
You can check what issues are currently in the database by going to:
.Bd -literal -offset indent
https://github.com/jmmv/kyua/issues/
.Ed
.Pp
If there is no existing issue that describes an issue similar to the
one you are experiencing, you can open a new one by visiting:
.Bd -literal -offset indent
https://github.com/jmmv/kyua/issues/new/
.Ed
.Pp
When doing so, please include as much detail as possible.
Among other things, explain what operating system and platform you are running
.Nm
on, what were you trying to do, what exact messages you saw on the screen,
how did you expect the program to behave, and any other details that you
may find relevant.
.Pp
Also, please include a copy of the log file corresponding to the problem
you are experiencing.
Unless you have changed the location of the log files, you can most likely
find them in
.Pa ~/.kyua/logs/ .
If the problem is reproducible, it is good idea to regenerate the log file
with an increased log level so as to provide more information.
For example:
.Bd -literal -offset indent
$ kyua --logfile=problem.log --loglevel=debug \\
    [rest of the command line]
.Ed
.Sh ENVIRONMENT
The following variables are recognized and can be freely tuned by the end user:
.Bl -tag -width COLUMNSXX
.It Va COLUMNS
The width of the screen, in number of characters.
.Nm
uses this to wrap long lines.
If not present, the width of the screen is determined from the terminal
stdout is connected to, and, if the guessing fails, this defaults to infinity.
.It Va HOME
Path to the user's home directory.
.Nm
uses this location to determine paths to configuration files and default log
files.
.It Va TMPDIR
Path to the system-wide temporary directory.
.Nm
uses this location to place the work directory of test cases, among other
things.
.Pp
The default value of this variable depends on the operating system.
In general, it is
.Pa /tmp .
.El
.Pp
The following variables are also recognized, but you should not need to set them
during normal operation.
They are only provided to override the value of built-in values, which is useful
when testing
.Nm
itself:
.Bl -tag -width KYUAXCONFDIRXX
.It Va KYUA_CONFDIR
Path to the system-wide configuration files for
.Nm .
.Pp
Defaults to
.Pa __CONFDIR__ .
.It Va KYUA_DOCDIR
Path to the location of installed documentation.
.Pp
Defaults to
.Pa __DOCDIR__ .
.It Va KYUA_MISCDIR
Path to the location of the installed miscellaneous scripts and data
files provided by
.Nm .
.Pp
Defaults to
.Pa __MISCDIR__ .
.It Va KYUA_STOREDIR
Path to the location of the installed store support files; e.g., the
directory containing the SQL database schema.
.Pp
Defaults to
.Pa __STOREDIR__ .
.El
.Sh FILES
.Bl -tag -width XXXX
.It Pa ~/.kyua/store/
Default location for the results files.
.It Pa ~/.kyua/kyua.conf
User-specific configuration file.
.It Pa ~/.kyua/logs/
Default location for the collected log files.
.It Pa __CONFDIR__/kyua.conf
System-wide configuration file.
.El
.Sh EXIT STATUS
.Nm
returns 0 on success, 1 on a controlled error condition in the given
subcommand, 2 on a general unexpected error and 3 on a usage error.
.Pp
The documentation of the subcommands in the corresponding manual pages only
details the difference between a successful exit (0) and the detection of a
controlled error (1).
Even though when those manual pages do not describe any other exit statuses,
codes above 1 can be returned.
.Sh SEE ALSO
.Xr kyua.conf 5 ,
.Xr kyuafile 5 ,
.Xr atf 7 ,
.Xr tests 7
.Sh AUTHORS
For more details on the people that made
.Nm
possible and the license terms, run:
.Bd -literal -offset indent
$ kyua about
.Ed