Generating Reports
Usually, test results from Fuego tests are examined in the Jenkins User interface. However, Fuego also provides the capability to generate reports outside of Jenkins.
A report consists of data from one or more test runs, collected and presented in the report in an organized fashion. The Fuego system includes the capability to generate lists of test results, providing a number of options to control the content, format, and location of reports.
The command used to generate reports in Fuego is ftc gen-report
ftc gen-report
The ftc gen-report
command gives you control over the results
that are reported, as well as the content (exact fields and headers)
and format of the report.
Using ftc gen-report
, you can:
select the test runs from which to report results
select the header fields to show in the report
select the data fields to show in the report
filter the data by results (for example to show only failures)
select the format of the report
select the location for the report output file
Overall command usage
To generate a report, you use ftc gen-report
and use
command line options to control the data and format of
the report.
The overall command usage is:
ftc gen-report [filter-options] [field-options] [format-option]
The online command usage (obtainable with ftc gen-report help
is shown in the section ftc gen-report usage help below).
Details about the options for generating reports is shown in the following sections.
Selecting data elements to include in the report
A report consists of data from one or more test runs, collected and presented in the report.
Each report has a header section and a body section.
Header section
The fields in the header section are meta-data about the tests included in the report, such as the set of test names, boards that the tests were run on, the kernel versions tested, the start_time of the tests, and the date that the report was generated.
You can control which fields are shown in the header section of
the report, using the --header-fields
option.
The list of header fields available is:
test
board
kernel
timestamp
report_date
Header fields that have multiple values will be shown as a comma-separated list (like the board names), or as a range of values with a start and end (e.g. for the timestamp field).
If no --header-fields
option is used, then the default
set of header fields displayed in the header section of the
resport (and their field order), is:
test,board,kernel,timestamp,report_date
Data section
The data section of the report has data about test results for the selected runs. This includes information such as the test names, board, testcase names, and results.
The data in the report is obtained from a set of test runs that are selected for the report (see below). The data consists of information about the test run (including, importantly, the status of each test), and can also contain detailed information about individual testcase results.
You can control which fields are shown (and the order they
are shown in) in the data section of
the report, using the --fields
option.
The list of fields available for inclusion in the report is:
test_name - full test name
test - short test name
spec - test spec used for the run
board - board that test ran on
kernel - kernel that test ran on
timestamp - time and date of test start
start_time - test start time (in seconds since the epoch)
tguid - test case identifier
tguid:result - test case result
status - test result (the summary result for the whole test, after the test criteria is applied)
duration_ms - the duration of the test in milliseconds
The order of the fields in each report line corresponds to the order
in which the fields are specified as arguments to the --fields
option. If no --fields
option is used, then the default
set of fields (and their order), is:
test_name,spec,board,timestamp,tguid,tguid:result
Difference between ‘status’ and ‘tguid:result’
For each test displayed in the report, the test’s status may be shown. The ‘status’ field for a test run is the overall result of the entire test. Only one of these is shown per test run.
Depending on the fields displayed in the report, a report may also display individual testcase names and results. In the report generator, a testcase name is called a ‘tguid’. This stands for “testcase globally unique identifier”. Each tguid consists of a string containing one or more elements indicating the test suite name, test set name, test case name and measurement name (in the case of benchmark measurement results)
The result for each test case is one of PASS, FAIL, SKIP or ERROR. The result for a measurement is always a number.
An example of a tguid for a measurement is:
IOzone.2048_Kb_Record_Write.Random_write.Score
The associated tguid for this particular test case is:
IOzone.2048_Kb_Record_Write.Random_write
These indicate that this test case and measurement are for the IOzone test suite, the 2048_KB_Record_Write test set, and the Random_write test case within that set. The actual benchmark measurement for this test case is named its ‘Score’ by this test.
A single Fuego test will only have a single ‘status’, but may have multiple test cases (tguid) that can appear in a report.
Selecting data to include in the report
You can filter the set of runs (and test cases within those runs) to
include in the report by using the --where
option. The string
following the the --where
option is called a “where clause”. It is
used to specify a condition that must be met for a run (or a piece of
test result data) to be included in the report. This is similar in
functionality to the WHERE used in the SQL database query language, but
with different syntax.
Multiple where clauses can be specified for a single report,
either as a comma-separated list following a single
--where
option, or via separate --where
options.
(See below for some examples.)
Where clauses are used to select the data that will be included in the report. This includes filtering the set of runs whose data will be included, as well as individual testcase items or results that will be included. For example, you can use a where clause to select test runs by board name, test name, or the ‘spec’ that was used for a run. You can also use where clauses to select tests by start time, batch_id, or build_number. You can filter by the result for a full test (known as it’s “status”), as well as filter the data from each test by individaul testcase names, and testcase results (both strings and numbers).
In order for an element to be included in the report (a run or a
result), it must match all specified where clauses. That is,
the filter is an AND
operation between all the where clauses that
are specified.
where clause syntax
A single ‘where clause’ consists of a field_name, an operator, and a value, like this:
board=beaglebone
Allowed field names are:
test - the full or shortened test name
type - the test type (value may be either “Functional” or “Benchmark”)
spec - the variant or ‘spec’ used for the run
board - the board
start_time - the time when the test was started
batch_id - a batch id string for the run
status - the overall result of a test run
build_number - the build number for the run
tguid - an individual testcase identifier (globally unique id)
tguid:result - an individual testcase result
Allowed operators are:
‘=’ - the field has a value equal to the specified value
‘<’ - the field has a value less than the specified value
‘<=’ - the field has a value less than or equal to the specified value
‘>’ - the field has a value greater than the specified value
‘>=’ - the field has a value greather than or equal to the specified value
‘!=’ - the field has a valud that is not equal to the specified value
‘=~’ - the field has a value that matches the specified regular expression
The ‘=~’ (match) operator is used with a regular expression that the field value must match, in order for the run (or data element) to be included in the report. The regular expression is expressed in python ‘re’ syntax. See https://docs.python.org/2/library/re.html#regular-expression-syntax for information about the regular expression syntax supported with the ‘=~’ (match) operator.
Here are some example where clauses:
example 1: --where test=LTP
example 2: --where board=beaglebone,test=bonnie
example 3: --where board=beaglebone --where test=bonnie
example 4: --where "start_time>2 hours ago"
example 5: --where batch_id=12
example 6: --where "tguid=~.*udp.*"
example 7: --where tguid:result=FAIL
example 8: --where test=fio,"tguid:result>10000"
Here are descriptions of these examples:
example 1 says to generate a report of LTP test runs
example 2 says to show test results for the ‘bonnie’ test on the ‘beaglebone’ board
example 3 says the same thing as example 2. Examples 2 and 3 are different ways of expressing multiple where clauses for a single report generation command. Their effect is identical.
example 4 says to show tests started in the last 2 hours
example 5 says to show tests with batch_id=12
example 6 says to show tests results where the testcase identifier includes the string ‘udp’
example 7 says to show test results that failed.
example 8 says to show test results for the ‘fio’ test, where the benchmark result was greater than 10,000.
Here are some additional tips for specifying where clauses:
When using a where clause that has spaces, enclose the value, or the whole expression in quotes, to avoid the shell splitting the command line argument
When using ‘>’ or ‘<’, enclose the expression in quotes, to avoid the shell interpreting the comparison operators as output or input redirection, respectively
When using shell wildcard characters (such as ‘*’ or ‘?’) in a ‘=~’ regular expression, enclose the expression in quotes, to avoid the shell interpreting the wilcards as file-matching strings.
start_time tricks
It can be a bit tricky to specify the right start_time for the where clause. Say, for example, you want to get the report for tests within the last few hours. This is possible; the parsing for start_time in the where clause is flexible. But it may be confusing.
In general, Fuego supports specifying the start_time value in a where clause using English strings that specify absolute or relative time. For example, the following strings are supported:
–where “start_time>2023-02-24 10:15”
–where “start_time>today at 10:00”
–where “start_time>oct 1 2022”
–where “start_time>1 hour ago”
The last of these examples uses a phrase that has relative time sense.
That is, the time is expressed relative to the time of invocation of
the ftc
.
In Fuego, start_time comparison operations are in absolute time sense, not in relative time sense, even if the time value is expressed using relative wording. Thus
--where "start_time>1 hour ago"
will include tests started “within the last hour”. That is: show tests with a
start_time time that has a value greater than the time 1 hour
before the ftc gen-report
command was run).
The where clause:
--where "start_time>2023-02-24 00:01"
means include tests with a start time after 12:01 am Feb 24, 2023.
As a help, if you use the -v
option with ftc gen-report
,
Fuego will tell you in human-readable form what the start_time
value is that Fuego interpreted for your where clause.
Here’s an example:
$ ftc -v gen-report --where "start_time>last year"
start_time value in 'where' is '2022-01-01 09:00:00.000002'
======================================================================
**** Fuego Test Report ****
...
Using batch_id to group runs for a report
Another way to group test runs for a report is by using a ‘batch_id’.
When you run a batch test in Fuego, all the individual tests in the
batch are assigned the same batch_id
. This batch id is printed
during the test run, or you can use ftc gen-report
with the
--fields
option to see the batch-id for any test or set of tests.
Using a ‘batch-id’ makes it easy to generate a report for a collection of tests, by using a where clause that specifies the batch_id.
Even if you are not using a Fuego batch test (where a batch_id
is assigned automatically), you can manually assign
a batch-id to a collection of tests, by setting the variable
FUEGO_BATCH_ID
to some unique string, before executing the
ftc run-test
command line, for a set of tests.
Here is an example of using a custom ‘batch_id’ for a collection of tests, to enable generating a report for that collection of test runs:
export FUEGO_BATCH_ID=CI-loop-26
ftc run-test -b beaglebone,minnowboard -t fuego_board_check
...
ftc run-test -b beaglebone,raspberry-pi -t fio
...
ftc run-test -b minnowboard -t cyclictest
...
ftc gen-report --where batch_id=CI-loop-26
...
Filtering the data by results
You can also filter the data for a test report by specifyihg where clauses that specify matches or comparisons for test results (status), individual testcase results or Benchmark test measurement results.
A common report filter is one where only errors, skips and failures are reported, and PASS results are ignored (that is, omitted from the report). This can be done by filtering on the field ‘tguid:result’
Here is an example:
ftc gen-report --where "start_time>last week" --where tguid:result!=PASS
This would generate a report with a list of all failing tests and testcase results, in the last week.
If you are not interested in ERROR or SKIP results, you might use a more specific result filter, looking only at FAIL results.
ftc gen-report --where "start_time>last week" --where tguid:result=FAIL
Or, you might only want a summary of tests
that failed in the last week, ignoring individual testcase results
within those tests. To do that, you need to specify a custom
field list that does not include tguid:result
:
ftc gen-report --where "start_time>last week" --where status=FAIL \
--fields test,spec,board,build_number,timestamp,status
Specifying the format and location for the report
Fuego supports outputting the report data in a few different formats.
By default, the report is output in text format, to the stdout
of the ‘ftc’ command. To specify a different
format for the report, use the --format
option.
The following ouput formats are supported:
txt - plain text
html - HTML format
rst - reStructured text
pdf - PDF file format
excel - Excel spreadsheet format
csv - comma separated values
The default format, if none is specified with the --format
option,
is ‘txt’ (plain text).
Each of the other formats are well-known document formats.
Extra text report format options
A few extra options are available to control the formatting of the output, when using the ‘txt’ output format.
-f - use fixed column widths in the text output format
-q - quiet mode: omit the header section and headings in text output format
By default, when outputting data in the text output format, Fuego will automatically adjust the column widths according to the sizes of the strings in that column. However, it may be desirable to present the values at fixed offsets in each line, so that the data may be more easily parsed by other tools. To use a fixed width of 20 characters for each column (starting at 2nd character of each line), use the ‘-f’ option. Here is an example of report output, without the ‘-f’:
$ ftc gen-report --where board=bbb,test=bc,build_number=1
======================================================================
**** Fuego Test Report ****
test : bc
board : bbb
kernel : 4.4.155-ti-r155
timestamp : 2022-03-09_19:07:49
report_date : 2023-03-15_03:12:19
======================================================================
---------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
---------------------------------------------------------------------
Functional.bc default bbb 2022-03-09_19:07:49 bc PASS
---------------------------------------------------------------------
And here is an example of same report, with the ‘-f’ option:
$ ftc gen-report -f --where board=beaglebone,test=bc,build_number=1
======================================================================
**** Fuego Test Report ****
test : bc
board : bbb
kernel : 4.4.155-ti-r155
timestamp : 2022-03-09_19:07:49
report_date : 2023-03-15_03:12:04
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.bc default bbb 2022-03-09_19:07:49 bc PASS
--------------------------------------------------------------------------------------------------------------------------------
You can get more concise output, in text mode, using the ‘-q’ option.
When you use ‘-q’ with ftc gen-report
and you are in text mode, only
the data section of the report is shown. The header section is omitted,
and the heading for the rows of data are also omitted. Only the fields
for each line of data in the report are printed. Also, the
report data starts in column 0, instead of column 2. (That is, the
report data is not indented by 2 spaces).
The purpose of ‘quiet mode’ (the ‘-q’ option) in report generation is to allow you to extract data from the Fuego test results, in a format that can easily be parsed by other tools.
Here is an example:
$ ftc gen-report -q --where board=beaglebone,build_number=1,tguid=Read.Seq.speed \
--fields tguid:result
21457.00
Output location
Fuego can output report data to stdout (the output of the ftc
command),
or save the data to a file. The filename for a report sent to a file
follows the pattern: “Test_report_<date_and_time>.<ext>”.
The default output location depends on the format used for the report.
The output directory can be changed using the -o
option.
Here is the default output location for the different output format types:
txt - output to stdout
html - output to stdout
rst - output to stdout
csv - output to /fuego-rw/reports/Test_report_<date-and-time>.csv
excel - output to /fuego-rw/reports/Test_report_<date-and-time>.xls
pdf - output to /fuego-rw/reports/Test_report_<date_and_time>.pdf
The ‘txt’, ‘html’ and ‘rst’ formats are output to stdout, unless a
report directory is specified (using the -o
option).
The ‘csv’, ‘excel’, and ‘pdf’ formats are
written to a file in the Fuego reports directory. By default the
report directory is /fuego-rw/reports
, inside the docker container.
This directory is visible on the host in the fuego-rw/reports
directory where you installed Fuego.
You can use “-o -” to use the default report directory for ‘txt’,
‘html’ and ‘rst’ formats. Instead of writing the report to stdout,
the Fuego will write reports using these formats
to fuego-rw/reports
, using the standard report filename and
an appropriate filename extension.
Important
When using a Fuego container, the report directory
that is used with the ftc gen-report
command will be relative
to the root of the container, not the host filesystem. That is,
if you run: ftc gen-report -o /tmp
, from outside the container,
the report file will be placed in the /tmp
directory inside
the container, NOT the /tmp
directory on the host.
This may cause confusion. When the Fuego reports directory
(fuego-rw/reports
) is used, this distinction is not a problem
since the fuego-rw directory is visible in both the container and the
host.
ftc gen-report usage help
Here is output generated when you type ftc gen-report help
:
ftc gen-report: Generate a report from a set of runs
Usage: ftc gen-report [--where <where-clause1>[,<where_clausen>]...] \
[-f] [-q]
[--format [txt|html|pdf|excel|csv|rst]] \
[--header_fields <field_list>] \
[--fields <field_list>] \
[--layout <report_name>]
[-o <report_dir>]
Generates a report from test run data as specified. The where option
controls which runs are included in the report. The format option controls
the text encoding of the report, and the layout specifies a report style.
If no arguments are provided, the defaults of "all tests", "txt" output,
and a layout consisting of summary results is used.
-f use fixed column widths in the text output format
-q omit table header and column headings in text output format
txt, html and rst formats are output to stdout, unless a report dir is
specified. pdf, excel and csv formats are written to a file in the
report diretory. By default the report directory is /fuego-rw/reports,
but this can be overridden with the -o option. Use "-o -" to use the
default report directory for txt, html and rst formats.
The --where option can be used to specify one or more 'where' specifiers
to filter the list of runs. Each where clause is separated by a comma.
A 'where clause' consists of a field_name, an operator and a value.
Allowed field names are: test, type, spec, board, start_time, result,
batch_id, status, build_number, tguid, and tguid:result.
Allowed operators are: '=','<','<=','>','>=','!=','=~'. The '=~' operator
means the value is a regular expression to match, for the indicated field.
Here are some example where options:
--where test=LTP
--where test=bonnie,board=beaglebone
--where "start_time>2 hours ago"
--where batch_id=12
--where tguid=~.*udp.*
--where tguid:result=FAIL
The --header_fields and --fields options allow specifying a list of
field names (comma-separated) for inclusion in the header and body
of the report, respectively. The default field lists, if none are
specified are: (for headers) test,board,kernel,timestamp,report_date
and (for fields) test_name,spec,board,timestamp,tguid,tguid:result.
Here is an example:
ftc gen-report --header_fields test --fields timestamp,tguid,tguid:result