Discussion:
What is a 'program' in Section 15.3.3.1?
(too old to reply)
Arthur Schwarz
2015-05-27 16:03:25 UTC
Permalink
15.3.3.1 Command-line arguments for test drivers

"The first non-option argument passed to the test driver is the program to
be run, and all the following ones are command-line options and arguments
for this program."

A few minor questions:
1: What is "the program to be run"?
2: What are the "arguments for this program"?
3: What is the format of the command line executing a custom driver?

I am looking at the tap-driver.sh and guessing what some of this means. I
have absolutely no idea that my guesses make sense, but:
1: Are the programs check/recheck from make check, make recheck
respectively?
2: Are the arguments the [AM_]_LOG_DRIVER_FLAGS driver options?
3: Is the format of the command line:
driver custom_driver_options [AM_]_LOG_DRIVER_FLAGS check|recheck

And, based on these guesses, there is no "program to be run" and there are
no "arguments for this program".

Could someone clarify this?

 
Maintenance turns design into chaos
Arthur Schwarz
2015-05-27 20:55:02 UTC
Permalink
In looking at tap-driver.sh there doesn't appear to be a place where a
'program' is accepted on the input command line. It appears that after all
options are read if the input command line '$#' is not zero then an error is
declared. So, is the TAP interface different from other Custom Test Driver
interfaces?



The failure of the past is the challenge of the present and the success of
the future.
Nick Bowler
2015-05-27 22:00:27 UTC
Permalink
Post by Arthur Schwarz
In looking at tap-driver.sh there doesn't appear to be a place where a
'program' is accepted on the input command line. It appears that after all
options are read if the input command line '$#' is not zero then an error is
declared. So, is the TAP interface different from other Custom Test Driver
interfaces?
I am guessing that you are referring to this line in tap-driver.sh:

test $# -gt 0 || usage_error "missing test command"

The error is only printed if $# is *equal* to zero (i.e., additional
arguments including the program name are mandatory).

Regards,
--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Bob Friesenhahn
2015-05-27 21:53:29 UTC
Permalink
Post by Arthur Schwarz
In looking at tap-driver.sh there doesn't appear to be a place where a
'program' is accepted on the input command line. It appears that after all
options are read if the input command line '$#' is not zero then an error is
declared. So, is the TAP interface different from other Custom Test Driver
interfaces?
The TAP interface is very different. It depends on subordinate
scripts printing 'ok' or 'not ok' (plus some optional extra diagnostic
info) rather than return codes. The subordinate scripts take full
responsibility for the set of tests to be performed, what "programs"
are invoked, and the options supplied to those programs.

For GraphicsMagick, I created a script which is sourced by actual test
scripts (".tap" scripts), which provides generic functions used by
those scripts. The functions support pass/fail detection based on
program exit status, but also support 'xfail' by being passed a set of
feature id strings which are matched which the actual available
features. If the test fails, but a necessary feature is missing, then
an extra diagnosis is added which Automake's TAP script will turn into
an informative XFAIL message:

not ok # SKIP requires foo support

See the implementation at

http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/scripts/tap-functions.shi

and

http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/tests/common.shi

and

http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/tests/rwfile.tap

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Arthur Schwarz
2015-05-28 01:35:18 UTC
Permalink
Thanks.

(Also see lists.gnu.org/archive/html/automake/2015-05/msg00051.html How are
data files input for to custom drivers?)
Post by Bob Friesenhahn
The TAP interface is very different.
TAP is supposed to be a custom test driver. Is the interface and manner of
calling different from other custom drivers, and the API and other comments
describing custom drivers in the manual?
Post by Bob Friesenhahn
It depends on subordinate
scripts printing
I'm very confused here. It looks to me like tap-driver.sh is a standalone
script and doesn't need any help scripts. The input data is processed in awk
and all the needed functions are defined in this context.
Post by Bob Friesenhahn
For GraphicsMagick, I created a script which is sourced by actual test
scripts (".tap" scripts), which provides generic functions used by
those scripts. The functions support pass/fail detection based on
program exit status, but also support 'xfail' by being passed a set of
feature id strings which are matched which the actual available
features. If the test fails, but a necessary feature is missing, then
an extra diagnosis is added which Automake's TAP script will turn into
I am confused. Using Automake the Developer can generate a reference to a
class of test cases defined in the TESTS variable. Each one of the scripts
is required to output the results of one of more tests that they run in
quasi-TAP format. The TAP script, tap-driver.sh, takes the output and
generates a trs file. Included with the tap-driver.sh there is a means to
generate XFAIL and XPASS, however this seems to be global to all subtests
executed in a TESTS defined test. Each ok or not ok returned will be
translated (as required) to XPASS or XFAIL depending on
--expect-failure=yes.

As an example:
TEST_EXTENSIONS =.abc
ABC_LOG_DRIVER = tap-driver.sh
TESTS = test1.abc test2.abc

Means that tap-driver.sh is called twice, once for test1.abc, once for
test2.abc. Each of these tests can return one or more of ok, not ok, skip,
bail, or a YAML-like text string defined by --diagnostic-string and
--comments.

Another point of confusion is that the options defined for the custom test
drivers have the format "a=b" if 'a' has a value, but tap-driver.sh does not
allow "a=b", it requires "a b".

Anyhow, thanks for the heads up, I will be looking at your files. And, as
always, I am confused.
Bob Friesenhahn
2015-05-28 03:46:53 UTC
Permalink
Post by Arthur Schwarz
TAP is supposed to be a custom test driver. Is the interface and manner of
calling different from other custom drivers, and the API and other comments
describing custom drivers in the manual?
I think that the method of calling is similar to other type of tests
except that the list of tests 'TESTS' is a list of scripts which
produce TAP output strings.
Post by Arthur Schwarz
Post by Bob Friesenhahn
It depends on subordinate
scripts printing
I'm very confused here. It looks to me like tap-driver.sh is a standalone
script and doesn't need any help scripts. The input data is processed in awk
and all the needed functions are defined in this context.
If the test program was a C program so it was a binary and printed
'ok' and 'not ok' outputs, and is able to find any input files without
assistance, then no extra shim script should be required.

In my case I was replacing perhaps 1000 individual classic Automake
test scripts and wanted to replace them with far fewer TAP scripts.
Post by Arthur Schwarz
I am confused. Using Automake the Developer can generate a reference to a
class of test cases defined in the TESTS variable. Each one of the scripts
is required to output the results of one of more tests that they run in
quasi-TAP format. The TAP script, tap-driver.sh, takes the output and
generates a trs file. Included with the tap-driver.sh there is a means to
generate XFAIL and XPASS, however this seems to be global to all subtests
executed in a TESTS defined test. Each ok or not ok returned will be
translated (as required) to XPASS or XFAIL depending on
--expect-failure=yes.
A TAP test program/script may contain many tests and it may output
multiple test lines. The script indicates how many tests it plans to
run by first printing a line like

1..532

to indicate that it plans to run 532 tests. Each test gets is own
PASS, FAIL, XFAIL indication.
Post by Arthur Schwarz
TEST_EXTENSIONS =.abc
ABC_LOG_DRIVER = tap-driver.sh
TESTS = test1.abc test2.abc
Means that tap-driver.sh is called twice, once for test1.abc, once for
test2.abc. Each of these tests can return one or more of ok, not ok, skip,
bail, or a YAML-like text string defined by --diagnostic-string and
--comments.
Yes, but tap-driver.sh interprets the output while the test
program/scripts runs and it can handle an arbitary number of
tests/results in the same test script.
Post by Arthur Schwarz
Anyhow, thanks for the heads up, I will be looking at your files. And, as
always, I am confused.
I am not surprised. :-)

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Continue reading on narkive:
Loading...