How do you set VERBOSE for parallel testin

Post by Arthur Schwarz
I'm trying to set VERBOSE in a parallel test setup and it doesn't seem to
VERBOSE = yes and
VERBOSE = 1

Where were you making that setting, in Makefile.am?

Post by Arthur Schwarz
But I get the normal output listing. The Automake manual (Section 15.2.3
Parallel Test Harness) says: " The output from failed tests is collected in
the test-suite.log file. If the variable 'VERBOSE' is set, this file is
output after the summary."

It works for me when I do:

make check VERBOSE=1

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Arthur Schwarz

2015-05-08 17:34:16 UTC

-----Original Message-----
Sent: Thursday, May 07, 2015 4:02 PM
Subject: Re: How do you set VERBOSE for parallel testin

I'm trying to set VERBOSE in a parallel test setup and it doesn't seem >
VERBOSE = yes and
VERBOSE = 1

Where were you making that setting, in Makefile.am?

But I get the normal output listing. The Automake manual (Section 15.2.3
Parallel Test Harness) says: " The output from failed tests is collected in
the test-suite.log file. If the variable 'VERBOSE' is set, this file is
output after the summary."

make check VERBOSE=1
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

===========================================================

Yes. Not on the command line but in the Makefile.am file.

I did see the comment on command line usage (from Stackoverflow?). I haven't
tried to see if it works, that is, there is a command line option
(--verbose) available and the given command line has VERBOSE as an argument.

So, if it appears I "don't know nuttin'" then it appears correctly.

art

Arthur Schwarz

2015-05-08 22:55:45 UTC

I just checked the latest iteration of my output listing (VERBOSE = 1) and
saw the rule for
Makefile:478 showing test-suite.log failed. Don't know why.

The output listing is below and the Makefile.am is included in an
attachment.

Heck, I don't know what I'm doing. If anyone can explain what I need to do
to remove the error messages I'd really appreciate it.

art

PASS: hello.py
SKIP: hello.pl
FAIL: test1.sh
XFAIL: test2.sh
XPASS: test3.sh
ERROR: test4.sh
make[4]: Entering directory
'/cygdrive/c/home/skidmarks/Projects/Test/automake/installDir/test-0.5/src'
make[4]: Nothing to be done for 'all'.
make[4]: Leaving directory
'/cygdrive/c/home/skidmarks/Projects/Test/automake/installDir/test-0.5/src'
============================================================================
Testsuite summary for test 0.5
============================================================================
# TOTAL: 6
# PASS: 1
# SKIP: 1
# XFAIL: 1
# FAIL: 1
# XPASS: 1
# ERROR: 1
============================================================================
See src/test-suite.log
============================================================================
Makefile:478: recipe for target 'test-suite.log' failed
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory
'/cygdrive/c/home/skidmarks/Projects/Test/automake/installDir/test-0.5/src'
Makefile:584: recipe for target 'check-TESTS' failed
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory
'/cygdrive/c/home/skidmarks/Projects/Test/automake/installDir/test-0.5/src'
Makefile:664: recipe for target 'check-am' failed
make[1]: *** [check-am] Error 2
make[1]: Leaving directory
'/cygdrive/c/home/skidmarks/Projects/Test/automake/installDir/test-0.5/src'
Makefile:415: recipe for target 'check-recursive' failed
make: *** [check-recursive] Error 1
=================================================================

2nd Law Of Software: Maintenance leads to Chaos.

Eric Blake

2015-05-08 23:01:01 UTC

Post by Arthur Schwarz
I just checked the latest iteration of my output listing (VERBOSE = 1) and
saw the rule for
Makefile:478 showing test-suite.log failed. Don't know why.

If any test causes a FAIL, XPASS, or ERROR condition, then 'make check'
will likewise fail. The output from make is telling you that your
testsuite caught failures that should not be present if the testsuite
were passing.

Post by Arthur Schwarz
The output listing is below and the Makefile.am is included in an
attachment.
Heck, I don't know what I'm doing. If anyone can explain what I need to do
to remove the error messages I'd really appreciate it.

Which error messages are you trying to remove? The only error message I
say in the output you posted is expected, given that you have a failing
test.

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Arthur Schwarz

2015-05-08 23:13:59 UTC

Post by Arthur Schwarz
I just checked the latest iteration of my output listing (VERBOSE = 1) and
saw the rule for
Makefile:478 showing test-suite.log failed. Don't know why.

My issue is that the output indicates that the rule for test-suite.log
fails. Is this expected? My thought (from the automake manual) is that even
if there is a failure there should be no difficulty creating or outputting
test-suite.log. The test harness created a test-suite.log (included as an
attachment) but failed to output it. Don't know why.

Which error messages are you trying to remove? The only error message I
say in the output you posted is expected, given that you have a failing
test.

Ok. If they're supposed to fail then fail they did. What about the
test-suite.log failing to output?

Eric Blake

2015-05-08 23:29:11 UTC

Post by Arthur Schwarz
I just checked the latest iteration of my output listing (VERBOSE = 1)

and

Post by Arthur Schwarz
saw the rule for
Makefile:478 showing test-suite.log failed. Don't know why.

Yes, this is expected. test-suite.log is created no matter what, but
the rules associated with creating it fail if the log contains any
failure reports, so that make will quit running and let you investigate
those failures.

As for why test-suite.log was not replayed as part of the make output,
I'm not sure that you had VERBOSE=1 set. So far, you've only attached
the generated Makefile (but not the source Makefile.am), and didn't show
what command you ran. You'll need to make your setup easier for others
to reproduce before we can see what you are attempting.

As for the test-suite.log not having much output, it is probably because
your tests are not very noisy. Running make captured all of the output
of your tests, but if your tests didn't output anything, then the log
has very little to show. Again, without knowing what your test scripts
are doing, it's hard to reproduce your setup.

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Arthur Schwarz

2015-05-12 22:42:15 UTC

Post by Eric Blake
If any test causes a FAIL, XPASS, or ERROR condition, then 'make check'
will likewise fail. The output from make is telling you that your
testsuite caught failures that should not be present if the testsuite
were passing.

If I understand this correctly, when executing test scripts the first
failure will cause make to discontinue execution (?). This doesn't happen.
Make continues until all tests in the variable TESTS are executed. Then
again, I must be misunderstanding you?

Post by Eric Blake
As for why test-suite.log was not replayed as part of the make output,
I'm not sure that you had VERBOSE=1 set. So far, you've only attached
the generated Makefile (but not the source Makefile.am), and didn't show
what command you ran. You'll need to make your setup easier for others
to reproduce before we can see what you are attempting.

I am including several files.
test.pdf console output showing test results
Makefile.am src/Makefile
Test-suite.log generated file
configure.ac ./configure.am
Makefile.am ./Makefile.am

Post by Eric Blake
As for the test-suite.log not having much output, it is probably because
your tests are not very noisy. Running make captured all of the output
of your tests, but if your tests didn't output anything, then the log
has very little to show. Again, without knowing what your test scripts
are doing, it's hard to reproduce your setup.

From the Automake Manual Section 15.2.3 Parallel Test Harness, " The output
from failed tests is collected in the test-suite.log file. If the variable
'VERBOSE' is set, this file is output after the summary."

This quote is in fact not fully accurate. The output from all tests are
collected in the test-suite.log file. I can not set VERBOSE to output the
test-suite.log file.

Your statement concerning the paucity of output does not preclude showing
any output. In fact, removing the generated "echo" statements from the
generated shell scripts in Makefile.am, and making no other change, will
show that the test harness generates an output for each test case executed
in the test-suite.log file.

I have tried to set VERBOSE in my Makefile.am to '1' and to 'yes' and "yes"
and just plain yes. The test-suite.log file does not output. Am I missing
something? How is VERBOSE supposed to be turned on?

Anyhow, I'm puzzled as to what to do and how to check this out. The general
comments probably apply to all variables which have a 'yes/no' value. What
should the value be to turn them on or off? A similar question occurs in
setting AM_COLOR_TEST to 'always'. What are the legitimate values?

Sorry about the delay in answering.
art

Peter Johansson

2015-05-13 06:55:50 UTC

Post by Eric Blake
Yes, this is expected. test-suite.log is created no matter what, but
the rules associated with creating it fail if the log contains any
failure reports, so that make will quit running and let you investigate
those failures.

Correct. All tests will run and create their corresponding .log files.

I am including several files.
test.pdf console output showing test results
Makefile.am src/Makefile
Test-suite.log generated file
configure.ac ./configure.am
Makefile.am ./Makefile.am

Nope, I just ran 'make check' in one of my projects and test-suite.log
only contains the output from the test that failed.

Post by Arthur Schwarz
I can not set VERBOSE to output the
test-suite.log file.
Your statement concerning the paucity of output does not preclude showing
any output. In fact, removing the generated "echo" statements from the
generated shell scripts in Makefile.am, and making no other change, will
show that the test harness generates an output for each test case executed
in the test-suite.log file.
I have tried to set VERBOSE in my Makefile.am to '1' and to 'yes' and "yes"
and just plain yes. The test-suite.log file does not output. Am I missing
something? How is VERBOSE supposed to be turned on?

Usually I run my tests with something like this:

make check -j8 VERBOSE=1

hth,
Peter

--
Peter Johansson

Arthur Schwarz

2015-05-13 15:20:19 UTC

Post by Peter Johansson
make check -j8 VERBOSE=1

Thanks Peter.

My question is is this the only way to use VERBOSE? The Automake Manual
seems to say that VERBOSE is a variable, not a make argument. And, as a
variable, if the user (you) can change it's value then the appropriate way
to do it is either:
env VERBOSE=1 make -e check
or
VERBOSE=1; export VERBOSE; make -e check

Have you tried "make check -j8 VERBOSE"? If this works then it means that
the existence of VERBOSE is sufficient to turn the option on. This is a
possible interpretation of 15.2.3 Parallel Test Harness "If the variable
'VERBOSE' is set, this file is output after the summary."

And as a simple question, why are you using "make -j8"? Shouldn't this be
part of the automake generated test code? Again, my interpretation of the
Automake Manual 15.2.3 Parallel Test Harness

"By default, Automake generated a parallel (concurrent) test
harness. It features automatic collection of the test scripts
output in .log files, concurrent execution of tests with
make -j, specification of inter-test dependencies, lazy reruns
of tests that have not completed in a prior run, and hard
errors for exceptional failures."

is that the generated Makefile executes "make -j1 testcase" for each test
case or "make -j# test1 test2 ... for all test cases. Unlike the serial test
harness, the parallel test harness does not output an indication of what
it's doing (the serial test harness outputs "make test1 test2 ...").

In my little test (see the attachments in my previous post) the
test-suite.log file lists all the SKIP, FAIL, and ERROR, a.k.a, HARD, test
case results. Which is an elaboration of your comment and thank you for
pointing out that I was soo wrong. Sigh.

Oh, to give you some context, I am rewriting Section 15 of the Automake
Manual. I have had so much difficulty in understanding it that I thought
that if I spent time in understanding what it says I might as well document
my understanding. Hopefully at least part of the effort will be incorporated
in future Automake Manuals. As to my errors. Well, just mark them up to
blind ignorance.

Art
2nd law of software dynamics: Maintenance turns Order into Chaos

Roman Neuhauser

2015-05-13 15:43:07 UTC

Post by Peter Johansson
make check -j8 VERBOSE=1

make arguments are a mix of targets and macro assignments. an argument
in the form of foo=bar is treated as a macro assignment and overrides
assignments found in the makefile. environment variables can override
macros with the same name if the macro is assigned with the ?= operator
(or if you use `make -e`).

Post by Arthur Schwarz
Have you tried "make check -j8 VERBOSE"? If this works then it means that
the existence of VERBOSE is sufficient to turn the option on. This is a
possible interpretation of 15.2.3 Parallel Test Harness "If the variable
'VERBOSE' is set, this file is output after the summary."

no, that's not a possible interpretation unless you're really confused
about makefiles and make behavior.

--
roman

Nick Bowler

2015-05-13 16:12:04 UTC

Post by Peter Johansson
make check -j8 VERBOSE=1

Look at the Makefile.in which is generated by Automake. VERBOSE is an
environment variable (at least with the parallel test harness), and the
test condition is that VERBOSE is set to a non-empty value.

The interaction between make variables and environment variables is
complicated.

There are 3 "normal" ways a make variable can be set (this is not the
complete picture but it will do):

(1) In the environment (FOO=bar make)
(2) In the Makefile (FOO=bar in the Makefile)
(3) on the make command line (make FOO=bar)

The priority is in that order: if a variable is set on the command line
it will override any definition in the Makefile, which in turn will
override any definition in the environment. You can use the -e option
to make to switch the order of (1) and (2), which will probably break a
lot of things.

Now, before make runs a command, it adds the make variables from (3) to
the environment. Some, but not all[1], make implementations will further
update existing environment variables [those from (1)], with assigned
values from (2).

So the takeaway is this:

- Never set VERBOSE to any value in Makefile.am. Such an assignment
will be useless at best.

- Use make VERBOSE=1 check to enable verbose mode when running tests.

[1] Results from a quick test: GNU make and Heirloom make update the
environment, while dmake, NetBSD make, and FreeBSD make do not.

Cheers,

--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Arthur Schwarz

2015-05-13 20:18:31 UTC

Post by Nick Bowler
There are 3 "normal" ways a make variable can be set (this is not the
(1) In the environment (FOO=bar make)
(2) In the Makefile (FOO=bar in the Makefile)
(3) on the make command line (make FOO=bar)

Before I go out on a limb and say something I'll regret, if I understand
this correctly any user variable (not developer variable) can always be
defined via (1) and (3). Is this correct?

Can I do this with lists, e.g.: make TEST="test1 test2..."

And for variables which have script commands associated with them?
For example: make AM_TESTS_ENVIRONMENT=". $(srcdir)/tests-env.sh; \
if test -d /usr/xpg4/bin; then \
PATH=/usr/xpg4/bin:$$PATH; export PATH; \
fi;"

If I have several variables do I write: make FOO=bar BAZ=snafu ...?

Suppose the variable has a generated Makefile component (TEST_LOGS) and a
user can change it. What changes, if anything? In order to modify such a
variable is the user required to use form (3)?

Does anything change if the user variable name is different from the
Makefile.am variable name, e.g. ext_LOG_FLAGS and AM_ext_LOG_FLAGS?

Does anything change if the user variable name is the same as the
Makefile.am variable name, e.g. TESTS?

Since I am trying to rewrite Section 15 on Support for Test Suites, and
variables are involved, should I pursue all options which are available for
setting variables or is the Automake Manual sufficient.

thanks

Nick Bowler

2015-05-14 13:40:09 UTC

Before I go out on a limb and say something I'll regret, if I understand
this correctly any user variable (not developer variable) can always be
defined via (1) and (3). Is this correct?

Not necessarily, because variables like prefix are going to have an
assignment in the Makefile.

Post by Arthur Schwarz
Can I do this with lists, e.g.: make TEST="test1 test2..."
And for variables which have script commands associated with them?
For example: make AM_TESTS_ENVIRONMENT=". $(srcdir)/tests-env.sh; \
if test -d /usr/xpg4/bin; then \
PATH=/usr/xpg4/bin:$$PATH; export PATH; \
fi;"
If I have several variables do I write: make FOO=bar BAZ=snafu ...?
Suppose the variable has a generated Makefile component (TEST_LOGS) and a
user can change it. What changes, if anything? In order to modify such a
variable is the user required to use form (3)?
Does anything change if the user variable name is different from the
Makefile.am variable name, e.g. ext_LOG_FLAGS and AM_ext_LOG_FLAGS?
Does anything change if the user variable name is the same as the
Makefile.am variable name, e.g. TESTS?

The answer to all of these questions is "Yes, the user running 'make'
can override any make variable whatsoever and set it to whatever he or
she likes." Whether this is a reasonable thing for the user to do is
another story. Overriding a variable that was not designed to be
overridden by the makefile author is probably going to break the
build rather badly.

Make provides quite a lot of options that drastically alter its
behaviour. Actually using these features will often break things.

But none of this really has much to do with Automake, except
tangentially because Automake is a tool to help generate portable
make programs.

Cheers,

--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Arthur Schwarz

2015-05-13 20:35:29 UTC

(3) on the make command line (make FOO=bar)

Sorry to be a pest on this but which (or both) formats are correct for a
test:

make check FOO=bar or
make FOO=bar check

Ditto for recheck and other make options.

Peter Johansson

2015-05-13 23:31:15 UTC

Post by Arthur Schwarz
My question is is this the only way to use VERBOSE? The Automake
Manual seems to say that VERBOSE is a variable, not a make argument.
And, as a variable, if the user (you) can change it's value then the
appropriate way to do it is either: env VERBOSE=1 make -e check or
VERBOSE=1; export VERBOSE; make -e check

well I use GNU make for which 'make VERBOSE=1' works as well.

The short answer is because I wanna use 8 cpus. There is no sensible way
Automake can know how many cpus I wanna use.

Parallel here means that the tests will run in parallel. The old serial
test-suite had a rule with something like this

check: all
for t in $(TESTS); do $(SHELL) $$t; done

which is not parallel and the run command $(SHELL) $$t depended on the
labguage etc; can't remember the details.

Cheers,
Peter

Andrew W. Nosenko

2015-05-14 10:06:08 UTC

Post by Arthur Schwarz
And as a simple question, why are you using "make -j8"? Shouldn't this be
part of the automake generated test code?

The short answer is because I wanna use 8 cpus. There is no sensible way
Automake can know how many cpus I wanna use.

$ make check TESTSUITEFLAGS=-j8

--
Andrew W. Nosenko <***@gmail.com>

Arthur Schwarz

2015-05-14 14:24:45 UTC

Post by Arthur Schwarz
And as a simple question, why are you using "make -j8"? Shouldn't this

be part of the automake generated test code?

Post by Peter Johansson
The short answer is because I wanna use 8 cpus. There is no sensible way

Automake can know how many cpus I wanna use.

Post by Andrew W. Nosenko
$ make check TESTSUITEFLAGS=-j8

I just searched the Automake.html manual and can't find TESTSUITEFLAGS.
Where is this documented? Is this part of a new release?

Unlike Serial Test Harness execution, the Parallel Test Harness does not
output how it executes its tests. No way to find out what's under the hood
without checking the code.

I haven't looked at the Makefile source - mainly, I suspect, because I'm not
a make guru (or even an aficionado). My guess was that The Makefile directed
test execution was either:

make -j# $(TESTS) or

for I in $(TESTS); do make -j $i; done

In either case, the selection of the number of processors to use was made by
make and it is unnecessary and counter-productive to attempt specification
by the user. If the "make -j8" or "make check TESTSUITEFLAGS=-j8" works then
it seems to look like the Parallel Test Harness is really a Serial Test
Harness with some extra capability. That is, if make doesn't select parallel
execution internally (using something like my pseudo-code) then the default
is serial execution. Then again, if either user option works, then the make
internal logic must selectively attempt to execute k test cases using N
processors and to 'do what?' if there are more tests than processors (k >
N)? If I sound utterly baffled, it's because I am.

Peter Johansson

2015-05-14 23:40:08 UTC

Post by Arthur Schwarz
And as a simple question, why are you using "make -j8"? Shouldn't this

be part of the automake generated test code?

Post by Peter Johansson
The short answer is because I wanna use 8 cpus. There is no sensible way

Automake can know how many cpus I wanna use.

Post by Andrew W. Nosenko
$ make check TESTSUITEFLAGS=-j8

Your guess is wrong. The key line in 'Makefile.in' is

$(TEST_SUITE_LOG): $(TEST_LOGS)

which means that before creating 'test-suite.log' the log file for each
test is generated. This structure *allows* parallel execution, but only
if the user chooses to. If the user run make single-threaded the test
suite will be run on a single cpu. This is by design. If a user issues
'make check' you want one single cpu to be used; it would be very
annoying if make spawned away processes. If a user issues 'make -jX'
then the structure allows make to run tests in parallel. This is not
possible in the old serial test suite, which basically just ran the
tests in a loop.

Cheers,
Peter

Arthur Schwarz

2015-05-15 00:36:31 UTC

The short answer is because I wanna use 8 cpus. There is no sensible way
Automake can know how many cpus I wanna use.

Just a some idle speculation, Automake knows two things at this
point, that the user wants to run a Parallel Test Harness and the
Total number of test cases to execute (TESTS). The user can modify
TESTS to suit allowing make / Makefile to determine the number of
test cases in the variable TESTS. Shouldn't this be enough to
establish how many processes are needed and to select a strategy
for allocating run time resources?

$ make check TESTSUITEFLAGS=-j8

Is TESTSUITEFLAGS a new variable for the next Automake release?
It appears undocumented.

make -j# $(TESTS) or

for I in $(TESTS); do make -j $i; done

Your guess is wrong. The key line in 'Makefile.in' is

$(TEST_SUITE_LOG): $(TEST_LOGS)

which means that before creating 'test-suite.log' the log file for
each test is generated. This structure allows parallel execution, but only
if the user chooses to.

TEST_LOGS contains the list of log files. The rule seems to say
after all log files are generated then the $(TEST_SUITE_LOG)
preconditions are satisfied and $(TEST_SUITE_LOG) is valid. This
doesn't seem to address how the log files get generated. From
the manual it seems that .log files are the output (stdout and
stderr) of a test case, and that there is no way for the user
to bypass this. The rule is neutral with respect to execution
time semantics. My conceptual pseudo-code seems to be valid
(at least the 'for' part).

If the user run make single-threaded the test suite will be run on a
single cpu. This is by design. If a user issues 'make check' you want one
single cpu to be used; it would be very annoying if make spawned away
processes. If a user issues 'make -jX' then the structure allows make to run
tests in parallel. This is not possible in the old serial test suite, which
basically just ran the tests in a loop.

Thanks. I didn't get this from my reading. I think you are saying
that if the user wants to execute the tests in parallel the user
must specify this on the command line.

And, if I guess correctly, the user can write
"make check TESTSUITEFLAGS=-jN"

As a nit-noy, don't you mean "processor" and not "cpu"? And
does -jN specify processors or processes or both?

Peter Johansson

2015-05-15 02:29:46 UTC

Post by Arthur Schwarz
And, if I guess correctly, the user can write
"make check TESTSUITEFLAGS=-jN"

I've never seen TESTSUITEFLAG before, so I don't know.

Post by Arthur Schwarz
As a nit-noy, don't you mean "processor" and not "cpu"? And
does -jN specify processors or processes or both?

Correct.

Cheers,
Peter

Eric Blake

2015-05-15 03:15:59 UTC

Post by Arthur Schwarz
And, if I guess correctly, the user can write
"make check TESTSUITEFLAGS=-jN"

I've never seen TESTSUITEFLAG before, so I don't know.

TESTSUITEFLAGS is the suggested variable name for someone using the
'autotest' setup from autoconf (which is yet another testing framework,
different from automake, where automake runs a single test called
'testsuite', but 'testsuite' itself can run multiple tests in series or
parallel).

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Andrew W. Nosenko

2015-05-15 07:31:35 UTC

Post by Arthur Schwarz
And, if I guess correctly, the user can write
"make check TESTSUITEFLAGS=-jN"

I've never seen TESTSUITEFLAG before, so I don't know.

Yes, sorry, I indeed forgotten that my testsuite uses autotest instead of
automake's framework.

Sorry for the noise.

--
Andrew W. Nosenko <***@gmail.com>

Arthur Schwarz

2015-05-21 00:00:57 UTC

15.2.3 Parallel Test Harness

You can set the TEST_LOGS variable. By default, this variable is computed at
make run time from the value of TESTS as described above. For example, you
can use the following:

set x subset*.log; shift
env TEST_LOGS="foo.log $*" make -e check

What does setting TEST_LOGS do? It is read only in the Makefile file.

There is an unclear explanation of the TEST_LOGS meaning (don't explain, I
think I've got it figured out). And it sort of makes sense. TEST_LOGS
contains the name of all .log files. I get that. What happens if the user
changes one or all the names. Suppose Makefile looks something like:

TESTS = foo.exe

Then
TEST_LOGS = foo.log

And the user sets this to
env TEST_LOGS = bar.log make -e check

What on earth does this mean?

And the examples with TEST_EXTENSIONS and duplicate base names (test cases
with extensions removed, like TESTS = test.sh test.pl and TEST_EXTENSIONS =
.pl .sh) grows even more complex.

Any idea?

And what does 'set x subset*.log; shift' have to do with anything. It looks
like something erroneously removed from an existing script and put in the
document.

Roman Neuhauser

2015-05-21 08:12:13 UTC

Post by Arthur Schwarz
15.2.3 Parallel Test Harness
You can set the TEST_LOGS variable. By default, this variable is computed at
make run time from the value of TESTS as described above. For example, you
set x subset*.log; shift
env TEST_LOGS="foo.log $*" make -e check
What does setting TEST_LOGS do?

it overrides the definition present in the Makefile.

Post by Arthur Schwarz
It is read only in the Makefile file.

no.

Post by Arthur Schwarz
TESTS = foo.exe
Then
TEST_LOGS = foo.log
And the user sets this to
env TEST_LOGS = bar.log make -e check
What on earth does this mean?

first, it would have to be

env TEST_LOGS=bar.log make -e check

(no ws around the equals sign).

second, it looks like a pilot error, and i would expect make to produce
an error.

Post by Arthur Schwarz
And what does 'set x subset*.log; shift' have to do with anything. It looks
like something erroneously removed from an existing script and put in the
document.

that's just defensive shell script programming. if no files match the
pattern, it will expand to nothing, and "set" with no options or arguments
produces a list of all parameters (variables) defined in the current
shell environment (instead of setting argc = 0, argv = {}. to prevent
this mishap you need to ensure that "set" sees at least one argument,
and then shift it off the beginning of argv.

--
roman

Gavin Smith

2015-05-21 10:57:04 UTC