automatically showing test-suite.log on failure?

[Karl Berry]

After make check runs, if there were any failures, I'm wishing for a way
to have automake to automatically show the relevant test-suite.log.

The post at https://stackoverflow.com/questions/20961959 suggests that
the only way to do this is to modify the test-driver script. Is that correct?
There isn't a target or variable that can be set to achive this,
like a "cat test-suite.log" command somewhere/somehow? I didn't see
anywhere to hook in in the generated Makefile[.in]s, but I certainly
could have missed it. (That post says the clean-local target never gets
run in the case of failure; I didn't confirm it ...)

If nothing else, I guess it could be done by writing my own little
LOG_DRIVER script that invokes the real test_driver and then looks at
the result. (I haven't tried it.) I certainly don't want to modify the
distributed test-driver, as that would not be future-proof ...

However, this seems like it would be fairly commonly useful and easy
enough to do in the canonical test-driver script. So, any chance of
adding it as a standard feature? Any reasonable way of enabling it would
be fine, e.g., a flag that can be added to [AM_]LOG_DRIVER_FLAGS.

BTW, as with that post, I'm primarily interested in this because of
automated build environments where all that is (easily) seen is the log.
Secondarily, in a big build tree, and with srcdir!=builddir, it can be
annoying just to navigate to the correct test-suite.log file. Thus it
would be nice to just have it up front.

Thanks,
Karl

[Bob Friesenhahn]

Post by Karl Berry
However, this seems like it would be fairly commonly useful and easy
enough to do in the canonical test-driver script. So, any chance of
adding it as a standard feature? Any reasonable way of enabling it would
be fine, e.g., a flag that can be added to [AM_]LOG_DRIVER_FLAGS.

Take care since some test logs could be megabytes in size.

Post by Karl Berry
BTW, as with that post, I'm primarily interested in this because of
automated build environments where all that is (easily) seen is the log.
Secondarily, in a big build tree, and with srcdir!=builddir, it can be
annoying just to navigate to the correct test-suite.log file. Thus it
would be nice to just have it up front.

It would be good to be able to enable this via a standard configure
script option.

Bob

[Bob Proulx]

Post by Bob Friesenhahn

Post by Karl Berry
However, this seems like it would be fairly commonly useful and easy
enough to do in the canonical test-driver script. So, any chance of
adding it as a standard feature? Any reasonable way of enabling it would
be fine, e.g., a flag that can be added to [AM_]LOG_DRIVER_FLAGS.

Take care since some test logs could be megabytes in size.

I want to put a point in that often users try to be helpful and email
that multi-megabyte log file as an email attachment to bug reporting
mailing lists that sometimes have a lot of subscribers. This is very
painful on the network bandwidth usage. And also every downstream
subscriber, perhaps on a metered cell modem data plan, pays the cost
of receiving it whether they can help with it or not.

If there is some way, clever or not, to encourange the user to
compress that log file before attaching it that would save a lot of
downstream pain. When compressed that log file is a small fraction of
the original size and rarely a problem.

Just mentioning it to keep it in the brain cache when thinking about
improvements here. :-)

Thanks!
Bob

[Eric Blake]

[adding autoconf, as this also affects testsuite.log generated by
projects using autotest instead of automake's built-in support]

Post by Bob Proulx

Post by Bob Friesenhahn

Post by Karl Berry
However, this seems like it would be fairly commonly useful and easy
enough to do in the canonical test-driver script. So, any chance of
adding it as a standard feature? Any reasonable way of enabling it would
be fine, e.g., a flag that can be added to [AM_]LOG_DRIVER_FLAGS.

Take care since some test logs could be megabytes in size.

I want to put a point in that often users try to be helpful and email
that multi-megabyte log file as an email attachment to bug reporting
mailing lists that sometimes have a lot of subscribers. This is very
painful on the network bandwidth usage. And also every downstream
subscriber, perhaps on a metered cell modem data plan, pays the cost
of receiving it whether they can help with it or not.
If there is some way, clever or not, to encourange the user to
compress that log file before attaching it that would save a lot of
downstream pain. When compressed that log file is a small fraction of
the original size and rarely a problem.
Just mentioning it to keep it in the brain cache when thinking about
improvements here. :-)

Indeed - I almost wonder if it would be worth patching the sequences
that generate testsuite.log to automatically generate a compressed
version in parallel, and in the output when a test fails, mention that
the user may inspect testsuite.log but should email the compressed
testsuite.log.XYZ to the developer. Determining the best compression
program to recommend is interesting - if the user is testing a package
that shipped only as foo.tar.xz, then testsuite.log.xz makes the most
sense to generate; but if foo ships both foo.tar.gz and foo.tar.xz, it
is not obvious which form the user uncompressed and would require a
configure or runtime probe for which compression engine to attempt.

If nothing else, autoconf should be patched where it states:

echo
if $at_debug_p; then
at_msg='per-test log files'
else
at_msg="\`${at_testdir+${at_testdir}/}$as_me.log'"
fi
AS_ECHO(["Please send $at_msg and all information you think might help:

in lib/autotest/general.m4, to mention the recommendation for compression.

[Peter Johansson]

Hi Karl,

Post by Karl Berry
After make check runs, if there were any failures, I'm wishing for a way
to have automake to automatically show the relevant test-suite.log.

I might be missing something, but I get that behavior in my automatic
builds by calling 'make check VERBOSE=1'.

Cheers,
  Peter

[Karl Berry]

I might be missing something, but I get that behavior in my automatic
builds by calling 'make check VERBOSE=1'.

Yes! Thank you!!

[Bob Friesenhahn]

Post by Peter Johansson
I might be missing something, but I get that behavior in my automatic
builds by calling 'make check VERBOSE=1'.
Yes! Thank you!!

This does not appear to have any effect when using the TAP framework.

Bob

[Karl Berry]

Post by Peter Johansson
I might be missing something, but I get that behavior in my automatic
builds by calling 'make check VERBOSE=1'.

This does not appear to have any effect when using the TAP framework.

So it seems. From grepping the installed automake, I see VERBOSE used
in exactly one line of code:

am/check.am:373: test x"$$VERBOSE" = x || cat $(TEST_SUITE_LOG);

(All the other occurrences are about Automake's V= verbosity feature, I
believe.)

But since in my project (TeX Live) we have always used the regular
driver, I am happy :). --best, karl.