automake: tap-driver.sh: cleanup and revival.

[Mike Mestnik]

I'm reading this script closely because I'd like to make a few changes
and am surprised it hasn't received any edits since 2012. I think the
output is missing ratio of *test's failed, currently only file level
stats are presented not test level, and also the following options
from prove.

--verbose Print all test lines.
--failures Show failed tests.

* This to me is a key feature of TAP and this script silently discards
the most useful parts of the protocol.

Somethings I'v spotted that are driving me nuts.
http://git.savannah.gnu.org/cgit/automake.git/tree/lib/tap-driver.sh#n100
Clearly indicates that "--color-tests=yes" is the only way to enable
color, there is no auto-detecting a PTY or any support for the
documented "--color-tests=always"

It's also crazy that "--color-tests=y" or "--color-tests=1" won't work
and like wise "--comments" "--no-comments" seems out of place or
rather a better way to implement Boolean arguments.

Is there much interest in keeping this script the way it is or can I
lobotomize and release it under a new name? Suffix of ng is popular
these days, tapng-driver.sh. or tap-driverng.sh

[Bob Friesenhahn]

Post by Mike Mestnik
I'm reading this script closely because I'd like to make a few changes
and am surprised it hasn't received any edits since 2012. I think the

Perhaps it has been working perfectly since 2012. Or perhaps no one
has volunteered to improve it.

I (using this script since October, 2012) did not realize that there
was a problem with the script until today.

Post by Mike Mestnik
output is missing ratio of *test's failed, currently only file level
stats are presented not test level, and also the following options
from prove.
--verbose Print all test lines.
--failures Show failed tests.
* This to me is a key feature of TAP and this script silently discards
the most useful parts of the protocol.

It would be useful if there was more test summary output.

Post by Mike Mestnik
Somethings I'v spotted that are driving me nuts.
http://git.savannah.gnu.org/cgit/automake.git/tree/lib/tap-driver.sh#n100
Clearly indicates that "--color-tests=yes" is the only way to enable
color, there is no auto-detecting a PTY or any support for the
documented "--color-tests=always"
It's also crazy that "--color-tests=y" or "--color-tests=1" won't work
and like wise "--comments" "--no-comments" seems out of place or
rather a better way to implement Boolean arguments.

While I like color in photos and nature, automatic colorization of
output often causes problems for me since it often renders the text
unreadable. It also introduces terminal escape codes into the text
output, which can cause issues if captured output is later displayed
on something other than a ANSI terminal.

If there is color support, it should be disabled by default and
enabled by the user.

Post by Mike Mestnik
Is there much interest in keeping this script the way it is or can I
lobotomize and release it under a new name? Suffix of ng is popular
these days, tapng-driver.sh. or tap-driverng.sh

It is normal for projects which use this script to copy the script
into their own source tree. In this case you are free to name it
anything you like.

If you have implemented improvements then you should submit a patch to
the Automake project.

Bob

[Warren Young]

Post by Mike Mestnik
It's also crazy that "--color-tests=y" or "--color-tests=1" won't work

While I like color in photos and nature, automatic colorization of output often causes problems for me since it often renders the text unreadable.

Why, exactly?

I ask because the default color scheme of some terminal emulators makes the dark blue on black text difficult for me to read, but that’s easily fixed by switching the color scheme.

It also introduces terminal escape codes into the text output, which can cause issues if captured output is later displayed on something other than a ANSI terminal.

Both problems are caused by programs that hard-code literal ANSI/VT codes into text output. While ANSI-family terminal protocols have taken over the world now, that’s still a bad practice which needs to be smacked down wherever it reappears.

Well-behaved programs (e.g. GNU ls --color=auto, colordiff…) suppress color output when stdout is not a terminal. They do that by making calls like isatty(3) if written in C or test(1) -t if written in shell.

As for the portability of ANSI terminal escape codes, it’s still best to delegate such things to curses or libraries like it, despite the near ubiquity of ANSI-family terminal emulators.

[Mathieu Lirzin]

Post by Warren Young

Post by Bob Friesenhahn

Post by Mike Mestnik
It's also crazy that "--color-tests=y" or "--color-tests=1" won't work

While I like color in photos and nature, automatic colorization of
output often causes problems for me since it often renders the text
unreadable.

Why, exactly?
I ask because the default color scheme of some terminal emulators
makes the dark blue on black text difficult for me to read, but that’s
easily fixed by switching the color scheme.

Post by Bob Friesenhahn
It also introduces terminal escape codes into the text output, which can cause issues if captured output is later displayed on something other than a ANSI terminal.

Both problems are caused by programs that hard-code literal ANSI/VT
codes into text output. While ANSI-family terminal protocols have
taken over the world now, that’s still a bad practice which needs to
be smacked down wherever it reappears.
Well-behaved programs (e.g. GNU ls --color=auto, colordiff…) suppress
color output when stdout is not a terminal. They do that by making
calls like isatty(3) if written in C or test(1) -t if written in
shell.
As for the portability of ANSI terminal escape codes, it’s still best
to delegate such things to curses or libraries like it, despite the
near ubiquity of ANSI-family terminal emulators.

Colors are already automatically used when possible [1] and can be
disabled with the AM_COLOR_TESTS environment variable.

[Warren Young]

Post by Mathieu Lirzin

Post by Warren Young
As for the portability of ANSI terminal escape codes, it’s still best
to delegate such things to curses or libraries like it, despite the
near ubiquity of ANSI-family terminal emulators.

I didn’t quite finish that thought: “…because at the very least, you can then say 'TERM=dumb myprogram’ to strip terminal escape codes on an ad hoc basis.”

Post by Mathieu Lirzin
Colors are already automatically used when possible [1] and can be
disabled with the AM_COLOR_TESTS environment variable.

Right on.

Any reason it doesn’t back that up with “test -t 1”, though?

[Mathieu Lirzin]

Post by Warren Young

Post by Mathieu Lirzin

Post by Warren Young
As for the portability of ANSI terminal escape codes, it’s still best
to delegate such things to curses or libraries like it, despite the
near ubiquity of ANSI-family terminal emulators.

I didn’t quite finish that thought: “…because at the very least, you can then say 'TERM=dumb myprogram’ to strip terminal escape codes on an ad hoc basis.”

Post by Mathieu Lirzin
Colors are already automatically used when possible [1] and can be
disabled with the AM_COLOR_TESTS environment variable.

Right on.
Any reason it doesn’t back that up with “test -t 1”, though?

I think it does. :-)

Here is a snippet from lib/check.am which handles the detection:

--8<---------------cut here---------------start------------->8---
am__tty_colors_dummy = \
mgn= red= grn= lgn= blu= brg= std=; \
am__color_tests=no

am__tty_colors = { \
$(am__tty_colors_dummy); \
if test "X$(AM_COLOR_TESTS)" = Xno; then \
am__color_tests=no; \
elif test "X$(AM_COLOR_TESTS)" = Xalways; then \
am__color_tests=yes; \
## If stdout is a non-dumb tty, use colors. If test -t is not supported,
## then this check fails; a conservative approach. Of course do not
## redirect stdout here, just stderr.
elif test "X$$TERM" != Xdumb && { test -t 1; } 2>/dev/null; then \
am__color_tests=yes; \
fi; \
if test $$am__color_tests = yes; then \
red='[0;31m'; \
grn='[0;32m'; \
lgn='[1;32m'; \
blu='[1;34m'; \
mgn='[0;35m'; \
brg='[1m'; \
std='[m'; \
fi; \
}
--8<---------------cut here---------------end--------------->8---

[Bob Friesenhahn]

Post by Mathieu Lirzin

Post by Warren Young
As for the portability of ANSI terminal escape codes, it’s still best
to delegate such things to curses or libraries like it, despite the
near ubiquity of ANSI-family terminal emulators.

Colors are already automatically used when possible [1] and can be
disabled with the AM_COLOR_TESTS environment variable.

Yes, and in fact I am seeing colors in my TAP test output and the
colors are readible using my current terminal background.

Bob

[Bob Friesenhahn]

Post by Warren Young

Post by Bob Friesenhahn

Post by Mike Mestnik
It's also crazy that "--color-tests=y" or "--color-tests=1" won't work

While I like color in photos and nature, automatic colorization of
output often causes problems for me since it often renders the text
unreadable.

Why, exactly?

There are always assumptions about the terminal colors. It is often
assumed that the terminal background is black but I use a light color
background. Bright greens and cyans are virtually illegible for me.

Not all displays behave the same so what looks good one one display
may be illegible on another.

Post by Warren Young
I ask because the default color scheme of some terminal emulators
makes the dark blue on black text difficult for me to read, but
that’s easily fixed by switching the color scheme.

Switching the color scheme takes valuable time. When one visits many
different remote systems using various client systems, it is a losing
proposition to attempt to customize colors.

Post by Warren Young
Well-behaved programs (e.g. GNU ls --color=auto, colordiff…)
suppress color output when stdout is not a terminal. They do that
by making calls like isatty(3) if written in C or test(1) -t if
written in shell.

I sometimes make a typescript using 'script' and in this case colors
will be output. Other times I use 'tee' to capture output while still
seeing what is going on.

Post by Warren Young
As for the portability of ANSI terminal escape codes, it’s still
best to delegate such things to curses or libraries like it, despite
the near ubiquity of ANSI-family terminal emulators.

Agreed. Yet, a mismatch wastes time. I would rather use something
which works 100% of the time and looks less pretty than something
which works 80% of the time but can be made to look immaculate.

Defaulting to being usable 100% of the time works for me. :-)

Bob

[Russ Allbery]

Post by Warren Young
As for the portability of ANSI terminal escape codes, it’s still best to
delegate such things to curses or libraries like it, despite the near
ubiquity of ANSI-family terminal emulators.

Does anyone really use a non-ANSI terminal to run Automake test suites? I
haven't seen one in probably twenty years. I'm sure they still exist in
odd corners of some data center, but I'm quite dubious they would be
running Automake-driven test suites, as opposed to being connected to some
ancient mainframe that's as obscure as the terminal.

I think this is pointless portability akin to testing whether the C
library supports memcpy. I suppose in theory one could use tput to get
the appropriate strings, but now you're trading a theoretical portability
issue (a terminal type that's so esoteric as to not support basic ANSI
escape codes) for a very real and practical portability issue (a system
that doesn't have the curses/ncurses binaries installed). This doesn't
seem like a benefit.

(I'm in favor of disabling color by default, though, and having the
default color enabled mode test whether stdout is a tty before adding
colors. Those address the much more common case of redirecting test
output to a file, where the ANSI codes cause a lot of problems.)

[Warren Young]

Post by Russ Allbery

Post by Warren Young
As for the portability of ANSI terminal escape codes, it’s still best to
delegate such things to curses or libraries like it, despite the near
ubiquity of ANSI-family terminal emulators.

Does anyone really use a non-ANSI terminal to run Automake test suites?

I’ve never even used the Automake test suite features, but I do currently have a Raspberry Pi connected to a VT220 clone as the serial console:

https://tangentsoft.com/pidp8i/wiki?name=Warren%27s+PiDP-8/I+System

I’ve run programs through it that have not done the right thing despite my TERM being set appropriately, suggesting they’ve got ANSI X3.64 — or more likely, xterm-color[256] — expectations hard-coded into them.

Also, I gave the “TERM=dumb ./myprogram” example in a later reply.

Although DEC’s VT series, TERM=linux, xterm[-color[256]], and more are all loosely related, the Venn diagram for it would probably be pretty messy.

This thread is about color. My Link MC3+ doesn’t even *have* color, but it will do bold, underline, and blink. Why should Autoconf care what my terminal can do? The operating system has a database mapping what my terminal can do to a common API. Let the library handle it.

[Bob Friesenhahn]

Post by Warren Young
This thread is about color. My Link MC3+ doesn’t even *have* color,
but it will do bold, underline, and blink. Why should Autoconf care
what my terminal can do? The operating system has a database
mapping what my terminal can do to a common API. Let the library
handle it.

I should point out that the "operating system" does not necessarily
have what you describe.

Autoconf and Automake support many environments, including some
which lack a terminal database.

Indeed, Autoconf and Automake may be configuring/building software
which adds support for a terminal database.

This does not mean that it can't be supported on systems which do have
terminal information, but it makes the task somewhat more challenging.

Bob

[Warren Young]

Post by Warren Young
The operating system has a database mapping what my terminal can do to a common API. Let the library handle it.

I should point out that the "operating system" does not necessarily have what you describe.

Yes, I’m aware that I’m handwaving away the question of how a shell script talks to ncurses or whatever. (Too bad POSIX never nailed that one down.)

Anyway, all the opinions are on the table now. Whoever ends up implementing this now gets to mull through it all and decide which opinions mesh best with his own and go with it.

[Bob Friesenhahn]

Post by Warren Young

Post by Warren Young
The operating system has a database mapping what my terminal can do to a common API. Let the library handle it.

I should point out that the "operating system" does not necessarily have what you describe.

Yes, I’m aware that I’m handwaving away the question of how a shell
script talks to ncurses or whatever. (Too bad POSIX never nailed
that one down.)

Yes, and the current build might be the ncurses build (possible
chicken and egg situation).

In the classic self-hosted build situation the platform should only
need to provide a certain level of POSIX and other classic standards
support. Then it should be possible to build the rest of the
operating system and user applications starting from that point.
This is how the GNU Project got to where it is today.

It should not be necessary to have a fully-populated system in order
to build software.

Bob

[Russ Allbery]

Post by Warren Young
I’ve never even used the Automake test suite features, but I do
currently have a Raspberry Pi connected to a VT220 clone as the serial
https://tangentsoft.com/pidp8i/wiki?name=Warren%27s+PiDP-8/I+System

VT220 is definitely not a non-ANSI terminal. It's a direct descendent of
pretty much the *canonical* ANSI terminal.

You would need a Tektronix graphics terminal or something similar. Or a
VT52, if you can find one outside of a museum.

Post by Warren Young
I’ve run programs through it that have not done the right thing despite
my TERM being set appropriately, suggesting they’ve got ANSI X3.64 — or
more likely, xterm-color[256] — expectations hard-coded into them.

Lots of programs get the ANSI standard completely wrong. Also, lots of
programs aren't familiar with what's guaranteed and what isn't. But the
base color sequences should be either accepted or ignored by pretty much
every terminal anyone actually uses. (I maintain some software that
handles this for Perl -- Term::ANSIColor -- without using ncurses because
ncurses is a very heavy dependency and isn't readily available on several
platforms on which Perl is available, and have gotten compatibility
information from a pretty large number of devices.)

Post by Warren Young
Although DEC’s VT series, TERM=linux, xterm[-color[256]], and more are
all loosely related, the Venn diagram for it would probably be pretty
messy.

Yes, but the ESC [ N ; N m sequence is definitely supported by all of
those.

Post by Warren Young
This thread is about color. My Link MC3+ doesn’t even *have* color, but
it will do bold, underline, and blink.

Does it correctly ignore the color sequences that it doesn't support?

If you have examples of terminals that neither support nor ignore those
sequences, I'd very much like to know so that I can update my
compatibility chart.

[Mathieu Lirzin]

Hello,

Post by Bob Friesenhahn

Post by Mike Mestnik
Is there much interest in keeping this script the way it is or can I
lobotomize and release it under a new name? Suffix of ng is popular
these days, tapng-driver.sh. or tap-driverng.sh

It is normal for projects which use this script to copy the script
into their own source tree. In this case you are free to name it
anything you like.
If you have implemented improvements then you should submit a patch to
the Automake project.

Indeed, contribution for improving tap-driver.sh would be welcome.

[Mathieu Lirzin]

Hello,

Post by Mike Mestnik
Somethings I'v spotted that are driving me nuts.
http://git.savannah.gnu.org/cgit/automake.git/tree/lib/tap-driver.sh#n100
Clearly indicates that "--color-tests=yes" is the only way to enable
color, there is no auto-detecting a PTY or any support for the
documented "--color-tests=always"

The detection is done in the generated Makefile which invokes
tap-driver.sh with the appropriate command line arguments. Duplicating
detection inside the test driver would make sense only in the context of
invoking it interactively.

Do you have a use case where we would want to invoke tap-driver.sh
outside of the 'make check' context?

Thanks.

[Mike Mestnik]

Post by Mathieu Lirzin
Hello,

Post by Mike Mestnik
Somethings I'v spotted that are driving me nuts.
http://git.savannah.gnu.org/cgit/automake.git/tree/lib/tap-driver.sh#n100
Clearly indicates that "--color-tests=yes" is the only way to enable
color, there is no auto-detecting a PTY or any support for the
documented "--color-tests=always"

The detection is done in the generated Makefile which invokes
tap-driver.sh with the appropriate command line arguments. Duplicating
detection inside the test driver would make sense only in the context of
invoking it interactively.

This is something important for me to be aware of, as I am planing on
making changes I'll need to understand the whole system.
I'll read check.am now.

Thanks.

Post by Mathieu Lirzin
Do you have a use case where we would want to invoke tap-driver.sh
outside of the 'make check' context?

I just needed to know why the script and the documentation were mismatched.

Post by Mathieu Lirzin
Thanks.
--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37