How to avoid stopping the recursive `check` target?

Discussion:

Philippe Proulx

2017-08-08 17:32:05 UTC

My project has the following directory structure (some files hidden for
clarity):

Makefile.am
tests/
Makefile.am
a/
Makefile.am
b/
Makefile.am
c/
Makefile.am

If I put all the tests to run that are in the a/, b/, and c/
subdirectories in the TESTS variable of tests/Makefile.am, then with
`make check` in tests/, all the tests run, even if one test in one of
the subdirectories fail. Assume this is the behaviour I want.

With this configuration, however, I cannot go to b/, for example, and
run `make check`: since there's no TESTS variable there, `make check`
does nothing.

So I put a TESTS variable in a/Makefile.am, b/Makefile.am, and
c/Makefile.am, and remove the TESTS variable from tests/Makefile.am. Now
I can go to a/, b/, or c/ and run `make check` to test only specific
parts of the project. However, since each individual `make check` can
fail, now the "global" `make check` in tests/ fails as soon as one
subdirectory fails, which is the expected behaviour of Make.

My current workaround is to use `make --keep-going check` in tests/, so
that, as per make(1):

Continue as much as possible after an error. While the target that
failed, and those that depend on it, can‐ not be remade, the other
dependencies of these targets can be processed all the same.

This seems to run all the individual `make check` and exit with
something else than 0 if one of them fails.

I can also wrap this `make --keep-going check` in a new target, for
example:

test:
$(MAKE) $(AM_MAKEFLAGS) --keep-going check

Now `make test` does what I want in tests/.

Is there anything more "Automaky" I could do to achieve the same goal?
Or is using `make --keep-going check` the expected method here if I
don't want Make to stop as soon as one subdirectory test fails?

Thank you for your time,
Phil

Mathieu Lirzin

2017-08-08 21:06:37 UTC

Permalink

Hello,

Post by Philippe Proulx
So I put a TESTS variable in a/Makefile.am, b/Makefile.am, and
c/Makefile.am, and remove the TESTS variable from tests/Makefile.am. Now
I can go to a/, b/, or c/ and run `make check` to test only specific
parts of the project. However, since each individual `make check` can
fail, now the "global" `make check` in tests/ fails as soon as one
subdirectory fails, which is the expected behaviour of Make.
My current workaround is to use `make --keep-going check` in tests/, so
Continue as much as possible after an error. While the target that
failed, and those that depend on it, can‐ not be remade, the other
dependencies of these targets can be processed all the same.
This seems to run all the individual `make check` and exit with
something else than 0 if one of them fails.
I can also wrap this `make --keep-going check` in a new target, for
$(MAKE) $(AM_MAKEFLAGS) --keep-going check
Now `make test` does what I want in tests/.
Is there anything more "Automaky" I could do to achieve the same goal?
Or is using `make --keep-going check` the expected method here if I
don't want Make to stop as soon as one subdirectory test fails?

I don't know if this is more "Automaky" but for that use case I would
use a non-recursive makefile with custom targets for each subset of
tests. Here is an untested snippet:

--8<---------------cut here---------------start------------->8---
foo_tests = foo/...
bar_tests = bar/...

TESTS = $(foo_tests) $(bar_tests)

check-foo:
$(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)"

check-bar:
$(MAKE) $(AM_MAKEFLAGS) TESTS="$(bar_tests)"
--8<---------------cut here---------------end--------------->8---

If you really want the ability to run 'make check' from sub
directories, you can create additional makefiles containing

--8<---------------cut here---------------start------------->8---
check:
$(MAKE) -C .. check-xxx
--8<---------------cut here---------------end--------------->8---

However I would not recommend it.

HTH,

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37

Philippe Proulx

2017-08-08 21:29:38 UTC

Permalink

Post by Mathieu Lirzin
Hello,

I don't know if this is more "Automaky" but for that use case I would
use a non-recursive makefile with custom targets for each subset of

Not a bad idea indeed.

Post by Mathieu Lirzin
--8<---------------cut here---------------start------------->8---
foo_tests = foo/...
bar_tests = bar/...
TESTS = $(foo_tests) $(bar_tests)
$(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)"

I guess you mean:

$(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)" check

Post by Mathieu Lirzin
$(MAKE) $(AM_MAKEFLAGS) TESTS="$(bar_tests)"

Same here?

Thank you,
Phil

Post by Mathieu Lirzin
--8<---------------cut here---------------end--------------->8---
If you really want the ability to run 'make check' from sub
directories, you can create additional makefiles containing
--8<---------------cut here---------------start------------->8---
$(MAKE) -C .. check-xxx
--8<---------------cut here---------------end--------------->8---
However I would not recommend it.
HTH,
--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37

Mathieu Lirzin

2017-08-08 22:10:18 UTC

Permalink

Post by Philippe Proulx

$(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)" check

Post by Mathieu Lirzin
$(MAKE) $(AM_MAKEFLAGS) TESTS="$(bar_tests)"

Same here?

Yes indeed. :)

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37