distcheck does 'chmod a-w' - leads to mkdir "permission denied"

[Paul Jakma]

Hi,

My make distcheck is broken and I can't figure out how to fix it or
where the problem lies. It seems to be a similar issue to:

http://gnu-automake.7480.n7.nabble.com/cannot-create-directory-am15097-Permission-denied-td209.html

Mine does:

...
chmod -R a-w quagga-1.2.0
chmod u+w quagga-1.2.0
mkdir quagga-1.2.0/_build quagga-1.2.0/_build/sub quagga-1.2.0/_inst
chmod a-w quagga-1.2.0
test -d quagga-1.2.0/_build || exit 0; \
...

before going into quagga-1.2.0/_build/sub to do an out of source-tree
build in _build/sub on the sources in ../.. or quagga-1.2.0. Except then
at some stage later it fails trying to do a mkdir in ../../../doc/ (i.e.
quagga-1.2.0/doc):

Making all in doc
make[3]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
make all-am
make[4]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
MAKEINFO ../../../doc/quagga.info
mkdir: cannot create directory ‘.am18743’: Permission denied
could not open ../../../doc/quagga.texi: No such file or directory

With V=99:

Making all in doc
make[3]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
make all-am
make[4]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
restore=: && backupdir=".am$$" && \
am__cwd=`pwd` && CDPATH="${ZSH_VERSION+.}:" && cd ../../../doc && \
rm -rf $backupdir && mkdir $backupdir && \
if (makeinfo --version) >/dev/null 2>&1; then \
for f in ../../../doc/quagga.info ../../../doc/quagga.info-[0-9]
../../../doc/quagga.info-[0-9][0-9] ../../../doc/quagga.i[0-9]
../../../doc/quagga.i[0-9][0-9]; do \
if test -f $f; then mv $f $backupdir; restore=mv; else :; fi; \
done; \
else :; fi && \
cd "$am__cwd"; \
if makeinfo -I ../../../doc \
-o ../../../doc/quagga.info ../../../doc/quagga.texi; \
then \
rc=0; \
CDPATH="${ZSH_VERSION+.}:" && cd ../../../doc; \
else \
rc=$?; \
CDPATH="${ZSH_VERSION+.}:" && cd ../../../doc && \
$restore $backupdir/* `echo "./../../../doc/quagga.info" | sed
's|[^/]*$||'`; \
fi; \
rm -rf $backupdir; exit $rc
mkdir: cannot create directory ‘.am1976’: Permission denied
could not open ../../../doc/quagga.texi: No such file or directory
/bin/sh: line 16: cd: ../../../doc: No such file or directory
Makefile:493: recipe for target '../../../doc/quagga.info' failed

I'm a bit mystified as to where to start looking. It don't see anything
obvious in the doc/Makefile.am. The only thing is we have a
'defines.texi' that is built by automake from defines.texi.in and
declared as:

BUILT_SOURCES = defines.texi

regards,

[Mathieu Lirzin]

Hi Paul,

Post by Paul Jakma
My make distcheck is broken and I can't figure out how to fix it or
http://gnu-automake.7480.n7.nabble.com/cannot-create-directory-am15097-Permission-denied-td209.html
...
chmod -R a-w quagga-1.2.0
chmod u+w quagga-1.2.0
mkdir quagga-1.2.0/_build quagga-1.2.0/_build/sub quagga-1.2.0/_inst
chmod a-w quagga-1.2.0
test -d quagga-1.2.0/_build || exit 0; \
...
before going into quagga-1.2.0/_build/sub to do an out of source-tree
build in _build/sub on the sources in ../.. or quagga-1.2.0. Except
then at some stage later it fails trying to do a mkdir in
Making all in doc
make[3]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
make all-am
make[4]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
MAKEINFO ../../../doc/quagga.info
mkdir: cannot create directory ‘.am18743’: Permission denied
could not open ../../../doc/quagga.texi: No such file or directory
Making all in doc
make[3]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
make all-am
make[4]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
restore=: && backupdir=".am$$" && \
am__cwd=`pwd` && CDPATH="${ZSH_VERSION+.}:" && cd ../../../doc && \
rm -rf $backupdir && mkdir $backupdir && \
if (makeinfo --version) >/dev/null 2>&1; then \
for f in ../../../doc/quagga.info ../../../doc/quagga.info-[0-9]
../../../doc/quagga.info-[0-9][0-9] ../../../doc/quagga.i[0-9]
../../../doc/quagga.i[0-9][0-9]; do \
if test -f $f; then mv $f $backupdir; restore=mv; else :; fi; \
done; \
else :; fi && \
cd "$am__cwd"; \
if makeinfo -I ../../../doc \
-o ../../../doc/quagga.info ../../../doc/quagga.texi; \
then \
rc=0; \
CDPATH="${ZSH_VERSION+.}:" && cd ../../../doc; \
else \
rc=$?; \
CDPATH="${ZSH_VERSION+.}:" && cd ../../../doc && \
$restore $backupdir/* `echo "./../../../doc/quagga.info" | sed
's|[^/]*$||'`; \
fi; \
rm -rf $backupdir; exit $rc
mkdir: cannot create directory ‘.am1976’: Permission denied
could not open ../../../doc/quagga.texi: No such file or directory
/bin/sh: line 16: cd: ../../../doc: No such file or directory
Makefile:493: recipe for target '../../../doc/quagga.info' failed
I'm a bit mystified as to where to start looking. It don't see
anything obvious in the doc/Makefile.am. The only thing is we have a
'defines.texi' that is built by automake from defines.texi.in and
BUILT_SOURCES = defines.texi

Can you provide a minimal example that allow others to reproduce the
problem?

Thanks.

[Paul Jakma]

Post by Mathieu Lirzin
Can you provide a minimal example that allow others to reproduce the
problem?

I have a reproducer, but far from minimal:

(cd /tmp/ && \
wget http://download.savannah.gnu.org/releases/quagga/quagga-1.2.0.tar.gz && \
cd quagga-1.2.0 && ./update-autotools && \
./configure && make distcheck)

It's something in the doc/ build obviously, to make quagga.texinfo, but
i don't know what...

The other post I found with a similar issue suggested it might be a bug
in texinfo automake macros?

regards,

[Mathieu Lirzin]

Post by Paul Jakma

Post by Mathieu Lirzin
Can you provide a minimal example that allow others to reproduce the
problem?

(cd /tmp/ && \
wget http://download.savannah.gnu.org/releases/quagga/quagga-1.2.0.tar.gz && \
cd quagga-1.2.0 && ./update-autotools && \
./configure && make distcheck)

I stopped digging when I saw that I need libcares which is not provided
by my distro.

Post by Paul Jakma
It's something in the doc/ build obviously, to make quagga.texinfo,
but i don't know what...
The other post I found with a similar issue suggested it might be a
bug in texinfo automake macros?

in "doc/Makefile.am" 'quagga_TEXINFOS' contains "defines.texi". By
default files in 'quagga_TEXINFOS' are distributed in the tarball.
since "defines.texi" is a generated file you don't want to distribute
it. Putting it in 'nodist_quagga_TEXINFOS' instead should fix your
issue.

The error message of 'make distcheck' is not really helpful to detect
this issue, I am trying to see if something can be done to improve that.

[Paul Jakma]

Post by Mathieu Lirzin
I stopped digging when I saw that I need libcares which is not provided
by my distro.

Ah, well, it's only needed for nhrpd, which can be disabled with
--disable-nhrpd. I'm not sure if the configure script handles it
correctly though.

Post by Mathieu Lirzin
in "doc/Makefile.am" 'quagga_TEXINFOS' contains "defines.texi". By
default files in 'quagga_TEXINFOS' are distributed in the tarball.
since "defines.texi" is a generated file you don't want to distribute
it. Putting it in 'nodist_quagga_TEXINFOS' instead should fix your
issue.

Even with defines.texi not distributed, or regenerated in the distcheck
build, it still fails. Cause quagga.info depends on it. :(

I tried nodist_quagga_TEXINFOS and got the same result.

The automake option to build the info in the builddir worked though!

Post by Mathieu Lirzin
The error message of 'make distcheck' is not really helpful to detect
this issue, I am trying to see if something can be done to improve that.

That would be great. :) It's very difficult to figure out the issue
without help.

Thanks!

Oh, I also had to add:

# do nothing for DVI, so we don't have to generate or distribute EPS
# figures
dvi: # nothing

to my doc/Makefile.am, or it would fail checking the 'dvi' target. Does
anyone use DVI anymore? It's been probably 20 years since I did. Does it
still need to be a default generated target?

Now I've got errors from 'distcleancheck', but those are definitely ones
I wanted to see. ;)

regards,

[Mathieu Lirzin]

Post by Paul Jakma

Post by Mathieu Lirzin
I stopped digging when I saw that I need libcares which is not provided
by my distro.

Ah, well, it's only needed for nhrpd, which can be disabled with
--disable-nhrpd. I'm not sure if the configure script handles it
correctly though.

Post by Mathieu Lirzin
in "doc/Makefile.am" 'quagga_TEXINFOS' contains "defines.texi". By
default files in 'quagga_TEXINFOS' are distributed in the
tarball. since "defines.texi" is a generated file you don't want to
distribute it. Putting it in 'nodist_quagga_TEXINFOS' instead
should fix your issue.

Even with defines.texi not distributed, or regenerated in the
distcheck build, it still fails. Cause quagga.info depends on it. :(
I tried nodist_quagga_TEXINFOS and got the same result.
The automake option to build the info in the builddir worked though!

OK.

Post by Paul Jakma

Post by Mathieu Lirzin
The error message of 'make distcheck' is not really helpful to detect
this issue, I am trying to see if something can be done to improve that.

That would be great. :) It's very difficult to figure out the issue
without help.
Thanks!
# do nothing for DVI, so we don't have to generate or distribute EPS
# figures
dvi: # nothing

I Will take a look.

Post by Paul Jakma
to my doc/Makefile.am, or it would fail checking the 'dvi'
target. Does anyone use DVI anymore? It's been probably 20 years since
I did. Does it still need to be a default generated target?

I don't know. :)

Post by Paul Jakma
Now I've got errors from 'distcleancheck', but those are definitely
ones I wanted to see. ;)

Thanks.

[Nick Bowler]

Hi,

Post by Paul Jakma
My make distcheck is broken and I can't figure out how to fix it or
where the problem lies.

[...]

Post by Paul Jakma
chmod -R a-w quagga-1.2.0
chmod u+w quagga-1.2.0
mkdir quagga-1.2.0/_build quagga-1.2.0/_build/sub quagga-1.2.0/_inst
chmod a-w quagga-1.2.0
test -d quagga-1.2.0/_build || exit 0; \

[...]

Post by Paul Jakma
Making all in doc
make[3]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
make all-am
make[4]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
MAKEINFO ../../../doc/quagga.info
mkdir: cannot create directory ‘.am18743’: Permission denied

One of the things 'make distcheck' tests it that it can run builds from a
read-only source tree. That is, remove all write permissions from the
unpacked tarball, then perform build actions (at the same time, it is
doing a VPATH build).

This includes running 'make dist' from such a source tree.

It looks to me like you have a problem where some build rule is trying
to write to srcdir (this is a common way to write rules when distributing
generated files) This probably means you have a timestamp problem in
your distribution tarball (e.g., some distributed files are older than
their source files).

Expected behaviour on a freshly unpacked tarball is that all such
generated files are up to date, and therefore no build rules will
attempt to update them.

Cheers,
Nick

[Paul Jakma]

Post by Nick Bowler
This includes running 'make dist' from such a source tree.

Yeah makes sense, and it's a good thing, and I'd like to fix it.

But I'm struggling to see where the issue is - whether it's something in
our rules, or some bug in some automake macro somewhere (e.g. texinfo,
as other post suggested).

Post by Nick Bowler
It looks to me like you have a problem where some build rule is trying
to write to srcdir (this is a common way to write rules when distributing
generated files) This probably means you have a timestamp problem in
your distribution tarball (e.g., some distributed files are older than
their source files).

Hmm, that's generated via 'make dist' though, so that shouldn't happen?

Post by Nick Bowler
Expected behaviour on a freshly unpacked tarball is that all such
generated files are up to date, and therefore no build rules will
attempt to update them.

It's doc/quagga.info it is trying to create - a generated file, which we
don't keep in the git repo. If I go back up to the main doc dir and
'make quagga.info' so it is guaranteed to be up to date, and then go
back up to the top-dir and do 'make distcheck' it fails again the exact
same way - but the file in the source _is_ up to date:

make[4]: Entering directory
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
MAKEINFO ../../../doc/quagga.info
mkdir: cannot create directory ‘.am8211’: Permission denied
could not open ../../../doc/quagga.texi: No such file or directory
/bin/sh: line 16: cd: ../../../doc: No such file or directory
..
$ cd doc # in the top-level repo
$ make quagga.info
make: 'quagga.info' is up to date.

Baffling?

regards,

[Paul Smith]

make[4]: Entering directory 
'/home/paul/code/quagga/quagga-1.2.0/_build/sub/doc'
   MAKEINFO ../../../doc/quagga.info
mkdir: cannot create directory ‘.am8211’: Permission denied
could not open ../../../doc/quagga.texi: No such file or directory
/bin/sh: line 16: cd: ../../../doc: No such file or directory
..
$ cd doc # in the top-level repo
$ make quagga.info
make: 'quagga.info' is up to date.
Baffling?

Run make (in a remote directory freshly unpacked from your dist file)
with the "-d" flag (redirect it due to voluminous output) and it will
tell you exactly why it decided to run that recipe.

No doubt it depends on some other file which has an unexpected
timestamp.

[Paul Jakma]

Post by Paul Smith

Post by Paul Jakma
Baffling?

Run make (in a remote directory freshly unpacked from your dist file)
with the "-d" flag (redirect it due to voluminous output) and it will
tell you exactly why it decided to run that recipe.
No doubt it depends on some other file which has an unexpected
timestamp.

Well, it's due to the distributed 'doc/quagga.info' being slightly older
than the (built by automake) doc/defines.texi.

Both are made by the 'make dist' and put in the dist tarball. Then
distcheck unpacks that to a subdir, makes it RO, and creates another
subdir for an out-of-tree build and builds in that. automake then makes
a new doc/defines.texi, but the dist source ../../../doc/quagga.info of
course is now older, so...

This is from within the out-of-tree builddir that make distcheck is
building in, with the unpacked RO dist sources at ../../..

$ ls -l ../../../doc/{quagga.info,defines.texi*}
-r--r--r-- 1 paul paul 503 Mar 3 16:34 ../../../doc/defines.texi
-r--r--r-- 1 paul paul 480 Feb 8 15:19 ../../../doc/defines.texi.in
-r--r--r-- 1 paul paul 9169 Mar 3 16:34 ../../../doc/quagga.info
$ ls -l defines.texi
-rw-rw-r-- 1 paul paul 503 Mar 3 16:44 defines.texi

defines.texi.in is how we get stuff like INSTALL_PREFIX into the TexInfo
documentation.

I guess we could find a way to /not/ distribute the built quagga.info,
but that would then require people to have texinfo to build the project.
Though, there's a risk whatever it is would still try build it as
../../../doc/quagga.info rather than doc/quagga.info.

I'm using fairly standard Automake macros to do this:

# Built from defines.texi.in
BUILT_SOURCES = defines.texi

info_TEXINFOS = quagga.texi

quagga_TEXINFOS =

Looking at the automake documentation I read:

"It is worth noting that, contrary to what happens with the other
formats, the generated ‘.info’ files are by default placed in ‘srcdir’
rather than in the ‘builddir’."

Could that be the source of the problem?

It goes on:

"This can be changed with the ‘info-in-builddir’ option."

And, indeed, when I add "AUTOMAKE_OPTIONS=info-in-builddir" to
doc/Makefile.am the above problem disappears!

This seems to be sub-optimal in automake? Any auto-generation of files
by automake used for the info documentation results in a broken
distcheck?

regards,

[Paul Smith]

Post by Paul Jakma
Well, it's due to the distributed 'doc/quagga.info' being slightly
older than the (built by automake) doc/defines.texi.

I don't understand this: if doc/defines.texi is needed by quagga.info
then why doesn't quagga.info depend on it?

And if it does depend on it, then how can doc/defines.texi be newer
without rebuilding quagga.info (when creating the dist file, not when
building the dist file)?

[Paul Jakma]

Post by Paul Smith

Post by Paul Jakma
Well, it's due to the distributed 'doc/quagga.info' being slightly
older than the (built by automake) doc/defines.texi.

I don't understand this: if doc/defines.texi is needed by quagga.info
then why doesn't quagga.info depend on it?

The dependencies seem correct:

$ make quagga.info
MAKEINFO quagga.info
$ touch defines.texi
$ make quagga.info
MAKEINFO quagga.info
$ touch defines.texi.in
$ make quagga.info
cd .. && /bin/sh ./config.status doc/defines.texi
config.status: creating doc/defines.texi
MAKEINFO quagga.info

Post by Paul Smith
And if it does depend on it, then how can doc/defines.texi be newer
without rebuilding quagga.info (when creating the dist file, not when
building the dist file)?

It's because doc/defines.texi gets generated in the builddir when
configure is run.

$ make distclean
test -z "quagga.dvi quagga.pdf quagga.ps quagga.html" \
|| rm -rf quagga.dvi quagga.pdf quagga.ps quagga.html
rm -rf .libs _libs
rm -rf quagga.t2d quagga.t2p
rm -f *.lo
rm -f vti.tmp* ./version.texi.tmp*
test -z "defines.texi" || rm -f defines.texi
test . = "." || test -z "" || rm -f
rm -f Makefile
$ ./configure > /dev/null
# defines.texi is back:
$ ls doc/defines.texi
doc/defines.texi

So:

- the dist tarball are unpacked to RO source dir

- configure is run in another build dir

- defines.texi is generated in the build-dir doc/ by automake (though it
was distributed, so that isn't necessary in the abstract)

- So quagga.info must be updated of course

- but automake defaults to building the info sources in the source-dir,
which is of course RO as part of distcheck

- Boom

Is what it seems like to me.

regards,

[Nick Bowler]

Hello,

Post by Paul Jakma
Well, it's due to the distributed 'doc/quagga.info' being slightly older
than the (built by automake) doc/defines.texi.
Both are made by the 'make dist' and put in the dist tarball. Then
distcheck unpacks that to a subdir, makes it RO, and creates another
subdir for an out-of-tree build and builds in that. automake then makes
a new doc/defines.texi, but the dist source ../../../doc/quagga.info of
course is now older, so...

So the problem seems to be that defines.texi is *also* generated, but by
your normal build process? configure outputs it? If the docs depend on
configure output then there is no point in distributing generated docs
at all, since every user will have to regenerate them anyway.

[...]

Post by Paul Jakma
I guess we could find a way to /not/ distribute the built quagga.info,
but that would then require people to have texinfo to build the project.

Well as you can see, texinfo is already required since the rule to build
documentation is being run in any case.

Post by Paul Jakma
Though, there's a risk whatever it is would still try build it as
../../../doc/quagga.info rather than doc/quagga.info.
# Built from defines.texi.in
BUILT_SOURCES = defines.texi
info_TEXINFOS = quagga.texi
quagga_TEXINFOS =
"It is worth noting that, contrary to what happens with the other
formats, the generated ‘.info’ files are by default placed in ‘srcdir’
rather than in the ‘builddir’."
Could that be the source of the problem?

Well yes, but probably not by itself...

Post by Paul Jakma
"This can be changed with the ‘info-in-builddir’ option."
And, indeed, when I add "AUTOMAKE_OPTIONS=info-in-builddir" to
doc/Makefile.am the above problem disappears!

The builddir is writable so generating docs there will not fail (well,
unless the user doesn't have makeinfo installed).

Personally I prefer to generate files into builddir and distribute from
there (so I like this option), but are other tradeoffs (this way can have
different subtle problems, usually related to unintentionally having
files of the same name in srcdir and builddir).

Post by Paul Jakma
This seems to be sub-optimal in automake? Any auto-generation of files
by automake used for the info documentation results in a broken
distcheck?

I would expect it to work out of the box provided that your texi files
are not modified by the build system.

Cheers,
Nick

[Paul Jakma]

Post by Nick Bowler
So the problem seems to be that defines.texi is *also* generated, but
by your normal build process? configure outputs it?

Or automake, as run by configure?

$ cat doc/defines.texi.in
@c -*- texinfo -*-
@c @configure_input@

@c Set variables
@set PACKAGE_NAME @PACKAGE_NAME@
@set PACKAGE_TARNAME @PACKAGE_TARNAME@
@set PACKAGE_STRING @PACKAGE_STRING@

<further static stuff>

Goes to:

$ cat doc/defines.texi
@c -*- texinfo -*-
@c doc/defines.texi. Generated from defines.texi.in by configure.

@c Set variables
@set PACKAGE_NAME Quagga
@set PACKAGE_TARNAME quagga
@set PACKAGE_STRING Quagga 1.2.0

<remaining static stuff>

Post by Nick Bowler
If the docs depend on configure output then there is no point in
distributing generated docs at all, since every user will have to
regenerate them anyway.
Well as you can see, texinfo is already required since the rule to
build documentation is being run in any case.

True.

I didn't realise that actually.

Post by Nick Bowler
I would expect it to work out of the box provided that your texi files
are not modified by the build system.

That's the nub of it.

I guess I need to get rid of defines.texi.in somehow.

Thanks,