More control over 'make dist'

[Michal Privoznik]

Dear list,

I'm a libvirt devel and I've ran into interesting problem. I'd like to
hear your opinions on it.

Libvirt is a virtualization library that uses XML to store a virtual
machine config. We have couple of tests in our repository too that check
whether XML configs are valid, and whether some operations change it in
expected way. Some operations don't change the XML at all, in which case
we just symlink the output file to point to the input file:

ln -s $xml.in $xml.out

However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved. I've traced down
the problem and found that autoconf is just hardcoding some of tar's
options. Namely -chf. Yes, it is -h that causes a symlink to be
dereferenced.

So my question is, what do you think of making -h configurable? We could
add new tar-* option to AM_INIT_AUTOMAKE, say tar-symlinks, which would
suppress -h on tar's command line.

What are your thoughts?

Michal

[Peter Rosin]

Post by Michal Privoznik
Dear list,
I'm a libvirt devel and I've ran into interesting problem. I'd like to
hear your opinions on it.
Libvirt is a virtualization library that uses XML to store a virtual
machine config. We have couple of tests in our repository too that check
whether XML configs are valid, and whether some operations change it in
expected way. Some operations don't change the XML at all, in which case
ln -s $xml.in $xml.out
However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved. I've traced down
the problem and found that autoconf is just hardcoding some of tar's
options. Namely -chf. Yes, it is -h that causes a symlink to be
dereferenced.
So my question is, what do you think of making -h configurable? We could
add new tar-* option to AM_INIT_AUTOMAKE, say tar-symlinks, which would
suppress -h on tar's command line.
What are your thoughts?

I believe that is for the benefit of supporting unpacking the release
tarball on systems that do not support symlinks, or where symlinks are
not as flexible as one might wish for.

Cheers,
Peter

[Warren Young]

Post by Peter Rosin

Post by Michal Privoznik
ln -s $xml.in $xml.out
However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved.

I believe that is for the benefit of supporting unpacking the release
tarball on systems that do not support symlinks, or where symlinks are
not as flexible as one might wish for.

The question, then, is whether libvirt would ever be unpacked on such a system.

I’m barely aware of what libvirt does, but I think I can come up with a plausible scenario: libvirt built from source on Cygwin.

It appears from the home page that libvirt already supports Hyper-V, so a naive user might decide to build it under Cygwin rather than whatever native Windows toolchain is currently used for that case. (The reason being that libvirt, coming from the Linux world, probably builds better under a Unix-like environment.) Since NTFS symlinks have a number of unfortunate limitations[1] and restrictions[2], the tarball almost certainly won’t unpack correctly.

[1] https://en.wikipedia.org/wiki/NTFS_symbolic_link#Restrictions
[2] https://en.wikipedia.org/wiki/NTFS_symbolic_link#Limitations

[Eric Blake]

[re-adding Michal; not sure he is a subscriber. It's usually best to
reply-to-all rather than assume that all readers are on-list]

Post by Warren Young

Post by Peter Rosin

Post by Michal Privoznik
ln -s $xml.in $xml.out
However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved.

I believe that is for the benefit of supporting unpacking the release
tarball on systems that do not support symlinks, or where symlinks are
not as flexible as one might wish for.

The question, then, is whether libvirt would ever be unpacked on such a system.
I’m barely aware of what libvirt does, but I think I can come up with a plausible scenario: libvirt built from source on Cygwin.
It appears from the home page that libvirt already supports Hyper-V, so a naive user might decide to build it under Cygwin rather than whatever native Windows toolchain is currently used for that case. (The reason being that libvirt, coming from the Linux world, probably builds better under a Unix-like environment.) Since NTFS symlinks have a number of unfortunate limitations[1] and restrictions[2], the tarball almost certainly won’t unpack correctly.

Cygwin unpacks and handles symlinks just fine as one of the many things
it emulates, so that's not the issue (note, however, that non-cygwin
apps are generally unable to recognize cygwin symlinks, which are
intentionally NOT done as NTFS symlinks because NTFS symlinks are not
POSIXy).

I'd be more worried if trying to unpack libvirt and build it with mingw,
where you no longer have cygwin symlinks in the mix, and would indeed be
limited to just NTFS symlinks. But I think that mingw ports of tar
already take care of that, by turning symlinks in the tarball into file
copies (where the link target was also in the tarball) or an error
message (where the link target is unknown) (although I haven't actually
tested what MSYS would actually do, so I welcome further comments from
anyone with experience).

And yes, libvirt is (cross-)built for mingw on Fedora already, as well
as me getting ready to prepare a Cygwin distribution build of libvirt
(both for the remote machine control aspect, and for the HyperV aspect).
I'm less familiar with cross-building for mingw to know for certain
whether symlinks in tarballs cause problems for mingw targets, or
whether Wine emulation on Linux even handles symlinks. So that agrees
with the notion of making tarballs as portable as possible by avoiding
symlinks.

On the other hand, maybe the GNU Coding Standards should be updated to
state that avoiding symlinks is no longer required of GNU packages (it's
entirely plausible that when automake first coded up the tar -h rule,
years ago, there really were Unixy systems where either the file system
or the tar program didn't handle symlinks as gracefully as desired,
which is a different beast than today's mingw situation).

[Michal Privoznik]

Post by Eric Blake
[re-adding Michal; not sure he is a subscriber. It's usually best to
reply-to-all rather than assume that all readers are on-list]

Thank you, I'll use your e-mail to reply to previous ones too.

Post by Eric Blake

Post by Warren Young

Post by Peter Rosin

Post by Michal Privoznik
ln -s $xml.in $xml.out
However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved.

I believe that is for the benefit of supporting unpacking the release
tarball on systems that do not support symlinks, or where symlinks are
not as flexible as one might wish for.

Well, I believe it's project developers to say whether they want to
support it or not. For instance if there's a project implementing a
linux kernel module I think it's fair to expect some reasonable
environment too. Moreover there are just a few filesystems that don't
support symlinks (including FAT which would be dead already if it wasn't
for all the portable electronics).

Post by Eric Blake

Post by Warren Young
The question, then, is whether libvirt would ever be unpacked on such a system.
I’m barely aware of what libvirt does, but I think I can come up with a plausible scenario: libvirt built from source on Cygwin.
It appears from the home page that libvirt already supports Hyper-V, so a naive user might decide to build it under Cygwin rather than whatever native Windows toolchain is currently used for that case. (The reason being that libvirt, coming from the Linux world, probably builds better under a Unix-like environment.) Since NTFS symlinks have a number of unfortunate limitations[1] and restrictions[2], the tarball almost certainly won’t unpack correctly.

Well, we've stopped providing official .exe a long ago (2011), but we
still allow users to compile client library using mingw (no tests are
run there though). But still - it is build under Linux with some
reasonable filesystem (i.e. one supporting symlinks).

Post by Eric Blake
Cygwin unpacks and handles symlinks just fine as one of the many things
it emulates, so that's not the issue (note, however, that non-cygwin
apps are generally unable to recognize cygwin symlinks, which are
intentionally NOT done as NTFS symlinks because NTFS symlinks are not
POSIXy).
I'd be more worried if trying to unpack libvirt and build it with mingw,
where you no longer have cygwin symlinks in the mix, and would indeed be
limited to just NTFS symlinks. But I think that mingw ports of tar
already take care of that, by turning symlinks in the tarball into file
copies (where the link target was also in the tarball) or an error
message (where the link target is unknown) (although I haven't actually
tested what MSYS would actually do, so I welcome further comments from
anyone with experience).
And yes, libvirt is (cross-)built for mingw on Fedora already, as well
as me getting ready to prepare a Cygwin distribution build of libvirt
(both for the remote machine control aspect, and for the HyperV aspect).
I'm less familiar with cross-building for mingw to know for certain
whether symlinks in tarballs cause problems for mingw targets, or
whether Wine emulation on Linux even handles symlinks. So that agrees
with the notion of making tarballs as portable as possible by avoiding
symlinks.
On the other hand, maybe the GNU Coding Standards should be updated to
state that avoiding symlinks is no longer required of GNU packages (it's
entirely plausible that when automake first coded up the tar -h rule,
years ago, there really were Unixy systems where either the file system
or the tar program didn't handle symlinks as gracefully as desired,
which is a different beast than today's mingw situation).

Agreed. I'm not advocating for every project out there using it, I just
think that it would be nice if developers are given the choice.

Michal

[Daniel Herring]

Hi Michal,

I've always handled this by distributing the single real file, along with
a script that creates missing symlinks later, either during configure or
as part of one of the make targets.

Its a bit quirky, but not too annoying. Since the dist target is for
making source packages, there are plenty of natural places to hide this.

Many related pieces of infrastructure, including Debian packages, have
the same basic symlink packing issue and workaround.

- Daniel

Post by Michal Privoznik
Dear list,
I'm a libvirt devel and I've ran into interesting problem. I'd like to
hear your opinions on it.
Libvirt is a virtualization library that uses XML to store a virtual
machine config. We have couple of tests in our repository too that check
whether XML configs are valid, and whether some operations change it in
expected way. Some operations don't change the XML at all, in which case
ln -s $xml.in $xml.out
However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved. I've traced down
the problem and found that autoconf is just hardcoding some of tar's
options. Namely -chf. Yes, it is -h that causes a symlink to be
dereferenced.
So my question is, what do you think of making -h configurable? We could
add new tar-* option to AM_INIT_AUTOMAKE, say tar-symlinks, which would
suppress -h on tar's command line.
What are your thoughts?
Michal