Integrating flexc++(1) and bisonc++(1) into Makefile.am

[Kip Warner]

Hey list,

I'm aware autoconf has M4 macros for traditional GNU flex(1) and
bison(1) implementations of POSIX lex(1) and yacc(1) respectively. I'm
also aware the manual contains information for integrating the two into
a typical autotool'd build environment.

My challenge is replicating their functionality for flexc++(1) and
bisonc++(1) in the absense of macros to make their usage easier in
Automake. These are C++ generators for a lexer and parser, analogous to
their C counterparts. Some of the available tutorials online contradict
some of the recommendations of the Autoconf / Automake documentation,
so I figured I'd ask the gurus.

What I am trying to accomplish is have Makefile.am run the two tools on
their two respective inputs and generate their source as part of the
build process of the rest of my project. This is normally done
externally like so:

$ flexc++ Lexer.lpp
(produces LexerBase.h, Lexer.h, Lexer.ih, and LexerBase.cpp)

$ bisonc++ Parser.ypp
(produces ParserBase.h, Parser.h, Parser.ih, and Parser.cpp)

$ g++ Parser.cpp LexerBase.cpp Main.cpp -o MyProgram

All of the generated files I consider derived for the purpose of my
project and I don't mind if the 'clean' target removes them. I figured
as I refine my grammar rules, the lexer and parser code will change a
lot anyways.

In trying to integrate the two tools into my build environment, I've
attempted the following in Makefile.am:

...

BUILT_SOURCES = \
Source/LexerBase.h \
Source/Lexer.h \
Source/Lexer.ih \
Source/LexerBase.cpp \
Source/ParserBase.h \
Source/Parser.h \
Source/Parser.ih \
Source/Parser.cpp

myprogram_SOURCES = \
... \
Source/LexerBase.cpp \
Source/Parser.cpp

...

EXTRA_DIST = \
... \
Source/Lexer.lpp \
Source/Parser.ypp

...

# Generate lexer source from regular expression token descriptions via flexc++...
Source/LexerBase.h:
Source/LogicLexer.h:
Source/LogicLexer.ih:
Source/LexerBase.cpp: Source/LogicLexer.lpp
$(FLEXCPP) --target-directory=$(top_builddir)/Source $< ; \
$(SED) -i '/#include "Lexer.ih"/a #include "ParserBase.h" \/\/ Token declarations...' $(top_builddir)/Source/LexerBase.cpp

# Generate parser source from Backus-Naur grammar rules via bisonc++...
Source/ParserBase.h:
Source/Parser.h:
Source/Parser.ih:
Source/Parser.cpp: Source/Parser.ypp
$(BISONCPP) --target-directory=$(top_builddir)/Source $<

FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.

This all works ok, but I suspect this is not an elegant solution and
there are some very good suggestions from this mailing list.

Yours truly,

[Nick Bowler]

Hello,

Post by Kip Warner
My challenge is replicating their functionality for flexc++(1) and
bisonc++(1) in the absense of macros to make their usage easier in
Automake

[...]

Post by Kip Warner
In trying to integrate the two tools into my build environment, I've

[...]

Post by Kip Warner
BUILT_SOURCES = \

[...]

Post by Kip Warner
Source/ParserBase.h \
Source/Parser.h \
Source/Parser.ih \
Source/Parser.cpp
myprogram_SOURCES = \

[...]

Post by Kip Warner
Source/Parser.cpp

[...]

Post by Kip Warner
# Generate parser source from Backus-Naur grammar rules via bisonc++...
Source/Parser.cpp: Source/Parser.ypp
$(BISONCPP) --target-directory=$(top_builddir)/Source $<
FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.
This all works ok, but I suspect this is not an elegant solution and
there are some very good suggestions from this mailing list.

There aren't really any "elegant" solutions. Make handles this kind of
tool quite badly. It is possible to get things to work but it is always
a tradeoff between flexibility of your build system and simplicity of
your rules.

If you are happy with this method then it is totally fine. Do make
sure parallel builds work by testing them routinely (both clean and
incrementally) -- I think listing everything in BUILT_SOURCES like you
do probably "resolves" any parallelism problems here, (by reducing
opportunities for parallelism).

The Automake manual has section on writing portable make rules for tools
that produce multiple outputs[1], with a discussion of various approaches
and their limitations. I generally prefer approaches using a dedicated
witness file.

Finally, consider whether you want to distribute the generated parser
sources. That way your users don't need these tools installed just to
build your package.

[1] https://www.gnu.org/software/automake/manual/automake.html#Multiple-Outputs

Cheers,
Nick

[Will Estes]

You might also have a look at the flex test suite to see various ways in which automake is abused to build odd things for flex itself. There's a lot of edge cases there and you might get ideas.

Post by Nick Bowler
Hello,

Post by Kip Warner
My challenge is replicating their functionality for flexc++(1) and
bisonc++(1) in the absense of macros to make their usage easier in
Automake

[...]

Post by Kip Warner
In trying to integrate the two tools into my build environment, I've

[...]

Post by Kip Warner
BUILT_SOURCES = \

[...]

Post by Kip Warner
Source/ParserBase.h \
Source/Parser.h \
Source/Parser.ih \
Source/Parser.cpp
myprogram_SOURCES = \

[...]

Post by Kip Warner
Source/Parser.cpp

[...]

Post by Kip Warner
# Generate parser source from Backus-Naur grammar rules via bisonc++...
Source/Parser.cpp: Source/Parser.ypp
$(BISONCPP) --target-directory=$(top_builddir)/Source $<
FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.
This all works ok, but I suspect this is not an elegant solution and
there are some very good suggestions from this mailing list.

There aren't really any "elegant" solutions. Make handles this kind of
tool quite badly. It is possible to get things to work but it is always
a tradeoff between flexibility of your build system and simplicity of
your rules.
If you are happy with this method then it is totally fine. Do make
sure parallel builds work by testing them routinely (both clean and
incrementally) -- I think listing everything in BUILT_SOURCES like you
do probably "resolves" any parallelism problems here, (by reducing
opportunities for parallelism).
The Automake manual has section on writing portable make rules for tools
that produce multiple outputs[1], with a discussion of various approaches
and their limitations. I generally prefer approaches using a dedicated
witness file.
Finally, consider whether you want to distribute the generated parser
sources. That way your users don't need these tools installed just to
build your package.
[1] https://www.gnu.org/software/automake/manual/automake.html#Multiple-Outputs
Cheers,
Nick

[Kip Warner]

Post by Will Estes
You might also have a look at the flex test suite to see various ways
in which automake is abused to build odd things for flex itself.
There's a lot of edge cases there and you might get ideas.

Great idea. Thanks Will.

[Kip Warner]

Post by Nick Bowler
Hello,

Hey Nick.

Post by Nick Bowler
There aren't really any "elegant" solutions.  Make handles this kind
of tool quite badly.  It is possible to get things to work but it is
always a tradeoff between flexibility of your build system and
simplicity of your rules.

I had suspected something along those lines due to the scarcity of
documentation that's well defined on this particular class of problem,
but feel better knowing my hack might be close to as good as it's going
to get for the time being.

Post by Nick Bowler
If you are happy with this method then it is totally fine.  Do make
sure parallel builds work by testing them routinely (both clean and
incrementally) -- I think listing everything in BUILT_SOURCES like
you do probably "resolves" any parallelism problems here, (by
reducing opportunities for parallelism).

Tested. It appears to work fine. Everything else will presumably be
able to build in parallel, but the flexc++(1) and bisonc++(1) derived
targets will have to be in serial.

Post by Nick Bowler
The Automake manual has section on writing portable make rules for
tools that produce multiple outputs[1], with a discussion of various
approaches and their limitations.  I generally prefer approaches
using a dedicated witness file.

I took a look at this and I think it's a good idea. But I also think
that it may not be worth it if I have to integrate these two tools in
only one place in the build environment, as opposed to with multiple
parsers in multiple sub projects or some such.

Post by Nick Bowler
Finally, consider whether you want to distribute the generated parser
sources.  That way your users don't need these tools installed just
to build your package.

I think that as long as I don't have to actually modify the generated
parser sources, I don't have a problem requiring the user to have the
two tools installed. Most of the compilation I'll be doing, and most of
what remains will probably be on a build server which will
automatically pull the required build dependencies (e.g. sbuild debs).

Thanks a lot for your help, Nick.

[Diab Jerius]

---------- Forwarded message ----------
From: Diab Jerius <***@cfa.harvard.edu>
Date: Wed, Jul 12, 2017 at 10:44 AM
Subject: Re: Integrating flexc++(1) and bisonc++(1) into Makefile.am

Post by Kip Warner
Hey list,
I'm aware autoconf has M4 macros for traditional GNU flex(1) and
bison(1) implementations of POSIX lex(1) and yacc(1) respectively. I'm
also aware the manual contains information for integrating the two into
a typical autotool'd build environment.
My challenge is replicating their functionality for flexc++(1) and
bisonc++(1) in the absense of macros to make their usage easier in
Automake. These are C++ generators for a lexer and parser, analogous to
their C counterparts. Some of the available tutorials online contradict
some of the recommendations of the Autoconf / Automake documentation,
so I figured I'd ask the gurus.
What I am trying to accomplish is have Makefile.am run the two tools on
their two respective inputs and generate their source as part of the
build process of the rest of my project. This is normally done
$ flexc++ Lexer.lpp
(produces LexerBase.h, Lexer.h, Lexer.ih, and LexerBase.cpp)
$ bisonc++ Parser.ypp
(produces ParserBase.h, Parser.h, Parser.ih, and Parser.cpp)
$ g++ Parser.cpp LexerBase.cpp Main.cpp -o MyProgram
All of the generated files I consider derived for the purpose of my
project and I don't mind if the 'clean' target removes them. I figured
as I refine my grammar rules, the lexer and parser code will change a
lot anyways.

Take a look at section 18.2 of the automake manual "Handling new file
extensions"

Essentially you want to tell automake to tell make that you have a
rule to create a .cpp file from a .lpp file.

As an example, here's what I have to compile Lua code

SUFFIXES += .lc .lua
AX_V_LUAC = $(***@AM_V@)
AX_V_LUAC_ = $(***@AM_DEFAULT_V@)
AX_V_LUAC_0 = @echo LUAC $@;
.lua.lc :
$(AX_V_LUAC)test "$(LUA_PATH)set" = set || export
LUA_PATH="$(LUA_PATH)" ;\
$(LUAC) -o $@ $<

It uses the automake silent rules convention to quiet things; see
section 21.3 "How Automake can help in silencing make"

It doesn't work with the automake subdir-objects option; I'm trying to
figure out why.

Post by Kip Warner
In trying to integrate the two tools into my build environment, I've
...
BUILT_SOURCES = \
Source/LexerBase.h \
Source/Lexer.h \
Source/Lexer.ih \
Source/LexerBase.cpp \
Source/ParserBase.h \
Source/Parser.h \
Source/Parser.ih \
Source/Parser.cpp
myprogram_SOURCES = \
... \
Source/LexerBase.cpp \
Source/Parser.cpp
...
EXTRA_DIST = \
... \
Source/Lexer.lpp \
Source/Parser.ypp
...
# Generate lexer source from regular expression token descriptions via flexc++...
Source/LexerBase.cpp: Source/LogicLexer.lpp
$(FLEXCPP) --target-directory=$(top_builddir)/Source $< ; \
$(SED) -i '/#include "Lexer.ih"/a #include "ParserBase.h" \/\/ Token declarations...' $(top_builddir)/Source/LexerBase.cpp
# Generate parser source from Backus-Naur grammar rules via bisonc++...
Source/Parser.cpp: Source/Parser.ypp
$(BISONCPP) --target-directory=$(top_builddir)/Source $<
FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.
This all works ok, but I suspect this is not an elegant solution and
there are some very good suggestions from this mailing list.
Yours truly,
--
Kip Warner | Senior Software Engineer
OpenPGP signed/encrypted mail preferred
http://www.thevertigo.com

[Kip Warner]

Post by Diab Jerius
Take a look at section 18.2 of the automake manual "Handling new file
extensions"
Essentially you want to tell automake to tell make that you have a
rule to create a .cpp file from a .lpp file.

Intuitively this seems to make sense. My issue is that I don't have
hundreds of .lpp files, and only anticipate one for the entire source
tree. Further, there's a little ad hoc magic I've got to do on the only
one in running its generated .cpp through sed to insert some code. If
you have an elegant suggestion for this, I'm all ears.

Post by Diab Jerius
As an example, here's what I have to compile Lua code
SUFFIXES += .lc .lua
        $(AX_V_LUAC)test "$(LUA_PATH)set" = set || export
LUA_PATH="$(LUA_PATH)" ;\
It uses the automake silent rules convention to quiet things; see
section 21.3 "How Automake can help in silencing make"

Excellent idea for extending the silent rules. I was actually going to
ask you guys that next. Now I just wish colorgcc recognized it ;)