What is MakeBuilder?

MakeBuilder is a convenient frontend to GNU Make with lots of cool features. Basically, it processes make.xml files located anywhere in the source tree and creates a Makefile.

It analyzes the source code and automatically determines the dependencies among targets (libraries and programs). Based on the dependency graph, it outputs which targets it can (not) build and creates appropriate commands for compiling and linking.

Furthermore, it is easily extensible for custom build steps.

Why another make frontend?

We would really like to rely more on a standard solution. However, we are not aware of a build tool that natively supports the important features we need. Customizations for another build tool we used to have (SCons) were more complex than the whole of MakeBuilder – and did not work that well. Notably, the build tool customizations in various other frameworks are also more complex (MakeBuilder is around 2000 LOC). Certain conventions in our source code allow for features and optimizations that would be rather difficult to achieve with a generic make frontend.

There are certainly a lot of things that can still be improved (smaller generated Makefiles, lookup of external libraries, packaging as extensible standalone tool, better name ). This happens from time to time. Nevertheless, we would argue that MakeBuilder is already pretty advanced in some areas.

Simple Usage

Call make. For instance,

make -j3 ravon

Possible targets

Possible targets are listed at the beginning of the Makefile.

• (e.g. finroc_plugins_structure) will build all the targets in this repository and all of its dependencies.
• will build a project and all of its dependencies.
• will build a library and all of its dependencies.
• .so will compile a single .so file and all of its dependencies.
• -bin will compile a single binary and all of its dependencies.
• clean removes all build directories.
• clean- removes build artifacts of specified target only
• build- builds that target without generating a new Makefile

make.xml files

The make.xml file format was originally inspired by Apache Ant, a convenient build tool for Java that we quite like. Nowadays, however, make.xml files only contain a set of targets (programs and libraries). Basically, they define which source files belong to which target and which external libraries are required.

Syntax is rather straightforward. Wildcard support is similar to ant: * can be any set of characters excluding /. ** can be anything.

Example:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<targets>
  <library libs="pcl ogre">
    <sources exclude="tests/**">
      **.cpp
    </sources>
  </library>

  <!-- executed as unit test; by convention unit tests are located in a subdirectory 'tests' -->
  <program name="transformations">
    <sources>
      tests/unit_test_transformations.cpp
    </sources>
  </program>
</targets>

Several formats are allowed.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <!-- example: complex target -->
 <program name="complex_target"
             libs="qt4"
             optionallibs="openmp"
             cxxflags="-fpermissive -I/usr/local/cuda/include -DSOME_DEFINE"
             cflags="-I/usr/local/cuda/include -DSOME_DEFINE"
             ldflags="-Llibraries/dialog_system -lamiasr"
             sources="**/t*.c* ~*.ui **.cu tDescriptions.h"
             exclude="test/**"/>

 <!-- example: same complex target with different formatting -->
 <program name="complex_target">
   <sources exclude="test/**">
     **/t*.c*
     *.ui
     **.cu
     tDescriptions.h
   </sources>
   <libs>
     qt4
   </libs>
   <optionallibs>
     openmp
   </optionallibs>
   <cxxflags>
     -fpermissive
     -I/usr/local/cuda/include
     -DSOME_DEFINE
   </cxxflags>
   <cflags>
     -I/usr/local/cuda/include
     -DSOME_DEFINE
   </cflags>
   <ldflags>
     -Llibraries/dialog_system
     -lamiasr
   </ldflags>
 </program>

</targets>

Compile Options

At the very beginning of the Makefile, several variables are defined. Some of them are intended to possibly be changed in a make call.

• CFLAGS (Extra) flags/options for the C/C++ compiler
• CC Allows specifying the C compiler. Default is gcc.
• CCFLAGS Flags for the C compiler only - Replaces CFLAGS.
• CXX Allows specifying the C++ compiler. Default is g++.
• CXXFLAGS Flags for the C++ compilter only - Replaces CFLAGS.
• LDFLAGS Additional options to pass to the linker
• GCC_VERSION Allows specifying the gcc version to use - e.g. -4.8.3 (string is appended to gcc and g++)
• NVCCFLAGS and NVCC are the same for the nVidia CUDA compiler nvcc.

A make call could look like this:

make -j4 GCC_VERSION=-4.8.3 CFLAGS='-O3 -D NDEBUG' ravon

Target Configuration Files

The compile options are typically set in a target configuration file. These files can be used to set (some of) the above options for a specific target. Several of them are included with Finroc. They can be found in the etc/targets directory and have the name of the target (e.g. linux_i686_release). They have a simple format (and are actually included from the Makefile). This, for example, is the content of the configuration file for the linux_i686_release target:

CFLAGS=-O3 -D NDEBUG -D RRLIB_LOGGING_LESS_OUTPUT -ftrack-macro-expansion=0
CXXFLAGS=$(CFLAGS) -std=c++11
NVCC_FLAGS=-arch=sm_20

External Library 'Database' and Library Checks

Finroc libraries and projects depend on various external libraries. Dependencies to those libraries are specified in make.xml via libs="library1 library2".

If information on these libraries is available via pkg-config, you can just use the name they have there.

Information about all other libraries is stored in make_builder/etc/libdb.raw. There is one line per library. Syntax is:

<library name in make.xml>: <raw compiler flags for this library>

The raw compiler flags are almost the flags that will be passed to the compiler whenever the library is used. The library check will attempt to locate the required include and library files - and possibly adds some paths. It transforms the system-independent libdb.raw to the system-specific libdb.txt. The latter will contain exactly the options/flags that will be passed to the compiler. During the transformation: * The libraries (-l...) will be located - and possibly some path (-L...) added. * The includes (-I<comma-separated list of includes that need be present in single directory>) will be located - and possibly some path (-I...) added. * If includes or libraries cannot be found, a N/A is placed in the libdb.txt.

Whenever a new library is required, an approriate line should be added to the libdb.raw.

Whenever the libdb.raw changes - a new library check will be initiated with the next make call. However, the system libraries themselves might change. This is not detected by make. In this case, make libdb needs to be executed manually.

Searching for relevant files is done in the libdb.search script. It outputs all potentially relevant files (absolute paths) to stdout. That means that updatelibdb will only find files (headers, libraries) etc. that are output of this script. It executes libdb.search.default and - if existent - libdb.search.local and /etc/make_builder/libdb.search (any local, system-specific additions should be made to these two; the latter is interesting to make system-wide modifications to the search paths). The libdb.search.default contains find commands that can be simply modified and used in other scripts. However, any command that outputs existing files with full paths may be added.

It is possible to have multiple lines in the libdb.raw for the same library - an example is the soqt library. In this case, searches for both variants of the library are performed separately. This can be used to support different variants of a library. If both variants are found, the first one is selected.

Sometimes, it can be necessary to make modifications to the libdb.raw for a local system only. To avoid problems when updating/committing with mercurial, a local libdb.raw.local can be created. Entries in the local version override entries in libdb.raw. The libdb.raw.local is also the preferred way, to select a specific version of a library when there are multiple installations - simply by hard-coding the paths.

Cross-compiling

To cross-compile Finroc for another platform, a target configuration file (see above) for this platform needs to be added to Finroc's etc/targets directory - called cross_<OS>_<architecture>_<mode>. Targets for e.g. the Raspberry Pi are included in Finroc - such as cross_linux-raspbian_armv6l_release:

PKG_CONFIG_EXTRA_PATH=/usr/lib/arm-linux-gnueabihf/pkgconfig
CFLAGS=-ggdb -ftrack-macro-expansion=0
LDFLAGS=-lrt
CXXFLAGS=$(CFLAGS) -std=c++11

CC=arm-linux-gnueabihf-gcc --sysroot=$(FINROC_CROSS_ROOT)
CXX=arm-linux-gnueabihf-g++ --sysroot=$(FINROC_CROSS_ROOT)
LD=arm-linux-gnueabihf-ld --sysroot=$(FINROC_CROSS_ROOT)

Typically, alternative compilers with a different system root directory (of the cross-compile environment) are specified. As the installation path of cross-compile environments may vary, they are specified via source scripts/setenv. To select the Raspberry Pi target above, you would type:

source scripts/setenv -o linux-raspbian -a armv6l -m release -c /path/to/raspbian/system/root

MakeBuilder Development

Concept of the MakeBuilder Tool

The MakeBuilder tool is based on an abstract concept: There's a set of BuildFileLoaders and a chain of SourceFileHandlers. There are some general ones and some Finroc-specific ones.

An arbitrary number of BuildFileLoaders parses build files (make.xml by default) and creates BuildEntity objects. Then, a chain of SourceFileHandlers transforms these objects. In this process, targets are added to the Makefile. A SourceFileHandler typically handles a certain class of source files.

This architecture allows to add or remove specific functionality cleanly.

The Finroc build process is currently made up of the following Loaders and Handlers.

Loader

• SConscriptParser (to handle some old MCA2 projects)
• MakeXMLLoader

Handlers

• Qt4Handler (responsible for calling Qt uic und moc)
• NvccHandler (for .cu source files)
• DescriptionBuilderHandler (everything related to MCA2 description-builder)
• EnumStringsBuilderHandler (generates string constants for all public enums defined in headers)
• PortDescriptionBuilderHandler (generates names for ports of Finroc components)
• PkgConfigFileHandler (generates pkg-config files for all libraries)
• FinrocSystemLibLoader (Handles any system-installed libraries)
• CppHandler (compiles and links C/C++ code)
• JavaHandler (compiles Java code)
• ScriptHandler (generates start scripts - used for Java targets)