BrettW's Tech Tips

My SCons build environment

Developers like creating the One True Build system that has all the bells and whistles that you might want. I’m not going to claim I have that, but I’ve made a system that I find quite nifty. It was pieced together from SCons tutorials on the web and my own trial-and-error. Hopefully you can make use of it.

Overview

I use this for my game The Day After and it’s suited to game development. The goals for this build environment is:

  • Debug, Release and Unit Test builds
  • Versioned builds
  • Doxygen documentation build
  • Building extra tools
  • Unit testing
  • Separate build directories (based on build type, version and platform)
  • Nice directory structure
  • Recursive builds so that you can turn on/off things easily.

Warning: this code is fairly bare-bones and hacky. You could potentially write it with some nicer Python and structuring.

Our directory structure

Our project will look like this:

  • PROJECT HOME
    src/
    The source code for the main project

    engine/
    The engine code
    game/
    The game code
    build/
    Where all the build artifacts (object files and executables) go

    (platform)
    The platform for this build

    (version)
    The build artifacts for that version

    debug/
    Debug objects
    release/
    Release objects
    test/
    Test objects
    current/
    Copies of the executables to mimic released builds
    data/
    Game data
    doc/
    Documentation
    etc/
    External dependencies (their includes, documentation and everything else)
    lib/
    Actual library modules for compilation

    debug/
    Debug library modules
    release/
    Release library modules
    test/
    Unit-test source code
    tools/
    Extra tools source code
    SConstruct
    The primary SCons file
    build_support.py
    Helper functions for the build
    thedayafter.doxyfile
    The Doxygen config file for the documentation build

Under build/(platform)/(version)/debug/ the directory structure mirrors that in src/ (and the same for release/). Similarly, build/(platform)/(version)/test/ mirrors test/.

The lib/ directory splits the libraries into debug/ and release/ in case the library provider doesn’t use filenames to distinguish the builds. It also keeps you honest.

I also use the doxygen.py that people have made for SCons.

Basic versioning

We use a very simple major, minor, build number setup. For example, 1.1.234 being version 1.1, build 234. We also allow for codenames.

# Version
major = 1
minor = 0
build = 0
codename = ''

# Create the version number
versionNum = '.'.join( str(i) for i in [ major, minor, build])

# Create the version name (which may contain the codename)
if codename:
    version = versionNum + ' (' + codename + ')'
else:
    version = versionNum

Nothing complicated going on here. You could alternatively grab these values from a core header file, but I like the simplicity of this.

Standard variables

Across all builds we want to specify some global variables. This is mostly include directories and our directory structure.

etc/Ogre-1.9/include/OIS/" )
boostIncludeDir = Dir( "etc/boost-1.54/")
ceguiIncludeDir = Dir( "etc/plugins/include/")

includePath = [ boostIncludeDir, ogreIncludeDir, oisIncludeDir, ceguiIncludeDir, sourceDir ]
baseTargetName = 'theDayAfter'

# Set up a basic environment
basicEnv = Environment( ENV = {'PATH' : os.environ['PATH'], 'TMP' : os.environ['TMP'] }, TARGET_ARCH="x86"  )
basicEnv.Append( CPPPATH = includePath )

Build helpers

We define a few tiny functions just to make some of our paths and version strings uniform. We put these in build_support.py in the same directory.

import os

# Write the build directory path
# eg. build/win32/v1.0.0/debug/

def getBuildDir( build, build_type, platform, version ):

    return os.path.join( build, platform, version, build_type )

# Write the build target name
# eg on Windows: theDayAfter-1.0.0-dbg.exe

def getTargetName( base, build_type, version ):

    if build_type == '':
        return '-'.join( [base, version])
    else:
        return '-'.join( [base, version, build_type] )

Customize these however you like.

Debug environment

For each build type we create a SCons build environment and set a bunch of related variables. This is done in a uniform way so that the Debug environment is almost identical in structure to the Release one. When we recursively wander the source tree, it’ll assume this basic structure and not have to deal with Debug/Release peculiarities.

Our debug environment is as below:

###############################
# Debug environment
debugBuildDir = getBuildDir( buildDir.path , 'debug', sys.platform, versionNum )
debugBuildTarget = getTargetName( baseTargetName, 'dbg', versionNum )

debugDefines = ['DEBUG']
debugFlags = [ '-Zi', '-EHsc', '/MDd' ]
debugLinkerFlags = [ '-debug']

# Libraries
debugLibs = ['OgreMain_d', 'OIS_d', 'CEGUIBase-9999_d', 'CEGUIOgreRenderer-9999_d' ]
debugLibDirs = [ Dir('lib/debug/') ]

# Platform-specific stuff
if sys.platform == "win32":
    debugLibs.append( 'user32' )

debugEnv = basicEnv.Clone( CPPDEFINES = debugDefines, CCFLAGS = debugFlags )

debugEnv.VariantDir( debugBuildDir, sourceDir, duplicate=0 )
(objList, mainObj) = debugEnv.SConscript( os.path.join(debugBuildDir, 'SConscript'), duplicate=0, exports={'env' : debugEnv } )

debug = debugEnv.Program( target = os.path.join( debugBuildDir, debugBuildTarget + '.exe'), source = objList, LIBS = debugLibs, LIBPATH = debugLibDirs, LINKFLAGS = debugLinkerFlags )

debugEnv.Install('current', debug)

Alias('debug', debug)
Default(debug)

This is currently set for the Microsoft Visual C++ compiler, but you can easily integrate a switch between compilers based on platform. Notice that the platform is determined by Python’s sys.platform. If you want to do cross-platform compiling, you need to break that check out into a function that allows you to override the platform if you want.

The debugEnv is a clone of the basicEnv one so you inherit the universal settings like include directories. I overwrite the CPPDEFINES and CCFLAGS with the debug versions, but you might want to append them.

The next bit is a tricky bit that took me forever to get right, despite its simplicity.

debugEnv.VariantDir( debugBuildDir, sourceDir, duplicate=0 )

To get a separate build directory from the source directory we use VariantDir. We specify the build directory (which uses getBuildDir() which incorporates the debug settings), the universal source directory and specify to not duplicate compiled objects in the source directory.

In short, this compiles the objects and puts them in the build directory in a directory structure that mimicks the source directory structure.

The next line:

(objList, mainObj) = debugEnv.SConscript( debugBuildDir + '/SConscript', duplicate=0, exports={'env' : debugEnv } )

Our recursive build works like this:

  1. Get a list of child directories.
  2. Get a list of objects from this specific directory. Add them to the current list of objects (objList)with the path clearly specified.
  3. For each child, move to that directory and recurse from Step 1, appending any objects given by the children.
  4. When we’re done, return the list of objects.

Each child directory has a SConscript file with a very simple structure.

The mainObj just gives us a handle on the main object, namely thedayafter.cpp where our int main() lives.

The final important bit is this:

debug = debugEnv.Program( target = os.path.join( debugBuildDir, debugBuildTarget + '.exe'), source = objList, LIBS = debugLibs, LIBPATH = debugLibDirs, LINKFLAGS = debugLinkerFlags )

This sets the debug artifact (theDayAfter-1.0.0-dbg.exe) under the right directory (build/win32/1.0.0/debug/). It uses the objList we built previously, and the debug settings for libraries and linker flags.

It then installs it into current/ for convenient use/testing.

The Alias allows us to build just the debug build if we want. But by default we build everything.

Release and test builds

The Release build is almost identical. Wherever we have “debug” in a variable name, we put “release”. Flags are changed to the appropriate things.

This means that release re-walks the source directory to pick up objects for objList. This is okay because we might want to do something different for the release build (like not include developer objects).

The test build is interesting – the variables are set up in the same way as debug and release. However we use the same objList from release without recompiling. But since we know that we want our own test front-end, we knock out mainObj:

testEnv = basicEnv.Clone(CPPDEFINES=testDefines, CCFLAGS=testFlags)

testEnv.VariantDir(testBuildDir, testSourceDir, duplicate=0)

# Knock out main() from release
objList.remove( mainObj )

We then walk the test/ directory to pick up all our unit-test code:

objList += testEnv.SConscript( os.path.join( testBuildDir, 'SConscript'), duplicate=0, exports={'env' : testEnv })

We then continue exactly as in the debug setup, but with debug replaced with test everywhere.

If you want to actually run the unit-test as part of the build, create an empty environment and force running the test executable:

unitTestEnv = basicEnv.Clone( )

#unitAlias = unitTestEnv.Alias('unittest', [test], test[0].path )
unitAlias = unitTestEnv.Alias('unittest', ['./current/' + testBuildTarget + '.exe'] )
AlwaysBuild(unitAlias)

You can also do this whole pattern for the game dev tools you have.

Documentation

Just quickly before we descend into the source directory, we build documentation using Doxygen:

###############################
# Documentation
docEnv = Environment( tools = ["doxygen"], toolpath = '.', DOXYGEN='"C:/Program Files (x86)/doxygen/bin/doxygen"' )
doc = docEnv.Doxygen("theDayAfter.doxyfile")

Alias('doc', doc)
AlwaysBuild(doc)
Default(doc)

This is a little hacky and doesn’t properly accommodate compiles on other platforms, but it should be clear what you need to change (just the DOXYGEN variable).

SConscript files

At each directory branch in our source tree we have a SConscript file.

Here’s the one in src/

# Import all environment variables
Import( 'env' )

# Subdirectories to descend to next
subdirs = ['engine', 'game']

# Source files in this directory to compile
sourceFiles = [ File('thedayafter.cpp') ]

# Create the mainObj for distinction
mainObj = env.Object( File('thedayafter.cpp') )

# Descend into lower subdirectories, recursively picking up objects
# The src_dir is very important to make sure the VariantDir build directory stuff works fine. As is duplicate=0.
# The exports makes sure the environment propagates downwards
for d in subdirs:
    sourceFiles += env.SConscript( dirs = d, src_dir = Dir('.').srcnode().path, exports=['env'], duplicate=0 )

# Create the base objList
objList = []

# For each source file, specify the corresponding compile Object using the current environment
for i in sourceFiles:
    objList.append( env.Object( i ) )

# Return the object list and the mainObj into the level above    
Return( 'objList', 'mainObj' )

Every directory is pretty much the same. Keep the same code, but just specify subdirs and sourceFiles. It’s vital to use File(sourcefile) to make sure the VariantDir works correctly.

Notice that the build environment is propagated downwards, so the debug/release specifications need only be made up in the SConstruct file. You could get lower levels to inject dependencies like libraries and include directories, but there’s a world of pain awaiting that approach. Just set all of them at top-level for simplicity. It also stops you going insane if you change libraries and don’t know why the old one keeps getting linked in.

As examples of lower branches, this is what is in the engine/SConscript:

Import('env')

subdirs = [ 'app', 'log', 'event' , 'text', 'options',  'random', 'utility', 'gamestate', 'graphics', 'gui', 'fileio', 'stats' ]

sourceFiles = []

for d in subdirs:
    sourceFiles += env.SConscript( dirs = d, src_dir = Dir('.').srcnode().path, exports=['env'], duplicate=0 )

Return('sourceFiles')

At a leaf node like src/engine/app/ the SConscript file looks simply like:

Import('env')

sourceFiles = [ File('GameApp.cpp') ]

Return('sourceFiles')

Wrap-up

So there you go. That’s how I do my build environment in SCons. It works quite nicely and is easy to tweak and extend without re-engineering the whole environment.


Posted on: 6:30 pm on 16 Jan 2014
Filed under: Programming
Tags:

No Comments »

Leave a Reply