How to speed up source automated builds

Often the first answer that people have is "better hardware" and yes it does make a difference but often people are trying to do way too much, or simply don't fully understand what they are trying to build. The following is an alternative view that I believe are a very good starting point before you also look at the hardware.


The short and sweet.

• Keep your source setup (including builds) consistent. Similar to any other code if you have a common repeatable way of doing something it makes it far easier to refactor when things change or you find a quicker way of doing something.
• Keep things simple. Maybe just have it as a straight build for CI and have a different process that runs nightly or on-demand that takes care of packaging and deployment. 
• Understand the dependencies of what you are building. Do you really need to rebuild the whole application or suite of applications for a UI change?

The expanded version.
Consistency is a good thing and keeping things simple makes the job much easier. Something I have seen a number of times with projects is they obviously start out simple and the build times are quick but as the project grows so does the complexity. Then what happens when another project comes along,often the setup is similar but different enough to add an maintenance overhead that often means the build and source infrastructure diverges further. An approach that I have used that covers the the above points and gives you VERY quick builds I will outline below. Overtime I will expand on the various area, or if you interested get in touch and I can talk about the areas you are interested in.

Structure of the buildable unit
First you need to agree on the structure of the build-able unit. What are the common artifacts that these units will have? An example would be as shown below where each of the headings would be represented by a solution folder. Once this is set it is easy to template and script the setup, so that next time you start a new project or split an existing one the infrastructure already exists.


Buildable unit

• Documents
• e.g. documentation relating to this unit such as UML diagram's
• Dependencies 
• build output from other units that this unit requires or where third party assemblies will be referenced from.
• Build
• any build files fore building of this module. I have the TFSBuild.proj and TFSBuild.rsp here
• Binaries 
• build artifacts for this unit will be output here.
• Source
• Source project(s)
• Tests
• Unit test project(s)
• Integration test project(s)

Setup of your buildable units within a branch

So that your not building the whole stack when you only have a simple UI change, or maybe you have certain areas of your code don't change that much, you can look at splitting your source into areas. These areas may still be grouped together, such as libraries or applications but this way you are cutting down on what you are building and you have a better understanding of what the dependencies are in your code. Your units maybe made up of the classifications shown below.

Build-able units

• Applications
• e.g. applications.deployment
• Services
• e.g. services.deployment, services.build
• Libraries
• e.g libraries.build, libraries.deployment, libraries.core

Your (faster) build script

I have found that 95 % of the time you are trying to do the same tasks with your builds, and because of this having as few build scripts as possible is a good thing for maintenance. Previously I have achieved this  

by having one main build script with all the required and any optional targets that are needed and then I import this into each of my modules TFSBuild.proj file. The optionals targets and any other parameters are then simply passed in the modules TFSBuild.rsp file.

The reasons you end out with faster and more consistent builds are

1. You only build what is changing.
2. You have the one build script so if there are any gains to be made in speed then they propagate through to all your modules. 
3. No more tracking down of mystery build issues for one area that no one touch's, or the build is so 'busy' that no one really understands it. They are all the same.

And the nice thing is the solution is scales very well too, you are developing another product in your suite, then add in a new module(s) and all the build things you always want to run already exist.