Nested (recursive) submodules in Git

Git submodules

Git submodules are the way to implement dependencies between projects.

Let's say you have a common libray 'mylib' shared between apps 'myapp1' and 'myapp2' then you will keep only 3 repositories (1 for each of them).

Both 'myapp1' and 'myapp2' will have 'mylib' declared as a submodule, which is cloned from / pushed to its own repository. And the repository of each parent app will only keep a reference to the commit to use from the submodule repository.

Nested Git submodules

Extending the same principle, 'mylib' could as well be dependant on 'somebaselib', declared as a submodule and hosted elsewhere.

There's plenty of documentation about how to clone or update such repositories (once they exist), using

   git clone ... --recursive
   git submodule update --recursive

But how to build one of those is not so intuitive.

Adding nested submodules: the problem

You may have used this procedure to add a submodule into the 'mylib' repository:

   cd ~/mylib
   git submodule add git@someurl/somebaselib baselib   

As the baselib folder does not exist, the previous command does a clone of the specified repository into that folder, and then adds it as a submodule.
You may think the same would to add the nested submodules to the superproject:

   cd ~/myapp
   git submodule add git@someurl/mylib lib

but unfortunatelly this does not work... because the cloning fails.

Adding nested submodules: the solution

The workaround is to manually clone the nested submodules into the subfolder with the --recursive option, before doing the 'submodule add'

   cd ~/myapp
   git clone git@someurl/mylib lib --recursive
   git submodule add git@someurl/mylib lib

Now, as the lib folder does exist, the 'submodule add' command does not try to clone it and just adds it as a submodule. You are all set!

Git labeller, branches and dynamic values

After some use of the Git labeller for CCNET described in my previous post, we were faced with an unexpected problem.

On a project following the basic workflow described here, we had the following scenario:

• At some point of time, release tag 'v1.1.200.0' was placed on 'qa' branch and that version went into production
• Later, while development continued on the 'master' and 'qa' branches, a fix was needed for the previous release, so we've created a branch 'release1.1.200' from the last release tag, commited the fixes into it, and placed a new tag 'v1.1.200.1' on the last commit of the release branch - pushing both the commits and the tags to the origin.
• On the CCNET automated build, when getting the code using the Git source control plugin, we specified 'release1.1.200' as the branch to use, and expected to get '1.1.200.1' as the CcNetLabel

But instead of that, we kept getting a '1.1.200.0' label  ¿?

By following the CCNET server log, we've found out that the sequence in which the Git source control plugin works is:

1. repository is cloned or updated
2. labeller is invoked
3. branch specified in CCNET configuration is checked out (!)

So when the labeller tries to get the last tag by invoking git describe, it receives something like 1.1.200.0-xxx where xxx specify the commits present after that tag.

Once the problem was clear, the solution was to add to the labeller configuration an (optional) element specifying a branch to ckeckout, and to do so within the labeler before issuing git describe.

As a final touch, we've also upgraded the labeller to support dynamic parameters, so the name of the release branch could be specified dynamically when launching the build. The updated labeller is available at its Gitub project .

Git tag labeller plugin for CCNET

When implementing the basic Git workflow described in my previous post, we needed a way to read the last version tag and other related info and pack it as the "build label" within CruiseControl.Net, the integration platform we use.


More precisely, we needed a valid version identifier to be built in two different scenarios:

• Release builds: take the release tag from Git (i.e. "v1.0.4.0"), skip the "v" prefix and return the rest as the version ID.
• QA builds: we do not tag each QA build, so we need to take into account the last tag and the number of commits ahead that, then merge them into something like "1.0.4.105" meaning "5 commits ahead of release 1.0.4.0".  Why 105 and not 5?  Well, we are reserving a block of  numbers (1.0.4.1 - 1.0.4.100) to accommodate eventual  production fixes on a dedicated release branch.

Enter Git tag labeller plugin for CCNet - we built a plugin to be used as a block in a CCNet configuration script and posted it on GitHub, you can find it here.

Besides the functionality we needed, we added a couple of features some one else may use, like a simple concatenation of the existing git tag with the commits ahead count; i.e. "ReleaseCandidate.6".

See the GitHub project for the most up-to-date documentation and code.

Get Going with a Minimalistic Git Workflow

Yeah, I know. The workflow in Git, as in any DVCS, is something to be taken seriously. Once your whole team is fluid in branching, merging and rebasing you are ready for a full blown Git workflow like git-flow, or the GitHub flow described by Scott Chacon.

But what if you project is small, and / or you've just migrated from SVN and are still trying to figure it all out, and / or you just don't want a model so complicated?

What would be a good starting point? If you need to, lets say:

1. commit or merge code into a main branch with CI
2. regularly set milestones for deployment in a QA or staging server
3. from time to time release a production version - that's what you are paid for, after all
4. do some maintenance and bug fixing on the released code.

After some research and some practice we choosed the following minimalistic workflow to solve those four needs.


1. The master branch is the place where all developers integrate their code.

This may be done:

• By committing their code directly into the master branch - when working in small teams or for small incremental changes.
• On a separate feature or develop branch that is later merged into master - when the circumstances require it.

CI tests should be running on this branch.


2. A fast-forward-only qa branch acts as a "sliding tag" following closely the master head.

This allows you to set temporary marks that will be used for deploying in a QA  or  staging environment. Some of them will later turn into releases - see the following section.


3. A series of annotated version tags signal each release that will go into production.

The version tag for a release (i.e. v1.0.2.0) is always placed on the commit at the current head of the qa branch (not from the master branch, which may be ahead of the last quality assured build).

Note that in this model, only an annotated tag is initially required for deploying a release into production. Only later, if required, a dedicated branch will be created from this tag.


4. Dedicated branches are created for each release that requires maintenance or fixes.

On early stages of an application development , releases may occur quite frequently and most of the times they will not be updated with fixes - the fix will be included in the next full release from the master / qa  branches.
In these cases, the version tag mentioned in section 3 will be enough.

However, when the app or site is in production and releases tend to be more sporadic, the requirements for fixes on the released version are quite common. On the first occasion that a fix is required on a newly created version - initially defined only by a tag - a dedicated branch for that specific release should be created (i.e. release1.0.2)

Each fix release for this major version will be created on the same dedicated branch, and will get a separate tag (i.e. v1.0.2.1, v1.0.2.2, ...)

Changes on the release branch may be merged back - as a whole or cherry picked - into the master tag; never into the qa branch that is following the master in a fast-forward way.

The next full release (i.e. v1.0.3.0) will be created as a version tag on the main qa branch, and later branched from there, if necessary.

Credits

The fast-forward qa branch and release tag concepts are well explained in the two last sections of this post from Rein Henrichs.

My friend Benjamin Eidelman was a key participant in the discussion and contributed many of the basic ideas behind this post.