How to Smoothly Release Split Repositories

Mark Scherer
Mark Scherer Senior Software Developer & Release Manager
13. March 2018 in

Technology

Following the challenges of major releases and how we tackle them, this article dives into the details of the Spryker Commerce OS release process and our Atomic Releases.

Do you also have a larger modular code base - or are you planning to move your monolithic structure into one? The following will be a compact guide to achieve exactly that. Plus, it will enable your code base to release your packages atomically.


How Do We Manage 300+ Modules?

We all know how long it takes, even with enabled cache, to pull hundreds of composer dependencies. With stable tags this is still way faster than using dev-master repository branches. However, ensuring speed when pulling composer dependencies is only one of the problems that becomes visible with such a high number of dependencies that are declared in the composer.json. Imagine you want to switch branches or rebase on top of someone else’s commits.


Larger changes on so many separate dependencies would also entail a large number of pull requests (PRs) to be merged and released. This leaves us at only one conclusion: it is not feasible.

Our customers will usually only use a small subset of all available modules, but in Spryker core development, naturally all modules must be considered. Our solution is using the Git subtree-split approach and working with a non-split repository for our core developmentThis turns all core changes into a single PR and makes the internal development a lot easier and faster. On core side, we have an internal repository for our demoshop that requires our core non-split repository. It contains the following code in its composer.json:

"require": {

"spryker/spryker": "dev-master"

}

 

This way, all core modules appear in the ‘vendor’ directory as one single repository. We can easily batch-add or -modify several modules at once when we release new features.

 

Kicking off the Release with Review and Merge

Once code changes are reviewed as pull request against master branch, we prepare the release. This involves defining the changed modules and their target semantic versions (patch, minor, major) using our internal release-app.


Here is one example of how the release overview can look like for a specific feature:

As you can see we also make sure, that any changed dependencies between the modules are outlined. If, for example, the CartVariant can only work based on the new Product minor, the tool will know that the modules’ composer.json will need an update like this:

"require": {

...
"spryker/product": "^6.1.0"

},

Firstly, we implemented this part of the release process as validation tooling. As a second step, we slowly automated with more tooling. Find out why tooling is so important further down the lines of this article. When the release is finally approved, the main core PR along with the demoshop code are merged and the composer.json is updated.

 

Doing the Split

Now the magic of our split script happens, which takes the merged non-split core repository, and splits each “module” and all new commits into the split repositories. In our case “spryker/cart-variant”, “spryker/product” and a new “spryker/product-checkout-connector” will have the new commits pushed into their master branch. The script runs in the background, as cronjob, and takes up to 1 minute for all modules and their commits to be placed into their split repositories.
While keeping a lookout for any new commits or branches in the non-split repository, the script creates a cache folder and applies any necessary changes in the split part.

In order to only allow master branches to be published, we have a special naming convention in place. The only exceptions are beta/ and hotfix/ prefixed branches, as they are needed on the split side to verify proper behavior prior to the actual release.

 

One-Click Core Releases

On the release-app, a monitor checks if all splits have been properly updated and are ready for release. It also ensures that our Continuous Integration system reports are green for current master to give the go for continuing the release process. At this point, the release itself is just a matter of a button click. Based on the above template, a “github release” is triggered on the split repository, which then tags it with the new expected version and adds an automated release note into it as content. At the same time, initiating the release process will also create an internal release summary for publication via email and in the Spryker Academy

We rely on Packagist for the listing of all our released modules. Each tag we set triggers an update on Packagist and makes the new module available for everyone as composer package.

Spryker_Release-App

Going Public

Remember I mentioned that we use a private modified Demoshop to require the non-split repository? The next step is to merge the changes in the public Demoshop. As final action point, the changes from the non-split demoshop are merged into the public one. We then update all split core modules that have been released. 

The release-app supports this process by generating a list of commands to execute. In our case composer require "spryker/product-checkout-connector":"^1.0.0" has to be executed as a command line call. 
The demoshop’s composer.json contains the above connector release as new dependency:

"require": {

...
"spryker/cart-variant": "^2.0.1",
"spryker/product": "^6.0.0",
"spryker/product-checkout-connector": "^1.0.0"
...

}

The other two modules - since we follow semantic versioning (semver) - do not actually need a composer.json update. A simple composer update "spryker/*" will automatically update the composer.lock file here and make sure those new module versions are included.
At this point, the new functionality in the public demoshop can be verified and completes the split release.

I hope you are all now clear on why the release management tooling mentioned earlier is so important. The modules’s composer.json files had no functional use up to this point. The release-app ensures that those files are valid and complete prior to the release and for tagging the splits. Any mistakes require us to make “patch release” fixes at this point in time, which trigger a completely new release-cycle.

 

Hotfix Releases

One use case that cannot be based on the above approach, is hotfixing older releases. In this case, we need to work on a per repository level as we need to verify the functionality based on newer releases of dependencies directly in the public demoshop.
But fortunately, this is a rare use case.

Our Learnings from Developing This Split Process

Subtree-Splitting enables single to multi repo releases and a slim workflow for hundreds of Git repositories while allowing flexible usage and updating on the split side.
Adding automation and validation tooling are essential, though. They reduce human error, remove repetitive tasks and speed up the process further.



Still got questions?

If you have any questions or would like to discuss your release process, stop by our forum.

Fancy something more personal? Secure a free consultation and find out more about improving the release process in your business today.

Get in Touch

 



mail-iconDon't miss out on any industry news!

Sign up for our newsletter

Still got questions?
Ask the author for further information.