How to manage common library dependencies to prevent cross-application breakage?

Unfortunately, if you make a number of essentially independent codebases dependent on the always-latest version of a common library, then you get two problems.

One, is that any change to the common library requires far more extensive analysis to determine whether the change is breaking or not towards the dependencies.

Two, the scale and complexity of the above analysis, and the rework of many dependencies that may be entailed all at once, means a relatively conservative approach must be adopted when modifying common libraries.

A traditional justification for common libraries is that improvements to the common code are automatically incorporated into all dependencies, but this is a fallacy.

Many (not all) improvements require a global/whole-system readjustment of programming, rather than the changes being local to the common library and the benefits propagating automatically. Determining whether and where such sympathetic changes are required across the system, is often the majority of the effort - the changes themselves may be relatively trivial to apply.

And regressions, or faults due to missing-but-necessary readjustments, propagate just as automatically as certain improvements might.

Sometimes copy-and-paste incorporation of common code, actually contributes to better management overall, precisely because divergence is the default and it avoids the entire system of dependencies having to be analysed at once for every change.

However it is often not clear in advance which approach will be the better one, and neither approach will suit all scenarios.

What is important is to acknowledge that both approaches may be required at certain times, and ensure that it is possible to take either approach as the circumstances require.

So if you link to a common library, you need on occasion be able to switch to linking to a fixed version or a specific branch, rather than always to the very-latest version. Or if you copy-and-paste (or start linking to fixed separate branches), you need a system for tracking each place where a copy is used, and control any unintended or excessive divergence between each place.

There isn't a purely technical fix to this. The projects will need to communicate and coordinate changes to the shared libs.

It's something we are aiming to do after resolving or making this issue less problematic.

That's backwards thinking. If you had test suites for all your applications, then each shared library can gate changes behind the tests being run. That in itself will make this issue less problematic, at least to the degree that Dev A will become aware they are breaking App B.

All the code lives inside a single per-device repository where every commit goes into a main branch.

Do you mean all the code, or only the per-device code? Dev A can't break App B if they don't commit to a repository used by App B. Does each App have to pull the HEAD of shared code? You can instead pin it to a known version, and periodically advance a given app to the current HEAD (or a known stable point) at a time convenient to that project.

Libraries seem to need independent milestone/feature versions

Minor changes like localized bug fixes are trivial. Breaking changes should deliver a new product, with test code, an explanation on the changes and what to do to migrate.

I might imagine, that here the organization is too isolated seeing the nature of embedded software development. Causing the changes not to be packed in large version steps, but with hacking small code changes to a library till one application works.

The solution is tagging larger breaking steps, probably making it a new library major version. With documentation, how-tos, test code. And especially with a list of using projects of this version.

Thus one can plan the migration of the version. How many versions one is back for this application, what are the benefits / improvements, how invasive are the changes, testing effort needed.

So instead of a amorphous stream of library versions, real mile stones. Organisatorical with as most budget feasible, documentation, peer-review, presentation (slides).

An example: there is a requirement to read a text. In a change request one wants to first try reading the text in full Unicode, with encoding UTF-8. When failing using the local platform encoding. Developing however it seems best to change reading an URL into buffering the text first as bytes. At this moment a regular minor version update becomes a major milestone update requiring more effort for documentation, unit tests and such. Also communication.

The codebase currently does not support automated testing. Making the code testable requires heavy refactoring that in the current situation we can't easily do because of the above It's something we are aiming to do after resolving or making this issue less problematic.

What strategy can we adopt to resolve the above problem?

By refactoring the codebase to facilitate automated testing.

You won't make this issue less problematic until you do. I'm sorry, but I need to toss out a frame challenge on this one. You are worried about productivity or stopping another team's work in order to do this refactoring, but the reality is you already experience regular stoppages of work. Where is the line?

You don't need to stop all work on all apps on all devices. Focus on refactoring the shared libraries first. Make the shared libraries testable in small pieces. I can't recommend concrete techniques because I would need access to your codebase — something impossible for you to do. Instead, this needs to be a group effort with a shared understanding of the challenges and solutions.

Some additional ideas:

• Identify parts of the library that see the most churn and prioritize those first.
• Use feature toggles or compiler flags to turn code on or off. This can quickly become a big ball of mud unless you plan carefully.
• Changes to these shared libraries need to be coordinated better. Each device might need its own mini change control board for changes to shared libs.
• Can you write emulators for the hardware? This would at least allow you to write a test suite in software, but the complexity of setting this up would need to be balanced with the cost of setting up an automated test suite with real hardware.

I just don't think you can make this situation tenable until teams communicate changes better, and these libraries are changed to make them isolatable for testing purposes. It will cause things to slow down, but introducing automated testing for these shared libraries is the only answer long term, and will pay dividends down the line. You should see breaking changes causing work stoppages less frequently, which will free up more time to continue evolving the codebase and team process to finish alleviating these problems.

Another thought just came to me, but this requires using Git for version control. If you don't use Git, this might be just the incentive to make the switch.

Git Submodules.

A Git submodule is a git repository inside a git repository. It's a folder inside a git checkout that contains another git checkout from another repository. Make the shared libs a separate Git repository from the device repository. This allows developers to create branches and commit to these shared libraries. You will need to integrate changes and coordinate with other app teams, but this at least gives you the isolation necessary to apply changes in a controlled manner.

Failing this, you're left with my frame challenge above, and Git submodules might give you just enough flexibility to do the frame challenge in the first place.

1. Once you get to a certain level of maturity, stop making breaking changes without talking to the other team members. Set up some kind of "approval board", even if that's only the application team leads


2. Develop on the trunk but make tags (or whatever they are called in your version control system) once you are at release candidate status.

Move the responsibility. If developer X needs changes to the library that he needs for his app A and these changes break B and C, then make it that developers job to fix B and C.

Why? Because otherwise this guy is seen as the hero who creates new libraries at speed while the others are seen as lame developers who spend their time fixing broken code instead of doing anything useful. So guess who gets a raise. He has no reason to work more careful until he pays the cost of his carelessness.

Meanwhile do code reviews. You would check if your app works with the new code, otherwise the change is rejected until the library works with B and C just fine. Alternatively, remove the changes to the library if that is needed to make B and C work. Again, the work is moved to the person who checked in broken code.

You may also consider changing your build systems so the apps use a specific version of the library instead of the latest.