I advocate for metrics tied to actual business value, like KPIs, but I sometimes see these miss the mark.

For example, I worked with an eCommerce software company and was tasked with meeting an SEO target of increasing the sites ranking to increase revenue. This was in early 2016 when Google and other search engines weren’t crawling single-page applications properly. So the site was not appearing high up, or at all, in searches for our product catalogue.

On the surface, this is a very reasonable target. We spent several weeks investigating rewriting parts of the single-page application to pre-render server-side to allow the search engines to crawl the data. This was going to be a radical rewrite which would take many months involving most of the engineering teams.

However, during this process, we started to look at the data we had around our user-profiles and the usage patterns on the site. The site sold high-end luxury fashion clothes and jewellery. The conversion rate was quite low but the average basket value was very high (> £30k).

By examining this data and publishing a survey attached to the registration and checkout processes we determined that almost all of these high value clients were not even searching for our site through search engines. The traffic to the site was driven evenly from fashion magazine advertising and social media posts.

Due to this revelation, we switched our efforts from SEO to streamlining our registration process, checkout process, and started to sponsor social media influencers. This increased conversion rate (albeit lowering the average basket value) and more efficiently pushed high-value clients to the site.

As a happy side effect less than four months after abandoning the attempts at increasing site ranking, Google’s crawler was updated to crawl single-page applications and our page ranking increased without us having to expend more time on it.

Sometimes, it’s worth challenging the hypothesis behind a target as not all KPIs are created equal.

_{If you want to discuss how to use hypothesis driven development please contact me through the comments, email or LinkedIn.}

_{Photo by Lukas}

I have had several clients who have asked me to carry out software architecture reviews to determine how to replace bespoke software developed in-house with a commercial off-the-shelf (COTS) solution.

What I’ve found with most of these clients has been that the COTS solution (even if customisable) only covers about 70-80% of the functionality of the bespoke solution that they wish to replace. It also covers a load of other areas and features that the client has no use for. In addition, they’re often not aware of the operational expenditure that the COTS solution will take and haven’t factored in the cost of customisations and maintenance so they’re not comparing like with like. They also tend to underestimate the cost and the timescale of the data migration work required to move to the replacement system (usually by a factor of 2.5).

Even when all this is taken into account a small cost difference in favour of the bespoke solution alone would not necessarily prevent me from recommending moving to an off-the-shelf solution as it can pay off in reduced opportunity cost due to freeing internal engineering capacity.

However, the biggest factor around how to progress is usually what makes your business unique. This is usually your value proposition and the reason you get the market share you do. This value proposition is often built into your bespoke solution but not something easily replicated in the off-the-shelf solution (as it’s unique!).

This is usually what leads me to recommend either not moving to the off-the-shelf solution or even taking a hybrid approach of replacing all but the unique value proposition and either keeping the bespoke solution or building a replacement for the value proposition only.

All of this can be summed up as don’t give away the Crown Jewels of your business to a vendor who can’t do it as well and can utilise your data to determine how to replicate it for your competitors.

Conversely, if the bespoke solution does not make your business unique then consider a commercial replacement (depending on the costs).

_{If you want to discuss how to determine how to migrate to new solutions please contact me through the comments, email or LinkedIn.}

_{Photo by Tembela Bohle}

You may think that AI is the solution but are you even making the best use of the information you already have?

Last week I was having a chat with my friend and entrepreneur Mike in a coffee shop the other day and the subject turned to AI.

During the discussion, we came to the conclusion that most businesses considering using AI don’t understand the data they already have or could have. Often these jewels of information are in disparate locations, difficult to use formats or not even captured.

As we both enjoy a good simile we analogised this data to reservoirs whose locations are not even known let alone used to power electric turbines.

Often these businesses jump to a solution (data warehouse, data lake, data lakehouse..) without even understanding what the data is, how it might be used and how much it will cost to get it fit for use, let alone the opportunity costs of not utilising it.

It sounds counter-intuitive but I tend to build data strategies, tactically, one question at a time, incrementing to build towards a coherent approach, learning as I progress.

I’ve worked with dozens of clients to:

1. Identify the key value propositions.
2. Identify a manageable number of metrics for this value.
3. Build systems to capture and broadcast these metrics.
4. Ingest and transform existing data sources.
5. Blend existing data and metrics to answer a specific and manageable number of questions.
6. Identify actions and implement them.
7. Repeat from 1.

Often this process can identify data you already have that’s not readily usable because:

1. You weren’t aware you already captured it.
2. It’s in a format and/or location that makes it unmanageable.
3. It’s missing a key dimension you’re not already capturing.

In a lot of cases, the missing technology that’s preventing you from getting value from your data is data engineering and not AI.

_{If you want to discuss how to release the potential of your data please contact me through the comments, email or LinkedIn.}

_{Photo by Pixabay}

I find it incongruous that in the software development industry we often talk about languages, frameworks, libraries and architectural patterns but we don’t often consider them as construction materials. We do consider some ‘structural properties’ like performance, resilience, security but we often make a decision based on these properties and apply it to our entire system.

If we were fabricating something physical (non-digital) would we choose one material for our entire project? Would we ignore other properties like malleability, texture, density? Would we require the same structural rigidity everywhere or would we pick materials that had more ‘give’ in places that require change and add rigidity in places under extreme stress?

We often think about programming languages and libraries as tools but they are part of our product. The choices we make affect the amount of change we can accommodate, the ‘texture’ of those languages, frameworks and libraries have a visceral impact on how our engineers work with them, to the point that they can make engineers leave to work on something more fun.

This is often the underlying reason why we see large tech companies that start with one language and then re-write parts (or all) of their system in other languages later, usually once the need for the product has been established, the problem areas identified and the stresses caused by changes, performance constraints, load, etc. measured and understood.

This is also another reason modularization patterns, such as micro services, have gained popularity. But these patterns in themselves are material choices. When we are proving our concept viable (startup mode) should we build lots of separate components in concrete, stainless steel and glass, then fit them together with different glues, screws and fitments (often bespoke) or should we shape something quickly out of polystyrene or wood and test it first?

We should think about computer science as a material science. We should consider the intrinsic properties of our building materials and consider what is suitable for our current environment and how we might evolve to different materials as our environment changes.

_{If you want to discuss how the ‘material science’ of software development has an impact on your software and your development teams please contact me through the comments, email or LinkedIn to discuss in more detail.}

_{Photo by Laura Tancredi}

As a start-up founder you probably lie awake worrying about “How will my technology cope with another 10 or even 100 thousand users overnight?”. Firstly, if you genuinely have this concern, congratulations it’s a great problem to have! Most start-ups never get to the stage of having to worry about exponential growth.

Secondly, your major concern at this stage shouldn’t be rapidly adding shiny new features but things not breaking catastrophically so you retain customers and don’t suffer irrevocable reputational damage.

Microservices, Function-as-a-Service, Serverless…?

Should you build your software as a number of microservices, should you use serverless database, should you use AWS Lambdas/Azure Functions/Google Cloud Functions, do we need an API Gateway…? Too many technologies, too many buzzwords, and too many choices.

In fact, you’re probably worrying about the wrong choices.

Not ‘Can my tech scale’ but ‘Can my Engineers scale my tech’?

If you’ve read my other writings you’ll recognise a theme, I like to think of software as existing within layers of ecosystems.

Scaling your technology successfully relies on the ecosystems that it’s developed within, which depend, incongruously, on constraints.

If you pick the right constraints for your technology team(s) you provide the right guard rails and support for your team(s) which in turn will provide the right technology and architecture to respond rapidly to change, like rapid user growth.

_{If you want to hear more about my thoughts on how to scale or software ecosystems let me know in the comments.}

Neither!

Well that was quick… on to the next subject!

Seriously though when I hear a startup/scaleup asking how to hire many Engineers quickly it’s usually symptomatic of a business whose current Engineers are swamped or a business that’s trying to tackle a founder/investor wish list of many targets at once.

What problem are you solving?

The first question I’d ask is why do you think you need 10 more engineers or an engineer superstar?

Often the answer is either;

• We have x many features we have to deliver, promised to investors/customers
• Our current engineering team is swamped with issues/bugs/rework

Or even both.

So if you have either or both these issues your problem is your team is context switching (what I less politely call “thrashing”). Context switching is highly inefficient so you’re not using your current engineering capacity effectively.

In general, doing something productive in software engineering takes concentrated focus and unbroken time for at least 2-3 hours, or about half a day. This is different from a manager who can gather information or make a decision in a 30 minute meeting. Engineers work on a different schedule to managers.

You don’t need to hire more engineers unless you’re hiring a consultant or senior engineer to coach your existing team to be more effective.

Stop thrashing

How can you stop context switching?

1. Identify the highest priority feature and focus an entire team of engineers on just that one feature to deliver something of value in 1-2 week, measure it’s effect, rinse and repeat until it’s not the highest priority then move on to the now highest priority.
2. If bugs and rework are sucking time from your team, be brutal about what bugs to address, only those losing you customers today! Carve out half of each day for addressing bugs. The other half should be looking to increase engineering efficiency. How? This requires another post to answer but the one liner is, start introducing practices that focus on continuous integration and deployment (not just tooling but culture and architectural decisions).
3. Reduce the number of meetings you pull engineers into. Schedule them to allow blocks of at least two hours between meetings.

I have a lot more detail on these subjects and if these are issues you struggle with let me know and I’ll write more.

PS

You never want to hire a so-called x10 Engineer, or at least the usual definition of a x10 Engineer, i.e someone who churns out loads of code and fast, is usually disruptive to a team and often produces code that’s hard to grok and harder to change. What you want is an engineer who multiplies your teams effectiveness by 5! I.e. not necessarily the best engineer technically but someone who identifies opportunities to communicate better and promotes quality and changability as first class concepts.

Ever since I took up Clojure as my preferred programming language in 2013 I’ve been asked or seen in the Clojure Community discussions around “Why is Clojure not widely adopted in the software development industry”?

If you want some perspective on the arguments in this debate here is a small selection of the discussions:

• Why is Clojure not popular?
• Why Clojure is not widely adopted like mainstream languages?
• Some thoughts about Clojure, its latent potential and adoption

I’ve engaged in some of these conversations over the years and I’ve put forward similar opinions as many of the respondents in the discussions above.

The Usual Suspects

The arguments over why Clojure is not more ‘popular’ range from the banal to the ridiculous:

• there is no framework ‘to rule them all’
• there is no ‘killer application’ to ‘rule them all’
• functional programming is hard
• Lisps are hard
• there are too many parenthesis

I’ve left off this list the most ridiculous argument because it’s less of an argument and more of a self-fulfilling prophecy “We can’t hire 100 Clojure developers”.

This last one is simultaneously re-enforcing a reason not to use Clojure by not providing demand for developers to learn or migrate to using Clojure but is also an example of misunderstanding Clojure’s benefits of being able to do more with less developers by assuming you even need 100 developers. Unless you’re Nubank sized you probably don’t!

However, although annoyingly silly this argument about a smaller pool of developer talent is probably the most destructive argument against increasing Clojure adoption as it’s one non-technical managers without a detailed understanding of the ‘economics’ of the constraints and benefits of different programming languages latch on to.

You can read in the many discussions online (some I mentioned above) the somewhat subjective analysis of the most common arguments, however in thinking about this recently I thought I’d attack the problem from the opposite angle.

Why do people adopt Clojure?

In order to get to the root of ‘Why’ people and organisations find Clojure interesting enough to adopt we need to look at who adopts Clojure.

Who adopts Clojure?

I will use a point of personal privilege by ‘speaking from authority’ by citing my personal experience of working with more than 7 organisations that decided to adopt Clojure, working on the programme committee of a Clojure conference for 4 years and spending over 7 years talking to Clojurians online.

My experience is that most people who were in the position to be decision-makers in adopting Clojure in an organisation are:

1. as decision-makers they tend to be senior in terms of job title and have > 8 years of software development experience,
2. they are exclusively technical and either still hands on with code or recently removed from coding,
3. they have extensive experience of working in some quite complex problem spaces,
4. they are not directly from academia although some have higher degrees their experience is grounded in solving real world problems,
5. they have extensive experience of building software with the object-oriented paradigm (and sometimes also procedural paradigm) but not necessarily a functional programming background.

Obviously my experience is not a scientific survey but these characteristics seem to be prevalent in the decision-makers who adopt Clojure.

What organisations adopt Clojure

For this question we have some slightly more objective data. The State of Clojure 2022 survey:

As we can see the majority of organisations using Clojure are < 100 people in size.

Perhaps more telling is the number of people in the organisation using Clojure. It tends to be either organisations with small engineering functions or just a few teams in the organisation.

What’s Clojure being adopted for?

What kind of applications is Clojure being put to?

As we can see most people are currently using Clojure for web development, building and delivering commercial services and enterprise applications. We can fairly assume that the Open Source development is supporting the building of the rest of the applications.

Programmers adopting Clojure

What about the kinds of people using Clojure? What does that tell us?

We can also see that most people moving to Clojure come from a programming language typically used to solve practical business problems. These languages are also typically used in web and mobile-based applications rather than more scientific or engineering problems. These languages are also primarily object-oriented in focus although several can be written in a more functional programming style.

Judging from the number of people who skipped this question (17 of 2373 = 0.7%) we can assume that very few people are new to programming. Almost none it seems.

Lastly, how long have people adopting Clojure been programming professionally?

and how long have they used Clojure for?

We can see from this that most people using Clojure have been programming for more than 5 years and these two results seem to reinforce the assumption that almost all have come from programming in another language first. Also most people adopting Clojure seem to stick with it (although that’s hard to determine solely from the people who’ve stopped using it as this is only the respondents to the Clojure survey who would be self selecting to be mainly Clojure users, the high number of people using Clojure for > 5 years is indicative of high retention).

Characteristics of those adopting Clojure

So, what does this tell us about ‘Who is adopting Clojure’?

1. they’re solving pragmatic business problems,
2. they’ve come from other programming languages (mostly object-oriented focused),
3. they’ve a lot of experience already before coming to Clojure

Why people adopt Clojure revisited

So, back to the earlier question, ‘Why do people adopt Clojure?’.

The answer I will give is obviously subjective to some extent but rooted in my personal experience in working with over 7 different organisations on over 35 different services or applications.

I believe that people adopt Clojure because they see in it’s philosophy and constraints a language that tends to naturally support:

1. simple code (as Rich Hickey will tell you definitely not the same as easy!),
2. code that’s easier to change,
3. code that’s easier to reason about with regard to state changes,
4. code that’s expressive of the problem to be solved (i.e. more code related to the business problem and less code related to just running the software)

So, why is Clojure not more widely adopted?

Again this is my subjective opinion, but I think the answer to that lies in the profile of who recognises the need to adopt the language. I think that you have to have suffered a number of failures and frustrations with other programming languages to recognise that the benefits far out weigh the alien syntax of a Lisp which has a functional programming paradigm and immutable data by default.

_{Photograph attributed to ~bostonbill~ under creative commons license.}

Back in 2018 I wrote a, somewhat, critical essay on Clojure and how a lack of developer discipline can easily build up in a Clojure codebase to make it hard to grok and hard to change.

I just wanted to revisit this as some time has passed and in the meantime I’ve worked on three more Clojure codebases as well as a Java one.

Although I still feel all the points that I made in the original post are still valid my recent experience has tempered my opinions.

My main critiques could be summarised as:

1. Clojure’s dynamic typing & lack of language enforced patterns can make building a mental model of the problem more difficult.
2. Clojure’s lack of ubiquitous frameworks also tends to increase the cognitive load on the developers understanding.

Clojure revisited

Having worked on three more Clojure codebases and another Java codebase I’d like to readdress my opinion on the cognitive load Clojure tends to introduce compared with that introduced by statically typed languages (or at least those not derived from the ML tradition of Hindley–Milner type systems).

Change trumps all

One of the things I noticed when working on these codebases was that the higher the rate of change the easier the cognitive load imposed by Clojure was compared with Java.

Initially I thought this was counter intuitive as I’d expected that the strict types of Java would reduce the amount of mental juggling I had to do to construct a model that needed to change. Although having types (classes) in Java helped me superficially to determine the shape of data in my mental model that initial at-a-glance definition wasn’t that much of a time saver.

Why?

Although Java classes show the data encapsulated in a concept;

• unless small they are not that fast to scan and absorb
• they tend to describe one small concept and not the concept in context
• often you’re only interested in one or two fields in the class and not the whole class
• changing one or two fields may subtly change the meaning of the class or conflate multiple meanings of the same class in different contexts
• even with refactoring tools it can be laborious to change a class to include/exclude data

Treating data as a generic data structure (a map [dictionary] or a vector [array]) means within a given context (i.e. within a call chain or a single function) you only have to change/add/remove a field and that only impacts that field and the access path of that field.

With good naming and use of something like Spec to describe the data at the boundaries of the system or components of the system generic data structures may add a few seconds extra to building a mental model but this is easily compensated by minutes or hours of time saved in changing types (classes/interfaces) or introducing new types (classes/interfaces).

In summary

Therefore, I think that Clojure’s ‘disadvantages’ as a dynamically typed language that doesn’t enforce a data definition burden on the developer up front are more than out weighed by the advantages of making changes faster and in a more isolated manner.

I’m old.

No, really, relative to most software developers I’m really old!

I’ve been a professional software developer for over 30 years and I’ve designed software systems for over 29 years.

This means I’ve seen software trends come and go, and then return again in a different guise…. multiple times. During that time I’ve had to forget many tools, techniques and languages and learn new ones. Recently this has lead me to thinking about what knowledge has endured over that time.

I can only think of a few subjects that I studied at degree level or below that are still highly relevant today. Setting aside basic mathematics and underlying computer science concepts, like logic, there is one field that I think not only holds relevant but has become even more relevant - Systems Theory.

What is Systems Theory?

People have literally written whole books on this so I can’t explain this in any depth in a blog but here’s a definition:

Systems Theory is the interdisciplinary study of systems, where a System is:

A set of elements or parts that is coherently organised and inter-connected in a pattern or structure that produces a characteristic set of behaviours, often classified as it’s “function” or “purpose”.

Obviously in software development we often think about systems. However, we often lose sight of a number of truisms.

• Our software is a representation of an incomplete, inaccurate model of the system.
• We, the people designing and building the software, are a part of that system and therefore we exert influence over it’s behaviours.

Systems are dynamic. They consist of stocks (‘reservoirs’ of information), flows (of information) and feedback loops (with delays).

Mapping systems theory to software

If we are trying to solve a problem using software one of the first things we need to do is draw a boundary around what it is that we are trying to solve: What is inside our problem and what is external?

In Domain-Driven Design this boundary is called ‘bounded context’.

Defining a bounded context is both critical to success and really hard. I’ve never worked on a software system where we got this context ‘right’ first time with the exception of a system re-write two years after the first system was written. Even if, as in that case, the bounded context reasonably contains the problem, the world moves on and the problem space changes.

This is the fundamental driver for ‘agile’ methods. The acceptance that things will change and our software needs to adapt to this change. As a result, we accept that we, the development team, are an integral part of the system over time. Our bounded context is an imperfect model of the problem our software is representing. The runtime environment, the development team, the organisation we work in and the market that organisation is operating in all have an effect on that model and are potentially affected by the models dynamics.

What levers do we have for changing systems?

So if we accept that the team, department, organisation and the market we operate in all influence our software how do we make changes to that larger system if we want to transform it? This is often the question organisations ask when starting a transformative journey.

Examples of these transformative journeys are often expressed in trite marketing phrases like; ‘Agile Transformation’, ‘Digital First Strategy’, ‘Customer-centric approach’. However, I would add ‘Evolutionary Software Architecture’ to that list.

So what are the levers we can use to affect change in whole systems?

In increasing order of effectiveness:

1. Numbers: Constants and parameters such as thresholds, targets and standards.
2. Buffers: The sizes of stabilising stocks relative to their flows e.g. Work in progress limits, team size, number of specialists, datastores, etc.
3. Stock-and-Flow Structures: Physical systems and their nodes of intersection e.g. Departments in an organisation, responsibilities across teams, services, etc.
4. Delays: The length of time relative to the rates of system changes e.g. milliseconds in runtime environment, days and weeks in teams changing software, months/quarters in organisations, quarter and years in markets. Shortening these delays has an effect on the system overall (sometimes in the wrong direction if not careful).
5. Balancing feedback loops: The strength of the feedbacks relative to the impacts they are trying to correct e.g. Rate limiting APIs, Circuit breakers, Fixing forward or rollback, Customer complaints feeding backlogs.
6. Reinforcing feedback loops: The strength of the gain of driving loops e.g. adding more developers adds more code with more complexity, more code needs more developers to change it...
7. Information Flows: The structure of who does and does not have access to the information. Missing information flows is one of the most common causes of system malfunction. For example: not measuring the impact of code quality can lead to seeing the code rot, thus slowing delivery this leads to adding more developers to increase delivery, focusing on new features and not improving code quality, which results in more code rot and slower delivery.
8. Rules: Incentives, punishments, constraints. For example: rewarding 'hero' behaviour over collaboration leads to individuals hoarding knowledge and becoming both a bottleneck and a single point of failure.
9. Self organisation: The power to add, change, or evolve system structure. Here's where evolutionary architecture sits. The ability to self organise is a key lever to responding to change. In this context, self organising means changing any aspect of a system already mentioned in this list. Enabling self organisation means inherently giving up some 'control' to allow for experimentation and diversity, the trick is to recognise when you are pulling the lever in the desired direction.
10. Goals: The purpose or function of the system. The goal of a system is what drives all the things already mentioned. Changing the fundamental purpose of a system by definition will remake the system to chase that new goal. The goal chosen has a huge impact. Most corporations would state their primary goal as "To make profits" but this is just a necessary condition. How different are two corporations that chose goals of "Increase the wealth of our shareholders" and "Improve the health, happiness and well-being of our customers and employees"?
11. Paradigms: The mind-set out of which the system - its goals, structure, rules, delays parameters - arises. Paradigms are the sources of systems. The ancient Egyptians built pyramids because they believed in a specific material afterlife. Paradigms can be both incredibly difficult to change and also changed in an instant. A change of leader of an organisation can start to impact the direction of that organisation overnight.
12. Transcending paradigms: If you keep yourself unattached in the arena of paradigms, stay flexible, realise that no paradigm is "true", this allows you to choose the paradigm that helps achieve your purpose or even defines your purpose.

So some of the levers in that list may appear to be beyond the scope of a development team or a software architect but it’s important to understand where pressure to resist change comes from and where we can use these same levers to affect change.

Scope of change

The order of effectiveness of the levers shown above is somewhat fluid. Depending on the context you may find some movement in the order but it’s generally limited to the swapping of the order of two ‘levers’ next to each other.

If we assume the ascending order of effectiveness shown we can use this to understand whether we are likely to be effective in driving change using one of these levers.

Assume that we are trying to change an organisation to react more quickly to change from the market it operates in, what is typically referred to as an ‘agile’ transformation.

Adding visibility on ‘stories’ delivered by a single development team through some kind of information radiator like a ‘sprint’ or ‘kanban’ board (information flows) is not likely to make a drastic change in the organisations responsiveness. We haven’t incentivized team members to collaborate (rules) and given the team the authority over their own tools & work (self organisation) and therefore the current hierarchy of communication and management structures will apply pressure to resist responsiveness to change.

In fact, the reason most ‘agile’ transformations fail, or at least only partially succeed, is that the lever you actually need to ‘pull’ to make the whole organisation more responsive is at least at the level of ‘goals’ for the entire organisation but more often that of a complete ‘paradigm’ shift for the entire organisation.

Note that not only is the lever further down the list but that the scope of the focus of the lever is much broader. Introducing agile information radiators to development teams is not going to address the way the work flows to the teams from other parts of the organisation, how the work is delivered or measured.

True agile transformations are a complete mind shift in the whole organisation, driving fundamental restructuring of all the systematic structures in the organisation.

Evolutionary architecture

As, by definition, evolutionary architecture is the concept of software design and implementation being responsive to change it is predicated on the goal of the organisation responding rapidly and effectively to change.

Evolutionary architecture is the art of self organisation of both the human subsystems that produce software and the automated systems that form the environment and materials for that software.

Therefore to be successful in producing architecture capable of evolution, which is predominantly at the level of ‘self organisation’, we need to already be in an organisation that embraces a paradigm of change.

For a more detailed discussion of the various potential tools and techniques of evolutionary architecture I suggest reading my blog post on Prerequisites for evolutionary architectures and/or read Building evolutionary architectures by Neal Ford, Rebecca Parsons & Patrick Kua.

Designing software that is flexible and changeable is arguably the most important architectural property. I often get other software architects saying “What about performance?” or “What about security?” I’m not saying these other properties are not important to consider early on. They are. However, if we optimise our architecture for change (evolvability), when we discover a performance issue or a security vulnerability we can change our system to help address it. The ability to respond quickly to issues like these is exactly what makes evolutionary architecture so essential.

What properties are important in evolution?

You can think of the way a species adapts to its environment in the same way that you think of evolutionary architecture. To be successful, animals need to produce new generations with advantageous traits, respond to feedback from the environment, and leave room for failure by falling back on what works.

Software is similar. You need to make sure it’s adaptable and that you’re making changes to your system based on what works. There are a few key ways that we can create these adaptable architectures: Pick constraints to support rapid change

• Separate the concepts of deployment from release
• Gather and share fast (appropriate) feedback
  • In development
  • In production
• Build a responsive culture

Pick your constraints to support changeability

In order to support evolution in software we need to be aware of the constraints of the software and the environment that the software operates in.

As software architects and developers we have control over some aspects of the environment we build and run software in. Here are some of the constraints we might want to consider to support change/evolution.

Pick the right building materials

At the start of my career, I believed that any Turing complete programming language was equivalent to any other and the language picked was not that important. As I’ve become exposed to more programming languages, paradigms, libraries, and frameworks I’ve realised that the ‘building materials’ we pick have a huge impact on the inherent properties of our software systems, especially on changeability.

When you start building a new system, consider the following:

• Favour languages, libraries, and approaches that allow you to parse the data you need from larger more complex structures.
• If you are using a strongly statically typed language consider using a language that infers type to reduce changes you need to make through your code base.
• If you use a weakly typed language, think about what libraries or idioms you may need to add constraints on the types of changes allowed and how they can occur (otherwise you are in the wild west and anything goes)
• Favour immutable data structures - immutability constrains the way state can change in your executing program thus simplifying reasoning about state, especially in a multi threaded environment. If your language doesn’t support immutable data structures as the default there are plenty of libraries out there for most languages (look for persistent immutable data structures that reuse memory).
• Favour declarative approaches over imperative ones.
• Do you understand the problem well enough to pick a framework early on or should you maintain the flexibility by constructing your solution using small libraries? Pick tools and approaches to support feedback and deployment

In order to evolve, our software needs to be easy and quick to release, and we need feedback about it’s appropriateness during development and while in production. Therefore we should pick tooling and approaches that support those properties. Here’s a non-exhaustive list of some things to consider:

• Continuous integration
• Continuous delivery
• Dark deployments
• Canary deployments
• Blue/green deployments
• Automated testing
• Automated alerting/monitoring

Though it may sound frightening, it can be useful to incorporate production-testing alongside these other testing methodologies. Sometimes testing less and allowing something to alert if it fails can be a risk worth taking or even be advantageous in detecting the actual problem in production. Production is the only real test environment. However, this is a risk judgement dependent on the problem, architecture etc.

Implementing some or all of these approaches can enable you to respond to a bug by fixing forward fast rather than taking a more defensive approach of testing excessively and reverting if a bug occurs.

Stay small as long as possible

For every additional person involved in writing a system, you exponentially increase the communication paths on your team. If you can keep your teams and the number of teams as small as practical you reduce the amount of communication and coordination required to implement each change.

I would extend this principle to keeping the number of teams as small as possible too. Working with a constraint of a limited number of (the right) people will result in innovative approaches to solutions. Just make sure that one of them is not to work longer hours (therefore consider an upper limit on working hours as another constraint!).

Organisation-wide systems thinking

Even the most ‘brick and mortar’ businesses do a lot if not the majority of their customer interactions via software (even if it’s B2B) and therefore your organisation should think of software as the primary means of revenue generation.

Forcing ‘project thinking’ onto software development is a bad idea. Trying to implement a number of ‘features’ to a deadline and budget is often necessary but if every change to your software happens this way then that short term focus never leads to longer term consideration of the product or platform and it’s quality properties.

You can use projects to manage budgets but always think about the product or platform when choosing what to implement.

Separate deployment from release

In order to evolve, our software has to generate a new ‘mutated’ generation. If you can deploy your changes in a canary deployment or even, depending on the change, a dark deployment you can test the change in the only realistic test environment, production.

Without mandating a specific architecture (e.g. microservices, event streaming, modular monolith) Domain Driven Development (DDD) and Event Storming are very useful in determining the boundaries of deployment units.

Don’t consider static modelling in isolation If you build a data, component or class model in isolation you often focus on the wrong attributes. For example, you can model hundreds of attributes about a student in a university but the bursary system probably only cares about identification, fees and payment information plus events that change the state of those attributes whereas student services care about much more personal information. Use the dynamic aspects of the system to guide what information is important to that context.

Thinking about events and flows often leads to discovering the components (deployment units) of the system. Each high level process that sends and/or receives a message is a potential component.

Here is a list of techniques that can be useful for separating deployment from release:

• Feature toggling
• Branch by abstractions
• Continuous Integration/Continuous Delivery
• Immutable servers – packaging aspects of the runtime environment in an immutable image makes deployment and rollback a much less risky proposition.
• Schema on Read DB’s – they make adding data in a deployment much easier provided you always ‘accrete’ (or add to) your schema.
• Consider always API ‘accretion’ – Always adding to your APIs and only logically deprecating functionality or data is easier for client applications to deal with and a ‘breaking’ change is really a new API so treat it that way. Think about how to deal with data returned if you don’t have full control over your clients and can’t trust they parse only the data they need (e.g. do all clients implement tolerant reader pattern?). This is where using GraphQL even in service to service calls can have benefits as it’s inherent constraints assume only returning what’s requested (and you get schema introspection).

Fast appropriate feedback

I like to think about software as ‘living’ inside increasing larger ecosystems in the same way that biological organisms do.

Illustration 1 shows the layers of ecosystems that our software ‘lives’ in. We can see from this illustration that the inner ecosystem can be affected by a change in one of the outer ecosystems but, conversely, the inner ecosystem can cause a change in the outer ecosystems.

Additionally not shown in this diagram is the concept of the frequency of the feedback.

• Software environment feedback is measured/sampled in microsecond/seconds/minutes/hours.
• Team/product environment feedback is likely to be measured/sampled in days or weeks.
• Organisational/departmental environment feedback is likely to be measured/sampled in weeks or months.
• World wide/market environment feedback is likely to be measured/sampled in quarters/bi-annually/annually.

Without going into an exhaustive list of metrics and techniques that might be used to provide feedback the following illustrations give you some ideas of what you might want to consider, but as always, there’s no silver bullet and YMMV.

The micro-ecosystem translates to the runtime environment and the software development practices used in developing the software. The illustration above gives some metrics and techniques that can provide feedback at that level.

The biological concept of a Biotope (or habitat) translates to the team and/or product that the software is a part of and Illustration 3 gives some examples of metrics and techniques for feedback at this level.

Illustration 4 shows some examples of metrics and techniques to provide feedback at the organisation or departmental level.

Finally, Illustration 5 gives examples of potential feedback mechanisms at the level of the target market or in other markets.

As you can see the example metrics I’ve suggested are a mix of measurements of the processes for producing/running software and the measurement of external factors that may impact or be impacted by that software. It’s important not to concentrate on only measuring the things you can change directly but also measure the factors that you only have indirect influence over to enable your software to evolve to those pressures too.

Although I’ve given a number of metrics you should start by identifying between 2 and 5 metrics in each ecosystem level. I also try to map lower-level metrics to metrics in the ecosystem above in order to ensure that a metric is driving the desired behaviour.

Building a responsive culture

Lastly, none of these techniques can be implemented effectively without a culture that embraces, seeks out, and thrives on change. The typical characteristics of this type of culture are all the things you see in agile and lean books/courses:

• No blame culture
• Empowerment of teams/individuals
• Delegating responsibility and authority for entire problems to teams
• Smaller teams to reduce communication networks
• Clear, high-level targets/goals with clear measurable objectives (see ‘Fast appropriate feedback’)
• etc.

However, I think it’s important to emphasise this culture is something that needs to be pervasive throughout the whole organisation. It’s all very well having a software development team or department that has this culture but if the rest of the organisation interfacing with that group is in a strictly hierarchical command and control culture this will cause friction and ultimately be much less effective in responding to change. So what do you do if the organisation is not on board with this culture?

Ideally you can convince the ‘C’ level management and the board that this culture is required and demonstrate how to achieve it through some ‘localised’ success by adopting some of the strategies suggested.

However, I have found some success in tying the metrics that the ‘C’ level management look at, which tend to be either in the organisational/department environment or world wide/market environment, to the metrics in the ecosystems below to show how ‘moving the needle’ in one metric impacts the others. I’ve also found that this helps start the conversation about software not being a ‘cost’ to the organisation but it’s primary means of revenue generation in the future for most organisations. Demonstrating that software is not only about automating processes but about creating new ways to interact with customers in new markets. This in turn can provoke a move away from project focused development towards product/platform focused development.

If you’re trying to convince the upper management/board of something, demonstrating how it impacts the metrics they care about is hugely powerful.

Summary

I’ve covered a lot of ground in this post. However, if there are only three things you take away from this I hope they would be:

• Pick a couple of properties in each ecosystem that you wish to change and identify metrics for them (preferably linking lower level metrics with those higher up the ecosystems ‘ladder’)
• Pick a couple of ‘constraints’ that you don’t yet implement that might help support the required changes above and think about how to implement those constraints
• Use the constraints you pick to guide the ‘materials’ you use and the ‘culture’ you need. For example, if you want to deal with changes in data/message structure more effectively you may decide to concentrate on structural typing and accretion only approaches to interfaces and data storage. This might lead you to picking certain languages/libraries that support structural typing, easy parsing of data, tolerant reader pattern and immutable databases.

There’s no one-size-fits-all solution when it comes to evolutionary architecture. Instead, it’s important to gather feedback over an appropriate timescale, adjust your approaches as you learn and grow, and don’t try to change everything all at once. I hope this post has given you some food for thought and a few practical approaches to try.