Large IT Projects as Complex Adaptive Systems - Appreciating the Complexity

Complexity science is a scientific study of complex systems, systems with many parts such that the behavior of the system cannot be easily explained by describing or studying the interactions between the parts. Examples of complex systems include cities, businesses, network systems, an ant colony, a flock of birds flying together and of course large IT projects. The scope of this blog is limited to decoding the science of complexity to start understanding the last one.

Large IT projects belong to the group of complex systems called Complex Adaptive Systems [CAS], the other two being, chaotic systems and non linear systems. One would agree that why only large; other IT projects could also be considered CAS. Well they could; it’s just easier to understand this better if you imagine a large one.

Research [1,4] indicates that some characteristics that help to describe a CAS system. Understanding of these is essential to appreciate the inherent complexity in these projects. Why we need to appreciate this complexity? Because many times we force solutions to this domain that attempt to take the problem from complex domain to a simple one. We hope to get “control”. We strive to set up “order”..This is not necessarily bad. The fact is by the time you set up control and order to things you know, the next thing hits you. And.. the cycle continues. Did we give a wrong solution or solved the wrong problem. Let’s see.

To better understand, let us take a look at what the attributes of a CAS are and how they apply to our projects:

1. Distributed control

There is no single centralized control mechanism that governs system behavior. Although the interrelationships between elements of the system produce coherence, the overall behavior usually cannot be explained merely as the sum of each part[1]

While we may want to believe that there is “control” in large software projects, in reality this could be an illusion or control limited to certain areas. There are complex relationships at play between the stakeholders – people as well as sub systems that make up the project. Not just that, inside each sub system that has a stake, there are inter relationships that affect how that sub system reacts and responds to certain situations and players. The system behavior is governed by all of these inter relationships and is clearly distributed control. No one player truly calls the shots. Note this does not necessarily mean there is no influence, just that there is not centralized control.

2. Connectivity

Complexity results from the inter-relationship, inter-action and inter-connectivity of the elements within a system and between a system and its environment. This implies that a decision or action by one part within a system will influence all other related parts but not in any uniform manner [1]

What happens when one of the players decides to go for a system upgrade. Without getting into the details of why that was needed, some players may not be affected at all, some will be affected directly and some eventually get an impact because one of the systems they depended on got a hit, not the original system that started the upgrade cycle in first place. Such cascading effects of actions and decisions are common place and it’s almost impossible to predict the connection till something actually hits you. In most cases, you would fully comprehend only in retrospect. It’s all connected.

3. Co-evolution

With co-evolution, elements in a system can change based on their interactions with one another and with the environment. Additionally, patterns of behavior can change over time[1]

Sub systems are observing one another and learning and adapting constantly. Take a scenario. A few scrum teams decide to go explore a new technology stack that improves their performance. They have initial success and the project ecosystem values this drive for engineering excellence highly. This reinforces their behavior and sets a positive trend for others to follow. In a similar scenario what will happen if the project ecosystem does not value this highly. They don’t put it down, but they don’t respect it too much. The initial teams may get demotivated losing the drive to innovate losing out on the opportunity for all to move forward. The sub systems evolve together – for good or worse.

4. Sensitive Dependence on Initial Conditions

CAS are sensitive due to their dependence on initial conditions. Changes in the input characteristics or rules are not correlated in a linear fashion with outcomes. Small changes can have a surprisingly profound impact on overall behavior, or vice-versa, a huge upset to the system may not affect it…. scientific certainty… is a property of “simple” systems (e.g, the ones used for electric lights, motors and electronic devices). Real systems, especially living organisms, are fundamentally unpredictable in their behavior. Long-term prediction and control are therefore believed to not be possible in complex systems [1]

Predictability is the buzz word these days, but predicting the fate of large IT projects with certainty is like going to the “The Oracle“. Most events that influence a large IT project would be non-deterministic , making it extremely hard to make predictions. You could see trends, you could see directions, you could feel where things are going. But to say “you know” is blasphemy. These systems are just so sensitive.

5. Emergent Order

Complexity in CAS refers to the potential for emergent behavior in complex and unpredictable phenomena… There is constant action and reaction to what other agents are doing, thus nothing in the environment is essentially fixed… From the interaction of the individual agents arises some kind of global property or pattern, something that could not have been predicted from understanding each particular agent… Any coherent behavior in a system arises from competition and cooperation among the agents themselves…. For many years, the second law of thermodynamics – that systems tend toward disorder – has generally been accepted. Ilya Prigogine’s work on “dissipative structures” in 1977 showed that this was not true for all systems. Some systems tend towards order not disorder and this is one of the big discoveries in the science of complexity…. Order can result from non-linear feedback interactions between agents where each agent goes about his own business… it appears that self-organization is an inherent property of CAS[1]

Suppose you have 10 scrum teams working in this project. It’s almost impossible to predict the behavior of the system looking at the trends of one or more of these teams. The system behavior is not a total of the individual behavior of teams, but a complex myriad of their intra team and inter team dynamics and interactions. Eventually with a lot of feedback (formal and informal, and sometimes with a lot of pain too), teams figure out a way to co-exist, and how fast they reach or where they reach is also sensitive to initial conditions created. They will reach some kind of order – a way for all of them to operate – together. Not necessarily this is the best way, but they reach some kind of alignment. Not necessarily always. If the initial conditions were not conducive and/or harmony is hard to find, they may well jump into chaos. It’s hard to predict at the outset. Order (or chaos) will emerge.

6. Far from Equilibrium

In 1989, Nicolis and Prigogine showed that when a physical or chemical system is pushed away from equilibrium, it could survive and thrive. If the system remains at equilibrium, it will die. The “far from equilibrium” phenomenon illustrates how systems that are forced to explore their space of possibilities will create different structures and new patterns of relationships…. it can be said that complex adaptive systems function best when they combine order and chaos in an appropriate measure.[1]

When a large system stabilizes and things become routine, there is always the risk of lethargy. Maintaining the status quo or state of equilibrium has high potential to create behaviors that resist change and ability to adapt to emerging situations. Individuals and teams could make compromises instead of taking measured risk that keeps them always updated and relevant. If they are at equilibrium, they may be caught off guard. A bit away from equilibrium, system is alert and awake.

7. State of Paradox

Research in CAS has indicated dynamics combining both order and chaos. This reinforces the idea of bounded instability or the edge of chaos that is characterized by a state of paradox: stability and instability, competition and cooperation, order and disorder.[1]

It is important that the system is operating at this boundary of order and chaos, not too lethargic that it does not improve, yet not to reckless to push it into chaos by wild experimentation.

This concludes our foray into understanding complexity science, specifically CAS applied to IT systems.

The insights do tell us one fundamental thing – trying to force control on such a system is absolutely not going to work. What will work, we don’t know – this is something that needs to be figured out as you go in a particular system context. There maybe patterns and set of tools that could help us, but that’s for another day..

However appreciating our systems, acknowledging and understanding that these are systems in the complex problem domain is the first big step in coming up with the right solutions.