Hung Up On Scaled Agile Dependency Management

Let's look together at the following photos.


Below you can see what's called SAFe Program Boards - an artefact (output) of a PI Planning meeting in SAFe.


I like the photos and the level of visualization. I think it is in general a great idea to visualize cross-team dependencies. And it is not a trivial job to do!

this one is from blog
this one is from blog

We value information radiators in agile project management due to their simplicity and power. So I appreciate that finally we have a creative and rather simple way of visualizing inter-team dependencies with colorful strings. They make us almost feel nice about those dependencies.


I'm personally a strong advocate of visual boards. And I do believe in power of visualization. Still...


Still every time I read another proud post from someone who has just done a "successful PI planning" - I feel there is something utterly wrong with it.


And I couldn't quite figure out why I felt like that until I found these images:

Finally I understood. Gosh!


Visualizing cross-team dependencies is like showing to the whole world how f**ked your company in fact is. You need to have some courage to do that! (that's why it is one of the Scrum values).


But this makes me feel sad and bad. Sad because I know how it feels being inside teams with too many dependencies. And bad - because I can't come and help.


I feel sorry for these companies.


I love Nepal and I feel sorry for Nepalis who have to deal with the complexity of their power system. There is quite some technical debt out there. It is also vulnerable and hard to debug. And for those who knows - power-cuts are not exceptional in Kathmandu.


I feel deep respect for people who are brave enough to share their Program Boards with the numerous strings on them. It shows the level of courage and also the depth of organizational debt. It is vulnerable and hard to debug.


Let's look at one more picture I've found:





Can you spot that there is something "wrong" with this board?


In fact every time I see a SAFe Program Board with no strings attached it makes me feel like the guys who made it just didn't get it. There shall be strings highlighting team dependencies otherwise you're doing it all wrong...


See how corrupt this thinking is?


SAFe made us believe dependencies are ought to be. There are always there. They are expected. They need to be well visualized and well managed.


And if there are no dependencies between the teams? Well you guys are not qualified for the scaled agility. You are too small, young, simple, startup-ish. Come back when you're grown! And bring some meters of strings with you.


I'm hung up on this.

Cut Them Loose, Not Hung Up On Them

What I'd like to see happenning on a PI Planning meeting (not after) is the following: after the teams do the initial string exercise, facilitate a workshop to help them try different re-structuring ideas and then repeat the string exercise to see if the dependencies got cut loose. And then repeat until there is a very minimal number of strings left. 


The remaining strings then need to be seen as a serious organizational impediment and to be taken care by management (facilitated by Scrum Masters).


Then it all will make a lot of sense. But visualization by itself doesn't provide a lot of value (although it may feel good).


Here is a great report how one can go about re-forming teams in a self-organized manner.


Talk: Organizational Complexity and Its Effects on Scaling Agility

A project manager walks into a bar. The bar is called “organizational design”.


He tries to order beer but can't figure out how. Then disappointed he is about to exit the bar, but can't locate the exit.


Apparently he looks around and notices a lot of other managers around in there. "That’s a trap." - he thinks. "It is a complex system" - echo the others.

This week I spoke at Agile Tour Vilnius 2016 and PM Day Kyiv 2016. In November I'm visiting Lean Kanban France 2016.


This is my third increment of this talk and now I'm happy how it goes: spaghetti, Japanese poetry and complexity seems to be a good combination for any talk.


Below are some photos and selected slides. Down below are a full slide deck. Video is in post-production. Enjoy.


Video From The Talk


Causal Loop Diagrams For Understanding Organizational Dynamics

Just last week I spoke at two conferences on Organizational Complexity and Its Effects on Scaling.


The talks went pretty well and I ended them up presenting three most commonly seen (by me) team structures that create an "interesting" dynamics.


Here they are:

Consumer-facing teams not owning the shared platform

Consumer-facing teams not owning the shared platform
Consumer-facing teams not owning the shared platform

This one is seen very often in gaming companies.


There is absolutely nothing wrong with reusing an engine (platform), that's actually a great idea. What I'm talking here is about the managerial decisions on forming teams around these architectural layers.


In this case a set of consumer-facing teams will be building "their" features (products) on top of a shared platform "owned" by another team (teams). If this is your case, you often hear terms like: "engine teams" or "platform teams" or sometimes "system teams".


It is exciting (for a system thinker) to observe how unaligned those two types of teams can be. How much information needs to be handed over from the consumer teams to the platform teams to get "their" back-end part right. How much over-processing and over-production can be created on the back-end side. And how hard it is at times to get the whole thing fly to production, once all teams say "we're done!"


Beware of the system dynamics such structures create.

Satellite product (e.g mobile) teams not owning APIs

Satellite product (e.g mobile) teams not owning APIs
Satellite product (e.g mobile) teams not owning APIs

This is a very common example for so-called "web" companies that had emerged in the pre-mobile era, had built their product(s) on web and later on then started the mobile adoption.


For such companies it is not uncommon to create the pure "mobile teams". And with a good intend: hard focus on mobile technologies, fast delivery of the apps, separation of responsibilities, reusing existing "ready" APIs... There are many good intentions why to separate the mobile teams form the rest of the "old" organization. And the road to waterfall is paved with them.


In such setting you often hear terms like "mobile teams", "iOS teams", "Android teams", "web teams", "API teams" - such names being used to identify specific teams, backlogs, product owners, roadmaps. As of course it is rational to let each of those teams do their own "scrums".


Guess, how flexible, rapid and agile such a constellation is when suddenly you need to launch a new product on both platforms: web and mobile!


Beware of the dynamics such locally optimized constellations creates.

Many Small Sub-products with Separate Backlogs and Product Owners Not Seeing the Whole

Many sub-products with their own backlogs and product owners not seeing the whole
Many sub-products with their own backlogs and product owners not seeing the whole

And lastly, this. The company embraced the idea of "full-stack", "feature" development teams - where each teams has all required specialization to be building the product features end-to-end.


But since the product is huge, it is of course rational to try simplifying the complexity by the old roman principles: divide and conquer. Anyone seen Romans recently?..


Indeed, it does feel right to break up a huge product into areas - that small that each can be "managed" by one product owner with a small development teams. Then each team has its easy-to-facilitate planning, review and a retrospective meeting. A ScrumMaster can be fully dedicated to the team and everyone is fully focused...


...Focused on a small piece of an overall big picture that no one is able to grasp anymore. The company turns into an unassembled puzzle. And it takes several layers of managers to start seeing the bigger picture... That's often an illusion of management, because no one is capable of understanding an unassembled puzzle.


Be ware of the dynamics your rational decisions create.

Studying System Dynamics

Among the many thinking tools, there is one called "Causal Loop Diagrams". Or the CLDs.


It is a way of modelling dynamics of a system through "variables" and their "relationships". The next illustrations should give you some high-level idea:

Why Modelling?

Modelling is a powerful way to get engaged in meaningful discussions.


And we all know from Alistair Cockburn's insights that no other way of communication beats several people talking while sketching on a whiteboard:

Richness and Effectiveness of Communication: talking at a whiteboards beats it all
Richness and Effectiveness of Communication: talking at a whiteboards beats it all

So, you wonder how you can get "you management" reconsider the structural decisions they have been making? Mobile teams, platform teams, small-product teams... That's hard and everyone needs to have her own epiphany to change her mind.


This takes time. But this as well can be facilitated and hopefully speed-up. For example, why not running an small in-house workshop with your decision-makers to let them explore some of the dynamics at play?


You don't have to call it a "system dynamics workshop" as this can scare mindfulness out of some of your people. Call it a retrospective. Call it a beer game. Call it anything you want. The goal is to get people talking about meaningful stuff.


The CLDs (or any other system thinking tool you know) might be of your great helpers.


As well as some food and beer to follow.

Slides from the Workshop


Scrum of Scrums Is Dead - Emerging Coordination Practices at Scale

Cross-team coordination is decided by the teams.

Prefer decentralized and informal coordination over centralized coordination.


Needless to say, that this article is heavily inspired by the principles behind the LeSS framework.

Scrum of Scrums is ... Born Dead

Scrum of Scrums ("SoS") is probably the most well-known coordination practice recommended for scaling Scrum beyond a one-team setting.


I've seen many variations of SoS and in fact recommended several of my clients to use it to help coordinate dependencies among their Scrum teams.


We all learn and I had to admit - it never worked too well. It was of course better than nothing... but those meetings were never fluent enough and fun to be at.


Still the idea somehow it brilliant - if Scrum with its feedback cycles work so great on the team level, then when having teams working together the same framework shall be applied on the higher level.


A great idea. And that also implies we need to have Scrum of Scrums, Sprint of Sprints, Backlog of Backlogs, Review of Reviews and Retrospective of Retrospectives... And don't tell me you haven't just thought of ScrumMaster of ScrumMasters.


Now you see why this thinking is going all wrong. 


What looks great on paper might not eventually turn out to be a good idea in practice. Remember, we are in a domain of complex things where it is not possible to draw conclusions on the future from thought experiments and past observations.


And in fact, after years of experiments and observations we (the agile community of practitioners who are still uncovering better ways...) are saying louder than ever before: Scrum of Scrums is not a recommended practice to use at scale. In fact, it should be your last resort, not the first one to start with.


Centralization of coordination implies the same "old thinking" that decisions need to be made "on top" based on the information gathered from the "bottom". That's blocking self-management that is the best known way to manage complexity. Such thinking results in over-complicated process based on compliance. Versus lightweight processes emerging from the workers wanting in solving specific issues at hand.


But what to do when you're in a multi-team (or scaled) environment with inter-team dependencies?


Let's see.

Coordination over Dependencies

Compare these two statements:

  1. We, the team-X, have a dependency on the team-Y.
  2. We, the team-X, need to coordinate with the team-Y.

Same situation. Two different mindsets.


In the first example when a team-X has a dependency, they becomes passive hostages of circumstances they are in. In the worst case they finish "their" part of the thing and then wait for the team-Y to deliver.


In the very worst case - they just don't start until the other team finishes. And if the team-Y is also "blocked" by being dependent on a team-Z...


In the second statement: "we need to coordinate" means an active step into dealing with the situation at hand. That means challenging how things are and making agreements with the other teams. That's an adult behavior. That's what we want to see in mature Scrum teams.


So let's stop calling things "dependencies". And let's call them "a need for coordination". That's a game changer.

Coordination and Integration

We need to do more Jenkins than Jira.
Ron Jeffries and Chet Hendrickson

Craig Larman and Bas Vodde in their last book on Large Scale-Scrum define a set of recommended coordination and integration practices (surely, the Scrum of Scrums is not listed as a recommended one).


Continuous integration of the overall product increment is in the heart of coordination practices at scale. Why? Because it creates a demand - a pull for coordination.


So integration pulls coordination.


And why do we coordinate? We coordinate in order to have integrated solutions.


So the result of coordination is integration.


As you can see these two channels (coordination and integration) strongly overlap, causing, supporting and enforcing each other.

Communicate in code

Continuous integration is integrating much more often than you think.

Craig Larman

And this doesn't mean writing comments to unclear code portions...


The modern version control systems (gitGitHub, BitBucket) and continuous integration tooling (like Jenkins, CircleCI) foster communication. If configured and maintained properly your CI server becomes your main enterprise signalling system. These signals visualize coordination that is needed to happen, for instance when a build has just failed because of red tests.


Of course, there are ways to abuse the tooling, no matter how great it is. The most common way that I see is following the guidelines of the "git flow" by introducing feature branches where each feature sits in a single branch until "done". 


Guess what happens next?



There are ways (life-hacks) to make your CI run parallel and independent builds for each of the feature branches, but I rarely see how this creates a need for coordination and prevents upcoming integration conflicts.


In fact this malpractice creates a collection of really bad engineering habits.


Recently I spent a week at a client digging into their build server. The guys did intensive pairing. But each of the pairs used to sit in "their" feature branch for days before merging. Their build server did a magnificent job of building and reporting on each commit in every  branches (I counted 35 branches for 7 pairs of developers). But since the branches were "owned" by a pair, no other pair cared if those branches are red or green. If no one cares, soon you'll stop caring. So 80% of builds in all branches were red.


At this firm since the red became the new norm, all pairs continued committing to their branches, producing continuously red builds - and that's a really bad habit. 


And that is a total opposite of continuous integration. They say: continuous integration is not a tool; it is a practice, a mindset. I can't agree more.



Then once a feature was finally "done", a pair spent few hours (and sometimes days) tweaking the tests to be green again. Removing "old broken tests" was not an exception.


So communication in code implies having the culture of shared code ownership across all teams (and all components and software layers) involved in development of the product.


That's definitely not a new practice at all. Still it is becoming even more important at scale as it creates a need for coordination that glues all teams, pairs and individual developers together better than any other centralized and top-bottom way of coordination (I imply of course the Scrum of Scrums).

More Decentralized Coordination Practices

To be continued with more coordination practices like:


  • Communities of Practice
  • Cross-Team Scrum Meetings
  • Component Mentors and Component Communities 
  • Open Spaces, Lean Coffees and Internal Conferences
  • Travellers, Scouts and Leading Teams  


Sprint Review Turned Into A Beer Fest

Today (XING office in Hamburg, July 2015) our sprint review was different. It didn’t look like any meeting at all. If you would enter the team areas you wouldn’t see anything drastically different to our normal working mode: people are talking in small groups, several people are discussing something by looking at their mobile phones, several others are leaning over tables and looking onto screens. Nothing like a meeting.

But two weeks ago and also every two weeks for the last year or so, this event looked pretty much different. We all would sit in one of our large meeting rooms with 40-50 people attending. One person would speak on a stage for about 7 minutes and others would watch the presentation. Then someone maybe ask one or two questions. Then people would applause and switch to the next presenter.


This was our “standard” way we used to run joint sprint reviews for the 3 teams in our XING mobile cluster. Then since our review was well attended, we invited several other teams to join. In the end we began to have one of the biggest sprint reviews at XING. Huge success! Really?


But despite the high attendance of the guests including the C-level executives, the feedback we started to get from the teams was rather on the negative side: the reviews were taking too much time to prepare; it was stressful to speak in front of so many people; not all of the presentations were uniquely valuable and engaging… and so on and so forth. But the main negative signal we heard was rather strong: the teams were receiving close to zero feedback (applause won’t count)


That was a refreshening “aha” moment for me (an Agile coach) when I came to realize the fact. Especially because I was one of the people who had stood behind creation of the joint review format. As a quick history tour: two years ago when I joined XING each team had a dedicated Sprint Review. But the teams were not receiving equal attention from the stakeholders and management. Also it was quite challenging for the stakeholders to attend all teams’ review. So we decided then to merge the reviews. What we ended up having wa a gigantic Sprint Review that was well-attended by stakeholders and the top company people.


But somewhere on the way apparently we’ve lost an important piece: feedback on the product. That was definitely a learning path.


But why feedback is so important? Well. Scrum is based on the mantra “inspect-and-adapt”. And that’s for a reason. Software development is a complex process that requires empirical process control. Which in simple words mean: one can’t just simply plan software development and then go and execute the plan (that would be a defined process). Instead you need to go a small distance, then stop for a moment, look around and based on what you see decide how you adjust your route. That’s empirical.


And that’s what the Scrum teams are supposed to do when running Sprint Review (and also Sprint Retrospectives) in Scrum: stop, inspect what they did and decide how to adapt it in order to improve.


Now back to our situation. We did the stops. But we didn’t get enough of useful insights on where to go next. So why to have Sprint Review at all? In the end, I’m sure that question was on many people’s minds…


So what we did was following the advice from the Large Scale Scrum practitioners: do the Sprint Review Bazaar (we, being a German company, like the “Beer Fest” metaphor better).

So that what we did:

  1. Asked each teams to have a booth where they will welcome guests (a desk with some candies is good enough).
  2. Make sure the teams have at least 2 people at the same time at the booth talking to the guests.
  3. Cross the fingers and let the folks self-organize.

Benefits we gained:

  • more interaction
  • easy to try things out
  • you can go into more details
  • you can ask now much more questions
  • cozy and intensive conversations
  • less preparations needed

There are of course things to improve, so more experiments to come. Inspect and adapt, right?


But what has become obvious is that the “Beer Fest” style of Sprint Reviews is the way to go. At least for the next few sprints until we find an even better idea.