One of the exercises we do during the Experience DevOps workshop is a ball flow scenario inspired by the Ball Point game. We set up different combinations and topologies of Dev teams and Ops teams, and then see how many balls the teams can pass in (say) 60 seconds.

When we ran the workshop recently in Bangalore, we had a large number of participants, which enabled some interesting experimentation with the topology of the teams. In the exercise, the ‘Dev’ team takes balls from the ‘backlog’, and eventually passes the balls to the ‘Ops’ team, who must ‘make the features live’ under some constraints designed to simulate real-world physical constraints.

With the large group of participants in Bangalore, we experimented with multiple value streams (or products). After ‘warm-up’ runs using a single ‘product’ (value stream) and therefore a single Dev team and a single Ops team, we split people into two separate ‘Dev’ teams (one team for each product, and a backlog each) but only a single Ops team servicing both Dev teams:

In this topology, the teams were able to roughly match the throughput and error rate from before the Dev people were split into two teams (around 16 balls per minute, with around 4 defects). Then we removed the single ‘Ops’ team, and instead aligned half of the ‘Ops’ people with one product (value stream) and half with the other, creating ‘product teams’ (or ‘service’ teams):

The results were very striking: across all teams the overall throughput more than doubled with end-to-end service teams compared to the situation with a shared Ops team. One service team managed 16 balls with no defects, and the second team managed 20 balls and one defect, for a total of 36 balls and a single defect. Compare this to the 16 balls and 4 defects that were managed by the 2x Dev + 1x Ops team:

It was clear that – in this scenario – the single Ops team acted as a bottleneck. Part of the reason was due to (simulated) shared infrastructure. When we split people into service teams, we also split the infrastructure too, so that each service team deployed to its own set of ‘servers’.

The simplicity of the exercise, and the speed with which different topologies and constraints can be tried out, makes this ball flow game very useful for exploring different team topologies in a DevOps context.