At the Unicom DevOps Summit event in London on February 28th 2014 we experimented with some extra audience/attendee participation by asking for questions on record cards and encouraged people to ‘dot-vote‘ on the questions most interesting to them. There were some good questions, but unfortunately we did not get chance to discuss many of them, so here are all the questions from the card board, along with some very brief attempts at answers.

• Should security be a part of DevOps?
• To what extent and how do you insist on standardisation for multiple Scrum + ‘DevOps’ teams with no separate Operations team?
• What’s the likely process flow of / disruptions to / duration of DevOps adoption?
• Where does ‘Operations’ sit in the ITIL model? All over the place? e.g. Service Transition?
• How about some example scenarios? Tangible comparison points would be useful.
• Where does DevOps start and finish (from a process perspective)?
• Is DevOps just a job title?
• Is co-location of resource necessary for successful DevOps?
• How essential is cloud technology to DevOps?
• How will the announcement that ThoughtWorks are ‘open sourcing’ their Go DevOps product affect other vendor products? Why pay for other products?
• There are a lot of open source DevOps/release/orchestration tools – is anyone using (or know about) the Windows equivalent?
• How do you overcome developer resistance to writing Run Book docs? Are the processes to drive adoption? Is it a sackable offence?

Should security be a part of DevOps?

Security should certainly  be part of a DevOps approach to building and operating software systems. In fact, done well, DevOps can improve security by highlighting problems sooner and enabling more rapid fixes through better monitoring and sharing of information, and improved testing and deployment automation. Gene Kim has written about this in his Visible Ops book and Nick Galbreath has spoken about InfoSec and DevOps at the SxSW conference. Gareth Rushgrove spoke on InfoSec and DevOps at Velocity EU in 2013, and mentions the Gauntlt toolkit; Steve Hall of ScriptRock also has some thoughts on InfoSec and DevOps.

To what extent and how do you insist on standardisation for multiple Scrum + ‘DevOps’ teams with no separate Operations team?

[I modified the question compared to the one in the image after speaking with the guys who posted the question and understanding more detail. This question really deserves a whole blog post.]

First, some background: Frank and Henk work at a large organisation in the Netherlands which is part-way through a major change in its ways of delivering software systems. The traditional ‘operations’ team has been entirely disbanded and merged into one of many independent product development teams (an extreme version of my Type 2 DevOps team topology), resulting in really two separate questions: (1) how can you effectively deal with Production incidents without a ‘first line’ service desk to triage incident reports? and (2) how and to what extent should you *force* product teams to adopt the same tools, processes, procedures, etc.?

As for (1) I think that most organisations will need some kind of unified service desk for doing the initial triage on Production incidents. I cannot see how you can effectively coordinate the diagnosis and remediation efforts of a non-trivial incident without a single service desk responsible for the whole system; this is basic good practice, espoused in ITIL. At the very least, I’d expect to see people from each product team rotate into the role of service desk incident triage for (say) a week at a time. It will be very interesting to see how organisations that choose to merge ‘Ops’ entirely into ‘Dev’ manage to deal with real incidents in a sustainable way without compromising availability and security.

The idea of forcing teams to adopt specific tools or processes (2) is to me very wrong, and the anti-thesis of the ‘learning organisation’, collaborative approach required. The organisation needs to be set up so that teams can experiment within a context of engineering excellence, sharing successes and failures with other teams, and learning which approaches seem good for the organisation; in this way you would aim to avoid a ‘free-for-all’ proliferation of tools and processes. You’d probably want to end up with no more than one or two main tools for each kind of activity (CI/CD, deployment, version control, provisioning, etc.) plus perhaps a third tool being prototyped for extreme edge cases or new techniques (e.g. Docker); the sharing and learning approach ought to achieve that.

[Thanks to Frank and Henk for a really stimulating conversation over coffee – I’m looking forward to hearing more. ]

What’s the likely process flow of / disruptions to / duration of DevOps adoption?

John Clapham writes of his experience of DevOps transformation at Nokia, including how they tracked the vital signs of DevOps adoption. It’s important to allow a ‘human-scale’ time for people to lose their fear of change and fear of failure, and begin to feel able to experiment as part of a learning organisation. Then again, perhaps in your organisation the decision-makers secretly hate DevOps and have no plans to introduce it?

Where does ‘Operations’ sit in the ITIL model? All over the place? e.g. Service Transition?

While I find many aspects of ITIL very useful for most organisations, some places choose to apply the patterns in a very rigid way, insisting on separate teams for Service Transition and Service Operation, for instance, which leads to hand-off and ownership loss. The ‘Operations’ part of ‘DevOps’ relates most strongly to the ‘Service Operation’ phase of ITIL, but also speaks to ‘Service Transition’ and ‘Continual Service Improvement’ (the latter maps to the ‘Third Way’ feedback loops characterised by Gene Kim). IT ‘Operations’ for me relates to ‘anyone with a primary focus on the Production system’.

How about some example scenarios? Tangible comparison points would be useful.

I have written about some possible team patterns for DevOps in my DevOps Team Topologies post, and some hints about mindsets needed in a post on DevOps interview questions (although this is a bit out of date now). The team setup at Spotify is worth understanding, with a fluid matrix-type grouping of skills and interests (‘tribes’, ‘guilds’, etc.)

Where does DevOps start and finish (from a process perspective)?

Logically, the ideas underpinning DevOps should be extended to encompass the entire organisation, not just IT, because DevOps really aims to solves a business problem, not a technical problem, as Damon Edwards points out. Patrick Debois has written about this, with ‘DevOps Lite’ being the Dev plus Ops bit and ‘DevOps’ including the whole org.

Is DevOps just a job title?

This was my naughty contribution: I agree with Pete Cheslock that DevOps recruitment is broken.  Yes, a temporary DevOps team might be a useful enabler, but a permanent DevOps team is another silo, and you cannot hire an “agile”, so you should not be looking for a “devop”.

Is co-location of resource necessary for successful DevOps?

Co-location might make DevOps easier to achieve, but it’s neither sufficient or completely necessary. I (as a Dev) have worked with Ops colleagues whom I trusted completely even though they were based in Bangalore or Texas or Heathrow while I was in London, and we delivered very effective software systems. Likewise, I have sat in the same office within 10 metres of Ops ‘colleagues’ who almost never shared what they were doing, building Production-only systems to satisfy their employee performance targets. If the incentives for collaboration are in place, then (with the wonders of Skype, Google Hangout, etc.) teams on different continents will find ways to collaborate as effectively as they can. With the wrong incentives in place, co-located teams will find ways to hinder or at best ignore each other. The time zone difference for distributed teams will always be a challenge which teams within the same time zone will not face, and organisations tend to underestimate the negative impact that time zone differences have.

How essential is cloud technology to DevOps?

I’ve come to view ‘cloud’ as orthogonal to DevOps, not essential for it. Yes, many of the shiniest examples of DevOps practices tend to include some use of Amazon EC2 (e.g. Netflix) but a culture of collaboration, automation, and shared metrics and data can be equally applied to building desktop software or mainframe software or any other software of non-trivial complexity and operational requirements. Vagrant, OpenStack, and other vistualisation tools certainly make aspects of DevOps easier, but there is much more to DevOps for me than a set of (amazing) tools.

How will the announcement that ThoughtWorks are ‘open sourcing’ their Go DevOps product affect other vendor products? Why pay for other products?

This really needs a separate blog post. ThoughtWorks GO is (as of February 2014) the tool with the most advanced deployment pipeline capabilities (not surprising since Jez Humble was the product owner for several years). I have been using GO heavily since 2011, and I have seen that its ability to visualise end-to-end deployment pipelines is hugely beneficial for engaging all people involved in product release: product owners, developers, testers, release people, and operations folk. No other tool currently comes close to the visualisation that GO provides. However, it must be said that GO lacks some of the Continuous Integration features of tools like TeamCity, lacks the deployment features of tools like Octopus (or even Chef or Puppet), and lacks direct is not as advanced in its integration with package and artefact repositories (Gems, NuGet, maven, Artifactory, Nexus, etc.) provided by tools like TeamCity and Bamboo. It will be interesting to see how the capabilities of GO develop as the pull requests start coming in via Github.

There are a lot of open source DevOps/release/orchestration tools – is anyone using (or know about) the Windows equivalent?

Jenkins and ThoughtWorks GO both run on Windows (although if you use msysgit server-side for git operations, YMMV, as there is a low-level thread wait bug in the ported code which means you’re better off on Linux – I’ll write about this some time if I get chance). TeamCity and Bamboo, although not open source, allow you to run a small CI/CD setup for free, and both these run on Windows too. Octopus is a great deployment tool for Windows. [This question seems to conflate ‘open source’ with ‘Linux’, not recognising that there is an increasing amount of open source tooling for Windows (such as NuGet and Chocolatey)]

How do you overcome developer resistance to writing Run Book docs? Are the processes to drive adoption? Is it a sackable offence?

This relates to my talk at DevOps Summit on Software Operability and Run Book Collaboration. I tried to emphasise that the most important aspect of getting developers to write a draft Run Book is that it encourages collaboration with Operations folk, and that the resulting information should *not* become documentation. In fact, I’d prefer that teams write a draft Run Book and then throw it away rather than it be seen as ‘more documentation’. Developers tend to be less resistive to the idea of the draft Run Book if (a) they know it’s not documentation and (b) they know they might be woken up for a P1 incident in the middle of the night! Threatening developers with losing their job for not collaborating with operations people seems rather extreme, and at odds with the culture of shared responsibility and learning that DevOps should foster. Focus should instead be on changing the organisational culture.

Thanks to everyone who submitted a question.

Edits: clarified details about package repository integration for GO (thanks, @capotribu).