Recently I read the interesting book – Leading the transformation – applying agile and DevOps principles at scale by Gary Gruver and Tommy Mouser. With this post I’ll summarize my insights and hope to inspire you to add it to your reading list too.

  • leave team with as much flexibility in operational choices as possible
  • first order effect – how teams come together, second order effect how individual teams work
  • the plan for transforming the organization should be kept as small as possible to reduce complexity but include all applications that have to work together in production
  • apply agile principles to the planning process and use DevOps to economically release smaller batches of changes on a more frequent basis
  • the DevOps approach provides the most significant improvements in aligning the work across teams
  • executives play a key role in communicating the advantages of the DevOps approach and in explaining how it differs from what is typically done in the industry
  • software development is such a discovery process that many of the assumptions made in planning stage quickly become obsolete during development
  • most of the real challenges are in organizational change management and shifting the culture
  • executives need to understand that the capacity of the organization to absorb change is the biggest constraint to rolling out these improvements
  • the agile transition should focus on the business breakthroughs
  • you want teams that can own the whole feature end to end
  • as leaders we feel it is important, wherever possible, to provide a framework with objectives and let the team have as much design flexibility in defining how the work will get done.
  • engage leaders in the agile transformation too – business objectives and running the continuous improvement process are important topics
  • many agile implementations fail because they focus on rolling out Agile teams the “right” way instead of applying Agile principles at scale. They try all the latest ideas without any clear reason why.
    • not having clear business objectives is a key source of the problem. Having a set of high level business objectives that the entire organization is focused on is the way to create cross organizational cooperation and partnership
    • management teams struggle to show bottom line business results

Two approaches to start the transformation

  • approach one – start with a clear understanding of how people are spending their time and the value of the software is intended to provide to your business – address the biggest cost and cycle time drivers that are not key to your business objectives
    • identify cost and cycle time drivers and identify waste in your process – to start with, check for the significant drivers
    • e.g look at the cycle time drivers
      • commit to trunk
      • number of builds per day
      • full (manual) regression test –>auto regression testing
    • and cost drivers
      • code integration –> continous integration
      • detailed planning –>agile planning
      • porting coded –>work on main branch
      • warranty
      • manual testing –>mostly automated testing
      • innovation capacity
    • integrating and getting it to the customer is by far the biggest opportunity for improvement
  • approach two –  focus on the areas that are typically the biggest sources of inefficiencies in most enterprise software development efforts: maintaining always-releasable code and your planning process
  • management needs to bring in their end-to-end view of the value chain and are best positioned to bring in  all the resources of the organization
    • create a culture of continuous improvement
    • define enterprise level iteration objectives
    • executives need to establish strategic objectives that make sense and that can be used to drive plans and track process at the enterprise level
      • should include key deliverables for the business and process changes
    • e.g. work with monthly iterations
    • establish metrics that enable tracking every objective down through the organization
    • the leadership team would then spend most of their days walking the floor trying to understand where we are struggling and why
    • become investigate reporters trying to understand what was working and what needed to be improved
    • …the engineers began sharing their ides and making their struggles visible because they realized that the executives where there to provide help and were willing to make changes
    • trust leads to more transparency leads to a greater ability to fix issues quickly

About planning

  • with detailed plans for software development we rob ourselves software’s greatest benefit (compared to hardware): flexibility 
    • with a relatively small investment in planning you can get a reasonable first pass at the plan (and do not start to invest even more in planning – as it will be inaccurate anyway)
    • once the code starts coming together, the teams learns a lot more about their assumptions and can better define their schedule for shorter time frames
    • trying to achieve more accuracy for planning eats up valuable capacity the organization could be using for identifying and delivering actual business value
Planning investment and accuracy of the estimation

Planning investment and accuracy of the estimation

  • it makes sense to capitalize on the flexibility of software and delay software feature decisions as late as possible so the commitment is based on the most current understanding of the market
  • (don’t lock the most flexible asset to commitments for features)
  • the planning process is used primarily to support different business decisions – so get clear about that decisions required and invest the least amount necessary to obtain enough information to support that decisions
  • ensure that the most important things are being developed first – otherwise individual teams are at risk of suboptimizing by developing capabilities they feel are important but are less important to the overall line of business
  • it needs a process of moving people or teams across initiatives to ensure the right overall business priorities are being developed first
  • build in flexibility with reserved capacity to respond to inevitable changes in the market and discoveries during development
  • measure requirements throughput by iteration
  • break down high level requirements in more details just when you know that they will get prioritized for development
  • don’t pile up requirements in a requirements inventory – and therewith minimize the risk of rework and investment in stories that will never get prioritized for development
  • you need to economically release small batches of new functionality to realize the full benefit of software
  • the objective of devops is to blur the lines between development and operations so that new capabilities flow easier from development to production – applying these concepts at scale is typically the source of the biggest breakthroughs in improving the efficiency and effectiveness of software development
    • CD tends to cover all the technical approaches and DevOps tends to be focussed on the cultural changes – DevOps as processes and approaches for improving the efficiency of taking newly created code out of development and to your customers

Five main objectives to consider for executives

  1. Improve the quality and speed of feedback for developers
    • as a real time process that helps them improve (rather than beating them up for making mistakes they don’t even remember)
    • if that feedbacks takes days or weeks it is of limited value to the developers learning
    • ensure that the feedback is coming form testing in an environment that is as much like production as possible
    • development and operatives need to use the same tools and automation to build, test and deploy
  2. Reduce the time and resources required to go from functionality complete to production
    • it requires automating your entire regression suite and implementing testing that can run every day to provide rapid feedback
    • aim for production-level trunk quality
  3. Improve the repeatability of the build, deploy and test process
    • it needs to be a structured, repeatable process – otherwise you end up wasting time and resources chasing issues you can’t find
  4. Develop an automated deployment process to quickly and efficiently find any deployment or environmental issues (before starting system testing and blurring the lines where and error occurred)
  5. Remove the duplication of work that comes from supporting multiple branches of similar code
    • branches are evil
    • check for branches that last more than a couple of days
  • have all teams continually integrating and testing their code in a production like environment
  • work on trunk – using versioning services (new version if it would break the exisiting functionality), re-architecture through abstraction, feature flags and evolutionary database design techniques
    • make keeping trunk stable the job 1 for everyone (it’s a cultural shift) – if you commit code you are required to stick around until the build was complete with automated acceptance tests passing green
    • don’t commit code into a broken build
    • introduce gated commits (if code does not pass a given quality gate it will not go to trunk … move offending code off the trackes) or auto reverts – to reduce binding capacity from the change management team
      • including proper information to affected changes for developers
      • keep flow on trunk and make it never blocking

Foundation

Clean architectures

That enable smaller teams to work independently in an enterprise and make it possible to find defects with fast running unit or subsystem tests

  • to quickly develop, test and deploy components of large systems independently
  • ideally clean and build the architecture first – in reality find a pragmatic balance between improving the development process and fixing the architecture
  • the challenge is to isolate the product variations so as much of the code as possible can be leveraged unchanged across the product line

Test automation

  • enable a fast feedback loop and provide an effective method for determining when to promote code forward in your pipeline
  • it requires strong coding skills and a devious mind to think about how to break the code
  • it requires proper time and attention
  • make it efficient to deal with thousands of tests on a daily basis
  • robust simulators and emulators that can be trusted to quickly find the majority of defects are a must
  • design tests for maintainability (just recording manual tests won’t work)
  • never ignore broken tests and ensure high test quality
  • best approach for automated testing – pair a really good development architec who knows the fundamentals of OO-programming with a QA engineer (see Cucumber & Cheese: A Tester’s Workshop by Jeff Morgan)
  • use a statistical approach to drive up tests passing rates and stability, group tests that are associated with different parts of code
  • design tests to quickly localize code issues (and don’t spend ages to find defects through unstructured and dirty test code)
    • ensure proper component based testing that fully exercise each component without relying on the functionality of other components
  • use test automation as documentation
    • use tools that focus on creating specifications in an executable form (see Specification by example e.g. using Cucumber)
  • build and the ability to manage different artifacts as independent components
    • break up build dependencies between different components
  • read Continuous delivery: Reliable Software Releases through Build, Test and Deployment Automation by Jez Humble and David Farley
    • CD approach to development and operations that automates as much of the configuration, deployment and testing process as possible and puts it all under a revision control
    • Increase build frequency  to force you to fix all the issues (otherwise the organization can still use brute force to work through these issues)
    • you need to know:
      • continuous integration
      • scripted environments
        • have every server configured of a common script that is under revision control
        • it ensures that every server has the exact same configuration
      • scripted deployments
        • to automate the deployment across all environments
        • ideally including a test of done deployments
        • also under version control
      • evolutionary database design
        • Refactoring Databases: Evolutionary database design by Scott W. Ambler and Pramod J. Sadalage
        • keep your application in an always releasable state – working with versions of columns instead of altering rows and columns
      • test automation
      • deployment pipeline and
        • defining how code is integrated into the system, deployed into different environments and promoted through various stages of testing
      • orchestrator
        • as a tool that coordinates all of this automation
        • enables you to call scripts to create new environments on a set of virtual machines with scripted environments, deploy code using scripted deployment, update the database using evolutionary database design, kick of automated testing and move ready versions to the next stage in your pipeline (up to production)
    • build automation scripts using an OO approach and avoid ending up with messy, big, long monolithic scripts
    • create a post deployment validation test suite to ensure the deployment was successful and use it on a server-by-server basis
      • use techniques to isolate code issues from deployment issues during the deployment process
      • use tests to ensure that the configuration has occurred correctly
      • checks must be done for every server and routing configuration

deploymentvalidation

  • let the pain of increasing the deployment frequency drive the priority of your technical changes – it will force you to fix issues in priority order
  • faste feedback with automated test developers can replicate locally to quickly identify problems and verify a fix before recommitting the code
  • more frequent build and test cycles means that fewer developers have committed code since the last build – ideally fully deployed and tested with every commit
  • testing layers
    • first step –  unit testing and static code analysis to catch as many defects as quickly as possible
    • second step – component or service layer testing at clean interfaces
    • final step – user interface based system testing (typically a few as they run slowly)
  • use a subset of tests for fast feedback before promoting the code forward to stages for more extensive testing
  • reduce the time window between test executions and localize issues to the smallest number of commits and developers
  • use service virtualization to break up large enterprise systems into manageable pieces – it allows independent development against interfaces and one can even have different development models (Agile, Waterfall,…)
  • define a set of build acceptance tests – defining the minimum level of stability that must be ensured for every build. If these tests fail, then getting them fixed is the top priority for the entire development organization and nobody is allowed to check in code until they are fixed. Recommended is to use your systems to enforce that behavior by automatically blocking code that does not pass the build acceptance tests.
  • aim for a loosely coupled architecture that enable small teams to independently develop, qualify and release code and work with much simpler deployment pipelines
  • the first step in improving stability and enabling smaller, more frequent releases requires understanding the work that occurs between release branch and production
    • have all information highly visible on one chart is very valuable to understand the characteristics of the project readiness
Visualize a trend of release readiness for your system

Visualize a trend of release readiness for your system

  • farming the build acceptance
Chart to visualize pass rates of build acceptance tests by component

Chart to visualize pass rates of build acceptance tests by component

  • identify the best candidate tests and promote them into the build acceptance test suite

Necessary cultural shifts

  • executives need to make sure they don’t let the magnitude of the overall change keep them from starting on improving the processes. Instead of worrying about the size of the challenge or how long it is going to take, executives juts need to make sure they start the enterprise level continuous improvement process.
  • necessary cultural shifts:
    • getting developers to own ensuring every check-in to trunk is stable in a production like environment as job priority one
    • getting development and operation teams using common tools and environments to align them on a common objective
    • the entire organization agrees that the definition of done at the release branch ist that the feature is signed off, defects-free, and the test automation is ready in term of test coverage and passing rates
    • getting the organization to embrace the unique characteristics of software and design a planning process that takes advantage of its flexibility

Further readings

DevOps on steroids

#NoEstimates approach