Category: Interview Series

In our recent conversation with Larry Maccherone, we explored the significance of Data, Metrics and DevSecOps. He shared his journey from software engineering metrics to adapting Agile. Larry has spoken in length about DevSecOps Transformation in Scale.

Lets’ read his thoughts;

Q1. Please share your journey from software engineering metrics to adapting to Agile.

Larry: It’s definitely been a journey but I wouldn’t pick those endpoints. I launched my first startup while I was still an undergrad in engineering at Virginia Tech. I grew it to 80 employees/$20M per year in sales. My first big learning was that while talent and passion is enough for a founder, you need a system once you get beyond founders. Even the best engineers do better with a system. In this sense, Continuous Improvement is a system. Spiral is a system. CMM and CMMI are systems. RUP is a system. XP is a system. Lean and Agile are systems. DevOps is a system. They all become outdated because the context (usually technical but not always) changes. Often the cost of something that was expensive approaches zero, like what the internet has done to the cost of delivering the actual bits to end-users. Systems identify what is valuable. It used to be that not having to send out technicians was valuable. Now, it’s much much more valuable to make sure you have product market fit. Agile captured this shift in value, among others. DevOps and DevSecOps crystalize another building upon Agile like Lean and Agile built on those that came before.

Q2. Data is the key to make well-informed decisions. What do you think about – let data be your guide in an Agile environment?

Larry: I did write that better measurement leads to better insight, which leads to better decisions, which leads to better outcomes. However, I also say that you should value qualitative insight that comes from your experience when making decisions.

Q3. Please define business value, metrics, and measurements. What should come first as per your experiences?

Larry: Business value is whatever will provide value to your business. It’s exceedingly difficult to measure. The words metrics and measurement are essentially synonyms in my opinion.

Q4. Forecasting is of great significance for any organisation. Do you have any surefire way of forecasting, that you can suggest to us?

Larry: My strongest message around forecasting is that you shouldn’t deliver a single date. Rather, you should deliver a probability distribution along the lines of, “we’re 95% sure we can deliver it by this date and 75% sure we can deliver by this date. However, if these identified risks hit, that will change to this date and this date.”

Q5. In recent times you have spoken about DevSecops Transformation in Scale. Please share your views on it.

Larry: Wow! I have a day long workshop worth of material now in DevSecOps. Let me hit the highlights. I think of DevOps as the successor to Agile. Among other things, Agile was able to break down the silos between the development team and QA on one end and Product Management/Business Analysis on the other end. DevOps continues that by breaking down the silo with Ops. DevSecOps adds Security to the mix. The idea is to have fully-empowered cross-functional teams. The definition of fully-empower cross-functional just keeps expanding.

The “Transformation” part of DevSecOps Transformation is essentially the same as Lean/Agile Transformation. You are trying to get new/improved practices/ceremonies/artifacts/mind-set shifts into the culture. The practices/etc. Just cover different subjects. 

For DevSecOps transformations, I follow the same approach as I used for large-scale Agile Transformations. First start by assuming the team wants to do the right thing. They really just need two things for that to happen: 1) They have to know what the right thing is and really internalize the value of it; and 2) The “right thing” needs to be the easy thing. 

For DevOps and DevSecOps there are often technical hurdles that can be solved for the entire organization that help with #2 so DevSecOps Transformation has a lot more technical enablement than most Agile Transformations. For instance, standing up a well-managed, sufficiently-resourced (memory, disk, CPU), multi-tenant CI/CD pipeline tool so teams don’t have to stand up their own. But this pendulum swing is also the normal progression. XP focused mostly on engineering practices. Agile that followed mostly on people and process. Looking back at history of software engineering evolution, you see this pattern over and over again. A technical-focused improvement that triggers a people/process oriented improvement, which is followed by a technical one, on and on.

OK, enough background. When you add at “… at scale” to the conversation things get very interesting. I have a three part framework that accomplishes this sort of large cultural transformation. It was proven doing Agile Transformations and I have evolved it and am using it for a large scale DevSecOps Transformation at Comcast. The three parts are: 1) Win the hearts and minds of developers. You do this with a set of guiding principles that align with their existing mindset but convinces them what the next step of improvement looks like… and using my Trust Algorithm (more on that later); 2) A shallow team-level improvement ramp. Make it easy for them to know what’s the next most valuable thing to improve. Conduct self-assessment workshops where you describe the ideal practices/process/culture and have the team say how close to that ideal they are. Visualize this in a way that tells them how far along they are towards this ideal. Then ask them to make a plan for some small set of improvements they will make in the next 90 days. Follow up every 90 days; and 3) Management visibility and goal setting. Get executive sponsorship by rolling up the visualization of team improvement opportunities and getting them to push the expectation down into their part of the organization.

Q6. It is that believed traditional cybersecurity practices do not provide actionable feedback to the developers. How does DevSecops approach excel here?

Larry: There is a massive lack of trust between security and development. Not sure that’s the cause or the result, but we also have a pattern where security thinks of their roles as gatekeepers and the problem largely remains with them. Developers sense this and have become comfortable with some other team “bolting security on” at the end. The key here is to get developers to “build security in” which involves them taking owner of the issue. You start down that path by having the security organization take this pledge…

We, the Security Team…

• Trust that Engineering Teams…
  • Want to do the right thing
  • Are closer to the business context and have the responsibility to make trade-off decisions between security and other risks
• Pledge to…
  • Provide information and advice so those trade-off decisions are more informed
  • Lower the cost/effort side of any investment in developer security tools or practices
• Understand that…
  • We are no longer gate keepers and police officers but rather tool-smiths and advisors

Talking about tools for a second…

Security has a tradition of spending big on security tools. Developers get their development tools mostly for free. So, the majority of security feedback tools target security groups. I like to say that putting “DevOps” lipstick on your tradition security tool pig does not make it DevSecOps. Some vendors are doing it right with great integration into the CI/CD pipelines that drive the DevOps team’s daily workflow and a federated model that provides high-level visibility with local flexibility. Others not so much. It’s a tricky chasm for many of them to cross because they don’t want to harm their relationship with their existing customers (security groups) while targeting a new group (development)… and the budgeting question is really hard. We’ve made it easier at Comcast by paying for DevSecOps tools out of security’s budget while putting it directly in the hands of developers.

Q7. Where do you see the Lean/Agile community heading in the next 2 years?

Larry: Business agility is what most agile leaders are now talking about but I’m more technical so I’ll emphasize DevOps and DevSecOps. Breaking down the silos of development teams vs ops or security. There are no dedicated QA departments anymore. Those folks all found other jobs or now work directly on a development team. But go back 12 or so years ago pre-agile and you had such dedicated departments. I see a similar thing happening with ops and security over the next decade.

Q8. One message for our readers…

Larry: Let me finish with my Trust Algorithm. The primary goal of a DevSecOps initiative is to get development teams to make certain mindset shifts and adopt security practices into their daily activities, which simply cannot be done without healthy collaboration and mutual trust. There’s the rub — there is typically a massive lack of trust between the security group and development teams, particularly at larger, more traditional organizations. It goes something like this:

Security people: “Those darn developers are cranking out crap that’s going to get us hacked!”

Developers: “Security is nothing but an obstacle. They don’t understand that we have lots of other concerns and the only ‘help’ they provide is to brow beat us.”

There are a lot of ways to explain this phenomenon, but it all boils down to the trust algorithm.

Where:

• Credibility = How well you actually know what you are talking about
• Reliability = How often and quickly do you do what you say
• Empathy = How much you show that you care about someone else’s interests
• Apparent self-interest = How apparent it is that your words and actions are in your own interest

The idea is that you want to maximize the terms in the numerator and minimize the term in the denominator. There is always some Apparent elf-interest to the denominator is never zero but the key is to emphasize shared interest like the company’s stock price. One way to maximize two of the terms in the numerator, Credibility and Empathy, is to make sure that you have developers working for the security organization (rather than security professionals) speaking with developers on the engineering teams rather than security professionals. Reliability is the same in any context. Do you do what you say you are going to do?

All of the elements of a high performance organization depend upon disciplined innovation says Kirk Botula, Founding CEO, CMMI – global leader in the benchmarking and elevation of organizational performance.

It had been an enriching experience to speak with Kirk on interesting topics of high-performance organisations.

Q1. How do you like to get introduced to the uninitiated?
Kirk: I am Kirk Botula, I was the founding CEO of the CMMI Institute. I am a serial entrepreneur with expertise in the commercialization of technology and  growing and scaling high performance organizations.

Q2. We all talk about continuous improvement and delivering quality. How do we know if we are improving?
Kirk: The only way to know if you are improving is through measurement. They key is to make sure your measures are relevant. 

Q3. What is the key difference between Maturity Levels and Capability Levels, when we speak about Capability Maturity Model Integration (CMMI)?
Kirk: In simple terms we think of maturity as applying to the whole organization as an organic system and capability levels as a functional thread within that system. 

Q4. Briefly explain the characteristics of high-performance organisations.
Kirk: There are a few characteristics of a high performance organization. First, it’s products and services are generally clearly differentiated in the marketplace and have a high degree of fit with customer needs. Generally a good indicator of this is a high Net Promoter Score. Secondly, the organization will deliver with a high degree of capital efficiency generally exhibited by better margins than their peers. Third, they will have high degrees of employee engagement and trust and can attract better talent and pay better. Fourth, they will operate transparently in a fact-based manner. Lastly, their key operating cycle times whether it is product or service development are faster than their peers. This means, for example, that for every two major releases a competitor does, the high performance organization can do three. Consequently, they are able to put their competitors in a reactive position. Another analogy would be playing a game of chess where one player is allowed to make two moves for every move their opponent makes. I think another good analogy of a high performance organization is a professional sports team. Individuals master their own skills and work to measure and improve their individual performance. Goals are clear. Everyone knows the playbook. Coaching supports the team. You keep score and win.

Q5. What are the three major threats of changing global landscape to the growing businesses?
Kirk: Technology is always a potential source of disruption to businesses. Regional barriers to entry still exist, but it is easier today for small companies to operate globally very rapidly. I tend to think of these kinds of threats more as opportunities. The high performance organization can accelerate innovation and always be working to disrupt itself thereby setting the pace for the whole market.

Q6. According to you, what is the significance of stable processes, governance and organisational structure to become an Agile organisation?
Kirk: All of the elements of a high performance organization depend upon disciplined innovation. If you do not have stable, measured processes, you have no basis upon which to know whether you are getting better or worse. The agile movement is really just the application of risk-based experimentation and validation to a company’s product lifecycle. I think this is most obvious in the Lean Startup movement where product management has become increasingly focused on the design of experiments to validate product market fit. 

The problem that “agile” organizations run into is that scrum deployed by itself to a single work unit is simply a local optimum. People are confused by the feeling that they are spinning their wheels and getting nowhere. As Eli Goldratt taught in the Lean movement, you have to optimize the throughput of the whole systems which means that agile teams have to work cross-functionally.

Q7. Share a tip to gauge organisational capability?
Kirk: The best way to initially gauge organizational capability is the lightweight CMMI evaluation appraisal that allows a trained individual to do a high level gap analysis of the organization. It is a great starting point for anyone’s journey to high performance.

Q8. A message for our readers to reduce the risk of a failed system.
Kirk: Design programs to drive out risk in order of likelihood and severity. For example, if the greatest risk to your next product is that no one will buy it, you should be trying to find customers to commit to it before bothering to spend anything on development. If the greatest risk is technical, you should prototype in order to drive out the technical risk.