Author: Noopur Pathak

In our recent conversation with Larry Maccherone, we explored the significance of Data, Metrics and DevSecOps. He shared his journey from software engineering metrics to adapting Agile. Larry has spoken in length about DevSecOps Transformation in Scale.

Lets’ read his thoughts;

Q1. Please share your journey from software engineering metrics to adapting to Agile.

Larry: It’s definitely been a journey but I wouldn’t pick those endpoints. I launched my first startup while I was still an undergrad in engineering at Virginia Tech. I grew it to 80 employees/$20M per year in sales. My first big learning was that while talent and passion is enough for a founder, you need a system once you get beyond founders. Even the best engineers do better with a system. In this sense, Continuous Improvement is a system. Spiral is a system. CMM and CMMI are systems. RUP is a system. XP is a system. Lean and Agile are systems. DevOps is a system. They all become outdated because the context (usually technical but not always) changes. Often the cost of something that was expensive approaches zero, like what the internet has done to the cost of delivering the actual bits to end-users. Systems identify what is valuable. It used to be that not having to send out technicians was valuable. Now, it’s much much more valuable to make sure you have product market fit. Agile captured this shift in value, among others. DevOps and DevSecOps crystalize another building upon Agile like Lean and Agile built on those that came before.

Q2. Data is the key to make well-informed decisions. What do you think about – let data be your guide in an Agile environment?

Larry: I did write that better measurement leads to better insight, which leads to better decisions, which leads to better outcomes. However, I also say that you should value qualitative insight that comes from your experience when making decisions.

Q3. Please define business value, metrics, and measurements. What should come first as per your experiences?

Larry: Business value is whatever will provide value to your business. It’s exceedingly difficult to measure. The words metrics and measurement are essentially synonyms in my opinion.

Q4. Forecasting is of great significance for any organisation. Do you have any surefire way of forecasting, that you can suggest to us?

Larry: My strongest message around forecasting is that you shouldn’t deliver a single date. Rather, you should deliver a probability distribution along the lines of, “we’re 95% sure we can deliver it by this date and 75% sure we can deliver by this date. However, if these identified risks hit, that will change to this date and this date.”

Q5. In recent times you have spoken about DevSecops Transformation in Scale. Please share your views on it.

Larry: Wow! I have a day long workshop worth of material now in DevSecOps. Let me hit the highlights. I think of DevOps as the successor to Agile. Among other things, Agile was able to break down the silos between the development team and QA on one end and Product Management/Business Analysis on the other end. DevOps continues that by breaking down the silo with Ops. DevSecOps adds Security to the mix. The idea is to have fully-empowered cross-functional teams. The definition of fully-empower cross-functional just keeps expanding.

The “Transformation” part of DevSecOps Transformation is essentially the same as Lean/Agile Transformation. You are trying to get new/improved practices/ceremonies/artifacts/mind-set shifts into the culture. The practices/etc. Just cover different subjects. 

For DevSecOps transformations, I follow the same approach as I used for large-scale Agile Transformations. First start by assuming the team wants to do the right thing. They really just need two things for that to happen: 1) They have to know what the right thing is and really internalize the value of it; and 2) The “right thing” needs to be the easy thing. 

For DevOps and DevSecOps there are often technical hurdles that can be solved for the entire organization that help with #2 so DevSecOps Transformation has a lot more technical enablement than most Agile Transformations. For instance, standing up a well-managed, sufficiently-resourced (memory, disk, CPU), multi-tenant CI/CD pipeline tool so teams don’t have to stand up their own. But this pendulum swing is also the normal progression. XP focused mostly on engineering practices. Agile that followed mostly on people and process. Looking back at history of software engineering evolution, you see this pattern over and over again. A technical-focused improvement that triggers a people/process oriented improvement, which is followed by a technical one, on and on.

OK, enough background. When you add at “… at scale” to the conversation things get very interesting. I have a three part framework that accomplishes this sort of large cultural transformation. It was proven doing Agile Transformations and I have evolved it and am using it for a large scale DevSecOps Transformation at Comcast. The three parts are: 1) Win the hearts and minds of developers. You do this with a set of guiding principles that align with their existing mindset but convinces them what the next step of improvement looks like… and using my Trust Algorithm (more on that later); 2) A shallow team-level improvement ramp. Make it easy for them to know what’s the next most valuable thing to improve. Conduct self-assessment workshops where you describe the ideal practices/process/culture and have the team say how close to that ideal they are. Visualize this in a way that tells them how far along they are towards this ideal. Then ask them to make a plan for some small set of improvements they will make in the next 90 days. Follow up every 90 days; and 3) Management visibility and goal setting. Get executive sponsorship by rolling up the visualization of team improvement opportunities and getting them to push the expectation down into their part of the organization.

Q6. It is that believed traditional cybersecurity practices do not provide actionable feedback to the developers. How does DevSecops approach excel here?

Larry: There is a massive lack of trust between security and development. Not sure that’s the cause or the result, but we also have a pattern where security thinks of their roles as gatekeepers and the problem largely remains with them. Developers sense this and have become comfortable with some other team “bolting security on” at the end. The key here is to get developers to “build security in” which involves them taking owner of the issue. You start down that path by having the security organization take this pledge…

We, the Security Team…

• Trust that Engineering Teams…
  • Want to do the right thing
  • Are closer to the business context and have the responsibility to make trade-off decisions between security and other risks
• Pledge to…
  • Provide information and advice so those trade-off decisions are more informed
  • Lower the cost/effort side of any investment in developer security tools or practices
• Understand that…
  • We are no longer gate keepers and police officers but rather tool-smiths and advisors

Talking about tools for a second…

Security has a tradition of spending big on security tools. Developers get their development tools mostly for free. So, the majority of security feedback tools target security groups. I like to say that putting “DevOps” lipstick on your tradition security tool pig does not make it DevSecOps. Some vendors are doing it right with great integration into the CI/CD pipelines that drive the DevOps team’s daily workflow and a federated model that provides high-level visibility with local flexibility. Others not so much. It’s a tricky chasm for many of them to cross because they don’t want to harm their relationship with their existing customers (security groups) while targeting a new group (development)… and the budgeting question is really hard. We’ve made it easier at Comcast by paying for DevSecOps tools out of security’s budget while putting it directly in the hands of developers.

Q7. Where do you see the Lean/Agile community heading in the next 2 years?

Larry: Business agility is what most agile leaders are now talking about but I’m more technical so I’ll emphasize DevOps and DevSecOps. Breaking down the silos of development teams vs ops or security. There are no dedicated QA departments anymore. Those folks all found other jobs or now work directly on a development team. But go back 12 or so years ago pre-agile and you had such dedicated departments. I see a similar thing happening with ops and security over the next decade.

Q8. One message for our readers…

Larry: Let me finish with my Trust Algorithm. The primary goal of a DevSecOps initiative is to get development teams to make certain mindset shifts and adopt security practices into their daily activities, which simply cannot be done without healthy collaboration and mutual trust. There’s the rub — there is typically a massive lack of trust between the security group and development teams, particularly at larger, more traditional organizations. It goes something like this:

Security people: “Those darn developers are cranking out crap that’s going to get us hacked!”

Developers: “Security is nothing but an obstacle. They don’t understand that we have lots of other concerns and the only ‘help’ they provide is to brow beat us.”

There are a lot of ways to explain this phenomenon, but it all boils down to the trust algorithm.

Where:

• Credibility = How well you actually know what you are talking about
• Reliability = How often and quickly do you do what you say
• Empathy = How much you show that you care about someone else’s interests
• Apparent self-interest = How apparent it is that your words and actions are in your own interest

The idea is that you want to maximize the terms in the numerator and minimize the term in the denominator. There is always some Apparent elf-interest to the denominator is never zero but the key is to emphasize shared interest like the company’s stock price. One way to maximize two of the terms in the numerator, Credibility and Empathy, is to make sure that you have developers working for the security organization (rather than security professionals) speaking with developers on the engineering teams rather than security professionals. Reliability is the same in any context. Do you do what you say you are going to do?