Your organization delivers software, so it should be obvious that improving your software delivery means improving your organization. But how to improve your software delivery organization? I’m going to tell you how, and you’ll be amazed at its simplicity: To improve an organization’s performance you need to improve all the components of the organization. And, you need to know how to improve each component.
An organization is made of…
An organization is composed of people, process, and artifacts.
An organization is made of three types of ingredients: people, process, and artifacts.
People
People includes you and your staff, the relationships among them and between them and others, and the culture that shapes these relationships. When we talk about “improving people,” we mean developing peoples’ skills and shaping the organizational culture.
Process
Process includes the path that work follows through the organization as value is added at each step along the way to the customer. Improving process focuses on optimizing this flow of value to the customer.
Artifacts
Artifacts are the inputs that flow into the processes, the outputs that result from them, and the tools and technologies utilized in processing.
A model for improvement
The Lean methodology for improvement: build, measure, learn, repeat.
The Lean Startup movement inspired a very effective model for rapidly improving in uncertain and risky business situations. By rapidly iterating through successive Build-Measure-Learn cycles, you systematically reduce risk and uncertainty. Each iteration through the cycle examines and eliminates the greatest remaining risk as the organization learns how to reduce its shape, size, and probability. Over time, the business carries less risk and delivers better products and services, with increasing precision (to the right customers), reliability (service quality), and control.
Improving the components of your organization
By applying the Lean model to each component of the organization, we can arrive at a prescription for improving all of them.
Build people
By “build people” we mean improving peoples’ skills and the overall culture. Several methods are effective:
Model new behavior. Leaders adopt, demonstrate, and reward the new behavior. This is the quickest way to change culture.
Exercise the new skills. This is the most effective way to acquire new skills.
Train.
Provide a mentor.
Measure people
The extent to which a person’s skills have delivered the expected results can be measured by:
Review by others.
Test.
Learn what to improve next in people
By “learn” we mean identifying the next area that needs improvement. Two ways are effective:
Conduct retrospectives and blameless postmortems.
Analyze the results of the reviews and the testing (performed in the “Measure” step).
Build process
To build—that is, to improve—process you mercilessly simplify it. For example:
Reduce redundancy, handoffs, and re-work.
Reduce batch sizes and cycle times so less work is in progress but is completed more quickly.
Increase visibility so the current status of all work is obvious.
Empower people to use their judgment.
Shift the drudgery to automated tools.
Process improvement cannot be delivered in isolation, however. The people and the artifacts needed to support the process must be developed in order for the process to have the desired result.
Measure process
To measure the extent to which a process has achieved its desired result, audit the process. Look for evidence that the process was followed and that the desired results were obtained.
Learn what to improve next in process
To identify the next area of process improvement to pursue:
Conduct retrospectives and blameless postmortems.
Analyze the results of the audit performed in the “Measure” step.
Build artifacts
To build (improve) your inputs, tools, and outputs:
Reduce variance.
Increase tolerances.
Raise standards (i.e., decrease margin of error).
Measure artifacts
To measure the extent to which your inputs, tools, and outputs have improved:
Test (ideally automated).
Survey customers, internal and external.
Learn what to improve next about artifacts
To identify the next area of process improvement to pursue:
Conduct retrospectives and blameless postmortems.
Analyze the results of the tests and surveys performed in the “Measure” step.
Applying it
Try the above model for improving your software delivery organization. Like my other clients, you will see your software delivery performance improve dramatically. Contact me if you need help.
You may have heard that the goal of DevOps is to rapidly deliver and reliably operate high quality software-based services. But that is only partially true. True, that is the desired result. But fostering the habits that produce that result consistently is integral to the goal.
Habits foster results. The goal is both.
In my work with clients I have found several cornerstone habits essential to creating DevOps results, and I have helped software organizations embrace these habits and earn the payoffs. Let’s explore the cornerstone DevOps habits and why these habits are as much a goal as the results they promote.
Habits come in two flavors, and the first flavor of habits is practical.
Habits of Practice
Habits of practice are the behaviors we routinely engage in, often without thinking about it. We buckle our seatbelt when sitting in a vehicle, or we stash our wallet in the same pocket or the same place in our purse. What today is an unconscious habit once began as a conscious choice to behave in a certain manner. Perhaps we were reminded by our parent to fasten our seatbelt, and we complied, consciously, until repetition forged an unconscious habit. Perhaps we were alarmed at the thought that we would forget our wallet or keys someplace, so we patted down our pockets, consciously, like Peter Falk in the final scene of The Princess Bride, until that, too, through repetition, became an unconscious habit. Whether conscious or unconscious, these habits of practice create the results that we are safer while sitting in the car, and that we do not forget our wallet.
Habits of practice lead to results.
Habits of practice create results. In DevOps, we want to deliver high quality software-based service rapidly. The habits of practice that create these results include, generally, automation, measurement, and sharing.
The particular practices of these general categories vary with each organization, as each has its own business situation to contend with. But it is impossible to deliver high quality, reliable service without these practices, especially as the scale of the operation grows to significant size.
We want to foster these habits of practice. In order to successfully develop them we need the other flavor of habits.
Habits of Mind
Habits of mind are the manners of thinking and the internal monologue we conduct inside our own heads. Just as our behavior can be habitual, so can our thought patterns. Habits of mind are important because they can create and change habits of practice. For example, we all know the feeling of frustration when something in our lives isn’t working for us. It grates at us, it taunts, and it refuses to budge. Frustration is a sign that we need to change something about our lives. Our internal monologue upon encountering frustration influences whether we will change and what kind of change we will make in response. When our thoughts reflect passivity, I always have bad luck, we are not motivated to change. When our thoughts are empowering, this is an unfortunate but temporary setback, we are moved to change something about our situation—to try again, to try in a different way, or perhaps to let off steam and simply get over it.
In order to consciously create a new habit of practice, you need to have certain essential habits of mind. The essential habits of mind drive you to change your behavior over time, until you have improved your habits of practice. Without the supporting habits of mind, habits of practice are created by default, without our awareness, and these default practices seldom are the most effective at reaching our desired goal. Even once you have changed your practice, you can only improve by honing those practices, and the drive to improve your practice stems from your habits of mind.
Habits of mind foster habits of practice, which create results.
In short, habits of mind create habits of practice create results.
Culture drives the practices of automation, measurement, and sharing, which yields DevOps results: High quality, reliable software-based service.
In the DevOps field the factors that influence practice have been collectively termed “culture.”
But culture is an opaque term that does not lend itself to scrutiny. It is more useful to consider the habits of mind that foster habits of practice.
Shlomo’s Four Cornerstone Habits of Mind for DevOps Practices
The four cornerstone habits of mind necessary to foster DevOps practices and DevOps results are agency, awareness, respect, and healthy selfishness. Read on for more details on each of these habits of mind.
Agency
Agency means thinking that you can change things, that you can improve your lot through your actions, that you are an agent in your own life. If you believe you are able to change things then you can summon the motivation to try to change your behavior. On the flip side, if you lack agency—if you believe nothing you do will change anything—then you will be disheartened and won’t bother taking any of the steps necessary to change behavior. The more agency is ingrained as a habit of mind, the more you will feel in control and able to change your behavior.
This is a cornerstone habit of mind for DevOps because DevOps habits of practice require constant refinement, and you must believe that you can change and improve the current situation in order to do so. In an organizational setting agency is sometimes called “empowerment.” Organizations, too, cannot improve their practices unless members believe that they can change the way things are done. This is why agency as a habit of mind is so important for cultivating DevOps habits of practice.
Awareness
Awareness is the mental habit of perceiving what is around you. We are not aware of everything that transpires in our lives, and often we do not recognize that our practices can be improved. We cannot change our habits of practice if we aren’t aware that they need changing. This is true in life in general and with DevOps in particular.
Respect
Respect means acknowledging the feelings and opinions of others. Organizations consist of, and exist to serve, individuals with varied feelings and opinions. If you want to change habits of practice you must acknowledge the feelings and opinions of others who are involved. This doesn’t mean you need to feel those feelings yourself; it only means you need to believe that those feelings exist and are valid.
Healthy Selfishness
Healthy selfishness is thinking of your interests so that you can help others. It means thinking about how to overcome your obstacles to success and enabling yourself to do the same for your teammates. It means wanting to make your world better so you can make the world at large better. It is the driving force behind economic and technological progress. And healthy selfishness is essential for DevOps because it is the motivating force behind improving habits of practice.
What About Empathy?
Empathy is the ability to understand and share the emotions of others. It requires lowering your emotional defenses and allowing yourself to be vulnerable to the stirrings of another’s heart. Within the context of a supportive friendship or life partnership, empathy is vital. Understanding and sharing your partner’s emotions is essential to support them through pain and adversity and to grow together with them—in fact it is the essence of such a long-term relationship.
But empathy’s importance for DevOps has been drastically overstated. At our jobs we do not need to open our hearts to feel pain. Demanding that someone exhibit empathy evokes alarm and raises their defenses. Even professionals who do directly confront emotional pain, the psychologists, counselors, and ministers, do not allow themselves to be moved by the emotions they are exposed to in the course of their work. They maintain their emotional defenses. For the rest of us who do not deal directly with pain at work, which includes everyone in the IT and software industry, respect—acknowledging feelings and opinions of others—is enough for the professional setting. Insisting on empathy at work is irresponsible.
The Habits Are the Goal
DevOps is the continuous improvement of practices to rapidly deliver and reliably operate high quality software-based systems. Continuous improvement means regularly, consciously, and intentionally developing and honing those skills. In other words, DevOps is more than just the goal of delivery and operations. It is also the habits—of mind and of practice—leading to continuous improvement.
Shlomo’s cornerstone habits of mind, agency, awareness, respect, and healthy selfishness, foster the habits of practice, automation, measurement, and sharing, that drive the results of DevOps: high quality, reliable software-based service.
Fostering the three DevOps habits of practice—automation, measurement, and sharing—requires the four cornerstone habits of mind: agency, awareness, respect, and healthy selfishness. In all, those are the Seven Habits of Highly Effective DevOps.
The classic formulation of DevOps, Culture, Automation, Measurement, Sharing—”CAMS”—is an expression of habits of mind and habits of practice: culture is a habit of mind, automation, measurement, and sharing are habits of practice. To succeed at a DevOps initiative, you and your team members must exhibit these four habits of mind. They are the keys to forming productive habits of practice and, ultimately, the results you want to achieve: rapidly delivering and reliably operating high quality software-based services.
Before you ask yourself “what can and can’t I do in the cloud,” stop to consider the larger picture. How will your customers know that you have adopted cloud computing? What resulting radical improvement will delight your customers? These questions help you focus on the impact you want to achieve with your initiative. The answers help you understand the facets that you’ll need to address – technology, skills, procedures, and behaviors – in order to achieve and measure that impact. Then you’ll be ready to talk nuts and bolts about vendors, tools, and cloud-appropriate workloads.
For example, my client, CIO of a media company, was enthusiastic about using cloud computing and wanted to get right down to details: what should he move into the cloud, how long would it take, who could help, and so on. After walking through the questions mentioned above we made several important discoveries:
External customers wanted a streamlined billing process.
Internal customers wanted to eliminate boring, error-prone manual work.
Neither group of customers cared what technology was used.
He didn’t know the effects of the current billing process on satisfaction, retention, and revenue.
It was clear that the project was more properly regarded as a customer retention initiative, not as a technology adoption effort. As a result, the CIO immediately knew what he had to do: measure the effects of billing on customer satisfaction, retention, and revenue; and he needed to recast the cloud computing adoption program as a program to substantially improve customer satisfaction. These guidelines provided the business context within which his staff was able to make intelligent, focused decisions about implementation details.
Next time you find yourself asking about what workloads to move to the cloud, think about the only thing that matters: what will delight customers?
Whether you operate IaaS, PaaS, or a SaaS service, watch out for these three pitfalls to the success of cloud offerings:
Encouraging the wrong thing. Measuring the success of the offering by looking only at utilization metrics damages sales. One cloud service provider measured the success of its IaaS service – and compensated its sales force – based on virtual machine usage. Little surprise that, for all the years that this practice continued, the provider could not land strategically important accounts. Like in any mature sales operation, your prospective cloud service clients should be evaluated based on several criteria, including:
strategic fit
urgency
reputation
visibility
cost of sale
short-term sales potential
long-term sales potential.
Make sure you evaluate all these elements, and that you prioritize your sales and marketing efforts toward prospects with the right combinations of all these factors.
Ignoring the relationship. On-demand and pay-as-you-go are not simply usage models; they are components of a new type of customer relationship based on transparency and immediacy. Make sure your customer relationships allow for:
Seeing usage and billing easily, and up-to-the-minute.
Setting custom notifications and enforceable consumption limits based on both usage and cost.
Getting quick notice of service degradations and their ongoing status.
Being reassured after service disruptions that you have identified the cause and taken steps to prevent a reoccurrence.
Your service may have other ways to build a customer relationship based on transparency and immediacy. Identify and exploit them.
Focusing on infrastructure reliability. Selling a cloud IaaS service that differentiates with higher level of service reliability is not sustainable; instead high-SLA workloads must be built using cloud-appropriate designs. Any IaaS provider who tries to compete against the commodity non-reliable infrastructure clouds by offering a higher SLA engages in an unwinnable battle: the cost structure will make it impossible to compete against commodity offerings. As computing services become more commoditized, previously established patterns of infrastructure usage (aka “best practices”) that called for highly reliable hardware and “N+1” architectures are architecturally moot. Following those patterns in the cloud will not increase overall workload reliability. Instead, workloads that use cloud infrastructure need to be designed for scale-out and rapid recovery from infrastructure failure, providing overall workload reliability via software. The overall cost of reengineering workloads for cloud-appropriate architecture will, in the end, be less than the cost of building and operating a cloud that delivers a “classic” hardware-based HA.
Don’t make these mistakes. They could be killing your cloud business.
Building and operating a private cloud ourselves is simple, right? Give our experienced IT folks a budget for hardware and software, a clear mandate, and deadlines, and we’re likely to succeed. That’s what we’ve always done, and it’s worked out tolerably well. Private cloud isn’t so different from running our own virtual machine environment, and we do that already, so why wouldn’t we succeed? I hear this sentiment expressed often so it seems to be a widely held belief. But it’s wrong precisely 97.73 percent of the time – I measured. Unfortunately, too many organizations act on this mistaken belief and experience unnecessary hurt as a result.
My good friend Keith Hudgins made this point cogently in his presentation at the CloudConnect conference in Chicago. Keith is one of the few people who really knows what it takes to build and run a cloud: he was part of the core team with me building KT’s public cloud service in South Korea, and he is currently Senior Cloud Engineer at enStratus where he helps enterprises get their private clouds to work properly. Keith’s presentation was sobering for the audience members, several of whom said they plan to reevaluate their private cloud initiatives as a result.
If you want to succeed at your own private cloud you need to consider these four critical aspects: your users, your competition, your skills, and your technology. Know these four and you have a fighting chance to make it. If not, get help – or prepare for disappointment.
Know your users
Different users need different applications, and your cloud must be designed to support the required applications. Private clouds might have three different types of users:
Regular desktop users, the kind of user most likely found in your Finance, HR, Sales, and Marketing organizations, will need your cloud to support the applications your IT department currently supports already: CRM, accounting, ERP, and business applications your IT department is already familiar with.
Software R&D staff will need your cloud to support the development environment, tools, and deployment processes they’re already using. This group will also likely need training to properly architect and operate reliable applications in a distributed environment.
IT staff will need your cloud to support core capabilities such as backup, directory management, and file sharing.
Ask your users what applications they are using and what they need the cloud to support.
Know your competition
Odds are your users already use public cloud services, and – like it or not – your own private cloud will be competing against those public alternatives. How well do public cloud services meet your users’ needs?
How reliable is their service? How does this compare to your own historical reliability?
How easy are they to use? Do public cloud service customers need multiple levels of corporate approval to acquire resources, or is the ordering and fulfillment process seamless and extremely short – a few minutes at most from order to delivery? Can you deliver the same or better usability?
How good is their support? Are there multiple sources for expertise and communities where users can share solutions and experience?
How rich is their ecosystem? How easy is it to employ a vast variety of tools and seamlessly integrated value-added services?
Determining how well your users’ needs are met by existing public cloud alternatives will give you a good idea of what you need to provide in order to achieve successful adoption of your private cloud.
Know your skills
The skills required to build and operate a cloud are not taught in classes, they’re learned by experience. These include:
System administration. Okay, this one is taught in classes, but the issues you’ll experience in a cloud environment are much more complicated and at a larger scale than you will normally encounter elsewhere. Keith suggested that OpenStack and Eucalyptus will be a challenge for your team if they lack linux administration skills.
Storage management skills are critical because storage is the most complicated technical aspect of a cloud. Keith emphasized the importance of NAS and iSCSI setup and management experience.
Networking is a key element of the cloud and networking expertise must be a part of your core cloud engineering team.
Automation will dictate how quickly you can recover from problems, so you must be experienced in its use and integrating automation tools into your workflow. Identical inputs (that is, identical hardware revisions) must always yield identical outputs, and that is only possible at a large scale with automation. Keith pointed out that automation takes time to produce. Make sure your team is proficient with the automation tools that work best with your technology choices.
Get these skills if you don’t have them. They will influence your choice of hardware and software.
Know your technology
Cloud is powered by technology, so you’ll need expertise in at least the following four technical areas to build and run it yourself.
Cloud Management Software is the layer that provisions cloud guests and sits between the API and the hypervisor. If you can’t explain the differences between the four major cloud management software options – CloudStack, OpenStack, Eucalyptus, and OpenNebula, then you do not have the expertise to choose the right one for your needs.
Storage, Keith pointed out, is by far the most complicated technical aspect to get right in a cloud. Local guest storage is the simplest to build and operate, but limits the range of applications that can be easily supported on the cloud. Attached storage increases flexibility at the cost of complexity. And object storage requires lots and lots of disks. Your options will be influenced by your choice of cloud management software, and any change to the default configurations will require serious development and testing to get it right.
Hardware and the way you manage it is the most important choice you can make. Of all the components in a cloud the hardware is what is going to break, so the ease with which you can replace hardware will dictate the maintenance headache. Keith provided some hard-won tips: make sure your hardware vendor can supply identical hardware down to the BIOS revision, create (automated) procedures to verify or reject hardware as it is delivered, and diligently avoid hardware inconsistencies since they greatly increase the complexity of the endeavor.
Your choices in the above three technology categories should be guided by your needs, but remember: you can only choose wisely if you have appropriate expertise in these technologies.
Do you have what it takes?
The above four aspects are critical to successfully building and operating your own private cloud. There are other success factors – know your strategy, know your business model, and use effective project management – that are part of any successful technological change. But the above four are critical in any private cloud initiative. So, how do you rate on these aspects?
Odds are, you don’t have the expertise required to do it yourself.
Like companies producing any kind of offering, software companies require three elements in order to successfully deliver: knowing what to build, knowing how to build it, and knowing how to deliver it to the customer. When each of these three functions – research, engineering, and delivery – does its job independently, you get slow progress and uninspired, sub-par products. But when all three functions work together in a customer-centric fashion, you get excellence – in implementation, innovation – in concept, and world-class results – by delivering those two to the customer.
With the explosion of internet-based services in the past decade delivery has by necessity become an ongoing activity. And so in recent years much attention has been devoted to integrating development with technical operations – DevOps. DevOps integrates into a single team the skills of how to build the service and the skills of how to deliver it. But, as pointed out earlier, DevOps teams often lack the most critical element: the customer’s involvement. DevOps teams can build and deliver really fast, but don’t know what to build.
Integrating development with the customer-facing functions (marketing, sales, etc.) without technical operations results in an organization that knows what to build and how to build it, but doesn’t know how to deliver it to the customer. This organization is full of ideas – even innovative ideas – and implements them, but they don’t see the light of day. I call this a “feasibility practice” – it can show you what is feasible but it can’t deliver it to customers.
Teams that know what to build and how to deliver it to the customer, but not how to build it, also do not provide value to the customer. Such teams are close to the customer both on the inbound (understanding the customer’s needs) and the outbound (explaining the company’s offering) directions. Marketing sits squarely in this space: it knows what to build and how to deliver but can’t transform that knowledge into value for the customer.
As an executive leading a software delivery effort, make sure your organization occupies the very center so it can understand, build, and deliver the right value to the customer. The customer is here in the center as well: he participates in validating the ideas, critiquing the implementation, and accepting the delivery. It is only here that your organization can achieve excellence, innovation, and world-class results.
Traditional descriptions of cloud computing and the various cloud operating models – IaaS, PaaS, SaaS – focus on the locus of responsibility for various layers of the system: facilities, network, storage, servers, etc. But these descriptions typically omit a critical element. Can you spot it in the diagram below?
The missing layer is the only layer that really matters – the customer. Ensuring that your application can actually be consumed – that delivery can be successful – is a critical part of providing your value. Your application’s facilities, network, and storage may be running properly, but will your customers actually benefit? Without a properly staffed, trained, equipped, and managed operations team, your service won’t last long enough for customers to care. If your customer is important, then so is your operations team.
