The Business of IT

Where is the value?

Back in the early days of computers, or so the legend goes, when an entire room housed less computing power than in today’s iPhone, a large government agency had a serious problem: their computer system stopped working. Though it was only three weeks since the system had been certified, the startup sequence refused to complete, leaving the agency lacking critical operational capabilities. Work on the project ground to a halt while every engineer on staff checked and rechecked wiring, tubes, and relays to find the issue. But two weeks of troubleshooting amounted to nothing, and the Director of the agency was beside himself with frustration. He decided to hire the worldwide expert on such computer systems, a retired professor who was known for his sense of humor.

The professor arrived on site in a white lab coat, was briefed by the Director, and immediately asked to be given access to the computer room. Once there, he proceeded to walk around the room from cabinet to cabinet. The Director and his staff watched with increasing curiosity as the professor paused before each cabinet for a moment, scratched his beard, muttered under his breath, and moved on to the next cabinet. Finally, after forty five long minutes of this, the professor stood up more erect, smiled at the Director and his staff, withdrew a piece of chalk from his coat pocket, walked over to one machine cabinet and marked a large “X” on the side of the cabinet. “Your problem is in here,” he said confidently. And, sure enough, as the technicians opened up the marked cabinet, they discovered a subtle wiring issue that had been missed by all previous inspections. Gratefully, they fixed the problem and were able to resume operation. “Please, send me your bill – whatever it is, I’ll see that it is paid promptly,” said the Director.

The professor sent a bill to the agency, a one-line invoice for “troubleshooting: $1000”. Unfortunately, as government policy dictated, the Director needed a detailed itemized invoice, and he was forced to delay payment and request the proper details from the professor. The professor’s corrected invoice arrived, and it read as follows:

One piece of chalk, with which to mark the problem: $0.01
Knowing where to place the mark: $999.99

The real value to your customers is in knowing how to fulfill their needs. All the rest is a commodity.

The Business of IT

Wanted: Change. Generous Reward.

Simon Wardley recently satirized the typical response of corporate IT departments to technology change as follows:  Ignore, ignore, ignore, “no”, “no”, “I said ‘no,’ dammit”, “Oh no”, “Oh, f**k”. It reminds me of the classic Toys-R-Us TV commercial from the 1980s featuring children singing about not wanting to grow up. Resistance to change is not unique to children or to IT departments – it is a feature of every organization. How can you help your organization avoid being stuck, and instead drive change before it’s unnecessarily painful to “grow up”? The key is urgency. Growing up is a good metaphor for organizational change: both are normal, both feature resistance, and in both the stakes increase over time. Immature people, like immature organizations, do not reliably achieve their goals. And the longer that inertia dominates, the further behind they remain. Failure to adapt – getting stuck – can be fatal, as it was for the Eastman Kodak Company. In personal as in organizational growth the pressure to change – the urgency – is fostered by discomfort. To increase urgency you need to cause people to feel discomfort with the current state of affairs. John P. Kotter’s seminal work Leading Change offers several kinds of tactics to increase urgency:

  • Show that the present isn’t working: Allow a crisis to happen, publicize poor results, force encounters with unhappy customers, publicize lost opportunities.
  • Change the metrics: Create performance targets that are high enough and/or broad enough they can’t be reached with business-as-usual.

These tactics to increase urgency all work by fostering discomfort with the present. A client found himself spending inordinate lengths of time evaluating opportunities and procrastinating a decision on which ones to pursue. When I showed him how his lengthy decision process was limiting the number and quality of opportunities that presented themselves, he realized that he could grow his business significantly by streamlining this process. He saw the urgency of addressing the issue, we worked together to fix it, and his business has grown threefold since.

Creating a sense of urgency is the first step in a successful change program. Kotter describes further steps to a successful change effort, but urgency underlies them all.

In short:

Ouch! Now, change.

The Business of IT

Being Good and Being Known for it

The owner of a home-based baking business once described his marketing strategy to me as “my cookies speak for themselves.” True, they were delicious cookies. But he completely neglected marketing so his customer base never grew beyond friends and family, and his business failed as a result.

At a recent client meeting I was reminded of that baker. The CEO of a fledgeling tech business said to me, “our differentiating factor will be our superior service quality.”

“Wonderful!” I said. “So I’d expect to see significant investment in engineering and quality control. Is that the case?”

“Yes! Here is our R&D and quality program…,” and he showed me.

“That looks reasonable,” I said, after glancing at it briefly. “Tell me about your marketing.”

“Oh, we hardly need any!” the CEO replied. “Our customers will love our service quality so we don’t need significant marketing.”

I was immediately concerned. “Let me show you something,” I said to him. And I drew the following:

Being good and being known for it are both essential. New businesses begin in the lower left quadrant. At first there is no product so it’s not good, and nobody knows about the company. Here customers are neutral about your offering: if they knew more they’d know it wasn’t good – and if it was good they still wouldn’t know it. In the upper left quadrant you’re no good and customers know it, so they leave. In the lower right you’re good but nobody knows it, so you lose opportunities to do business. The goal is to be in the upper right quadrant, where you’re good and customers know it, so you grow.

A business will traverse the playing field as its quality and reputation change over time. Sometimes quality will suffer or improve, sometimes reputation will have ups or downs. The key factor in achieving success is the ability to nimbly correct course, heading always toward being good and people knowing it. Identify improvements and deliver them to your customers quickly – and make them aware of the improvements. You’ll need agile product development, minimizing the time between concept and delivery. You’ll need to measure your customers’ perception of your quality. You’ll need to empower every employee to satisfy the customer as quickly and completely as possible at every level. The net result of enabling quick course correction will be that you get good and you get known for it – and you grow.

Thankfully, this CEO was convinced by my explanations. He agreed that “being known for being good” is as important as “being good” and I helped him craft his marketing, development, and customer support efforts accordingly. The cookies will speak, and he will make sure people hear them.

The Business of IT

How I helped by refusing work

A very good repeat client introduced me to his own customer, CEO of a fast-growing company, with the words “you should talk to Shlomo – he does wonders.” When I get an enthusiastic referral like that – which happens often, as most of my business is via referrals – I move fast to see how I can help.

The CEO and I met and began to explore important areas of his business that I typically help with: usage of cloud resources, leveraging the data collected, streamlining processes and team dynamics. The discussion was progressing quickly – in fact, too easily, too rapidly – toward defining a potential project. I sensed something was wrong. He couldn’t describe why he wanted these things to get done; all he could say was, “this is the next issue we need to tackle.”

“Tell me what your rate is for doing these things and we’ll get moving immediately,” he finally said. In the past I would have jumped at the offer – what could be easier than business that practically lands itself in my lap after only thirty minutes of discussion? But, thankfully, I’m constantly learning from the past. My motto is “make new mistakes,” and I had learned the perils of undertaking work without comprehending the motivation behind it – without understanding the “why.”

“Joe,” I said, “I’m not going to work with you on any of that.” He stared at me in wide-eyed shock for a moment. Had I not known he was an elite long-distance runner, in top cardiovascular shape, I would have feared the worst. I continued: “You’re too close to the business to see what’s really going on and what areas need the most urgent attention. That’s why it’s hard for you to explain why these items are important. With your permission, I’ll lead you through a brief series of questions to help us both ascertain the areas of true priority for your organization. Then we’ll be able to jointly determine the best way to work together to achieve those priorities.”

And so we did. We began with these three questions:

  • How much of your own time is spent resolving day-to-day issues versus establishing and communicating direction and priorities?
  • If your business volume were to triple tomorrow, what about your organization would break?
  • If you were to ask your engineers, your support staff, your accounting staff, etc., what their biggest headache was, what would they answer, and would you be surprised by their answers – and why?

These questions helped Joe step out of his day-to-day role and think about the business from the outside, as a doctor would observe a patient. From this vantage point we observed the symptoms that the patient exhibited – the areas in which the organization was not healthy. There were many. We both agreed on the diagnosis: the underlying problem was the organization’s inability to take on new customers because its accounting functions could not handle the additional work. Then we discussed the course of treatment. Joe suggested introducing an automated billing system.

An automated billing system would certainly have alleviated the problems, but I sensed that there was still something more fundamental to explore. I asked Joe, “how can your billing system be directly beneficial to your customers? Why should they care?” Joe thought for a moment, and then he said excitedly, “you’re so right! We need to integrate billing into the customer interface!” Joe’s product is all about simplifying the use of multiple aggregated services, and therefore simplified billing is part of his product’s key value proposition. He realized that he was missing a core element of his product, one that would also enable the company to grow profitably.

We had come a long way from “help me implement these next few things we need done,” and were now discussing how to help his company achieve its strategic objective: to grow profitably. The tasks we had been discussing originally would not have contributed to this goal significantly, and Joe acknowledged that those immediate issues of reducing cloud usage costs and leveraging collected data were illusorily urgent. And he thanked me for pushing back on his request to tackle those issues.

Joe and I are currently discussing various options for helping him build his critical billing capability. He would not have realized the real priority, the true “why” he needs to address, if I had said “yes” to his initial proposals. And my involvement in helping him achieve strategic objectives is much more valuable, and much easier to demonstrate, than extensive work on non-strategic tasks. I prefer the former, and so does Joe, and all my other clients as well.

Cloud Developer Tips The Business of IT

How to Work with Contractors on AWS EC2 Projects

Recently I answered a question on the EC2 forums about how to give third parties access to EC2 instances. I noticed there’s not a lot of info out there about how to work with contractors, consultants, or even internal groups to whom you want to grant access to your AWS account. Here’s how.

First, a Caveat

Please be very selective when you choose a contractor. You want to make sure you choose a candidate who can actually do the work you need – and unfortunately, not everyone who advertises as such can really deliver the goods. Reuven Cohen’s post about choosing a contractor/consultant for cloud projects examines six key factors to consider:

  1. Experience: experience solving real world problems is probably more important than anything else.
  2. Code: someone who can produce running code is often more useful than someone who just makes recommendations for others to follow.
  3. Community Engagement: discussion boards are a great way to gauge experience, and provide insight into the capabilities of the candidate.
  4. Blogs & Whitepaper: another good way to determine a candidate’s insight and capabilities.
  5. Interview: ask the candidate questions to gauge their qualifications.
  6. References: do your homework and make sure the candidate really did what s/he claims to have done.

Reuven’s post goes into more detail. It’s highly recommended for anyone considering using a third-party for cloud projects.

What’s Your Skill Level?

The best way to allow a contractor access to your resources depends on your level of familiarity with the EC2 environment and with systems administration in general.

If you know your way around the EC2 toolset and you’re comfortable managing SSH keypairs, then you probably already know how to allow third-party access safely. This article is not meant for you. (Sorry!)

If you don’t know your way around the EC2 toolset, specifically the command-line API tools, and the AWS Management Console or the ElasticFox Firefox Extension, then you will be better off allowing the contractor to launch and configure the EC2 resources for you. The next section is for you.

Giving EC2 Access to a Third Party

[An aside: It sounds strange, doesn’t it? “Third party”. Did I miss two parties already? Was there beer? Really, though, it makes sense. A third party is someone who is not you (you’re the first party) and not Amazon (they’re the counterparty, or the second party). An outside contractor is a third party.]

Let’s say you want a contractor to launch some EC2 instances for you and to set them up with specific software running on them. You also want them to set up automated EBS snapshots and other processes that will use the EC2 API.

What you should give the contractor

Give the contractor your Access Key ID and your Secret Access Key, which you should get from the Security Credentials page:

The Access Key ID is not a secret – but the Secret Access Key is, so make sure you transfer it securely. Don’t send it over email! Use a private DropBox or other secure method.

Don’t give out the email address and password that allows you to log into the AWS Management Console. You don’t want anyone but you to be able to change the billing information or to sign you up for new services. Or to order merchandise from using your account (!).

What the contractor will do

Using ElasticFox and your Access Key ID and Secret Access Key the contractor will be able to launch EC2 instances and make all the necessary configuration changes on your account. Plus they’ll be able to put these credentials in place for automated scripts to make EC2 API calls on your behalf – like to take an EBS snapshot. [There are some rare exceptions which will require your X.509 Certificates and the use of the command-line API tools.]

For example, here’s what the contractor will do to set up a Linux instance:

  1. Install ElasticFox and put in your access credentials, allowing him access to your account.
  2. Set up a security group allowing him to access the instance.
  3. Create a keypair, saving the private key to his machine (and to give to you later).
  4. Choose an appropriate AMI from among the many available. (I recommend the Alestic Ubuntu AMIs).
  5. Launch an instance of the chosen AMI, in the security group, using the keypair.
  6. Once the instance is launched he’ll SSH into the instance and set it up. He’ll use the instance’s public IP address and the private key half of the keypair (from step 3), and the user name (most likely “root”) to do this.

The contractor can also set up some code to take EBS snapshots – and the code will require your credentials.

What deliverables to expect from the contractor

When he’s done, the contractor will give you a few things. These should include:

  • the instance ids of the instances, their IP addresses, and a description of their roles.
  • the names of any load balancers, auto scaling groups, etc. created.
  • the private key he created in step 3 and the login name (usually “root”). Make sure you get this via a secure communications method – it allows privileged access to the instances.

Make sure you also get a thorough explanation of how to change the credentials used by any code requiring them. In fact, you should insist that this must be easy for you to do.

Plus, ask your contractor to set up the Security Groups so you will have the authorization you need to access your EC2 deployment from your location.

And, of course, before you release the contractor you should verify that everything works as expected.

What to do when the contractor’s engagement is over

When your contractor no longer needs access to your EC2 account you should create new access key credentials (see the “Create a new Access Key” link on the Security Credentials page mentioned above).

But don’t disable the old credentials just yet. First, update any code the contractor installed to use the new credentials and test it.

Once you’re sure the new credentials are working, disable the credentials given to the contractor (the “Make Inactive” link).

The above guidelines also apply to working with internal groups within your organization. You might not need to revoke their credentials, depending on their role – but you should follow the suggestions above so you can if you need to.