How to Castrate a Bull - the Blog - by Dave Hitz

Always Bet AGAINST Your Employees

Recently a group of NetApp managers tried to motivate their employees by betting that the team could meet a particular project milestone. If the team missed the date, the managers promised to dye their hair bright colors. (Silly hair bets have a long history at NetApp.)

Unfortunately, the team missed the date, and as a result, the managers not only had to deal with bad news, but they had brightly colored hair to advertise it. When people would ask, “What happened to your hair,” the managers had to say, “I bet my team that they would hit their schedule, but they missed.”

Even if there are perfectly good reasons for the miss, the listener might still conclude: You not only look like a clown, but you manage a whole team of clowns.

Much better to bet that your team will fail. When I first started running engineering (back in 1999), product reliability was an issue, so I bet my employees that they could not improve quality by a factor of 10. If they did, I would dye my hair any color that they wanted.

Imagine how this played out. The team succeeded (as measured by a man who cared so deeply about quality that I knew he would never cheat), and I ended up with hair that was bright magenta, blue, and red. To get hair these colors you can't put it on top of brown; you have to start by bleaching to white. So I looked like a freak, but at least, when people asked me about it, I had a good answer: “Let me tell you about my awesome team. I should never have bet against them.”

The conclusion: You look like a clown, but at least you manage a good team.

So I say, always bet against your employees. Of course, it helps if they understand it’s a bet you want to lose.

Cloud/Grid/Utility: Definitions Drift Because IT Is In Denial About Outsourcing

Why did the term cloud computing so quickly lose its original meaning? At first, cloud computing was about how to not build a data center, but it quickly morphed into an architectural description of how you should build a data center. The first definition is about accessing IT services over the Internet from computing resources that somebody else owns and operates. The second definition is about building your own hyper-efficient data center based on virtualization, shared resources, and dynamic provisioning. These definitions are so different – contradictory even – that it takes modifiers like external, public, internal, and private to tell them apart. To many, it seems that we have sunk into a morass of confusion.

My point here, though, is not to debate definitions or technology. (I believe that both internal and external clouds will be wildly successful for many years to come.)

My question today is why the definition got muddled so quickly, especially since this isn’t the first time it has happened. The original idea of utility computing was that computing should be like electricity. In the early days, companies had their own generators, but over time, centralized power companies replaced them. The theory was that IT should evolve in the same way. This same metaphor also inspired grid computing, but as with cloud computing, the definitions quickly shifted to data center architectures. I remember many data center tours where the proud owner of rack after rack of Linux nodes would say, “Check out my compute grid.”

What is going on here? I believe that IT is in denial. CEOs and CIOs are like ships passing in the night.

CEOs ask, “Why can’t we just convert to cloud computing?” What they mean is this: “I’m tired of expensive data centers, high capital costs, and hard to manage infrastructure. Why can’t someone else do all of that for us and we just buy it as a service over the Internet? You know, like Yahoo! email or Salesforce.com?” In other words, CEOs would like to outsource big chunks of IT, just like they have already outsourced big chunks of manufacturing.

And then the CIO comes back and says, “I figured it out. We can convert to cloud computing, but the good news is that we still get to build the data center and buy the IT equipment ourselves.” Ships in the night.

IT departments are so averse to the idea of having their jobs outsourced – who wouldn’t be! – that whenever someone tries to define a term to mean exactly that, they redefine it to mean a new thing that they get to build and run themselves. Perhaps soon there will be a creative new definition of external cloud that somehow means you build it yourself.

As I said above, I believe that both internal and external clouds will be wildly successful. For at least the next five years, more likely ten, most CIOs will run a hybrid model consisting of three main parts: (1) Traditional silos, where an application, a server, and storage are purchased and installed together; (2) Internal clouds, which will initially run less critical apps and grow over time; and (3) External clouds, which will also start low and move up.

Why Is NetApp's New Data Center So Efficient? Big Fans and Hot Air

NetApp just finished building a new data center in RTP, and it is one of the most efficient in the world. I mostly focus on innovation around storage and data management, so I was surprised by how much room there is for interesting innovation in data center design.

The big problem of data centers is getting the heat out. In a typical data center, for every kilowatt you put in to run your equipment, you have to spend another kilowatt on air-conditioning to pull the heat out. In other words, it actually takes 2 kW of energy to run 1 kW of equipment. This is a Power Utilization Effectiveness (PUE) of 2.0.

NetApp's new data center has a PUE of 1.2, which cuts the total power requirement almost in half. At current electricity prices, we'll save over $7 million a year. We use many techniques to achieve this, but I have two favorites: big fans and hot air.

First for the big fans: our data center moves enough air to fill the Goodyear Blimp in three seconds. To get a mental picture of this, imagine an endless series of blimps flying out of the building – every three seconds another blimp. The harder you blow, the faster you cool.

The hot air technique is more subtle. Most data centers cool air to 55 or 60 degrees, but our team figured out how to get effective cooling with 74-degree air. Less cooling equals less energy. The trick is to manage airflow carefully. Rack mounted equipment sucks air in the front and blows it out the back maybe twenty degrees hotter. Most data centers have rows and rows of racks in a big open room, so the air gets all mixed up. It works better to deliver cooler air directly to the front of the rack and collect the hot air from the back. In our previous data center, we experimented with plastic shower curtains to separate the “cool aisles” at the front of the racks from the “hot aisles” at the back, and it worked wonderfully. In the new data center, we went a step further and used drywall to build airtight cool aisles, which we pressurize to speed airflow through the equipment and ensure that the hot air can’t get back around to the front. The hot aisle can get up to 95 degrees, which is uncomfortable, but that’s okay – it’s optimized for equipment and energy, not for people.

In RTP, the outdoor temperature is 74 degrees or less 67% of the time, which means we can usually use outside air with no cooling at all. Our data center has more cooling capacity than the Empire State Building, but our goal is to leave it off.

It didn’t take exotic technology to achieve this result. The design and airflow are unusual, but we used ordinary air conditioning units, heat exchangers, and so on. In fact, we reduced the cooling capacity by about 20% so this technique is not only cheaper to run but also cheaper to build. If you want to learn more, we welcome visitors.

What Moves Markets: The Seven Hundred Million Dollar Man

Last week we had our Analyst Day in New York City, which means that we met with a theater full of Wall Street financial analysts and explained our company to them. You can learn a lot about how Wall Street thinks by watching how our stock price changed during the different presentations. (You can watch it yourself here.)

The first four speakers described our strategy, why we have been winning recently, and why we believe we are very well positioned to take advantage of trends in the market like cloud computing and industry consolidation – Tom (our new CEO) on the big picture, me on our vision of data centers and clouds, Manish on our product strategy, and Rob on our sales strategy.

Finally, Steve Gomo, our Chief Financial Officer, gave projected financial results for the next two quarters. In particular, he showed a slide saying that by Q3, we expect to be back to our normal operating profit of 16%.

We tracked the stock ticker throughout the presentations, and here’s how Wall Street responded. After the four of us did our absolute best – two hours of details – to explain what we are doing and why we will win, the stock had moved a total of three cents. In the first ten minutes of Steve’s talk, the stock went up a dollar. It’s clear what matters to Wall Street. Never mind technology, long term strategy, or market position, what moves markets is short term earnings. (Not new news, of course, but this is a graphic illustration.)

By the next day, the stock was up about two dollars. Perhaps the extra dollar came from the other four presentations, and it just took a while for the analysts to digest the details, but – realistically – that was probably Steve’s doing as well. Since we have 350 million shares, Steve’s short talk – maybe just that one slide – drove NetApp’s value up by seven hundred million dollars. If you look just at the first ten minutes of his talk, when Steve got the first dollar, he was increasing our market cap at the rate two billion dollars an hour. What power: mover of markets and creator of value. Steve Gomo, the seven hundred million dollar man!

And yet, when he got home from the meeting, late that night after a cross country flight, the first words Steve heard were: “Honey, I need you to take a look at this sink. The spray hose is leaking like crazy.” From star to plumber in six seconds.

Oracle is the “Crazy Ivan” of the IT Industry

In The Hunt for Red October, Sean Connery plays a mad genius submarine commander called “Crazy Ivan”. He stops American subs from sneaking up behind him by turning at random times in random directions. Follow too closely and you may get hit. I’m telling this story because people keeping asking me what I think Oracle will do with Sun. I think there’s no way to know.

If IBM had purchased Sun, things would have been easier to predict. I imagine the conversation in IBM’s boardroom going something like this: “They have a RISC chip; we have a RISC chip. Let’s phase theirs out. They have a UNIX; we have a UNIX. Let’s phase theirs out. They have Java, we have – uh – let’s keep that one!” And so on.

With Oracle, the outcome is not at all clear. Oracle is so different from Sun that they could keep pretty much everything. What parts of Sun are important depend on what kind of company Oracle is trying to become. Do they want to become a full-line systems vendor competing head-to-head with IBM and HP? Do they just want Java and MySQL, which are so closely related to their current product lines and strategies? Perhaps something more secret and clever than anyone else has thought of yet?

There’s just no telling, because Oracle is – and I mean this in the most positive and respectful way – the Crazy Ivan of the IT industry.

Finding Opportunity in Pain

I recently wrote an editorial for the Financial Times about how, counter-intuitively, good economic times can sometimes block opportunities, and bad economic times can expose them. It would be bad form (and probably a copyright violation) to post that editorial here, but here’s the link.

Can You Afford Good Company Culture In Bad Economic Times?

I wrote a guest blog for CNBC that addressed the question of whether it’s possible to maintain a good company culture in bad economic times. I think it would be bad form to re-post that blog here, but if you are interested, here’s the link.

Does Anyone Have The Opposite Point of View?

I got a great lesson in how to lead group discussions from Bill Barnett, a professor at Stanford’s business school. Many years ago he agreed to facilitate a strategy meeting for NetApp. There were eighty or a hundred people, and Bill was going around the room, asking people to share their thoughts on an important strategy decision. People raised their hand to talk, and hand after hand, comment after comment, everyone was in favor. It seemed that there was no dissent at all.

After quite a while of this, Bill said, “Okay, I think we’ve fleshed out the argument in favor of the proposal well enough. Does anyone have a completely different point of view?”

To my surprise, a dozen hands went up. Now it was hand after hand from people against the proposal. At least as many comments against as we had previously heard in support. What was going on here?

My first reaction was frustration. If these people disagreed, why hadn’t they said anything before? We could easily have adopted the plan thinking that we had full support from everyone in the room. To be honest, people who won’t speak up piss me off. How can we make good decisions if people won’t point out the flaws they see? Some problems may be so deadly that they kill the plan we are considering. Even small flaws may need to be addressed for the plan to succeed. Perhaps some people think that raising concerns is disloyal. They’d rather go with the flow than be seen as troublemakers. I disagree. If you spot a problem that others don’t know about, I think it’s irresponsible to keep quiet. In fact, keeping quiet is disloyal because it sets your group up for failure.

My second reaction, though, was admiration for Bill’s simple technique. Despite my frustration, I’ve come to accept that some people won’t speak up, and now I use Bill’s technique myself. It’s amazing how often his simple question—“Does anyone have the opposite point of view?”—triggers a very different discussion. I think it works because it acknowledges that other opinions may also be valid, and it gives very specific permission for people to express dissent.

This technique can also help “flush out agreement.” Have you ever been in a meeting where people keep talking and talking even though they all seem to be saying the same thing? Sometimes it’s because the dissenters are keeping quiet, but sometimes it’s because everybody really does agree. In that case, asking whether anyone has the opposite opinion can help move the meeting forward. Nobody says anything, and it becomes (hopefully) obvious that it’s time to stop talking.

I still wish that people would speak up on their own, but now I have a tool to encourage them. Plus, I have a fantasy that using this technique repeatedly over time can help change a company’s culture, to teach people that it is okay to speak up.

Does anyone have the opposite point of view?

Three Ten-Year-Trends That Are Driving IT Change

A trick I use to predict the future is to identify ten-year-trends. These are really big trends that take place over a decade or more. At the moment, there are three ten-year-trends that interest me, but before I get into that, let me give some historical examples to clarify what I’m talking about.

In 1981, there were only an elite and nerdy few of us who had computers on our own desks. (I built my first computer in 1977, at age fourteen, but I am … different.) There were more people with some kind of remote terminal, but the majority of business people had no personal access to computers at all. Then, in 1982, IBM introduced the PC, and by 1992, ten years later, pretty much everybody in business had a computer on their desk. The ten-year-trend in IT during that decade was “put a computer in front of everybody’s face.”

During the 1990s, the ten-year-trend was “wire all of those computers into one big network.” At the beginning of the nineties, many computers weren’t networked at all, and networks that did exist were often small, connecting just a handful of computers in nearby offices. By 2000, the Internet was ubiquitous. IBM was running TV ads with nuns on camels in the middle of the desert sending e-mails to each other.

When I see lists of “this year’s top trends,” they often feel small and short-sighted to me. Any trend that only lasts a year is too insignificant to be important. It’s probably mostly over by the time you spot it. The really interesting trends, the ones that drive big change, last many years—a decade or more.

The three ten-year-trends that I see in IT today are: 1) Cloud/Outsourced Computing, 2) Server Virtualization, and 3) Flash Memory.

The big question behind cloud computing (or outsourced computing) is whether a company should build or expand its own data center, or whether it should access computing resources remotely, over the Internet. People are already using lots of cloud/outsourced computing today for things like blogging and web training, but I expect it to move up-market over time. This won’t happen overnight, but ten years from now I believe that many medium-sized businesses and a few large enterprises will have cloud-outsourced pretty much all of their computing infrastructure. (The term “cloud computing” is confusing because so many definitions are floating around. I’ll give mine in an up-coming blog.)

For people who do build data centers (either because they ignore the cloud/outsourcing trend or because they provide cloud computing to others), server virtualization will radically change how those data centers are built. Server virtualization is at the center of the trend, but it pulls a boatload of other trends along in its wake, including blade farms, network virtualization, and (my favorite) storage virtualization. Data centers built in ten years will look radically different than they look today.

Flash memory is amazing stuff. It is the first technology in decades with the potential to create a pervasive new layer in the storage hierarchy. Many technologies over the years, like bubble memory, have been touted as “disk-drive killers,” and none of them ever panned out, but flash memory is emerging as an enormous force, and there is no question that storage systems will look radically different in ten years. I’m not predicting that flash memory will eliminate disks any time soon, certainly not in five years and probably not in ten years, but I do think that it will relegate disks to an increasingly tape-like roll as flash absorbs more and more of the I/O intensive loads that disks handle today.

These are the three ten-year-trends that I see driving change in IT, but I’d love to hear from readers if they think I’ve missed some, or that the ones I’ve chosen won’t drive as much change as I think they will.

I Hate Beige

If you ask a committee to choose a color, they will always choose beige. I’ve seen this for the color scheme in a new building, for new carpets, and even for new systems.

Here is my theory. Everyone in the committee has a favorite color, and they hope the group will choose it. And they have interesting favorites, like turquoise, magenta, or even electric lime. The problem is, most people also have colors they hate, and usually they hate the same interesting color that someone else loves. One person says, “I’m allergic to citrus: no tangerine,” and another says, “I lost my promotion to a Harvard grad: no crimson.” So it goes, through all the interesting colors. Nobody likes beige, but nobody hates beige either, so that’s what you end up with.

My input to committees is this: I hate beige!

As a result of this strategy, the color scheme in our headquarters building is avocado and purple. Not what I would have chosen myself, but it’s not beige, so I love it. I would much rather make a choice that some love and many like, even if it pisses a few people off. If your goal is to offend nobody, then you will appeal to nobody. I believe this style of committee-think is why so many big companies produce boring pablum.

Colors are my example, but projects and strategies can also be beige. I hate that too.