Lessons of Failure

Outsourcing:  The word American developers love to hate.  There are lots of stories out there about failed efforts that involve offshore development (“offshoring”).  I even have a few myself.  But this post is not about bashing outsourcing countries, the cheapskates that hire them, or the project managers who can’t control the resulting chaos.  This is about understanding why we have such a difficult time making offshore development work before any of those folks get involved.

Airline Disasters and PDI

What got me thinking about this subject was the book Outliers:  The Story of Success, by Malcolm Gladwell.  In it, Gladwell discusses the reason for a series of catastrophic airline failures.  Without repeating most of his excellent dialogue in the book, here’s the Cliff Notes version:

• There was a study by Geert Hofstede, where he tracked the Power Distance Index (PDI) among selected world countries.
• The Power Distance Index is an incredibly important measure of how a person in a country would generally react to an authoritarian figure.
• Countries with high PDI would have more people willing to accept an authoritarian power figure in a paternalistic sort of way.  Like how you might defer to your father’s decisions, or to a king’s requests, for example.  People in higher PDI countries are less likely to question authority and more willing to accept instructions from those in higher positions of society.
• Lower PDI countries are characterized by subordinates being more willing to question the orders of a superior.  Low PDI countries have people that tend to view themselves more like those in power than not.  In other words, you might judge yourself to be much like your boss in a low PDI country.

Crashes Caused By Power Differences

The NTSB regularly investigates airplane crashes to determine causes, but investigators were unnerved in the late 1990s to find a pattern of fatal crashes were found to be very airline (and also country) specific.  These two airlines with frightening records were Korean Airlines and Colombian Airlines.  The fatal crashes all had common attributes:

• The pilots didn’t make any single fatal error.  They all started with several smaller errors that gradually built up to a catastrophic failure.
• For any airline, the First Officers are always trained to double-check the Captain’s orders.  This is a safety protocol that prevents single-person failures from happening.  It is the First Officer’s duty to question and even override the Captain if the order he gives is unsound or improper.
• In these particular cases, the black-box recordings indicated that the First Officers were hinting at problems, but were not strongly identifying them in a way that would make it very obvious.  Nor did they explicitly tell the Captain that he was acting against normal procedures.

Power Means Fear of Communication

In the aftermath, an astute researcher matched up the Hofstede PDI data with these two countries and found that both Korea and Colombia are high on the PDI scale.  What were the implications of this?

• Clearly, a power difference would have existed in the cockpit between the First Officer and Captain.
• The manner of communication between superiors and subordinates is very different if you are from a high PDI country than a lower one.
• In a high PDI country, a subordinate must use the proper language, body posture, facial expressions and tone when communicating with a person of higher status or power.  In Korean, for example, there are no less than six distinct ways to address someone from the most formal to the least.  In the United States (a lower PDI country), we have a much looser style (“Sir”, which could apply to a General as much as a Lieutenant in the military, or even your father in some situations).
• This disparity of power created hesitance on the part of the First Officer to embarrass his Captain when the Captain made mistakes (and in each of these cases, at least 3 mistakes were made).  So instead of using a direct method of communication (“This weather is really bad, we should turn back immediately or regroup for another airport”), he used a more subtle, formal and proper one (“Sir, look how it is raining outside.”).
• The implication of a high PDI country is that there is a rich subtext going on between the two communicators.  But that subtext assumes that both sides are alert, paying attention and can clearly understand the implicit signals.
• In the case of the airlines, the Captain was almost always sleep deprived and exhausted by the time the situation arose, making that communication impossible.

The researcher concluded that the pilots were inside a cultural framework that dictated how they should behave at a time when those behaviors turned out to be detrimental and outright dangerous to the safety and welfare of the passengers.  In other words, the fact they came from high PDI countries made it impossible for the proper communication to take place when it was most necessary to be plain and step outside the traditional power roles of these cultures.

Past Outsourcing Blames

What does all this have to do with outsourcing?  First, we need to understand why outsourcing has traditionally “failed”, according to both buyers (those who purchase outsourced services) and providers (those who perform the service in their local country, or send people to other countries to perform services).  Here’s a graph of combined data about a survey regarded failed projects from 2004.  The data are still relevant to today’s discussion.  (Source:  The Outsourcing Center)

Notice the even distribution of reasons once the provider & buyer survey data is combined.  This is interesting because neither party can clearly point to a single, differentiating causation factor in the failure of outsourced projects.  But I believe that’s because they asked the wrong questions in the survey.

The Real Issue with Outsourcing is Power Difference

If you have a buyer from a lower PDI country and a provider from a higher PDI country, there are already implicit consequences to your interaction that neither party will know about without prior outsourcing experience or natural cultural awareness(1).  And even with that experience, it’s not a given that they will understand the reasons behind the challenges of outsourcing.  Let me create an example from my own personal experience:

Suppose you had an American company (Buyer) and an Indian company (Provider).  The American company contracts with the Indian one to provide offshore outsourced software development at a fixed price per developer.  Certain key performance indicators are agreed upon by both parties and the game is afoot.  Let’s also assume the Indians agree to a six month project to write a content management system for the Americans.

A typical scenario of engagement might follow like this:(2)

• The first month, everyone hammers out the requirements documents and in a great ball of fury, declares them sound and ready for implementation.  The American company at this point would typically reduce the daily oversight on the project to something more reasonable, like weekly updates.
• The second, third and maybe even fourth months pass with little fanfare.  The Indian developers are quietly building the specified software and the Americans are receiving updates about it that are all positive and sound great.
• At some point, the American company asks for a demo of the progress to date.  The Indians put together something after a bit of negotiation (since the Americans neglected to mention the demo as a deliverable before the end).  The Americans see the actual software and fly off the handle.  Performance is awful, the screens don’t look anything like what they want, and the software appears to be behind schedule.
• Further code reviews by American developers indicate that the code quality is fairly poor, lacking in comments, unit tests, and filled with copy-paste blocks of duplicate code.  The Americans immediately demand the project be put under different management.
• The project falls off of the rails somewhere after this.  It will either be canceled, brought back in house, or will be delivered extremely late after extensive modification to the original requirements.

There’s lots to pick on here on both sides of the table.  I would like to point out that the fact that I picked on Americans and Indians is actually irrelevant here. You could easily substitute “British” for Americans (3), and “Filipinos” for Indians with the same results.  But why are they so interchangeable in this fashion?  It’s because of PDI and the inherent cultural communication issues that come with it.

Here is a list of the top 10 Outsource Providing countries in 2009, and their PDI scores.

1. India (77)
2. Thailand (64)
3. Mexico (81)
4. China (80)
5. Indonesia (78)
6. Malaysia (104)
7. Philippines (94)
8. Jordan (no data)
9. Egypt (80)
10. Bulgaria (no data)

For reference, the United States is 40 on the scale.  Western countries can run the gamut as high as Belgium (60) to as low as Austria (11).  The scale is from 1-120, where 120 is extremely high PDI.  You can see all the countries’ measurements in the original study on this colorful world map of PDI indices.  The gray countries are ones that weren’t measured.  India would be considered moderately high PDI at 77 (in the 61-80 range).

So what happens when you bring a low PDI buyer together with a high PDI provider?

In a word:  Disaster.

Cultural Context Matters In Communication

Each side expects a certain subtext to go on during a conversation because of their own cultural context.  Like this:

Low PDI Manager: So, is the new website ready for launch by Friday?

Low PDI Developer: No, and it’s going to be another 2 weeks because we need the new servers to arrive, for QA to finish with testing after they do, and then release the code.

Pretty straight question, pretty straight answer if you’re from a low PDI country like the United States.  There is little assumption about the subtext because a low PDI communicator is used to “speaking his/her mind” about it.  The information is supposed to be in the conversation as spoken words.  If it’s not there, it’s ignored.

But what if we change that a bit?  Assume the High PDI and Low PDI Developers BOTH have access to the same information and are equally competent:

Low PDI Manager: So, is the new website ready for launch by Friday?

High PDI Developer: Yes, it may be ready by then.  We are looking into it.

That seems like a bad answer if you’re from a low PDI country (mostly because we know the context from the first scenario), but it may be taken at face value because the Low PDI Manager expects straight conversation.  If there was a problem, the Low PDI Manager expected the developer would say something specific about it.  When they didn’t, the Low PDI Manager assumes that Friday will be the date.

And what about the High PDI developer?  He didn’t want to offend the Low PDI Manager, because that’s what you are careful to do in a high PDI country.  The High PDI Developer assumed that the Low PDI Manager would understand his subtext “may be ready” and either ask further questions, or understand that Friday wasn’t necessarily a realistic date.

This is just the tip of the iceberg.  If this happens on a simple conversation about a deadline, what about really big stuff like:

• Requirements
• Deliverables
• Quality control testing
• Development standards
• Documentation

The implications are literally staggering.  In fact, I’d go so far as to say the fact that every outsourced project hasn’t failed is something of a miracle.  It’s a testament to having the right people who naturally and instinctually bridge these gaps through extra communication.

The Survey, In A New Light

Getting back to the survey questions, if you look at all of them and how they would be viewed relative to PDI, it’s arguable that PDI differential is the one, single, leading cause that relates to how providers and buyers have a hard time seeing eye-to-eye during the outsourcing process.  Of the eight named factors, I can see 6 of them that directly relate to PDI differential:

• Buyer’s unclear expectations up front (buyer assumes he is understood when the provider stops asking questions, but that’s a typical low-high PDI interaction)
• Poor governance (see my deadline example above)
• Poor communication (again, the deadline example)
• Poor cultural fit (again, the deadline example)
• Interests become misaligned over time (you don’t understand each other’s communication needs and are frustrated)
• Not mutually beneficial (you can’t work together because you don’t understand how to interact…)

Adding those 6 factors up, 72% of project failure reasons can be connected to PDI differential. If both sides understood that single factor going in to the process, everyone would be better served in the end.  Think I’m just making this all up?  It’s not just the buyers that complain.  High PDI country providers say the same things.

So what’s so hard about outsourcing?  It’s hard because of the cultural baggage we bring to the table on both sides, and neither side necessarily realizes it because of assumed interactions.  We need to be more aware of the cultural assumptions going in to projects like this, or we’re doomed to repeat them ad absurdum.

(1) I think it’s fair to say that most other countries would NOT say most Americans are blessed with “natural cultural awareness”.  🙂

(2) Before I get lots of angry comments from Indian readers about the interaction above, yes, there are other potential outcomes and perhaps you’ve been on projects where they are all successful.  I have a mixed bag of experience on this, and it’s not about bashing Indian developers.  Like American ones, they run the gamut–good, mediocre, and what-the-hell-are-you-doing-coding.  I’ve run into all three in about the same proportions as American developers, more or less.

(3) I’m sure one or more British readers are horrified at thinking they are interchangeable with Americans at this point.