Jedi Software Training–Part 2 - Lessons of Failure

So far we’ve established a few things from the last post on Jedi Software Training:

We already implicitly and perhaps accidentally practice the Apprentice/Master model in software development today.
The route to Software Master is paved with ambiguity. Most who achieve it can’t tell you how they got there.
A Software Apprentice often flails around in a sea of information without much guidance unless they are insanely lucky in their first job.
We need a systematic way to train apprentices and grow the craft in a reproducible manner.

Someone at this point might be saying, “Don’t we already have internships? Isn’t that enough?” Let’s distinguish an internship from an apprenticeship.

	Internship	Apprenticeship
Exploring career option?	Yes	No
Short time frame?	Yes	No
Project-based opportunity for career exposure?	Yes	No
Employer is looking for permanent hire?	No	Yes
Provides training in all aspects of field?	No	Yes

The important difference here is about goals and time frame. The internship is about checking things out and deciding interest, the apprenticeship is about getting serious and learning the craft from start to finish with the intent of employment. So no, the internship doesn’t address the need in the same manner.

To create successful apprentices, we need several things:

A road map to understand the most important aspects of the craft
A master who already understands the map
A place where the master practices and can freely train the apprentice

Who, what and where. The how is being addressed here, and the why was already established in the last post. Let’s focus on who now.

The Ideal Master

Taking a page from the book of Star Wars: To train a Jedi Apprentice, we first must find a Master. The master clearly must know software engineering inside and out. They should be versed in every aspect of project development from soup to nuts: analysis, requirements, design, development, testing, release and maintenance. Ideally, they should have practiced for 15+ years and on a wide variety of projects–from small to large, from internal to highly public, and from successful to dismal failure.

A master should be fluent in many tools, languages and frameworks. A good master is productive no matter what tool they use once they practice for a short time, given their vast prior experience on other tools.

A master must be able to communicate difficult concepts with simple and easy-to-understand analogies or other descriptions that even the newest apprentice can grasp. True mastery of material means the ability to describe it in multiple ways. The master should have infinite patience to describe things in various ways to give the apprentice many different ways to understand new concepts.

Most importantly, a master must be fluent in the business of software, since many software decisions are driven through the business, not the craft of software itself. A master that cannot understand the balance of these two forces is no master at all.

The Path to Mastery

To turn an apprentice into a journeyman, we need a training program. Electrical apprentices currently mix actual practice on the construction site with nighttime course work in electrical theory, such as the study of Ohm’s Law. A similar program for software could be useful:

How to use the tools of the trade (e.g. IDEs, makefiles, testing scripts, automated build & testing tools, source control, etc)
Understanding the theoretical side of software engineering: requirements analysis, functional decomposition, design practice, project scheduling, developer estimation, and so on.
Putting these theories into practice using a small scale, but real, project

Some of you may be thinking that this sounds precisely like a university program for computer science (at least in part). Most computer science curricula are geared toward creating academics who are skilled in the field of computer science. The fact that software engineers might pop out on the other end of the coursework is an afterthought. It’s the same distinction between physics majors and mechanical engineers: science vs. applied knowledge to practical situations. Clearly we want the practical here.

The Master’s Workshop

There seems to be no perfect environment in which to train the apprentice in a modern setting. This, to me, is the crux of the problem. There are several potential possibilities, but each fraught with peril. Let’s examine each of them individually:

Universities
Open source projects
Companies of various sizes

The University Setting

The first and most obvious is university setting. But universities already have problems as a training ground for three major reasons:

Most professors are professional academics without the benefit of real-world software engineering experience in a business setting. This makes them less than ideal Masters, even though they may be quite accomplished in their own right.
Their goal as academics is to create more academics, not to create masters in the field. Generally, they prefer the theory to the practical aspects.
The end result of a professor’s research is a published paper, not a finished software product. Often time, prototypes are more than enough for them. Prototypes are not good examples of production grade software.

Dude, what about Summer of Code?

An excellent question: What about Open Source projects such as Google’s Summer of Code or the Apache Project? There’s an unlimited number of them, creating limitless opportunity for apprentices. They sure seem like a great place to start, but:

Most of these projects are working remotely, using distributed development techniques. A master/apprentice relationship functions best when the master and apprentice are close in proximity to keep the apprentice from getting too far into the weeds. SoC and other such setups do not offer that proximity and many apprentices are generally unprepared for that level of freedom, at least initially.
An apprentice is trying to develop confidence in their skills. These projects often assume some self-taught skill with the tools involved. Newcomers are often ridiculed and derided for lack of understanding of even basic tools. That doesn’t foster confidence in apprentices.
Some are treated more like internships (e.g. Google Summer of Code).

Be A Company (Wo)man!

That leaves only one place left as fertile ground for training: technology companies. But which ones–large, medium, small, or startup? Let’s make a case for each.

Large companies (more than a few thousand employees) seem to be a good choice: well funded, plenty of good facilities, and larger numbers of job openings per year. Yet I don’t think this setting would enable the craft of software to flourish and expand for several key reasons:

Large companies tend to hire large development groups. Inside these large groups, mediocre and completely untalented engineers can reside undisturbed for years because of the inertia of large organizations to get rid of them. This unseemly presence of incompetence could give the apprentice the wrong impressions of development techniques if left unchecked.
Large companies tend to foster painful bureaucracies that would stifle the creation or gestation of such apprenticeship programs.
CEOs and technical managers of such companies are highly focused on budgets and value-add for every person in the company. Apprentices are the very definition of unproductive learners, taking up substantial resources up front in order to become productive. Managers would have to place personal reputations on the line in order to start and maintain such programs in the face of budgetary pressure.
Large companies have very siloed groups, sometimes with company-specific processes, tools or standards, that would expose an apprentice to a very narrow view of development practices, making them less marketable to the broader economy as software engineers.

Medium size companies (say between a few hundred and a few thousand employees) would face some of the same pressures as large ones, but the right company with the foresight to create such a program might be successful. It comes down to the courage of the management and the strength of will of the development team to make it happen in the face of current company policy. However, in this current economic climate of 2009-2010, few companies would take such a chance, in my opinion.

I think the biggest bang for the buck will come from small companies (less than 200 employees, but more than a dozen) for several reasons:

Small companies because of their size must attract the best talent in their development teams. Small companies don’t have time or money to waste with unproductive people. But they are usually willing to develop good potential talent because of the value proposition.
Small companies tend to be nimble in their thinking and can change their internal culture to support such a movement the easiest of the 3 described so far.
A smaller company can use an apprentice more easily across groups and teams, making more efficient use of the apprentice’s time and energy, and exposing the apprentice to more of the business of software.

But it’s not all sunshine and roses either:

Small companies fight tooth and nail for whatever money they get. Budgets for developers are tough. Add an apprentice to this and it gets even tougher.
Many small company developers are often over tasked in their current roles, making it harder for them to act in the role of a master developer. A part-time master is a difficult and potentially inconsistent kind of teacher.

Finally, we’re left with startups (less than 20 employees). While they have some of the small company advantages such as agility and talent attraction, I believe that the chaotic environment of startups is best left to more experienced engineers: journeymen or at the very least, previously trained apprentices. Startups rarely have the time to spend training new people in techniques beyond those that implement the vision of the founders. And what meager monetary resources exist in small companies, even fewer are available with startups.

In short, the apprenticeship idea requires tremendous courage on the part of the company that fosters it. Small companies are used to taking big risks with substantial payouts in the future. I believe small companies are the ideal place to grow this notion and allow it to root.

The Bottom Line

Creating an apprentice program would require jumping a number of substantial obstacles, not the least of which have been enumerated here. But the company that creates such an environment would become a magnet for the latest talent in the industry because so few places offer the right opportunities to train new graduates into real software engineers through a systematic approach.