Jedi Software Training–Part 1

Software engineering is perhaps the youngest of all the engineering disciplines (and some would even argue, we don’t practice an engineering discipline at all).  But like all disciplines, an engineer must be trained in order to achieve a level of competence to practice their craft with any proficiency.

The Fresh Out Of College Problem

Software engineers run the gamut–amazing, mediocre, and step-away-from-that-IDE-before-you-hurt-someone levels.  Some are naturals and some probably will never be able to program their way out of a wet paper bag.  In terms of how we get trained, it’s all very informal in most cases.  Universities rarely focus on any software engineering courses as a serious part of a Computer Science program.  Indeed in my own Alma mater, Computer Science was part of the Engineering School but did not constitute an official engineering discipline.  The focus is entirely on data structures, programming, algorithms, and nuts-and-bolts sorts of topics.

Don’t get me wrong–those topics are incredibly important, but almost every college graduate I’ve ever worked with directly out of school (within the first 3 years of their graduation) generally has poor knowledge of how to run multi-engineer, moderate-size software projects.  I’m talking about simple things that are bread and butter for a successful complex project: source code control, project module management, requirements definition, functional design, and documentation.  This seems like a huge gap in training, considering that everyone knows these things are absolutely essential when they get out of school.  Employers are frustrated because graduates are unable to take the reins of a software project in a meaningful way without starting out in (very) junior positions that are rarely available and only modestly tolerated by most companies.  Graduates are frustrated at their lack of opportunities because they don’t have enough ‘real world’ experience to qualify them for real software development positions.

The question is, why are we training our engineers so sloppily today?

The Apprentice Model

Engineering training and Star Wars Jedi Knights have something in common:  they both start out as apprentices.  Let’s look at the history of this practice and see how it can apply today.

Back in the Middle Ages, if you wanted to practice a craft, such as blacksmithing, baking, masonry, or butchery, you needed to become an apprentice to a master craftsman.  That master had been practicing for years, had a demonstrated level of mastery, and often belonged to a guild of other masters who judged this master to be fit enough to earn that title of master in the first place.

Apprentices lived a hard life back then.  They were exploited as cheap labor for the master for a long period of time (5-7 years was typical).  The master provided tutelage in the craft, food, and lodging in exchange for their work.  After this period of time, the apprentice was shown the door and expected to fend for themselves as a newly minted journeyman.  Journeymen could either practice in solitude or work under a willing master (if available).  After some long period of time, the journeyman could try to produce a masterpiece of his or her craft in an attempt to demonstrate mastery, join a guild and attain the rank of master.

Some guilds were comically harsh in the training methods.  For example, up until 1791 in France an apprentice worked under a master for a long period of time through their journeyman rank.  If they failed to produce a masterpiece during their journeyman period, they were subsequently executed.  I, for one, am glad we don’t train people in this manner anymore.  The labor shortages this could create aside, the pressure to produce would be excruciating.  And in truth, not everyone is cut out to be a master.

Modern Apprentices

Apprenticeships are still used as a model for vocational training in many professions in a number of countries (in America, we still use this for plumbers, electricians, and carpenters among many other trades.  Across Europe, this is true as well), this training serves to bring new workers into the trade, give them baseline skills in a controlled environment under a specific instructor, and a route to advancement in the field.  But this formalization happened over years when best practices were easily formulated into codified documents and classroom formats, allowing the apprentice to reproduce a master’s work in a repetitive and formulaic manner.

If we characterize the differences between the three levels, you might see the following:

• An apprentice can take a specific set of instructions, created by journeymen or masters, and faithfully reproduce the steps to create a result of lesser quality than a master or journeyman.  An apprentice is unable to work without a concrete framework of rules to abide by while practicing the craft.  Apprentices generally need constant guidance and intervention from others in order to complete a project successfully.
• A journeyman is an apprentice who has completed their training period and achieved a baseline level of skill in the trade.  They are competent enough to work alone, but most often seek the continued education under a master to improve their skills.  Journeymen are comfortable in a wide variety of techniques as taught by the master, and many of these techniques are now second nature to the journeyman.  However, journeymen lack the ability to create new skills or perform the baseline skills in the effortless manner of a master.  Journeymen generally can do a skill, but have trouble expressing the exact reasons why one skill is preferable over another in a certain context outside of the rules given by his or her master.
• The master is the culmination of years of practice into “effortless skill”.  The master’s abilities require no conscious thought to manifest.  Masters have the ability to see patterns in projects and skills, arbitrarily combining things into new and unique ways of using them, often pioneering new techniques as a result of this artful mixing.  Masters understand the rules to the point where they can break them at will, knowing which limits are completely arbitrary.  A truly excellent master has the ability to teach their skills to apprentices in a way that creates excellent apprentices.  These masters are exceedingly rare.

The apprenticeship method is a time-tested, battle-worn method for bringing people of low-to-no skill into a field and training them to a level of success. Dozens of fields practice this, and almost all formal engineering disciplines have a similar model (engineer-in-training and P.E. (professional engineer) certification).  Academics have similar models to produce scholars, with undergrads promoted to graduate students, then post-doctorates and finally becoming associate professors, full professors, and professor emeritus.

My question is:  Why don’t we do this with software engineers?

Apprentice, Journeyman, Master – The Path Less Traveled

The path from apprentice to master must be reproducible and documented through time in a formal manner by previous masters.  Developers today are largely left to hack out their own destinies at random with high variability in the results.  If they inadvertently become masters, they attribute it to their own skills rather than the lucky circumstances they managed to find themselves in early on in their careers.

Transferring that mastery becomes incredibly difficult when a master’s path lacks any formal system to follow along the way.  Software masters are revered for their superhuman abilities to swoop in to save the day on a doomed project through superhuman coding effort via nights, weekends, major refactoring on a level that ultimately undermines the morale of the rest of the team.  This “cult of the software hero” approach ultimately shrouds the master’s path rather than illuminates it for others to follow.

We know this model is still at work in software today, albeit informally.  We see the various levels at work in our own teams and workplaces.  Here’s a simple OO experience hierarchy that you probably can relate to:

• An apprentice OO programmer will struggle with encapsulation and polymorphism, but can put together systems that have been well-specified.  They need highly detailed object designs to see the various interactions of a system come together with elegance.  Left without such guidance, they will inevitably create God Objects, Yo-yo Object Hierarchies, and other monstrosities.  Design patterns are something of a mystery to them.  Simple algorithms will require large amounts of effort to understand and encode into a language.  Their skills with the tools of the trade will be low to moderate (e.g. IDEs, source code control, automated build systems, bug tracking) and significant effort will be expended in using/learning them.  Design and architecture are generally beyond them.
• A journeyman OO developer will understand the value of data hiding and be able to participate in interface designs for obvious parts of the system.  They are comfortable with the tools of their trade and know many time-saving shortcuts, along with several flavors of tools.  Journeymen are typified by rigid adherence to systems, tools, frameworks or architectures because of their comfort level with them (“The Golden Hammer” effect).  They will recognize some design patterns and have regularly applied them, but may not know all cases when they are applicable, or apply new ones with ease.  Journeymen dabble in architecture with increasing skill and ease, but complex problems are often met with complex, obscure solutions when designed by journeymen.
• Master OO engineers can hear domain problems and suddenly see object models dancing in their heads before the description is finished.  Architecture is second nature and design patterns require little to no effort to apply or recall.  A master OO developer generally tends towards tool, language, and framework agnosticism because they understand their relative weaknesses and strengths, choosing only those tools that stay out of their way, make the most sense for the domain and allow for ease of creation.  A master’s trademark is the ability to create a simple and elegant solution to a complex problem.

So the $64,000 question becomes:  How do we create an apprenticeship model in software development that can succeed in cultivating better engineers? We’ll start to answer that in Part 2.