Specialization Versus Agile Development or, How to create high-performance teams, and love the generalists that can save your tail

One of the less talked about goals of some Agile approaches is to eliminate distinctions between engineers and make each team member a solid generalist. Ideally, anyone on the team should be able to take on the current highest priority task in the backlog, regardless of the knowledge and skill level required to complete it.

Benefits:

• Everyone writes better product code when they understand thorough testing. Everyone writes better tests when they understand the numerous pathways that exist in the code, and when testing is understood as a multi-dimensional and demanding discipline of its own.
• When they understand the overall system, its various components, and the end user's objectives, knowledge, and work-flow, engineers can make richer contributions to the product than when they just know "their part."
• People write better software on one side of an API when they also have experience on the other side.
• Not having to share specialists across teams reduces scheduling dependencies and delays, and allows each team to be truly self-sufficient, or whole. Additionally, this leads to a stronger sense of responsibility, to more complete team bonding, and to more team learning and efficiency.
• The closer we get to everyone being able to do all the different kinds of work there are, the higher our staff utilization, the more robust we are, and the more quickly we get high quality product to market.

It's all good. Except for the fantasy part. That is, it's a great idea, but since I have no expectation of attaining it, I want to talk about what we can realistically achieve.

First, why we won't get there: We have just described a significant subset of what it means to be a — cue the blaring trumpets — Great Software Architect which is not where everyone in the industry is going, so we know empirically that this goal will not be reached. Understanding why is this will help us guide our actions.

One reason is that not everyone wants to get there. To list just a few of many possible reasons, some want to only be responsible for the work they do with their own hands, some really like being a specialist, and some want to go into management, or product management. But an all-generalist team would be a poor plan even if everyone on your staff did want to become a great architect.

The kind of thinking that runs through single-threaded, interrupt driven, dynamically typed, front-end JavaScript with brilliance might not be adept at multi-threaded, stateful and process-oriented, type-safe, back-end Java programming. The brain that intuitively perceives end-user goals, relevant metaphors, order of user operations, and powerful presentation of results is unlikely to also do outstanding work in performance, scalability, extensibility, and security. A person skilled at building great general-purpose back-end business logic and library code is not automatically great at creating applications that handle even basic usability considerations.

Our vocational experience contributes significantly to what kind of thinking we do best. That might seem to imply that broader experience will automatically produce a broader range of thinking styles, but the data do not support this hypothesis. Whether due to biological brain differences, early life influences, acquired personal preferences, or declining mental flexibility, most painters do not bounce back and forth between impressionism and art deco, and most database people are not the ones you want creating your UI, nor do they want to do it.

When the rare someone is great at some key aspects of software product development and pretty good at several others, they are strong generalists, and they have the potential to become great architects. For combined reasons of different interests, different brains, and different accidents of experience, this is a small subset of those staffing our engineering efforts. Thus, for most products, and certainly most enterprise products, we should plan on hiring several people, with different skills, and sustaining a culture that helps people with different kinds of brains and experiences to work together effectively. Oh, damn. This is about social facilitation, not engineering.

It's about people with all their complexities, not inanimate things we can shuffle around without consequence. Several studies, including one done recently at Google, have found that the best-performing teams are not those with the most relevant expertise(!), nor those that work the longest hours, nor those with strong leaders. They are those in which unplanned communication is most fully shared - does everyone in a team meeting speak a similar amount of time, and to similarly engaged listeners.

These studies conclude that we need to talk with each other with a lot of social and vocational safety - it has to be safe to not know, to be unsure, to be wrong, to investigate and experiment, and to disappoint, in order to achieve greatness. Most engineers and most companies simply won't tolerate this. Most are achieving less than great results. Hmm.

The Google study also says team members need to be able to depend on each other, operate with clear goals, roles, and plans, and care about the meaning and impact of their work. In too many companies, "dependable" means you do what I ask, when I ask, regardless of the infeasibility, or cost to your family life, clear goals, roles, and plans comes down to the same thing, and meaningful work with impact means whatever it is that this company does — you're lucky to work here, so suck it up.

By contrast, the best-performing teams are dependable because they communicate in a safe environment in which they feel so respected, valued, and supported by other people's efforts that they don't want to let anyone down — that's a positive social bond, not the pressure and fear that are the driving forces in so many companies.

High-performing teams have clear goals, roles, and plans, because they create them together, and as the work progresses and these things all inevitably flex or change outright, safe, high-bandwidth communication keeps everyone aligned.

People on these teams might find meaning and impact in their work because of the company's mission, but what is likely to matter much more is whether they enjoy the problem-solving challenges in the work they do, feel pride in their increasing accomplishments, and find their relationships with co-workers and their boss meaningful. How many times have we heard that people don't quit jobs, they quit bosses and work environments where they don't feel that they are valued, or where they feel they have little left to learn?

This was a longer digression than I would have liked, but perhaps it helps us see that if we want to have a high-performing team, our goal should not be to make engineers into the interchangeable inanimate objects that they will not become. Rather our goal is:

Use cross-training in support of a culture in which people with significantly different types of brains are likely to develop genuine appreciation of each other's work, and of each other.

Our cross-training programs, when they have existed, have had the wrong goal, and have largely failed. The goal — my goal anyway — is not to get the UI guy to be a drop-in replacement for the database gal. That's well, um, stupid. The goal is for the UI guy to understand just enough about the engineering strengths and constraints of database programming to ask better questions, make better suggestions, ask for and provide better interfaces, and do better work as a UI guy .

How do we hit this more realistic target? The battle is largely won just by communicating that this believable result is the goal. In practical terms, use pair-programming to cross-train the UI guy to do just enough database work to have a feel for what is going on down there. Do just enough for the database gal to have an understanding of UI and UX challenges. Ensure this leads to communication between these layers becoming more efficient and less error-prone, and communication between these people more informed, respectful, trusting, and synergistic. That's teamwork, that's tactically and strategically valuable, and that's achievable. It's also the larger part of what managers should be measured on, as everything bottom-line flows from this, and everything in "soft skills" is embodied in this.

Let's begin by having our architects and technical leads describe the system neither in superficial "marketecture" terms, nor in details only specialists will understand. Let them use cross-discipline analogies and note cross-discipline differences. Charge them with the goal of making each specialization interesting and respect-worthy instead of strange and not my thing or not real engineering .

Benefits:

• Most staff members should perform better in their existing jobs
• Some will become able to work skillfully in multiple areas
• Some will end up ranking among your most valuable employees — only because you gave them this chance to grow

Do not expect cross-training (and certainly not the absence of cross-training) to result in everyone becoming interchangeable at management's convenience.

Note one exception to this general line of reasoning:

Go for deep cross-training between traditional testers and traditional product developers. If this does not yield better thought-out code with fewer defects, and better tests within a few months, management needs to figure out who needs what additional coaching, or perhaps who needs a new job!

That's it. Modest goal, modest investment, immodestly better teamwork and resulting product.

See how I snuck that claim in at the end there with no data to support it? I don't have any data other than my own casual observation. If this seems like a lot of hooey to you, ignore it. If this has seemed pretty sensible to you, take this approach and get people talking about the results that you obtain. Make it better. Share your experience more broadly.

You might suspect me of not believing in the generalist brain - so let me be explicit about that. A generalist mind is itself a kind of specialization. Generalist minds gravitate to the thematic, to patterns that often go undetected amidst noisy details, to semi-universals and how to know which to apply in a given case.

It would seem generalists are where eventual great architects are most likely to come from. After all, great architects are not people who are only good at one or two things, or at taking only one or two perspectives into account, although many such people are given the Architect job title. Great architects see patterns across diverse domains.

Require good performance but don't require the generalist mind to be your best performer in every category — perhaps not even any one category. Judge your generalists by their work product results, not by the difficulty some generalists have relaying details when questioned. Demanding details of a mind that constantly abstracts patterns seems reasonable from the outside, but can be paralyzing to some generalists.

Learn to work with them so they want to keep working with you. Honor his or her interest in moving through a range of jobs, synthesizing the knowledge and judgment required to gradually become one of the most valuable people you can have on your team.

Note too that a strong generalist who never makes it to the great architect level can still be one of the most valuable people on your team — often able to assist in diverse situations, and frankly, to cover your ass in innumerable emergencies. Some generalists are brilliant decision makers, and inspirational communicators, even when they can't articulate the details that have actually led them to their conclusions.

I recommend the following advice for all employees, but most especially for generalists:

Train people well enough so they can leave; treat them well enough so they don't want to.

Richard Branson

Since I cast this in opposition to a common Agile goal, I want to give a small sidebar: I am strongly in favor of Agile in general, and scrum in particular. I most emphatically recommend that people stop customizing its project management model, and then complaining that "Agile doesn't work" when they have declined to change their own most counter-productive practices. But the idea that all players on the team should be interchangeable is oddly surrealistic and out of place in a line of thinking otherwise grounded in empirical evidence.

Different kinds of work are best performed by people with different kinds of minds and different personal goals — these are real differences, not arbitrary divisions of labor. We should move in the direction of cross-team awareness (not de-specialization), in expectation of improved individual and team performance.

Featured:

May/Jun 2016