Spotify | An Emergent Organization
The mechanisms that enable self-management also balance freedom and control
[Originally posted on July 30 2019 at On The Horizon. Porting over all the Medium materials, and I am semi-vacationing, so an older post that many may have not seen before.]
Over the past few years, I’ve read a bit about Spotify’s rethinking of their engineering organization, but I hadn’t really looked closely. I was aware of Spotify’s squads, small (less than a dozen members), self-organizing teams focused on a single feature of Spotify’s system.
However, I came across a BBC piece by Jared Lindzon about Squads, and I read this [emphasis mine]:
As Spotify began to scale, it wanted to maintain the same culture of innovation that enabled its early success — but it’s hard to operate like a disruptive start-up when your employee count balloons into the hundreds, then thousands. That’s why, in 2012, Spotify began organising employees into groups of about six to 10 people, each with a single task or assignment. Although team members don’t necessarily have the same expertise as their squad-mates, each squad has the combined expertise necessary to tackle that challenge.
Squads operate like their own start-up within a tech giant, choosing their own leaders, timetables and working methods. The Agile framework is disrupting other huge, matrix-style organisations, upending the traditional working structures of major companies including Apple, Netflix, HP and Bank of America. That’s almost certainly just the start.
What I had read made me think of Amazon’s two-pizza teams (see Platforming | What Jeff Bezos Thinks) and Haier’s microenterprises (see Evolution of the Platform Organization: 3 Haier, Rendanheyi, and Zhang Ruimin’s Vision, Lee Bryant on Digital Leadership, and Hamel and Zanini on The End of Bureaucracy). More importantly, the Spotify story of organization evolution follows the same pattern of Amazon and Haier, winding up with what I am now calling the emergent organization.
I will recap the Spotify story, offering a bit more context than Lindzon does, and then I’ll sketch the general dynamics and structure of the emergent organization, and line that up with Spotify’s example.
Spotify Squads, Tribes, Chapters, and Guilds
Henrik Kniberg is one of the driving forces behind the Spotify model. In 2012 he was working on scaling the agile development model at Spotify, and wrote Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds with Anders Ivarsson.
A squad is a small, self-organizing, self-managing team focused on a single well-defined feature, such as building and improving the Android Spotify app, or the Spotify Radio experience, or integrating payments. As Kniberg and Ivarsson state, a squad is ‘designed to feel like a startup’. The principle behind Spotify’s model is ‘loosely coupled, tightly aligned’ squads that balance autonomy with accountability.
The product owner (PO) is ‘responsible for prioritizing the work done by the team’. Teams have design leads and are supported by agile coaches. The squad members decide how to undertake the work, what methods to use, and how to coordinate. Leadership in a squad is emergent and informal, not assigned from without. The squad has direct contact with the product owner and other stakeholders in other organizational units or squads.
As shown above (taken from a video by Kniberg) the squad’s mission is detailed, informed by the overall product strategy, and each squad is operating against quarterly short-term goals. The arrow represents the path taken by the squad to deliver the feature.
Multiple squads working on related features can make up a tribe, consisting of 40–150 people. Each tribe has a tribe leader, who does not direct the work within squads, but is responsible for supporting the work of the various squads, and working to coordinate shared initiatives across the squads, as well as providing an environment to work.
The 150 people limit is derived from the so-called Dunbar Constant, a principle of anthropology from Robin Dunbar, who noted that hunter-gatherer societies stabilized around groups of that size. He hypothesized that it was a size where all members of the group could know all the others and their social relationships. The premise is that when groups grow beyond 150, hierarchy and bureaucracy appear.
Kniberg states that the tribe is the likely ‘incubator’ for the squad, creating them as needed to meet the goals of the functional domain.
Chapters and Guilds
A chapter is a network of individuals with similar skills working in similar competency areas, for example, testing, database, or machine learning. The chapter lead acts somewhat like a line manager, developing chapter members skills, and notably setting salaries based on the standards for those kinds of skills. So people can switch squads while ‘reporting’ to the same chapter lead. Note that the setting pay scale aspect of the chapter lead role is quite a departure from the startup orientation of squads since in an actual startup, the founder/owner would set salaries. (In Haier’s microenterprises (MEs), the ME leader and team are almost totally responsible for that.)
Kniberg explains the difference between the product owner and the chapter lead as the difference between ‘what’ and ‘how’:
This matches the “professor and entrepreneur” model recommended by Mary and Tom Poppendieck. The PO is the “entrepreneur” or “product champion”, focusing on delivering a great product, while the chapter lead is the “professor” or “competency leader”, focusing on technical excellence.
Note also that chapter leads are generally developers themselves, working within a squad. Being a chapter lead is a part-time function.
A guild is more like a community of practice, a diffuse network of people sharing knowledge, tools, code, and practices. While chapters are local to a tribe, guilds can extend across tribe boundaries. Guilds don’t have ‘leads’ but instead coordinators.
Dependencies and Supports
Squads can come to be highly dependent on each others’ work, which can make it necessary to refactor the way the squads are structured or what their goals are.
Since one of the motivations for the Spotify model is to allow scaling of the team and the software without slowing down, they decoupled the architecture of the Spotify platform, so that squads could release their features without coordinating with other squads. It’s a micro-services architecture, and an internal open-source approach is employed. This means anyone can make changes to anything, although that’s done with some coordination. (At Amazon, teams go so far as to create second versions of services if they want to, to speed up development.)
Some dependencies are inherent, like the relationship of development (people creating new versions of the software platform) to operations (people installing the new versions on the operational code, like the web application and various devices).
Spotify sidesteps the potential friction of ops versus dev by creating an operations squad whose job it is to ‘pave the road’ for dev squads to release code themselves, creating tools to make that easy.
Spotify and the Emergent Organization
Spotify is another example of an Emergent Organization, like those I discussed in On Emergent Leadership. I was building on findings from three MIT researchers, Deborah Ancona, Elaine Backman, and Kate Isaacs, who wrote
First, we identified three distinct types of leaders. Entrepreneurial leaders, typically concentrated at lower levels of an organization, create value for customers with new products and services; collectively, they move the organization into unexplored territory. Enabling leaders, in the middle of the organization, make sure the entrepreneurs have the resources and information they need. And architecting leaders, near the top, keep an eye on the whole game board, monitoring culture, high-level strategy, and structure.
I sidestepped their ‘lower levels’ of the organization — so difficult to escape the top-down mindset, even when researching innovative, non-hierarchical companies — by casting their model into the concentric circles representation I have been using to understand Haier’s Rendanheyi operating philosophy:
I have jiggled a version of this Emergent Organization chart, superimposing Spotify’s leadership model on it.
Entrepreneurial leaders have three complementary variants at Spotify, which in other organizational models may all be played by one individual:
Squad leaders are informally selected by the squad, and work with the team to accomplish the squad’s task: the ‘how’. In some squads, this may be a shared role among two or all of the squad.
The product owner sets the ‘what’ for the squad.
Chapter leads support individual members in their advancement and providing performance feedback, including setting compensation.
Enabling leaders at Spotify include two flavors:
Tribe leads — who support the activities of perhaps ten or more squads, not by telling them what to do by enabling and coordinating across the groups, and creating a productive working environment. They often incubate squads.
System owners — who have final say on the ‘fitness’ and integrity of changes being made over time to platform systems.
Architecting leaders at Spotify include the chief architects, who maintain a vision of the platform across multiple systems.
Very little has been written about how the rest of the Spotify organization works, outside of engineering. My sense is that functions like people operations (HR), finance, and so on, operate like shared services do at Haier, and that senior executives operate as the chief architects of the company’s strategic direction, operating philosophy, and culture.
I expect that the general lineaments of emergent organizations share key characteristics, in particular, these:
Balancing self-management with accountability. At Spotify the squad is the nexus of this balance.
Increasing innovation through decoupling. At Spotify, the loose coupling within tribes and the tight alignment within squads — and the platform architecture to support it — sidesteps bottlenecks and friction that could hold back innovation.
Independent thought but shared awareness. Spotify squads work independently, but members transit from one squad to another frequently, and the guild and chapter structures, and the open-source model together support shared knowledge, best practices, history of decisions made, and technology trends.
As Ancona, Backman, and Isaacs noted, emergent organizations square the circle:
And here’s the real beauty of the system: The mechanisms that enable self-management also balance freedom and control. The companies function efficiently and exploit new opportunities quickly even as they minimize bureaucratic rules.