May 18, 2021
TLDR; In the 08th episode of our podcast series “Spotify: A Product Story”, we share stories and lessons from our hometown developer portal, backstage creation and open sourcing. Hear why a developer-friendly, market-based platform like Backstage can only be built on Spotify (where autonomy is valued, not a top-down mandate) and why it makes Backstage flexible for other companies. Listen to the episode now and get all our hard-earned lessons for the entertaining podcast form অথবা or read on for episode highlights and learn more about this important time of Spotify’s growth.
How it all started: “Like a cold shower”
The story began five years ago when Spotify had a problem: we were growing fast. Really, really fast. This should be a great problem, instead of increasing our speed without it, adding new fares is actually slowing us down.
As Pia Nielsen, engineering director at the podcast, explained, a metric from SpotFi’s platform team that was used to measure productivity: How long did it take the new engineer to integrate their tenth pool request into Spotify?
The answer was no good – more than 60 days. That is, from the day an engineer walks through the door of Spotify, it will take them another two months to contribute the code in the form of a tenth pool request.
But numbers alone do not capture the whole feeling. Gustav Söderström, Spotify’s chief R&D officer and podcast host, asked Pia what it was like to see the “60 Days” metric for the first time:
Gustav: Was it like, “Maybe okay”? Or was it like, “It seems too long”?
Also: Having spent 15 years as an engineer at other companies, it was like a cold shower.
Brrr. So the first thing Pia’s team had to do was find out what the new tenant was feeling cold. Why do productivity decrease as headcount increases?
Engineers are also users
When it comes to their own employees, companies often avoid researching users – after all, why ask when you can only direct?
But Spotify’s platform team sees Spotify’s developers as their customers. Their priority is our priority. Their point of pain is the problem we solve. So, to know what stuck to our engineers, the first thing to ask our engineers.
According to Peer, two common causes of productivity decline have emerged:
- Context switching: “People are constantly interrupted … new entrants had to tap someone’s shoulder because there was hardly any documentation.”
- Discoverability: “People can’t find things. It was as simple as that. It took forever to find the right service. There were many Almost Duplicate – not pure imitation – because people are very smart and they will recognize it.
There will be 15 different versions of the same service, each talking about slightly different needs of different groups. And what if a new team needs similar services? Instead of sorting through all these versions … they will simply create another version of the same service for them.
Similarly, this is what worked before for Spotify: small, autonomous teams are building fast. But that basic agile approach reached its limits. More team means more confusion, as evidenced by our onboarding metrics. New hires don’t even know where to start – leave it to our “spaghetti” codebase to explain to other engineers without tapping on the shoulder. It was a way of working that was becoming so common, we gave it a name – “rumor-driven development”.
And as Spotify continues to grow, the problem gets worse.
Speed, scale, autonomy … choose two?
Now that the problem was clear, the solution was also clear: centralization. But just as it was clear that a centralized team would always be much slower than a much smaller team. Need to trade speed for Spotify scale?
As it turns out, the question was irrelevant. Working to restore productivity, the platform team realized that a top-down, centralized approach would not work for others in Spotify, for many more fundamental reasons: it was not just part of Spotify’s DNA. As Pia explains in the podcast:
“So we basically knew we couldn’t create a centralized solution. It will never work. No one will use it. And really none of us even believed it. We joined Spotify because we all liked autonomy. We thought it was brilliant to liberate people. So the culture really spoke to us there: “Well, you have no choice but to build something central and force everyone.”
What made Spotify Engineering great is slowing it down now: too much autonomy. But that culture of autonomy will lead to better solutions than a simple technical requirements list or top-down mandate. As the VP of Spotify’s engineering, Tyson Singer says, “Backstage To be successful with our engineers, this is not the only solution, it needs to be a better solution:
“Most of the time, if we go out and we tell people to do X, they just shake their heads and do it wherever they want. So we really have to sell to them. Basically we have to improve their lives with what we do. And so on [our culture] If we want to take control of this divisive problem of our technology ecosystem, we have really talked about our approach. ”
Spotify wanted something that could give us everything: speed, scale – and a new concept of Spotify – aligned autonomy. And so backstage conception and birth took place.
How it works: Not only accepted, but embraced
So if we can’t get someone to use it, how do we know it’s working? Every day, we see 280 engineering teams inside Spotify manage more than 2,000 backend services, 300 websites, 4,000 data pipelines and 200 mobile features using backstage.
Contribution numbers are even more impressive. More than 200 engineers have contributed features backstage inside Spotify. We now have 120+ plugins created by our 50+ team. And 80% of the contribution came from Spotfire outside the backstage core team.
People can find what they need without constantly interrupting their fellow developers. Any Spotfire – not just engineers, but members of the compliance and security team can easily discover all the software in our ecosystem, see who owns it, and access technical documentation in a centralized location. In a decentralized environment like speed optimized and spotted, this information makes all the difference in being easily accessible.
For a fast-growing organization like ours, it’s a game-changing improvement for both productivity and developer happiness যা which we believe is at hand. And we know that the open source version will be able to transform other technology companies as well. As a product, backstage is when you treat your developers as thoughtfully as your users. According to our company-wide survey, 0% of our internal users are satisfied with backstage.
Want to know what happens next? How many bone-chilling “days0 to tenth consecutive requests” onboarding metrics have we been able to do? How did our domestic developer portal become Spotify’s largest open source project? And the significance of this humble GIF?
To know the full story – listen to episode 08 – “When to create vs. buy – and when to open source”. You will hear from Gustav, Tyson and Pia, as well as Jeremiah Loin, CEO of Prefect.IO, a company known as the “Open Core” model. Now streaming on Spotify – or wherever you listen to podcasts!
Want to know more about how Spotify was created, directly from those who created it? The podcast series “Spotify: A Product Story” shares the stories behind the most important product strategy lessons we’ve learned at Spotify, which were actually told to the people who were there.
In each episode, Spotify’s chief R&D officer, Gustav Söderström, is joined by Spotify internal and special guests, from Metallica’s Lars Ulrich and Napster’s Sean Parker to ML legend Andrew Ng.
How did P2P networking and local caching create a sense of magic in the first Spotify app? How did we go from stashing servers in a cupboard to running Google Cloud’s biggest dataflow of all time? What exactly does it mean to create an ML-first product? And what are the next boundaries for manufacturers and audio formats? You can find all the podcast episodes here.