November 16, 2021
At Spotify, we’re constantly working to create the best developer experience possible for our iOS engineers. Build Time Improvement is one of the most common requests for infrastructure teams and as such, we constantly strive to improve our infrastructure toolchain.
We’re excited to be open source XCRemoteCache, the library we’ve created to mitigate long local builds. As the name implies, this library is a remote caching implementation for iOS projects that aims to reuse Xcode target artifacts created on Integrated Integration (CI) machines. It supports the Objective-C, Swift, and Objective + Swift goals and can be easily integrated with existing Xcode projects, including those run by Cocopods or Carthage.
After all, XCRemoteCache results in a 70% reduction in clean build time (We classify as a build Clear When at least 50% of all targets compile at least one file).
Using our Xcode Build Metrics (see our open source XCMetrics project for more details on how this works), we’ve found that it often takes our developers more than 10 minutes to create a major Spotify iOS application. Although the number of these buildings is relatively low (less than 3% of all buildings), they take up more than 50% of the global building time (Figure 1).
After some investigation, it was revealed that chronic buildup usually occurs after rebusing or assembling distant branches. Implementing a remote cache solution was the perfect fit.
Remote Cache Policy
A popular strategy for remote caching is to use the “compile once, use everywhere” method to speed up the creation of large applications. Unless all input files and compilation parameters are the same, instead of creating a target locally, one can download artifacts created and shared from another machine. One of the key successes for remote caching is finding an optimal caching layer. Caching units that are very granular, where every single part of the compilation step can be cached, broad network traffic overhead, which can offset CPU savings. On the other hand, placing the entire codebase in a cacheable unit can significantly reduce the caching hit rate; Each single local change eliminates remotely available cache artifacts, triggering a complete build locally.
The main Spotify iOS application is highly modularized and contains over 400 independent modules configured as separate Xcode targets. Applying goal-level caching was normal, and as we learned later, the right decision.
Designing remote cache solutions
At the design stage, our goal was to come up with a simple solution that could be applied to a wide range of iOS applications, with minimal or no project changes. Given how the Excode build system works was an ambitious goal. Before moving on to the solution applied directly, let’s consider how an Xcode build actually works.
How does XCRemoteCache work?
In general, all caching processes use a fingerprint of input files to identify whether build products can be reused. However, finding a specific set of these files is a daunting task. The Xcode build system is very generous when it comes to reliability features. It hopefully tries to find a definition of dependency (header file,
.swiftmodule, Etc.) in all available search paths – either provided by the developer in Xcode project settings or in the current build product directory. As a result, developers do not have to specify all dependencies that use a target, but only make sure that those dependencies are placed in the correct position. Before Xcode actually needs them. The fact that compilers are able to inherently find the necessary dependencies prevents easy fingerprint creation by hashing out all the files available in the title and framework search paths – a list of files to consider in fingerprinting will often be too extensive.
On the other hand, we have noticed that Xcode works quite well for local incremental builds and implements only a narrow subset of the steps that have been affected since the last build. In other words, the Xcode build system knows which files are the actual input files for the compilation, but that list is created as the output of the compiler (.d file) and is not available before a compilation.
XCRemoteCache uses a unique method to automatically detect all input files in a compilation based on the git history and dependency list provided as a compilation output. The side of the generation is called Producer mode, Along with the compilation product, also uploads a meta file. That file contains a list of all the compilation files used by the compiler and the complete SHA-1 (Secure Hash Algorithm 1) commit identifier against which it was created. Producer mode should run in CI for each primary branch (e.g.
develop) Commit, as part of the post-merge episode.
Towards the consumer (aka Consumer mode), XCRemoteCache finds the promise of the most recent history for which the remote server has build artifacts and creates a fingerprint based on the input file provided in the meta file. Imagine two developers,
B, Works at their local branch
featureB, Branch out
Commit3, Respectively (Figure 2). A CI job that creates and uploads cache artifacts to the central server only finishes work for Commit 1, 2 and 4. For some reason,
Commit3 Artwork is not ready – either construction is in progress or has failed. The developer will reuse the patterns created for A’s machine
Commit1, When the developer takes them from B.
Commit2 – Tried it
Commit3, But they are not ready yet.
With that method, XCRemoteCache gets a strict list of input files almost for free
Assuming we have an app divided into a few independent targets and local branches that aren’t too far removed from the primary branch, the caching hit rate will be higher, only subtracting those targets that contain changes compared to the promise of using remote patterns.
Create works of art
Another problem to consider is “Build Artifact Portability” between multiple machines. Different types of compilation output files contain absolute paths, so for complete compatibility, some kind of normalization is required. For iOS projects, such a step is required
.swiftmodule File and debug symbols.
The project has been cloned
.swiftmodule Files that do not match the byte level. Swiftmodule represents an inter-module API (.h to Swift Counterparts) that can be included in a list of fingerprinting files. If two machines do not have the same absolute source root path, XCRemoteCache carries an additional fingerprint file (called a fingerprint override) to avoid false cache misses.
.swiftmodule Which includes a fingerprint of all the files used in the compilation step. A fingerprint override is the path agnostic, so the opposite
.swiftmodule File, it can be used as a byte-level stable fingerprint of a Swift target.
Debug symbols, other path-sensitive files, are added to the binary package to attach machine code to the corresponding source location when debugging. XCRemoteCache LLDB gains support for rewriting using runtime paths
settings set target.source-map. Both the producer and the consumer pass
debug-prefix-map The parameters of the Swift and Clang compilers create a simple, single-line command to LLDB source mapping to align the source route of all debug symbols.
At Spotify, we have a fast, well-optimized CI function that no longer slows down our development feedback loop (please explain how we gave our macOS CI superpowers to read more about this) so we’re focused on implementing XCRemoteCache on local machines. I did. Note that the tool is capable of working on CI machines as well as accelerating PR tasks.
Under control, XCRemoteCache was able to reduce the time spent creating the first iOS Spotify application by 85%. It was a great achievement, but in reality, the developers introduced changes locally and some goals had to be compiled locally. Our goal was to assess the real-world situation to understand their real impact. To estimate this, we rolled out the remote cache of 50% of our developers for a week and compared all the build metrics.
The results exceeded our expectations – we noticed a huge improvement in local construction time: Medium clean build and incremental build time decreased by 70% and 6%, respectively. We are classified as build Clear When at least 50% of all targets compile at least one file. Other builds that compile at least one file Growing.
So, we enabled XCRemoteCache in our main application over a year ago and since then, it has worked flawlessly. We get very positive feedback from our developers and as a side effect, we’ve seen almost twice as much clean build part over pre-remote cache time. Developers nowadays are twice as likely to re-establish their functional branches, eventually leading to less conflict when creating a pull request.
How to integrate XCRemoteCache into your existing project
Now that XCRemoteCache is open source, you can apply it to your own project with minimal effort. It supports multiple project setups, including those managed by CocoaPods, Carthage, or any other custom dependency management. Keep in mind, however, that for optimal results, your project should be split into several goals or you may run the risk of frequently caching failing as described above.
For seamless integration, we are open sourcing a CocoaPods plugin and providing an automated script to modify an existing .xcodeproj project.
XCRemoteCache works with any HTTP server that supports PUT, HEAD and GET requests. You are free to choose the server that works best for you, along with two popular storage options offered by Amazon S3 and Google’s Google Cloud Storage. We also provide a simple docker image that hosts a local server, perfect for development episodes.
With it, you’ll be able to try XCRemoteCache in minutes. For a list of integration steps, go How Division at GitHub Repo.
Contribute to XCRemoteCache
The XCRemoteCache tool is written entirely in Swift, so iOS developers can easily familiarize themselves with the codebase and potentially contribute to it. We’ve tried to cover most common situations but, keeping in mind that Xcode projects may have very custom setups, some of them may not be compatible at the moment. Therefore, any input from the community, both raising issues or requesting requests, is very welcome. We believe that, together, we will be able to take the project further and support a wider audience of iOS developers.
If you would like to contribute to CodeBase, be sure to check out our Development Guide. And if you would like to work full-time on such tools, please visit our open job positions.
I would like to personally thank Eric Camacho for his help in preparing this blog post.
Xcode is Apple Inc. It is a trademark, registered in US and other countries