Hosting the Swift Package Index
Posted Tuesday, March 30, 2021.
Making any open-source project sustainable for the long term is challenging. You need to find time to work on the code, plan out features, respond to feedback, and if the project is not only code but a live web site, it also needs hosting and regular maintenance.
The Swift Package Index has slightly bigger hosting problems than most web sites, too. In addition to web and database servers, we need a significant amount of processing power to monitor and analyse constant package releases and run our build machines that deal with the ~30,000 Swift builds per month that process through our compatibility build system. đ
Weâre delighted to say that as of today, all of the Swift Package Indexâs hosting problems are taken care of thanks to the generosity of two companies that believe in open-source.
First, MacStadium. The site has run on their Mac mini infrastructure since the public launch through their open-source programme. They have accommodated every request we have made, including giving us access to a DTK, which enabled Apple silicon compatibility results months in advance of M1 machines becoming available. They are a pleasure to work with and consistently supportive. Thank you to Heather, Brian, and all of the support team at MacStadium.
However, while running the web hosting, the database, package update monitoring, Linux builders, and all the Apple builders all on the same hardware is possible, itâs not ideal. Our build queue had become quite delayed because we were asking those Mac minis to do too much.
Thatâs where Microsoft Azure stepped in. We have now eased the load by migrating our web hosting and Linux builders to Azure. The site is load balanced and redundant, which not only increases speed and availability, it also decreases the maintenance effort required. Working with Microsoft has been an absolute pleasure, and their only priority was to see the project succeed. Nothing was too much trouble. Thank you to Stormy, Thomas, Shelby, and Candice.
With the web servers, database, package update monitoring, and Linux builders hosted at Azure and the macOS, macOS ARM, iOS, tvOS, and watchOS builders at MacStadium, the hosting for this project is secure and stable for the foreseeable future. Thank you so much to both companies for their help. Running this site wouldnât be possible without them.
Note: Itâs worth mentioning that this blog post is not a condition of any agreement between us and either of these companies. All this support came without any expectation of anything other than enabling an open-source project. We are writing this purely because we are grateful for their help. â¤ď¸
Whatâs next? February 2021
Posted Tuesday, February 2, 2021.Itâs been about seven months since we launched the Swift Package Index and weâve been hard at work implementing a few major updates. Most significantly, real-world package compatibility, a comprehensive set of RSS feeds and Twitter updates, and most recently inline README files.
But weâre far from done! đ
So, in the spirit of letting you all know what weâre planning for this project next, we wanted to start a regular series of blog posts about the direction weâre heading with feature development. Weâll post every couple of months, giving a quick recap of what weâve achieved along with where weâre heading next. Itâs not a formal roadmap. Itâs more a declaration of where our focus will be for the next couple of months.
So without further ado, what are we focusing on next?
Intermediate Search Page
Currently, search results are displayed inline on the home page, which makes it hard to share package searches (thereâs no URL for a search), or integrate with productivity tools like Alfred or Raycast, or even DuckDuckGo through their bangs feature. We want to fix that.
Package Collections
If youâve been following the package collection work in the SPM project, then it might have struck you that the Swift Package Index would be a great source of package collections, when they are added. We agree! Weâve already been working hard to prepare for this feature and want to have something publicly available as soon as the feature becomes available in SPM.
Design updates
The site design suited our feature set at launch but has become a little overloaded as we added features. This work has already started, with a new package page layout rolling out last Friday. Thereâs more to do though, including improvements to the home page. The lists of recent package additions and recent package releases are fine, but we can do better!
The Swift Package Index macOS App
Weâve always wanted to create a Swift Package Index Mac app of some kind. The appâs main purpose would be to allow you to easily try packages out in a Swift Playground (utilising the wonderful Arena tool). But it might end up being more over the long term. We want to define what this app will do, and then get something out that works towards that definition. We talked about the interesting story behind the âTry this packageâ feature on John Sundellâs podcast last year if youâre interested!
And MoreâŚ
This is not a comprehensive list of things weâll do. This is a young project, and we want to keep development agile. If we have a great idea, it may go into the plan ahead of everything you read above!
Thank you!
Weâd also like to use this update post to say a huge thank you to our generous sponsors. We wouldnât be able to put in the time and effort needed by this project without the generous support of Alexandre Colucci, Artem Loenko, Bartosz Dolewski, Boris BĂźgling, Byron Davies, Chris, Dominik Hauser, Federico Zanetello, Gleb Shevchenko, James Sherlock, Jason Bruder, Jeffrey Macko, Joseph Heck, Joshua Kaunert, Juanpe CatalĂĄn, Jules Coynel, Kyle Lee, Leo Picado, Marcos Griselli, Massimo Biolcati, MatÄj KaĹĄpar JirĂĄsek, Max Desiatov, Nathan Harris, Oliver Drobnik, PatrikTheDev, Paul Farnell, Paul Hudson, Peter Steinberger, PSPDFKit, Riley Testut, Robin Kunde, Sam Symons, Scott Robbins, Stefan-Cosmin Romanescu, Thomas Dohmke, Tim Condon, Umar Haroon, Vadim Shpakovski, and Xavier Lowmiller. Thank you all! â¤ď¸
Would you like to be part of this list next time we post an update like this? Sponsoring the Swift Package Index is done through GitHub Sponsors, and your support will allow us to continue to build this site into something special.
Inline README Files!
Posted Thursday, December 24, 2020.If youâve visited the main package index site over the last week or so, you might have noticed something new. Scroll down a little, and youâll see the README file for the package, right below all the other information that we aggregate.
When we consider adding a new feature, at the heart of our decision has to be the question âDoes this feature allow people to make better decisions about the package theyâre considering?â Rather embarrassingly, the README file is probably the most important piece of information when picking a package dependency. Letâs not focus on that though, and instead be thankful that we finally got around to adding it! đ
Weâre aware of a couple of problems with README files that contain embedded HTML or make use of in-page anchor links that are often used to build a table of contents. Weâll get those fixed over time, but decided that it was worth rolling the feature out even with a couple of remaining issues.
Of course, weâve not been able to test every single README file, so If you notice any other issues, please let us know by opening an issue.
Thank you all for your support this year, and we hope you enjoy this feature that brings our year of development on the Swift Package Index to an end. We canât wait to show you what we have planned for 2021!
Keeping Up To Date with Swift Packages
Posted Wednesday, November 25, 2020.How do you keep up with new releases of the packages you use? How do you discover new packages? Itâs a tough challenge to keep up to date with everything that the community releases, so we have some announcements today that can help you stay informed!
Before we get started with the announcements, we want to credit James Sherlouk for his invaluable contributions to building the Twitter updates feature. He was instrumental in getting this feature up and running and has been making other improvements in several areas of the site since we launched. Thank you, James!
Updates via Twitter
You may already be following the @SwiftPackages account where we tweet occasional updates about the project. But as of yesterday, weâre also tweeting on the @PackageFirehose account. As you might guess by the name, this is a high volume account that posts every time thereâs a new package added to the index, and every time thereâs a new release of a package that we track. Itâs around 20 tweets a day, if that counts as high volume!

While weâre on the subject of Twitter, weâd love to know what youâd like to see us do with the @SwiftPackages account. Weâve set up a short survey if you have a couple of minutes. Weâd love to hear your opinions.
Updates via RSS
Weâve had RSS support for a while now, but weâve not talked about it on the blog before, so itâs worth mentioning. If youâd prefer to keep up to date via a feed reader, or if you want fine-grained control over what type of package updates youâd like to be notified of then subscribe to one of our RSS feeds:
- New Packages â A feed with packages that are new to the Swift Package Index.
- All Package Releases â A feed of all new package version releases.
One advantage that these feeds have over Twitter is that they can be filtered with query string parameters. The package releases feed accepts four parameters; major
, minor
, patch
, and pre
. Pass true
to any of these parameters to filter the feed on that part of the semantic version number. So, you can subscribe to only major package release, or major and minor package releases combined, or if you only want to know about pre-release versions, you can find those here. Use any permutation of the parameters to get exactly the feed that you want.
We hope you find both of these mechanisms for keeping up to date useful!
The Swift Package Index Metadata File â First Steps
Posted Friday, October 30, 2020.The vast majority of the information you see in the package index is generated from a single URL, the location of the packageâs git repository.
From that URL we gather data from the repository itself, the package manifest, the GitHub API (yes, itâs GitHub only for now, but we do plan to support other hosts), and from running builds of the package to check compatibility with operating systems and Swift versions.
Edge cases
Of course, as with everything in software development, itâs not long before you find yourself talking about edge cases.
While creating the build system, we quickly encountered packages where watchOS targets failed to build. We use Xcodeâs automatic scheme creation functionality while running builds, but Xcode generates schemes that always include test targets, and XCTest
isnât available on watchOS. Using automatic schemes for watchOS builds wasnât going to work.
The only way to have these builds succeed is for our build system to use a specific scheme, rather than an automatically created one. Many packages already have schemes for this purpose, but as package authors can give schemes any name, itâs not possible to easily discover them.
Enter the Swift Package Index configuration file
Rather than trying to keep the build system 100% generic, we needed a mechanism for package authors to specify some configuration information, such as the scheme name to use to build watchOS targets.
We support this via a configuration file, .spi.yml
which we will look for in the root of your Swift package repository.
Schemes for watchOS
You can see the popular package swift-composable-architecture
by Pointfree making use of this config file:
version: 1
builder:
configs:
- platform: watchos
scheme: ComposableArchitecture_watchOS
Our build servers look for this file and use any scheme information we find for each platform. If a platform isnât listed, we use our default heuristics to determine the scheme as outlined in our build FAQ.
Images for Linux
Custom schemes arenât all we support though. Package authors can also use the .spi.yml
file to configure custom docker base images for our Linux builds.
We build packages for Linux with docker commands, selecting between the various base images that Apple provide.
/usr/local/bin/docker run --rm -v "$PWD":/host -w /host swiftlang/swift:5.2.4 swift build --enable-test-discovery
This works well unless a package requires an OS-level dependency, like OpenSSL.
We may eventually provide our own set of base images for supported Swift versions that also include common dependencies like OpenSSL. For now, we are asking package authors whose builds fail due to missing dependencies to create their own for the Swift versions they support.
For security reasons, we donât allow package authors to specify arbitrary images in the configuration file, so if you need to use this feature, please raise an issue asking for us to support a custom base image.
The AWS library Soto is an example of a package that makes use of this feature in its .spi.yml
file:
version: 1
builder:
configs:
- platform: linux
swift_version: '5.0'
image: adamfowlerphoto/aws-sdk-swift:5.0
- platform: linux
swift_version: '5.1'
image: adamfowlerphoto/aws-sdk-swift:5.1
- platform: linux
swift_version: '5.2'
image: adamfowlerphoto/aws-sdk-swift:5.2
- platform: linux
swift_version: '5.3'
image: adamfowlerphoto/aws-sdk-swift:5.3
Future direction
Itâs early days for this configuration file. We created it to solve issues which package authors were facing trying to make sure that the language and platform compatibility we report reflects reality.
We expect this file to evolve, which is why it carries a version number. Here are some of the things we are planning to add over time, so that package authors can control them:
- Author metadata. Full name, blog URL, Twitter account, etc.
- Package keywords or categories.
- The location of any hosted documentation for a package.
- Sponsorship or funding information for a package.
More metadata possibilities are under discussion in this GitHub issue. This is a great place to chime in if you have suggestions.
File format
We chose YAML
as the file format because it is an uncontroversial, universally beloved format that is entirely free of issues. đŹ
Seriously though, we chose YAML as itâs relatively easy for a human to read and write, and itâs well supported in Swift. While it has some downsides compared to JSON, we believe itâs a more practical format given that package authors need to create this file manually.
Of course, weâll let adoption be the guide as to how well it is working in practice!
Funding the Future of the Swift Package Index
Posted Monday, October 12, 2020.Itâs been almost four months since we launched the Swift Package Index, and weâve been so happy to see so many positive reactions to it across the community. Thank you all for checking it out and spreading the word.
Itâs still early days for Swift Package Manager adoption but we are confident that it will become the dominant technology for Swift dependency management, and we want the Swift Package Index to grow right alongside it.
We believe weâre off to a great start with what we have so far, but this is only the beginning. We have so many ideas for âbug fixes and improvementsâ! đ
Many open-source projects are just code and donât need constant maintenance and monitoring, but the Swift Package Index is a little different. Itâs a hosted instance of the site with a reasonably complex automated build system that has run over 600,000 builds so far (increasing by 7,000 more every day!). It takes significant time and effort to keep things running smoothly.
We want to find a way to make sure that we can continue to give this site the time and attention it needs, but we donât really want to go down the route of including advertising. Instead, we are opening up sponsorship on GitHub.
If you (or your company) would like to support the site, we would be thrilled to be able to make this a community-funded effort for years to come.
Also, a huge thanks to Oliver, who beat us to even announcing this by sponsoring the project yesterday! Thanks for being our very first sponsor. đ
Thank you!
Dave & Sven
Building 3,238 Swift Packages for Apple Silicon
Posted Wednesday, September 9, 2020.As part of the Swift Package Index build system, we have processed what must be the most extensive test of Apple Silicon compatibility outside of Apple.
At the time of writing, we currently have 12,942 Apple Silicon builds in our database covering 3,238 packages. Why so many? We test up to three versions of each package and compile each version with both xcodebuild
and swift build
.
Iâm sure that many of the Apple Silicon DTK machines Apple shipped around the world are working very hard, testing apps for compatibility, but Iâm not sure any of them have worked quite as hard as ours has. đ
What did we find?
As you might expect, thereâs plenty of good news. For packages with successful Intel builds, the vast majority also successfully build for Apple Silicon. đ
There are failures, though, and the most interesting failures are where the build fails on Apple Silicon while the corresponding Intel build succeeds. In cases like this, that indicates a compatibility issue that will potentially need some attention from the package author. Weâve identified 139 packages (~4%) where this is the case, and have compiled a list of them for reference.
There are a few common errors that weâre observing in these failures, and most of them are from one of the following problems:
- Any inline assembly will fail, obviously!
Float80
is unavailable.x86_thread_state64_t
is unavailable.__darwin_arm_thread_state64' has no member '__rsp'
.cannot convert value of type 'CVaListPointer' to expected argument type '__darwin_va_list?'
.- Sometimes, the compiler crashes!
Even though 139 failing packages might sound like a lot, itâs not that bad, especially when you consider that many of the failures all fail due to an issue in a common base dependency, PerfectLib
.
Where do we go from here?
Even though the results are generally positive, it does show that thereâs a bit of work that the community need to do to ensure the transition to Apple Silicon is as smooth as possible.
If youâre a package author and do not have a DTK machine at your disposal, please feel free to use ours via the Swift Package Index build system! Make changes to your packages, push them to a pre-release tag (for example, X.X.X-arm-beta-X) and the build system will pick up that tag and build it. You should see new build results, usually within a couple of hours of pushing a tag.
If you have any problems with the build system, thereâs a couple of ways to contact us. Weâre in the #swiftpackageindex
channel over at the SwiftPM Slack, or if it looks like there might be something wrong, please raise an issue on GitHub.
Launching Language and Platform Package Compatibility
Posted Thursday, August 20, 2020.Whatâs the first question you need an answer to after finding a package that fits your needs?
âDoes this package work with the Swift version and platform that my app uses?â
When we initially launched the Swift Package Index, we attempted to answer this question with the metadata available in the package manifest. Namely the swiftLanguageVersions
and platforms
properties.
The problem is that neither of those properties is perfect. swiftLanguageVersions
isnât granular enough, only officially allowing values of v4
, v4_2
, and v5
. The platforms
property is better, but doesnât let package authors declare compatibility with non-Apple operating systems such as Linux.
Wouldnât it be fantastic if you could see a matrix like this for every package? đ

Look at how information-rich that matrix is. You can instantly see that the latest stable version of PromiseKit is compatible with every version of Swift back to 4.2, and every platform, including Linux. Then, you can see that the alpha version in development drops support for iOS, tvOS, and watchOS, and Swift 4.2. That seems suspicious, right? Keep looking, and youâll see that the default branch fixes all those issues and restores compatibility. Iâm confident looking at that matrix that when 7.0.0 is released, itâll have green ticks across the board, but I also know to not depend on this current alpha. Thatâs practical, actionable information.
When we started thinking about how best to solve this problem, the obvious best solution was to build the packages! What better way to see if a package is compatible with Swift 4.2 than to build it with the version of xcodebuild
that shipped with Xcode 10.1.
So thatâs what we did, and itâs available right now. Why not give it a try by searching for a few of your favourite packages? đ
Accurate, real-world compatibility data
Itâs a little more complicated than âjust build each packageâ though. A package might build with Swift 5.2 on iOS, but that same build might fail using Swift 5.2 on macOS due to a UIKit dependency, or other macOS specific issue. Whatâs needed is a matrix of builds to generate an accurate picture of compatibility.
So, if we run builds using Swift 5.1 on iOS, macOS, tvOS, watchOS, and with Linux and any of them pass, itâs compatible with Swift 5.2. If any version of Swift builds without failure on iOS, then the package supports iOS.
We ended up with a platform list of:
- iOS using
xcodebuild
- macOS using
xcodebuild
- macOS using
swift build
(there are good reasons whereswift build
would pass in circumstances wherexcodebuild
might fail) - macOS using
xcodebuild
on Apple Silicon (yes, compiled using a DTK!) - macOS using
swift build
on Apple Silicon - tvOS using
xcodebuild
- watchOS using
xcodebuild
- Linux using
swift build
We then decided on a list of Swift compiler versions weâd like to check compatibility with:
- Swift 4.2 (4.2.1)
- Swift 5.0 (5.0.1)
- Swift 5.1 (5.1.3)
- Swift 5.2 (5.2.4)
- Swift 5.3 (beta)
Thatâs up to 32 builds per package, but thatâs just the beginning. What if thereâs a stable release and a beta release? The stable version might support Swift 4.2 and higher, and the new beta might drop support for anything less than Swift 5.2. Thatâs information which would be important when choosing a package, so we need to show it. As we also track the status of the default branch, we must build that too, and weâve quickly arrived at a place where we might need to run 96 builds per package! With almost 3,200 packages in the index, thatâs potentially more than 300,000 builds! đ
In practice, itâs less than that as most packages donât have a current beta release, but itâs still a lot of builds. Weâve processed more than 200,000 builds as I write this, and weâre not quite finished. As of today, weâre at 99% though, so we almost made it before launch! đŹ
If youâve been following these tweets, it should be obvious what all that processing was! Letâs take a look at the last 30 days of CPU graphs for our production server, a 2018 Mac mini with 32Gb RAM and a 6-core i7 CPU:

You can see a few of our final test runs in that graph, and then we started processing for real. Since then, weâve kept the CPU completely pegged for more than two weeks. Weâve also had our staging Mac mini, a spare 2016 MacBook Pro, and a DTK working on builds too.
Everyone loves badges
Providing compatibility information on this site is one thing, but everyone loves adorning their packages pages with shields.io badges, donât they? If you maintain an open-source project, wouldnât you like to show off real compatibility status in your README file, like this?

If youâre a package author, click the âCopy badgeâ button below each of the compatibility matrices and youâll have a Markdown image link in your clipboard, ready to use.
Your users will always see live, accurate compatibility information that updates whenever you release a new version.
Credit where itâs due!
First of all, weâd like to thank our kind friends at MacStadium for providing the significant hosting resources for this project as part of their open-source programme. At one point we were a little concerned that we might melt their machines, and we very glad that we didnât. Theyâve performed incredibly.
We also want to say thank you to Ankit Aggarwal and Boris BĂźgling of Apple. Their tireless help and support on the SwiftPM Slack saved us countless hours figuring out the correct way to approach this problem.
Finally, weâd love to say thank you to everyone who provided help and feedback along the way as we built this feature. We couldnât have done it without any of you.
Wrapping up
Some package authors set up continuous integration for their packages and of course, that includes a build step. That usually only covers one version of Swift though, and the information gets hidden away in a different place in each repo. We think that by centralising this data and making it available for all packages, we should be able to help this community make better decisions about their dependencies, and thatâs what this site is all about.
We hope you love this feature as much as we do! â¤ď¸
Say Hello to the Swift Package Index Blog
Posted Sunday, August 16, 2020.Hello! đ
Itâs always hard to know when to ship v1 of a project. Which features get included? Are there things that can wait? Itâs a balancing act. Youâre excited to show the world what you built, but is it good enough?
One decision we made when shipping the Swift Package Index was that a blog could wait. Looking back now, it feels like that was the right decision. The launch went smoothly, you all reacted very kindly and positively to it, and we went back to work fixing bugs and working on new features.
However, we always knew weâd need somewhere to write about what we are doing with this project. Weâd love to tell you about some of the interesting technical challenges we faced, but most of all we want to tell you about the new features weâve been building...
What features? Youâll have to wait just one more week to find out what weâve been working on, but weâre pretty excited about it, and thatâs why now was the right time to launch this blog.
Now we just need to keep writing on it! đŹ