The Swift Package Index Blog

Hosting the Swift Package Index

Making any open-source project sustainable for the long term is challenging. You need to find time to work on the code, plan out features, respond to feedback, and if the project is not only code but a live web site, it also needs hosting and regular maintenance.

The Swift Package Index has slightly bigger hosting problems than most web sites, too. In addition to web and database servers, we need a significant amount of processing power to monitor and analyse constant package releases and run our build machines that deal with the ~30,000 Swift builds per month that process through our compatibility build system. 😅

We’re delighted to say that as of today, all of the Swift Package Index’s hosting problems are taken care of thanks to the generosity of two companies that believe in open-source.

First, MacStadium. The site has run on their Mac mini infrastructure since the public launch through their open-source programme. They have accommodated every request we have made, including giving us access to a DTK, which enabled Apple silicon compatibility results months in advance of M1 machines becoming available. They are a pleasure to work with and consistently supportive. Thank you to Heather, Brian, and all of the support team at MacStadium.

However, while running the web hosting, the database, package update monitoring, Linux builders, and all the Apple builders all on the same hardware is possible, it’s not ideal. Our build queue had become quite delayed because we were asking those Mac minis to do too much.

That’s where Microsoft Azure stepped in. We have now eased the load by migrating our web hosting and Linux builders to Azure. The site is load balanced and redundant, which not only increases speed and availability, it also decreases the maintenance effort required. Working with Microsoft has been an absolute pleasure, and their only priority was to see the project succeed. Nothing was too much trouble. Thank you to Stormy, Thomas, Shelby, and Candice.

With the web servers, database, package update monitoring, and Linux builders hosted at Azure and the macOS, macOS ARM, iOS, tvOS, and watchOS builders at MacStadium, the hosting for this project is secure and stable for the foreseeable future. Thank you so much to both companies for their help. Running this site wouldn’t be possible without them.

Note: It’s worth mentioning that this blog post is not a condition of any agreement between us and either of these companies. All this support came without any expectation of anything other than enabling an open-source project. We are writing this purely because we are grateful for their help. ❤️

What’s next? February 2021

It’s been about seven months since we launched the Swift Package Index and we’ve been hard at work implementing a few major updates. Most significantly, real-world package compatibility, a comprehensive set of RSS feeds and Twitter updates, and most recently inline README files.

But we’re far from done! 😅

So, in the spirit of letting you all know what we’re planning for this project next, we wanted to start a regular series of blog posts about the direction we’re heading with feature development. We’ll post every couple of months, giving a quick recap of what we’ve achieved along with where we’re heading next. It’s not a formal roadmap. It’s more a declaration of where our focus will be for the next couple of months.

So without further ado, what are we focusing on next?

Intermediate Search Page

Currently, search results are displayed inline on the home page, which makes it hard to share package searches (there’s no URL for a search), or integrate with productivity tools like Alfred or Raycast, or even DuckDuckGo through their bangs feature. We want to fix that.

Package Collections

If you’ve been following the package collection work in the SPM project, then it might have struck you that the Swift Package Index would be a great source of package collections, when they are added. We agree! We’ve already been working hard to prepare for this feature and want to have something publicly available as soon as the feature becomes available in SPM.

Design updates

The site design suited our feature set at launch but has become a little overloaded as we added features. This work has already started, with a new package page layout rolling out last Friday. There’s more to do though, including improvements to the home page. The lists of recent package additions and recent package releases are fine, but we can do better!

The Swift Package Index macOS App

We’ve always wanted to create a Swift Package Index Mac app of some kind. The app’s main purpose would be to allow you to easily try packages out in a Swift Playground (utilising the wonderful Arena tool). But it might end up being more over the long term. We want to define what this app will do, and then get something out that works towards that definition. We talked about the interesting story behind the “Try this package” feature on John Sundell’s podcast last year if you’re interested!

And More…

This is not a comprehensive list of things we’ll do. This is a young project, and we want to keep development agile. If we have a great idea, it may go into the plan ahead of everything you read above!

Thank you!

We’d also like to use this update post to say a huge thank you to our generous sponsors. We wouldn’t be able to put in the time and effort needed by this project without the generous support of Alexandre Colucci, Artem Loenko, Bartosz Dolewski, Boris Bügling, Byron Davies, Chris, Dominik Hauser, Federico Zanetello, Gleb Shevchenko, James Sherlock, Jason Bruder, Jeffrey Macko, Joseph Heck, Joshua Kaunert, Juanpe Catalán, Jules Coynel, Kyle Lee, Leo Picado, Marcos Griselli, Massimo Biolcati, Matěj Kašpar Jirásek, Max Desiatov, Nathan Harris, Oliver Drobnik, PatrikTheDev, Paul Farnell, Paul Hudson, Peter Steinberger, PSPDFKit, Riley Testut, Robin Kunde, Sam Symons, Scott Robbins, Stefan-Cosmin Romanescu, Thomas Dohmke, Tim Condon, Umar Haroon, Vadim Shpakovski, and Xavier Lowmiller. Thank you all! ❤️

Would you like to be part of this list next time we post an update like this? Sponsoring the Swift Package Index is done through GitHub Sponsors, and your support will allow us to continue to build this site into something special.

Inline README Files!

If you’ve visited the main package index site over the last week or so, you might have noticed something new. Scroll down a little, and you’ll see the README file for the package, right below all the other information that we aggregate.

When we consider adding a new feature, at the heart of our decision has to be the question “Does this feature allow people to make better decisions about the package they’re considering?” Rather embarrassingly, the README file is probably the most important piece of information when picking a package dependency. Let’s not focus on that though, and instead be thankful that we finally got around to adding it! 😅

We’re aware of a couple of problems with README files that contain embedded HTML or make use of in-page anchor links that are often used to build a table of contents. We’ll get those fixed over time, but decided that it was worth rolling the feature out even with a couple of remaining issues.

Of course, we’ve not been able to test every single README file, so If you notice any other issues, please let us know by opening an issue.

Thank you all for your support this year, and we hope you enjoy this feature that brings our year of development on the Swift Package Index to an end. We can’t wait to show you what we have planned for 2021!

Keeping Up To Date with Swift Packages

How do you keep up with new releases of the packages you use? How do you discover new packages? It’s a tough challenge to keep up to date with everything that the community releases, so we have some announcements today that can help you stay informed!

Before we get started with the announcements, we want to credit James Sherlouk for his invaluable contributions to building the Twitter updates feature. He was instrumental in getting this feature up and running and has been making other improvements in several areas of the site since we launched. Thank you, James!

Updates via Twitter

You may already be following the @SwiftPackages account where we tweet occasional updates about the project. But as of yesterday, we’re also tweeting on the @PackageFirehose account. As you might guess by the name, this is a high volume account that posts every time there’s a new package added to the index, and every time there’s a new release of a package that we track. It’s around 20 tweets a day, if that counts as high volume!

While we’re on the subject of Twitter, we’d love to know what you’d like to see us do with the @SwiftPackages account. We’ve set up a short survey if you have a couple of minutes. We’d love to hear your opinions.

Updates via RSS

We’ve had RSS support for a while now, but we’ve not talked about it on the blog before, so it’s worth mentioning. If you’d prefer to keep up to date via a feed reader, or if you want fine-grained control over what type of package updates you’d like to be notified of then subscribe to one of our RSS feeds:

• New Packages – A feed with packages that are new to the Swift Package Index.
• All Package Releases – A feed of all new package version releases.

One advantage that these feeds have over Twitter is that they can be filtered with query string parameters. The package releases feed accepts four parameters; major, minor, patch, and pre. Pass true to any of these parameters to filter the feed on that part of the semantic version number. So, you can subscribe to only major package release, or major and minor package releases combined, or if you only want to know about pre-release versions, you can find those here. Use any permutation of the parameters to get exactly the feed that you want.

We hope you find both of these mechanisms for keeping up to date useful!

The Swift Package Index Metadata File – First Steps

The vast majority of the information you see in the package index is generated from a single URL, the location of the package’s git repository.

From that URL we gather data from the repository itself, the package manifest, the GitHub API (yes, it’s GitHub only for now, but we do plan to support other hosts), and from running builds of the package to check compatibility with operating systems and Swift versions.

Edge cases

Of course, as with everything in software development, it’s not long before you find yourself talking about edge cases.

While creating the build system, we quickly encountered packages where watchOS targets failed to build. We use Xcode’s automatic scheme creation functionality while running builds, but Xcode generates schemes that always include test targets, and XCTest isn’t available on watchOS. Using automatic schemes for watchOS builds wasn’t going to work.

The only way to have these builds succeed is for our build system to use a specific scheme, rather than an automatically created one. Many packages already have schemes for this purpose, but as package authors can give schemes any name, it’s not possible to easily discover them.

Enter the Swift Package Index configuration file

Rather than trying to keep the build system 100% generic, we needed a mechanism for package authors to specify some configuration information, such as the scheme name to use to build watchOS targets.

We support this via a configuration file, .spi.yml which we will look for in the root of your Swift package repository.

Schemes for watchOS

You can see the popular package swift-composable-architecture by Pointfree making use of this config file:

version: 1
builder:
  configs:
  - platform: watchos
    scheme: ComposableArchitecture_watchOS

Our build servers look for this file and use any scheme information we find for each platform. If a platform isn’t listed, we use our default heuristics to determine the scheme as outlined in our build FAQ.

Images for Linux

Custom schemes aren’t all we support though. Package authors can also use the .spi.yml file to configure custom docker base images for our Linux builds.

We build packages for Linux with docker commands, selecting between the various base images that Apple provide.

/usr/local/bin/docker run --rm -v "$PWD":/host -w /host swiftlang/swift:5.2.4 swift build --enable-test-discovery

This works well unless a package requires an OS-level dependency, like OpenSSL.

We may eventually provide our own set of base images for supported Swift versions that also include common dependencies like OpenSSL. For now, we are asking package authors whose builds fail due to missing dependencies to create their own for the Swift versions they support.

For security reasons, we don’t allow package authors to specify arbitrary images in the configuration file, so if you need to use this feature, please raise an issue asking for us to support a custom base image.

The AWS library Soto is an example of a package that makes use of this feature in its .spi.yml file:

version: 1
builder:
  configs:
  - platform: linux
    swift_version: '5.0'
    image: adamfowlerphoto/aws-sdk-swift:5.0
  - platform: linux
    swift_version: '5.1'
    image: adamfowlerphoto/aws-sdk-swift:5.1
  - platform: linux
    swift_version: '5.2'
    image: adamfowlerphoto/aws-sdk-swift:5.2
  - platform: linux
    swift_version: '5.3'
    image: adamfowlerphoto/aws-sdk-swift:5.3

Future direction

It’s early days for this configuration file. We created it to solve issues which package authors were facing trying to make sure that the language and platform compatibility we report reflects reality.

We expect this file to evolve, which is why it carries a version number. Here are some of the things we are planning to add over time, so that package authors can control them:

• Author metadata. Full name, blog URL, Twitter account, etc.
• Package keywords or categories.
• The location of any hosted documentation for a package.
• Sponsorship or funding information for a package.

More metadata possibilities are under discussion in this GitHub issue. This is a great place to chime in if you have suggestions.

File format

We chose YAML as the file format because it is an uncontroversial, universally beloved format that is entirely free of issues. 😬

Seriously though, we chose YAML as it’s relatively easy for a human to read and write, and it’s well supported in Swift. While it has some downsides compared to JSON, we believe it’s a more practical format given that package authors need to create this file manually.

Of course, we’ll let adoption be the guide as to how well it is working in practice!

Funding the Future of the Swift Package Index

It’s been almost four months since we launched the Swift Package Index, and we’ve been so happy to see so many positive reactions to it across the community. Thank you all for checking it out and spreading the word.

It’s still early days for Swift Package Manager adoption but we are confident that it will become the dominant technology for Swift dependency management, and we want the Swift Package Index to grow right alongside it.

We believe we’re off to a great start with what we have so far, but this is only the beginning. We have so many ideas for “bug fixes and improvements”! 😂

Many open-source projects are just code and don’t need constant maintenance and monitoring, but the Swift Package Index is a little different. It’s a hosted instance of the site with a reasonably complex automated build system that has run over 600,000 builds so far (increasing by 7,000 more every day!). It takes significant time and effort to keep things running smoothly.

We want to find a way to make sure that we can continue to give this site the time and attention it needs, but we don’t really want to go down the route of including advertising. Instead, we are opening up sponsorship on GitHub.

If you (or your company) would like to support the site, we would be thrilled to be able to make this a community-funded effort for years to come.

Also, a huge thanks to Oliver, who beat us to even announcing this by sponsoring the project yesterday! Thanks for being our very first sponsor. 😍

Thank you!
Dave & Sven

Building 3,238 Swift Packages for Apple Silicon

As part of the Swift Package Index build system, we have processed what must be the most extensive test of Apple Silicon compatibility outside of Apple.

At the time of writing, we currently have 12,942 Apple Silicon builds in our database covering 3,238 packages. Why so many? We test up to three versions of each package and compile each version with both xcodebuild and swift build.

I’m sure that many of the Apple Silicon DTK machines Apple shipped around the world are working very hard, testing apps for compatibility, but I’m not sure any of them have worked quite as hard as ours has. 🚀

What did we find?

As you might expect, there’s plenty of good news. For packages with successful Intel builds, the vast majority also successfully build for Apple Silicon. 👍

There are failures, though, and the most interesting failures are where the build fails on Apple Silicon while the corresponding Intel build succeeds. In cases like this, that indicates a compatibility issue that will potentially need some attention from the package author. We’ve identified 139 packages (~4%) where this is the case, and have compiled a list of them for reference.

There are a few common errors that we’re observing in these failures, and most of them are from one of the following problems:

• Any inline assembly will fail, obviously!
• Float80 is unavailable.
• x86_thread_state64_t is unavailable.
• __darwin_arm_thread_state64' has no member '__rsp'.
• cannot convert value of type 'CVaListPointer' to expected argument type '__darwin_va_list?'.
• Sometimes, the compiler crashes!

Even though 139 failing packages might sound like a lot, it’s not that bad, especially when you consider that many of the failures all fail due to an issue in a common base dependency, PerfectLib.

Where do we go from here?

Even though the results are generally positive, it does show that there’s a bit of work that the community need to do to ensure the transition to Apple Silicon is as smooth as possible.

If you’re a package author and do not have a DTK machine at your disposal, please feel free to use ours via the Swift Package Index build system! Make changes to your packages, push them to a pre-release tag (for example, X.X.X-arm-beta-X) and the build system will pick up that tag and build it. You should see new build results, usually within a couple of hours of pushing a tag.

If you have any problems with the build system, there’s a couple of ways to contact us. We’re in the #swiftpackageindex channel over at the SwiftPM Slack, or if it looks like there might be something wrong, please raise an issue on GitHub.

Launching Language and Platform Package Compatibility

What’s the first question you need an answer to after finding a package that fits your needs?

“Does this package work with the Swift version and platform that my app uses?”

When we initially launched the Swift Package Index, we attempted to answer this question with the metadata available in the package manifest. Namely the swiftLanguageVersions and platforms properties.

The problem is that neither of those properties is perfect. swiftLanguageVersions isn’t granular enough, only officially allowing values of v4, v4_2, and v5. The platforms property is better, but doesn’t let package authors declare compatibility with non-Apple operating systems such as Linux.

Wouldn’t it be fantastic if you could see a matrix like this for every package? 😍

Look at how information-rich that matrix is. You can instantly see that the latest stable version of PromiseKit is compatible with every version of Swift back to 4.2, and every platform, including Linux. Then, you can see that the alpha version in development drops support for iOS, tvOS, and watchOS, and Swift 4.2. That seems suspicious, right? Keep looking, and you’ll see that the default branch fixes all those issues and restores compatibility. I’m confident looking at that matrix that when 7.0.0 is released, it’ll have green ticks across the board, but I also know to not depend on this current alpha. That’s practical, actionable information.

When we started thinking about how best to solve this problem, the obvious best solution was to build the packages! What better way to see if a package is compatible with Swift 4.2 than to build it with the version of xcodebuild that shipped with Xcode 10.1.

So that’s what we did, and it’s available right now. Why not give it a try by searching for a few of your favourite packages? 🚀

Accurate, real-world compatibility data

It’s a little more complicated than “just build each package” though. A package might build with Swift 5.2 on iOS, but that same build might fail using Swift 5.2 on macOS due to a UIKit dependency, or other macOS specific issue. What’s needed is a matrix of builds to generate an accurate picture of compatibility.

So, if we run builds using Swift 5.1 on iOS, macOS, tvOS, watchOS, and with Linux and any of them pass, it’s compatible with Swift 5.2. If any version of Swift builds without failure on iOS, then the package supports iOS.

We ended up with a platform list of:

• iOS using xcodebuild
• macOS using xcodebuild
• macOS using swift build (there are good reasons where swift build would pass in circumstances where xcodebuild might fail)
• macOS using xcodebuild on Apple Silicon (yes, compiled using a DTK!)
• macOS using swift build on Apple Silicon
• tvOS using xcodebuild
• watchOS using xcodebuild
• Linux using swift build

We then decided on a list of Swift compiler versions we’d like to check compatibility with:

• Swift 4.2 (4.2.1)
• Swift 5.0 (5.0.1)
• Swift 5.1 (5.1.3)
• Swift 5.2 (5.2.4)
• Swift 5.3 (beta)

That’s up to 32 builds per package, but that’s just the beginning. What if there’s a stable release and a beta release? The stable version might support Swift 4.2 and higher, and the new beta might drop support for anything less than Swift 5.2. That’s information which would be important when choosing a package, so we need to show it. As we also track the status of the default branch, we must build that too, and we’ve quickly arrived at a place where we might need to run 96 builds per package! With almost 3,200 packages in the index, that’s potentially more than 300,000 builds! 😅

In practice, it’s less than that as most packages don’t have a current beta release, but it’s still a lot of builds. We’ve processed more than 200,000 builds as I write this, and we’re not quite finished. As of today, we’re at 99% though, so we almost made it before launch! 😬

If you’ve been following these tweets, it should be obvious what all that processing was! Let’s take a look at the last 30 days of CPU graphs for our production server, a 2018 Mac mini with 32Gb RAM and a 6-core i7 CPU:

You can see a few of our final test runs in that graph, and then we started processing for real. Since then, we’ve kept the CPU completely pegged for more than two weeks. We’ve also had our staging Mac mini, a spare 2016 MacBook Pro, and a DTK working on builds too.

Everyone loves badges

Providing compatibility information on this site is one thing, but everyone loves adorning their packages pages with shields.io badges, don’t they? If you maintain an open-source project, wouldn’t you like to show off real compatibility status in your README file, like this?

If you’re a package author, click the “Copy badge” button below each of the compatibility matrices and you’ll have a Markdown image link in your clipboard, ready to use.

Your users will always see live, accurate compatibility information that updates whenever you release a new version.

Credit where it’s due!

First of all, we’d like to thank our kind friends at MacStadium for providing the significant hosting resources for this project as part of their open-source programme. At one point we were a little concerned that we might melt their machines, and we very glad that we didn’t. They’ve performed incredibly.

We also want to say thank you to Ankit Aggarwal and Boris Bügling of Apple. Their tireless help and support on the SwiftPM Slack saved us countless hours figuring out the correct way to approach this problem.

Finally, we’d love to say thank you to everyone who provided help and feedback along the way as we built this feature. We couldn’t have done it without any of you.

Wrapping up

Some package authors set up continuous integration for their packages and of course, that includes a build step. That usually only covers one version of Swift though, and the information gets hidden away in a different place in each repo. We think that by centralising this data and making it available for all packages, we should be able to help this community make better decisions about their dependencies, and that’s what this site is all about.

We hope you love this feature as much as we do! ❤️

Say Hello to the Swift Package Index Blog

Hello! 👋

It’s always hard to know when to ship v1 of a project. Which features get included? Are there things that can wait? It’s a balancing act. You’re excited to show the world what you built, but is it good enough?

One decision we made when shipping the Swift Package Index was that a blog could wait. Looking back now, it feels like that was the right decision. The launch went smoothly, you all reacted very kindly and positively to it, and we went back to work fixing bugs and working on new features.

However, we always knew we’d need somewhere to write about what we are doing with this project. We’d love to tell you about some of the interesting technical challenges we faced, but most of all we want to tell you about the new features we’ve been building...

What features? You’ll have to wait just one more week to find out what we’ve been working on, but we’re pretty excited about it, and that’s why now was the right time to launch this blog.

Now we just need to keep writing on it! 😬