The Swift Package Index Blog

Launching Swift Package Collections

It's probably not news that this is a big week for Swift developers! On Monday, Apple kicked WWDC off by releasing the first beta of Xcode 13, including Swift 5.5. We've been following the release of the Swift Package Manager in Swift 5.5 closely as it implements package collections, a significant new feature.

What are package collections? JSON descriptions of Swift package metadata. What does the Swift Package Index have in droves? Metadata about Swift packages! As soon as we heard about the proposal, we knew we had to support it. For a comprehensive description of the feature, check out the post over on the official Swift blog.

We have big plans for package collections, but we wanted to ship something alongside this first beta release. We chose to implement author collections, a list of all packages from the index grouped by repository owner. For example, you can get a package collection containing all packages from Alamofire or all packages from Point-Free.

We expected this feature to roll out as part of the Swift Package Manager command-line tool, and it did. What we didn't expect was full support for package collections in Xcode 13! What a pleasant surprise that was. 😍

We've written up the details of how to access collections on the Swift Package Index, including how to add them to Xcode and use them from the command line. We'd love for you to check it out.

However, that wasn't the only surprise this week had in store for us! As we watched Nicole Jacque talk about What's new in Swift, we were amazed to see a screenshot of the Swift Package Index appear on the screen and hear Nicole start to talk about it. Remarkable!

We're so happy that Apple sees what we've been working on here as important enough to mention in a WWDC session, and we'd like to thank everyone who was involved in that decision for including us. It's a great honour and one that we do not take lightly.

What’s Next? - May 2021

You might just have finished reading our progress report from the last three months. Now it’s time to talk about what we’re planning for the next few months!

Like last time, this is a list of areas that we’ll focus on over the next few months, not a to-do list. Other features and bug fixes will crop up and might take priority, and that’s OK! It’s good to have a rough direction in mind, though.

Author Information

We feel strongly that open-source package authors should be acknowledged for their work. The very first designs for the package index had author information right at the top of the package metadata, but it didn’t quite make it into our feature set at launch. We want to change that, adding author information prominently at the top of our package metadata section.

We’ll collect the author information via the .spi.yml file we already use for build configuration.

Package Keywords

We’ve long debated whether packages should have categories that come from a pre-determined list or have freeform keywords. After giving it some thought we’re going to go with freeform keywords rather than categories. We’ll collect keyword metadata from GitHub topics and show the keywords on package pages.

Dependency Information

We’d also like to start working towards displaying the dependencies that a package would bring into your project if you use it. For example, we might start by adding a simple count of top-level and total dependencies.

Additional Package Metadata

We’d also love to pick up a few more data points about every package. Does a package have tests? Is source code available? Is it a fork of another package? All of these are good indicators when considering dependencies.

Home Page Redesign

Using the author information and keyword information mentioned above, we’d love to work on a much more information-rich home page. The first part of this is to think about it properly and put together a design, which is why we’re not going to talk about precisely what you can expect today. However, we’ll be working on this.

Package Collections

Finally, we’d like to finish off our support for Package Collections and put it live to coincide with the release of Swift 5.5. Initially, we’ll support collections that show all packages authored by their repository owner. So, for example, a collection of these packages from Apple, or these packages from the Vapor project.

Progress Report - May 2021

Around three months ago, we published a list of areas that we wanted to focus on next. We’re just about to update that list for the next few months, but before we do that, let’s check in with how well we stayed on track since February!

• Intermediate Search Page. Implemented. We re-worked the search function on the home page, added a new search box in the header. You can also now share search results.
• Package Collections. Mostly Implemented. We have a basic implementation of Package Collections up and running. However, we didn’t activate it yet, mainly because the feature will go live in Swift Package Manager with Swift 5.5.
• Design Updates. Mostly Implemented. The package page has the new design, and the README files are now in so much better shape, with anchor links, relative links, and images all working reliably. The only thing we didn’t get to was the home page, where we’re waiting until we have package categories implemented before tackling it.
• The SPI Playgrounds macOS App. Implemented. You can read all about it here.

We’re pretty happy with that progress, and that list doesn’t cover all of the other things we did, such as adding build support for Swift 5.4. 🚀

Of course, none of this would be possible without the generous support of our GitHub Sponsors. We’d first like to thank all of our new sponsors since February. Adam Fowler, Gaston Glyn Rampersad, Joakim Hassila, Joohee Kim, Loopwerk, Marius, Omar Albeik, Point-Free, Raymund Beyer, and Thomas Prezioso Jr.

Of course, we also want to show our continued thanks to our ongoing sponsors! Alexandre Colucci, Artem Loenko, Bartosz Dolewski, Boris Bügling, Byron Davies, Chris, Dominik Hauser, Federico Zanetello, Gleb Shevchenko, James Sherlock, Jason Bruder, Jeffrey Macko, Joseph Heck, Joshua Kaunert, Juanpe Catalán, Jules Coynel, Kyle Lee, Leo Picado, Marcos Griselli, Massimo Biolcati, Matěj Kašpar Jirásek, Max Desiatov, Nathan Harris, Oliver Drobnik, PatrikTheDev, Paul Farnell, Paul Hudson, Peter Steinberger, PSPDFKit, Riley Testut, Robin Kunde, Sam Symons, Scott Robbins, Thomas Dohmke, Tim Condon, Umar Haroon, Vadim Shpakovski, and Xavier Lowmiller. Thanks so much for your continued support. ❤️

Would you like to support the development of the Swift Package Index? We're almost half way to our first goal of 100 supporters!

So, what’s next? Read on to find out.

Using the SPI Playgrounds app to file better bug reports

We recently released the SPI Playgrounds app, which allows you to try out Swift Packages in a Playground in Xcode. You can use it to quickly get up and running with a new package when evaluating dependencies. However, it can also be helpful when working with dependencies you already use in your project. One such use case is creating reproducible bug reports.

It’s often tricky for open-source maintainers to help people when they report issues, especially if they are in a large app, where reproducing a bug might require a significant amount of set-up. It can be especially challenging for Swift on the server projects, where you might have dependencies like databases and other services.

To help open-source maintainers to help you, it can be useful to prepare something they can run in a standalone way. How about in a playground? It doesn’t get much simpler than that!

Recently, looking at one of the issues in the Swift Package Index, it turned out that this was the perfect opportunity to use Arena to create a playground where we could reproduce the issue.

Note: Arena is the underlying tool used in the SPI Playgrounds app. Think of it as the command-line version of the app. We’re using it here as it allows you to create a Playground with more than one dependency embedded, which the SPI Playgrounds app doesn’t yet support.

We started by running Arena as follows:

arena https://github.com/vapor/vapor https://github.com/vapor/fluent https://github.com/vapor/fluent-postgres-driver -o ssl-error-repro

This command produced a playground which imported Vapor, Fluent, and the Postgres driver. We knew the issue happened when accessing an Azure Postgres database via TLS, so we set up a test database with a single record and read the record via Fluent.

The whole snippet to reproduce the bug was about 30 lines:

import Vapor
import Fluent
import FluentPostgresDriver

final class Test: Model, Content {
    static let schema = "test"

    @ID(key: .id)
    var id: UUID?
}

func run() throws {
    let app = Application(.testing)
    defer { app.shutdown() }

    let tlsConfig: TLSConfiguration = .forClient()
    app.databases.use(.postgres(hostname: "pgnio-debug.postgres.database.azure.com",
                                port: 5432,
                                username: "test@pgnio-debug",
                                password: "<ask me for the password>",
                                database: "test",
                                tlsConfiguration: tlsConfig), as: .psql)

    let db = app.db

    let t = try Test.find(UUID("cafecafe-cafe-cafe-cafe-cafecafecafe"), on: db).wait()
    print("t: \(String(describing:  t))")
}

run()

In the end, it took more time to set up a new test database for the issue. What’s more, it was trivial from that point on to iterate and try more things as instructed. Much more so than it would have been had it still been part of the Swift Package Index server project.

In some ways, creating this playground helped get this bug fixed. If we had not taken the time to reproduce the issue in a standalone way, it would have been difficult for the maintainers to be sure it was a problem they could fix.

Thanks again to Fabian Fett, Gwynne Raskind and Johannes Weiss for their help and Fabian for the quick turnaround on the fix, and we hope this post also helps you file better bug reports!

Launching the Swift Package Index Playgrounds app for macOS

From the very beginning, the main goal of this site was to do more than echo Swift package metadata. We want to enable better decisions about which dependencies to use. That’s why we include information about how actively maintained a package is, and it’s why we check compatibility with the different platforms by building against them.

Once you’ve decided that a package might be a good fit for your project, what’s your next step? Do you import it and start coding against it?

There’s a better way.

What if you could click a button and open Xcode with a Swift playground that has the library you’re thinking of using already imported? That’s what we’re launching today with the Swift Package Index Playgrounds app.

Once you have installed the app, click “Try in a Playground” from any package page.

Then click "Create Playground" when the app launches.

Choose where to save it, and a few seconds later, the playground will open in Xcode, ready for experimentation and testing! We’d love to know if this is a helpful feature, so be sure to let us know on Twitter.

There’s also an amusing story behind this feature, which prompted us to write up the story behind “The Button”.

How the Swift Package Index project got started because of a button

It’s been one year since we made the first commit on the Swift Package Index repository, and we think that deserves a little celebration!

Did you enjoy the cake? 😂 No? Well, how about we also tell you the story of how this project started?

Note: We usually write blog posts here in the first person plural as this project is very much a joint effort, but this is a story with lots of references to both Dave and Sven individually, so I’ll write it in the first person singular. Dave

Before the Swift Package Index, there was the SwiftPM Library. The goal of that site was the same as this one, to provide a comprehensive index of Swift packages that helps you make better decisions about your dependencies.

A little while after the launch, I had a message from some members of the Vapor Discord. They had been chatting about creating something similar as an open-source Swift/Vapor project and wondered if I might be interested in collaborating on it. I’ll admit that I wasn’t hugely excited about rewriting a site that I had just launched, but I was in favour of it becoming open-source. However, the reality of an open-source project focused on indexing Swift packages being a Ruby project would make finding contributors challenging, so I put aside my doubts and decided to go for it. I started learning Vapor and got started. Unfortunately, circumstances worked against the rewrite being a success at that point. I found the Vapor learning curve tough while also maintaining the existing site, and the folks from the Vapor Discord were busy too. The project started to progress, but slowly.

That’s when I got an email from Sven. He had seen the SwiftPM Library and wondered if I might be open to integrating his open-source project, Arena, with it. His idea was to add a button next to each search result that, when clicked, would create a Swift Playground that imported the library, ready for experimentation. It was a great idea, and we ended up on a quick audio call a few days later to discuss it. While we chatted, I mentioned the potential rewrite of the SwiftPM Library into Swift/Vapor, and it turned into a much larger conversation. Sven was interested in the project, experienced working with Vapor, and keen to help. A winning combination! 🥇

Things progressed very quickly after that. I wrote up some details of how the existing project worked, and Sven worked his Vapor magic. Before too long, we had a prototype up and running, and the project started to feel real. There wasn’t time to add “the Arena button” as we needed to get the basics implemented first, but it remained firmly on the to-do list.

Sven and I ended up working really well together. We had different, complementary skill sets, and we progressed quickly towards getting the new project finished. It had been open-source from day one, and we were working in public, but no one had noticed, and we started to gear up for the launch of the Swift Package Index.

There was still no sign of “the button”, though, even many months after the launch. 😬

It wasn’t through any malice or deliberate neglect. It was just that there was always something else that took a slightly higher priority. We worked on language and platform compatibility reporting, Apple silicon builds, gathering additional metadata, funding the project, RSS feeds, inline README files, finding suitable hosting, and many more things that didn’t make it to the blog.

I’m delighted to say that today sees the launch of “the button”! But what does the button do? Read more about that in the official launch post.

That deserves more than a cake! 🥂🍾

I’m happy that we finally got to bring Sven’s original idea to fruition through this project, but this celebration is about much more than that! Working with Sven on this project has been a pleasure from the beginning. We work incredibly well together and are making something that feels significant and important. The button is part of that, but the potential of the site is so much more.

It’s also worth mentioning that if you want to hear Sven and I talk about this story in person, we discussed it on John Sundell’s podcast last year.

Here’s to year two!

Hosting the Swift Package Index

Making any open-source project sustainable for the long term is challenging. You need to find time to work on the code, plan out features, respond to feedback, and if the project is not only code but a live web site, it also needs hosting and regular maintenance.

The Swift Package Index has slightly bigger hosting problems than most web sites, too. In addition to web and database servers, we need a significant amount of processing power to monitor and analyse constant package releases and run our build machines that deal with the ~30,000 Swift builds per month that process through our compatibility build system. 😅

We’re delighted to say that as of today, all of the Swift Package Index’s hosting problems are taken care of thanks to the generosity of two companies that believe in open-source.

First, MacStadium. The site has run on their Mac mini infrastructure since the public launch through their open-source programme. They have accommodated every request we have made, including giving us access to a DTK, which enabled Apple silicon compatibility results months in advance of M1 machines becoming available. They are a pleasure to work with and consistently supportive. Thank you to Heather, Brian, and all of the support team at MacStadium.

However, while running the web hosting, the database, package update monitoring, Linux builders, and all the Apple builders all on the same hardware is possible, it’s not ideal. Our build queue had become quite delayed because we were asking those Mac minis to do too much.

That’s where Microsoft Azure stepped in. We have now eased the load by migrating our web hosting and Linux builders to Azure. The site is load balanced and redundant, which not only increases speed and availability, it also decreases the maintenance effort required. Working with Microsoft has been an absolute pleasure, and their only priority was to see the project succeed. Nothing was too much trouble. Thank you to Stormy, Thomas, Shelby, and Candice.

With the web servers, database, package update monitoring, and Linux builders hosted at Azure and the macOS, macOS ARM, iOS, tvOS, and watchOS builders at MacStadium, the hosting for this project is secure and stable for the foreseeable future. Thank you so much to both companies for their help. Running this site wouldn’t be possible without them.

Note: It’s worth mentioning that this blog post is not a condition of any agreement between us and either of these companies. All this support came without any expectation of anything other than enabling an open-source project. We are writing this purely because we are grateful for their help. ❤️

What’s Next? - February 2021

It’s been about seven months since we launched the Swift Package Index and we’ve been hard at work implementing a few major updates. Most significantly, real-world package compatibility, a comprehensive set of RSS feeds and Twitter updates, and most recently inline README files.

But we’re far from done! 😅

So, in the spirit of letting you all know what we’re planning for this project next, we wanted to start a regular series of blog posts about the direction we’re heading with feature development. We’ll post every couple of months, giving a quick recap of what we’ve achieved along with where we’re heading next. It’s not a formal roadmap. It’s more a declaration of where our focus will be for the next couple of months.

So without further ado, what are we focusing on next?

Intermediate Search Page

Currently, search results are displayed inline on the home page, which makes it hard to share package searches (there’s no URL for a search), or integrate with productivity tools like Alfred or Raycast, or even DuckDuckGo through their bangs feature. We want to fix that.

Package Collections

If you’ve been following the package collection work in the SPM project, then it might have struck you that the Swift Package Index would be a great source of package collections, when they are added. We agree! We’ve already been working hard to prepare for this feature and want to have something publicly available as soon as the feature becomes available in SPM.

Design updates

The site design suited our feature set at launch but has become a little overloaded as we added features. This work has already started, with a new package page layout rolling out last Friday. There’s more to do though, including improvements to the home page. The lists of recent package additions and recent package releases are fine, but we can do better!

The Swift Package Index macOS App

We’ve always wanted to create a Swift Package Index Mac app of some kind. The app’s main purpose would be to allow you to easily try packages out in a Swift Playground (utilising the wonderful Arena tool). But it might end up being more over the long term. We want to define what this app will do, and then get something out that works towards that definition. We talked about the interesting story behind the “Try this package” feature on John Sundell’s podcast last year if you’re interested!

And More…

This is not a comprehensive list of things we’ll do. This is a young project, and we want to keep development agile. If we have a great idea, it may go into the plan ahead of everything you read above!

Thank you!

We’d also like to use this update post to say a huge thank you to our generous sponsors. We wouldn’t be able to put in the time and effort needed by this project without the generous support of Alexandre Colucci, Artem Loenko, Bartosz Dolewski, Boris Bügling, Byron Davies, Chris, Dominik Hauser, Federico Zanetello, Gleb Shevchenko, James Sherlock, Jason Bruder, Jeffrey Macko, Joseph Heck, Joshua Kaunert, Juanpe Catalán, Jules Coynel, Kyle Lee, Leo Picado, Marcos Griselli, Massimo Biolcati, Matěj Kašpar Jirásek, Max Desiatov, Nathan Harris, Oliver Drobnik, PatrikTheDev, Paul Farnell, Paul Hudson, Peter Steinberger, PSPDFKit, Riley Testut, Robin Kunde, Sam Symons, Scott Robbins, Stefan-Cosmin Romanescu, Thomas Dohmke, Tim Condon, Umar Haroon, Vadim Shpakovski, and Xavier Lowmiller. Thank you all! ❤️

Would you like to be part of this list next time we post an update like this? Sponsoring the Swift Package Index is done through GitHub Sponsors, and your support will allow us to continue to build this site into something special.

Inline README Files!

If you’ve visited the main package index site over the last week or so, you might have noticed something new. Scroll down a little, and you’ll see the README file for the package, right below all the other information that we aggregate.

When we consider adding a new feature, at the heart of our decision has to be the question “Does this feature allow people to make better decisions about the package they’re considering?” Rather embarrassingly, the README file is probably the most important piece of information when picking a package dependency. Let’s not focus on that though, and instead be thankful that we finally got around to adding it! 😅

We’re aware of a couple of problems with README files that contain embedded HTML or make use of in-page anchor links that are often used to build a table of contents. We’ll get those fixed over time, but decided that it was worth rolling the feature out even with a couple of remaining issues.

Of course, we’ve not been able to test every single README file, so If you notice any other issues, please let us know by opening an issue.

Thank you all for your support this year, and we hope you enjoy this feature that brings our year of development on the Swift Package Index to an end. We can’t wait to show you what we have planned for 2021!

Keeping Up To Date with Swift Packages

How do you keep up with new releases of the packages you use? How do you discover new packages? It’s a tough challenge to keep up to date with everything that the community releases, so we have some announcements today that can help you stay informed!

Before we get started with the announcements, we want to credit James Sherlouk for his invaluable contributions to building the Twitter updates feature. He was instrumental in getting this feature up and running and has been making other improvements in several areas of the site since we launched. Thank you, James!

Updates via Twitter

You may already be following the @SwiftPackages account where we tweet occasional updates about the project. But as of yesterday, we’re also tweeting on the @PackageFirehose account. As you might guess by the name, this is a high volume account that posts every time there’s a new package added to the index, and every time there’s a new release of a package that we track. It’s around 20 tweets a day, if that counts as high volume!

While we’re on the subject of Twitter, we’d love to know what you’d like to see us do with the @SwiftPackages account. We’ve set up a short survey if you have a couple of minutes. We’d love to hear your opinions.

Updates via RSS

We’ve had RSS support for a while now, but we’ve not talked about it on the blog before, so it’s worth mentioning. If you’d prefer to keep up to date via a feed reader, or if you want fine-grained control over what type of package updates you’d like to be notified of then subscribe to one of our RSS feeds:

• New Packages – A feed with packages that are new to the Swift Package Index.
• All Package Releases – A feed of all new package version releases.

One advantage that these feeds have over Twitter is that they can be filtered with query string parameters. The package releases feed accepts four parameters; major, minor, patch, and pre. Pass true to any of these parameters to filter the feed on that part of the semantic version number. So, you can subscribe to only major package release, or major and minor package releases combined, or if you only want to know about pre-release versions, you can find those here. Use any permutation of the parameters to get exactly the feed that you want.

We hope you find both of these mechanisms for keeping up to date useful!

The Swift Package Index Metadata File – First Steps

The vast majority of the information you see in the package index is generated from a single URL, the location of the package’s git repository.

From that URL we gather data from the repository itself, the package manifest, the GitHub API (yes, it’s GitHub only for now, but we do plan to support other hosts), and from running builds of the package to check compatibility with operating systems and Swift versions.

Edge cases

Of course, as with everything in software development, it’s not long before you find yourself talking about edge cases.

While creating the build system, we quickly encountered packages where watchOS targets failed to build. We use Xcode’s automatic scheme creation functionality while running builds, but Xcode generates schemes that always include test targets, and XCTest isn’t available on watchOS. Using automatic schemes for watchOS builds wasn’t going to work.

The only way to have these builds succeed is for our build system to use a specific scheme, rather than an automatically created one. Many packages already have schemes for this purpose, but as package authors can give schemes any name, it’s not possible to easily discover them.

Enter the Swift Package Index configuration file

Rather than trying to keep the build system 100% generic, we needed a mechanism for package authors to specify some configuration information, such as the scheme name to use to build watchOS targets.

We support this via a configuration file, .spi.yml which we will look for in the root of your Swift package repository.

Schemes for watchOS

You can see the popular package swift-composable-architecture by Pointfree making use of this config file:

version: 1
builder:
  configs:
  - platform: watchos
    scheme: ComposableArchitecture_watchOS

Our build servers look for this file and use any scheme information we find for each platform. If a platform isn’t listed, we use our default heuristics to determine the scheme as outlined in our build FAQ.

Images for Linux

Custom schemes aren’t all we support though. Package authors can also use the .spi.yml file to configure custom docker base images for our Linux builds.

We build packages for Linux with docker commands, selecting between the various base images that Apple provide.

/usr/local/bin/docker run --rm -v "$PWD":/host -w /host swiftlang/swift:5.2.4 swift build --enable-test-discovery

This works well unless a package requires an OS-level dependency, like OpenSSL.

We may eventually provide our own set of base images for supported Swift versions that also include common dependencies like OpenSSL. For now, we are asking package authors whose builds fail due to missing dependencies to create their own for the Swift versions they support.

For security reasons, we don’t allow package authors to specify arbitrary images in the configuration file, so if you need to use this feature, please raise an issue asking for us to support a custom base image.

The AWS library Soto is an example of a package that makes use of this feature in its .spi.yml file:

version: 1
builder:
  configs:
  - platform: linux
    swift_version: '5.0'
    image: adamfowlerphoto/aws-sdk-swift:5.0
  - platform: linux
    swift_version: '5.1'
    image: adamfowlerphoto/aws-sdk-swift:5.1
  - platform: linux
    swift_version: '5.2'
    image: adamfowlerphoto/aws-sdk-swift:5.2
  - platform: linux
    swift_version: '5.3'
    image: adamfowlerphoto/aws-sdk-swift:5.3

Future direction

It’s early days for this configuration file. We created it to solve issues which package authors were facing trying to make sure that the language and platform compatibility we report reflects reality.

We expect this file to evolve, which is why it carries a version number. Here are some of the things we are planning to add over time, so that package authors can control them:

• Author metadata. Full name, blog URL, Twitter account, etc.
• Package keywords or categories.
• The location of any hosted documentation for a package.
• Sponsorship or funding information for a package.

More metadata possibilities are under discussion in this GitHub issue. This is a great place to chime in if you have suggestions.

File format

We chose YAML as the file format because it is an uncontroversial, universally beloved format that is entirely free of issues. 😬

Seriously though, we chose YAML as it’s relatively easy for a human to read and write, and it’s well supported in Swift. While it has some downsides compared to JSON, we believe it’s a more practical format given that package authors need to create this file manually.

Of course, we’ll let adoption be the guide as to how well it is working in practice!

Funding the Future of the Swift Package Index

It’s been almost four months since we launched the Swift Package Index, and we’ve been so happy to see so many positive reactions to it across the community. Thank you all for checking it out and spreading the word.

It’s still early days for Swift Package Manager adoption but we are confident that it will become the dominant technology for Swift dependency management, and we want the Swift Package Index to grow right alongside it.

We believe we’re off to a great start with what we have so far, but this is only the beginning. We have so many ideas for “bug fixes and improvements”! 😂

Many open-source projects are just code and don’t need constant maintenance and monitoring, but the Swift Package Index is a little different. It’s a hosted instance of the site with a reasonably complex automated build system that has run over 600,000 builds so far (increasing by 7,000 more every day!). It takes significant time and effort to keep things running smoothly.

We want to find a way to make sure that we can continue to give this site the time and attention it needs, but we don’t really want to go down the route of including advertising. Instead, we are opening up sponsorship on GitHub.

If you (or your company) would like to support the site, we would be thrilled to be able to make this a community-funded effort for years to come.

Also, a huge thanks to Oliver, who beat us to even announcing this by sponsoring the project yesterday! Thanks for being our very first sponsor. 😍

Thank you!
Dave & Sven

Building 3,238 Swift Packages for Apple Silicon

As part of the Swift Package Index build system, we have processed what must be the most extensive test of Apple Silicon compatibility outside of Apple.

At the time of writing, we currently have 12,942 Apple Silicon builds in our database covering 3,238 packages. Why so many? We test up to three versions of each package and compile each version with both xcodebuild and swift build.

I’m sure that many of the Apple Silicon DTK machines Apple shipped around the world are working very hard, testing apps for compatibility, but I’m not sure any of them have worked quite as hard as ours has. 🚀

What did we find?

As you might expect, there’s plenty of good news. For packages with successful Intel builds, the vast majority also successfully build for Apple Silicon. 👍

There are failures, though, and the most interesting failures are where the build fails on Apple Silicon while the corresponding Intel build succeeds. In cases like this, that indicates a compatibility issue that will potentially need some attention from the package author. We’ve identified 139 packages (~4%) where this is the case, and have compiled a list of them for reference.

There are a few common errors that we’re observing in these failures, and most of them are from one of the following problems:

• Any inline assembly will fail, obviously!
• Float80 is unavailable.
• x86_thread_state64_t is unavailable.
• __darwin_arm_thread_state64' has no member '__rsp'.
• cannot convert value of type 'CVaListPointer' to expected argument type '__darwin_va_list?'.
• Sometimes, the compiler crashes!

Even though 139 failing packages might sound like a lot, it’s not that bad, especially when you consider that many of the failures all fail due to an issue in a common base dependency, PerfectLib.

Where do we go from here?

Even though the results are generally positive, it does show that there’s a bit of work that the community need to do to ensure the transition to Apple Silicon is as smooth as possible.

If you’re a package author and do not have a DTK machine at your disposal, please feel free to use ours via the Swift Package Index build system! Make changes to your packages, push them to a pre-release tag (for example, X.X.X-arm-beta-X) and the build system will pick up that tag and build it. You should see new build results, usually within a couple of hours of pushing a tag.

If you have any problems with the build system, there’s a couple of ways to contact us. We’re in the #swiftpackageindex channel over at the SwiftPM Slack, or if it looks like there might be something wrong, please raise an issue on GitHub.

Launching Language and Platform Package Compatibility

What’s the first question you need an answer to after finding a package that fits your needs?

“Does this package work with the Swift version and platform that my app uses?”

When we initially launched the Swift Package Index, we attempted to answer this question with the metadata available in the package manifest. Namely the swiftLanguageVersions and platforms properties.

The problem is that neither of those properties is perfect. swiftLanguageVersions isn’t granular enough, only officially allowing values of v4, v4_2, and v5. The platforms property is better, but doesn’t let package authors declare compatibility with non-Apple operating systems such as Linux.

Wouldn’t it be fantastic if you could see a matrix like this for every package? 😍

Look at how information-rich that matrix is. You can instantly see that the latest stable version of PromiseKit is compatible with every version of Swift back to 4.2, and every platform, including Linux. Then, you can see that the alpha version in development drops support for iOS, tvOS, and watchOS, and Swift 4.2. That seems suspicious, right? Keep looking, and you’ll see that the default branch fixes all those issues and restores compatibility. I’m confident looking at that matrix that when 7.0.0 is released, it’ll have green ticks across the board, but I also know to not depend on this current alpha. That’s practical, actionable information.

When we started thinking about how best to solve this problem, the obvious best solution was to build the packages! What better way to see if a package is compatible with Swift 4.2 than to build it with the version of xcodebuild that shipped with Xcode 10.1.

So that’s what we did, and it’s available right now. Why not give it a try by searching for a few of your favourite packages? 🚀

Accurate, real-world compatibility data

It’s a little more complicated than “just build each package” though. A package might build with Swift 5.2 on iOS, but that same build might fail using Swift 5.2 on macOS due to a UIKit dependency, or other macOS specific issue. What’s needed is a matrix of builds to generate an accurate picture of compatibility.

So, if we run builds using Swift 5.1 on iOS, macOS, tvOS, watchOS, and with Linux and any of them pass, it’s compatible with Swift 5.2. If any version of Swift builds without failure on iOS, then the package supports iOS.

We ended up with a platform list of:

• iOS using xcodebuild
• macOS using xcodebuild
• macOS using swift build (there are good reasons where swift build would pass in circumstances where xcodebuild might fail)
• macOS using xcodebuild on Apple Silicon (yes, compiled using a DTK!)
• macOS using swift build on Apple Silicon
• tvOS using xcodebuild
• watchOS using xcodebuild
• Linux using swift build

We then decided on a list of Swift compiler versions we’d like to check compatibility with:

• Swift 4.2 (4.2.1)
• Swift 5.0 (5.0.1)
• Swift 5.1 (5.1.3)
• Swift 5.2 (5.2.4)
• Swift 5.3 (beta)

That’s up to 32 builds per package, but that’s just the beginning. What if there’s a stable release and a beta release? The stable version might support Swift 4.2 and higher, and the new beta might drop support for anything less than Swift 5.2. That’s information which would be important when choosing a package, so we need to show it. As we also track the status of the default branch, we must build that too, and we’ve quickly arrived at a place where we might need to run 96 builds per package! With almost 3,200 packages in the index, that’s potentially more than 300,000 builds! 😅

In practice, it’s less than that as most packages don’t have a current beta release, but it’s still a lot of builds. We’ve processed more than 200,000 builds as I write this, and we’re not quite finished. As of today, we’re at 99% though, so we almost made it before launch! 😬

If you’ve been following these tweets, it should be obvious what all that processing was! Let’s take a look at the last 30 days of CPU graphs for our production server, a 2018 Mac mini with 32Gb RAM and a 6-core i7 CPU:

You can see a few of our final test runs in that graph, and then we started processing for real. Since then, we’ve kept the CPU completely pegged for more than two weeks. We’ve also had our staging Mac mini, a spare 2016 MacBook Pro, and a DTK working on builds too.

Everyone loves badges

Providing compatibility information on this site is one thing, but everyone loves adorning their packages pages with shields.io badges, don’t they? If you maintain an open-source project, wouldn’t you like to show off real compatibility status in your README file, like this?

If you’re a package author, click the “Copy badge” button below each of the compatibility matrices and you’ll have a Markdown image link in your clipboard, ready to use.

Your users will always see live, accurate compatibility information that updates whenever you release a new version.

Credit where it’s due!

First of all, we’d like to thank our kind friends at MacStadium for providing the significant hosting resources for this project as part of their open-source programme. At one point we were a little concerned that we might melt their machines, and we very glad that we didn’t. They’ve performed incredibly.

We also want to say thank you to Ankit Aggarwal and Boris Bügling of Apple. Their tireless help and support on the SwiftPM Slack saved us countless hours figuring out the correct way to approach this problem.

Finally, we’d love to say thank you to everyone who provided help and feedback along the way as we built this feature. We couldn’t have done it without any of you.

Wrapping up

Some package authors set up continuous integration for their packages and of course, that includes a build step. That usually only covers one version of Swift though, and the information gets hidden away in a different place in each repo. We think that by centralising this data and making it available for all packages, we should be able to help this community make better decisions about their dependencies, and that’s what this site is all about.

We hope you love this feature as much as we do! ❤️

Say Hello to the Swift Package Index Blog

Hello! 👋

It’s always hard to know when to ship v1 of a project. Which features get included? Are there things that can wait? It’s a balancing act. You’re excited to show the world what you built, but is it good enough?

One decision we made when shipping the Swift Package Index was that a blog could wait. Looking back now, it feels like that was the right decision. The launch went smoothly, you all reacted very kindly and positively to it, and we went back to work fixing bugs and working on new features.

However, we always knew we’d need somewhere to write about what we are doing with this project. We’d love to tell you about some of the interesting technical challenges we faced, but most of all we want to tell you about the new features we’ve been building...

What features? You’ll have to wait just one more week to find out what we’ve been working on, but we’re pretty excited about it, and that’s why now was the right time to launch this blog.

Now we just need to keep writing on it! 😬