The Swift Package Index Blog

The Swift Package Index Metadata File – First Steps

The vast majority of the information you see in the package index is generated from a single URL, the location of the package’s git repository.

From that URL we gather data from the repository itself, the package manifest, the GitHub API (yes, it’s GitHub only for now, but we do plan to support other hosts), and from running builds of the package to check compatibility with operating systems and Swift versions.

Edge cases

Of course, as with everything in software development, it’s not long before you find yourself talking about edge cases.

While creating the build system, we quickly encountered packages where watchOS targets failed to build. We use Xcode’s automatic scheme creation functionality while running builds, but Xcode generates schemes that always include test targets, and XCTest isn’t available on watchOS. Using automatic schemes for watchOS builds wasn’t going to work.

The only way to have these builds succeed is for our build system to use a specific scheme, rather than an automatically created one. Many packages already have schemes for this purpose, but as package authors can give schemes any name, it’s not possible to easily discover them.

Enter the Swift Package Index configuration file

Rather than trying to keep the build system 100% generic, we needed a mechanism for package authors to specify some configuration information, such as the scheme name to use to build watchOS targets.

We support this via a configuration file, .spi.yml which we will look for in the root of your Swift package repository.

Schemes for watchOS

You can see the popular package swift-composable-architecture by Pointfree making use of this config file:

version: 1
builder:
  configs:
  - platform: watchos
    scheme: ComposableArchitecture_watchOS

Our build servers look for this file and use any scheme information we find for each platform. If a platform isn’t listed, we use our default heuristics to determine the scheme as outlined in our build FAQ.

Images for Linux

Custom schemes aren’t all we support though. Package authors can also use the .spi.yml file to configure custom docker base images for our Linux builds.

We build packages for Linux with docker commands, selecting between the various base images that Apple provide.

/usr/local/bin/docker run --rm -v "$PWD":/host -w /host swiftlang/swift:5.2.4 swift build --enable-test-discovery

This works well unless a package requires an OS-level dependency, like OpenSSL.

We may eventually provide our own set of base images for supported Swift versions that also include common dependencies like OpenSSL. For now, we are asking package authors whose builds fail due to missing dependencies to create their own for the Swift versions they support.

For security reasons, we don’t allow package authors to specify arbitrary images in the configuration file, so if you need to use this feature, please raise an issue asking for us to support a custom base image.

The AWS library Soto is an example of a package that makes use of this feature in its .spi.yml file:

version: 1
builder:
  configs:
  - platform: linux
    swift_version: '5.0'
    image: adamfowlerphoto/aws-sdk-swift:5.0
  - platform: linux
    swift_version: '5.1'
    image: adamfowlerphoto/aws-sdk-swift:5.1
  - platform: linux
    swift_version: '5.2'
    image: adamfowlerphoto/aws-sdk-swift:5.2
  - platform: linux
    swift_version: '5.3'
    image: adamfowlerphoto/aws-sdk-swift:5.3

Future direction

It’s early days for this configuration file. We created it to solve issues which package authors were facing trying to make sure that the language and platform compatibility we report reflects reality.

We expect this file to evolve, which is why it carries a version number. Here are some of the things we are planning to add over time, so that package authors can control them:

• Author metadata. Full name, blog URL, Twitter account, etc.
• Package keywords or categories.
• The location of any hosted documentation for a package.
• Sponsorship or funding information for a package.

More metadata possibilities are under discussion in this GitHub issue. This is a great place to chime in if you have suggestions.

File format

We chose YAML as the file format because it is an uncontroversial, universally beloved format that is entirely free of issues. 😬

Seriously though, we chose YAML as it’s relatively easy for a human to read and write, and it’s well supported in Swift. While it has some downsides compared to JSON, we believe it’s a more practical format given that package authors need to create this file manually.

Of course, we’ll let adoption be the guide as to how well it is working in practice!

Funding the Future of the Swift Package Index

It’s been almost four months since we launched the Swift Package Index, and we’ve been so happy to see so many positive reactions to it across the community. Thank you all for checking it out and spreading the word.

It’s still early days for Swift Package Manager adoption but we are confident that it will become the dominant technology for Swift dependency management, and we want the Swift Package Index to grow right alongside it.

We believe we’re off to a great start with what we have so far, but this is only the beginning. We have so many ideas for “bug fixes and improvements”! 😂

Many open-source projects are just code and don’t need constant maintenance and monitoring, but the Swift Package Index is a little different. It’s a hosted instance of the site with a reasonably complex automated build system that has run over 600,000 builds so far (increasing by 7,000 more every day!). It takes significant time and effort to keep things running smoothly.

We want to find a way to make sure that we can continue to give this site the time and attention it needs, but we don’t really want to go down the route of including advertising. Instead, we are opening up sponsorship on GitHub.

If you (or your company) would like to support the site, we would be thrilled to be able to make this a community-funded effort for years to come.

Also, a huge thanks to Oliver, who beat us to even announcing this by sponsoring the project yesterday! Thanks for being our very first sponsor. 😍

Thank you!
Dave & Sven

Building 3,238 Swift Packages for Apple Silicon

As part of the Swift Package Index build system, we have processed what must be the most extensive test of Apple Silicon compatibility outside of Apple.

At the time of writing, we currently have 12,942 Apple Silicon builds in our database covering 3,238 packages. Why so many? We test up to three versions of each package and compile each version with both xcodebuild and swift build.

I’m sure that many of the Apple Silicon DTK machines Apple shipped around the world are working very hard, testing apps for compatibility, but I’m not sure any of them have worked quite as hard as ours has. 🚀

What did we find?

As you might expect, there’s plenty of good news. For packages with successful Intel builds, the vast majority also successfully build for Apple Silicon. 👍

There are failures, though, and the most interesting failures are where the build fails on Apple Silicon while the corresponding Intel build succeeds. In cases like this, that indicates a compatibility issue that will potentially need some attention from the package author. We’ve identified 139 packages (~4%) where this is the case, and have compiled a list of them for reference.

There are a few common errors that we’re observing in these failures, and most of them are from one of the following problems:

• Any inline assembly will fail, obviously!
• Float80 is unavailable.
• x86_thread_state64_t is unavailable.
• __darwin_arm_thread_state64' has no member '__rsp'.
• cannot convert value of type 'CVaListPointer' to expected argument type '__darwin_va_list?'.
• Sometimes, the compiler crashes!

Even though 139 failing packages might sound like a lot, it’s not that bad, especially when you consider that many of the failures all fail due to an issue in a common base dependency, PerfectLib.

Where do we go from here?

Even though the results are generally positive, it does show that there’s a bit of work that the community need to do to ensure the transition to Apple Silicon is as smooth as possible.

If you’re a package author and do not have a DTK machine at your disposal, please feel free to use ours via the Swift Package Index build system! Make changes to your packages, push them to a pre-release tag (for example, X.X.X-arm-beta-X) and the build system will pick up that tag and build it. You should see new build results, usually within a couple of hours of pushing a tag.

If you have any problems with the build system, there’s a couple of ways to contact us. We’re in the #swiftpackageindex channel over at the SwiftPM Slack, or if it looks like there might be something wrong, please raise an issue on GitHub.

Launching Language and Platform Package Compatibility

What’s the first question you need an answer to after finding a package that fits your needs?

“Does this package work with the Swift version and platform that my app uses?”

When we initially launched the Swift Package Index, we attempted to answer this question with the metadata available in the package manifest. Namely the swiftLanguageVersions and platforms properties.

The problem is that neither of those properties is perfect. swiftLanguageVersions isn’t granular enough, only officially allowing values of v4, v4_2, and v5. The platforms property is better, but doesn’t let package authors declare compatibility with non-Apple operating systems such as Linux.

Wouldn’t it be fantastic if you could see a matrix like this for every package? 😍

Look at how information-rich that matrix is. You can instantly see that the latest stable version of PromiseKit is compatible with every version of Swift back to 4.2, and every platform, including Linux. Then, you can see that the alpha version in development drops support for iOS, tvOS, and watchOS, and Swift 4.2. That seems suspicious, right? Keep looking, and you’ll see that the default branch fixes all those issues and restores compatibility. I’m confident looking at that matrix that when 7.0.0 is released, it’ll have green ticks across the board, but I also know to not depend on this current alpha. That’s practical, actionable information.

When we started thinking about how best to solve this problem, the obvious best solution was to build the packages! What better way to see if a package is compatible with Swift 4.2 than to build it with the version of xcodebuild that shipped with Xcode 10.1.

So that’s what we did, and it’s available right now. Why not give it a try by searching for a few of your favourite packages? 🚀

Accurate, real-world compatibility data

It’s a little more complicated than “just build each package” though. A package might build with Swift 5.2 on iOS, but that same build might fail using Swift 5.2 on macOS due to a UIKit dependency, or other macOS specific issue. What’s needed is a matrix of builds to generate an accurate picture of compatibility.

So, if we run builds using Swift 5.1 on iOS, macOS, tvOS, watchOS, and with Linux and any of them pass, it’s compatible with Swift 5.2. If any version of Swift builds without failure on iOS, then the package supports iOS.

We ended up with a platform list of:

• iOS using xcodebuild
• macOS using xcodebuild
• macOS using swift build (there are good reasons where swift build would pass in circumstances where xcodebuild might fail)
• macOS using xcodebuild on Apple Silicon (yes, compiled using a DTK!)
• macOS using swift build on Apple Silicon
• tvOS using xcodebuild
• watchOS using xcodebuild
• Linux using swift build

We then decided on a list of Swift compiler versions we’d like to check compatibility with:

• Swift 4.2 (4.2.1)
• Swift 5.0 (5.0.1)
• Swift 5.1 (5.1.3)
• Swift 5.2 (5.2.4)
• Swift 5.3 (beta)

That’s up to 32 builds per package, but that’s just the beginning. What if there’s a stable release and a beta release? The stable version might support Swift 4.2 and higher, and the new beta might drop support for anything less than Swift 5.2. That’s information which would be important when choosing a package, so we need to show it. As we also track the status of the default branch, we must build that too, and we’ve quickly arrived at a place where we might need to run 96 builds per package! With almost 3,200 packages in the index, that’s potentially more than 300,000 builds! 😅

In practice, it’s less than that as most packages don’t have a current beta release, but it’s still a lot of builds. We’ve processed more than 200,000 builds as I write this, and we’re not quite finished. As of today, we’re at 99% though, so we almost made it before launch! 😬

If you’ve been following these tweets, it should be obvious what all that processing was! Let’s take a look at the last 30 days of CPU graphs for our production server, a 2018 Mac mini with 32Gb RAM and a 6-core i7 CPU:

You can see a few of our final test runs in that graph, and then we started processing for real. Since then, we’ve kept the CPU completely pegged for more than two weeks. We’ve also had our staging Mac mini, a spare 2016 MacBook Pro, and a DTK working on builds too.

Everyone loves badges

Providing compatibility information on this site is one thing, but everyone loves adorning their packages pages with shields.io badges, don’t they? If you maintain an open-source project, wouldn’t you like to show off real compatibility status in your README file, like this?

If you’re a package author, click the “Copy badge” button below each of the compatibility matrices and you’ll have a Markdown image link in your clipboard, ready to use.

Your users will always see live, accurate compatibility information that updates whenever you release a new version.

Credit where it’s due!

First of all, we’d like to thank our kind friends at MacStadium for providing the significant hosting resources for this project as part of their open-source programme. At one point we were a little concerned that we might melt their machines, and we very glad that we didn’t. They’ve performed incredibly.

We also want to say thank you to Ankit Aggarwal and Boris Bügling of Apple. Their tireless help and support on the SwiftPM Slack saved us countless hours figuring out the correct way to approach this problem.

Finally, we’d love to say thank you to everyone who provided help and feedback along the way as we built this feature. We couldn’t have done it without any of you.

Wrapping up

Some package authors set up continuous integration for their packages and of course, that includes a build step. That usually only covers one version of Swift though, and the information gets hidden away in a different place in each repo. We think that by centralising this data and making it available for all packages, we should be able to help this community make better decisions about their dependencies, and that’s what this site is all about.

We hope you love this feature as much as we do! ❤️

Say Hello to the Swift Package Index Blog

Hello! 👋

It’s always hard to know when to ship v1 of a project. Which features get included? Are there things that can wait? It’s a balancing act. You’re excited to show the world what you built, but is it good enough?

One decision we made when shipping the Swift Package Index was that a blog could wait. Looking back now, it feels like that was the right decision. The launch went smoothly, you all reacted very kindly and positively to it, and we went back to work fixing bugs and working on new features.

However, we always knew we’d need somewhere to write about what we are doing with this project. We’d love to tell you about some of the interesting technical challenges we faced, but most of all we want to tell you about the new features we’ve been building...

What features? You’ll have to wait just one more week to find out what we’ve been working on, but we’re pretty excited about it, and that’s why now was the right time to launch this blog.

Now we just need to keep writing on it! 😬