swift-symbolicate

This is a fairly simple tool to add/update symbols into backtraces generated by the built-in crash logger in the swift runtime, that is paired with the helper tool swift-backtrace from /usr/libexec/swift/.

Usage

swift-symbolicate [-s <symbol-path>] [--gdb-symbol-servers <url>] [--windows-symbol-servers <url>] [-o <output-file>] [<input-file>]

Options:

• -s, --symbol-additional-paths, --symbols — Additional path(s) to search for/cache symbol files. Can be specified multiple times.
• -o, --output-file — Where to send the symbolicated output (defaults to stdout on mac/Linux; required on Windows).
• --gdb-symbol-servers — Debuginfod symbol server base URL(s), tried in order. Can be specified multiple times.
• --windows-symbol-servers — Windows symbol server base URL(s), tried in order. Can be specified multiple times.

There are environment variables to allow ease of use for some of these options. See below.

By default, with no options, the tool will read standard input and output to standard output.

It scans the stream efficiently for json or plain text crash dumps in the above described formats and attempts to symbolicate them inline. All other input is passed through unchanged.

When running on macOS, the tool can symbolicate Mach-O, PE/COFF or ELF crash logs as needed and it will attempt to pick the right one and use the corresponding framework (e.g. CoreSymbolication for Mach-O).

Note: the tool is unable to symbolicate Mach-O crash logs when running on platforms other than macOS as CoreSymbolication is a platform dependency.

Environment Variables

All CLI options for symbol paths and servers can also be set via environment variables. Values are semicolon-separated lists.

• SWIFT_SYMBOLICATE_SYMBOL_PATHS — Equivalent to --symbol-additional-paths. Merged with any CLI-specified paths.
• SWIFT_SYMBOLICATE_GDB_SERVERS — Equivalent to --gdb-symbol-servers. Merged with any CLI-specified servers.
• SWIFT_SYMBOLICATE_WINDOWS_SERVERS — Equivalent to --windows-symbol-servers. Merged with any CLI-specified servers.
• SWIFT_SYMBOLICATE_SERVERS_DEBUG — Set to 1 to enable debug output for symbol server fetch operations.
• SWIFT_SYMBOLICATE_CACHE_UPDATE — Controls when cached symbol files are refreshed from the server. Values:
  • never (default) — If a cached file already exists locally, use it without contacting the server.
  • newer — Contact the server with an If-Modified-Since header; only download if the server has a newer copy.
  • always — Always download from the server, ignoring any locally cached file.

Backtrace formatting and symbolication behaviour can be tuned via the SWIFT_BACKTRACE environment variable (same key=value format as the Swift runtime uses), e.g.:

SWIFT_BACKTRACE="threads=crashed,demangle=no,sanitize=yes,registers=all" swift run swift-symbolicate crash.txt

Building

swift-symbolicate can be built using swift-package-manager.

Note that at the time of writing, the package requires a recent toolchain. On macOS, download a recent nightly toolchain from swift.org and install it, then use export TOOLCHAINS=swift to activate the toolchain before swift build, for example. Also, when running, you'll probably need to make sure the correct version of the Runtime module is used by swift-symbolicate, using a command such as export DYLD_LIBRARY_PATH=${HOME}/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift/macosx. On Windows, you'll need a recent Swift installer for the same reason. On Linux, you'll likely need to use something like swiftly to get an up-to-date swift toolchain, or an equivalent approach for the images if you're running in a container.

Testing

Use swift test to run the unit test suite. These are stand alone and work on macOS and Linux.

Deployment

The following subsidiary executables are used for testing and development only. They would not typically be deployed alongside swift-symbolicate:

• crashMe — generates a single-threaded crash for testing
• crashMeMultithreaded — generates a multithreaded crash for testing
• crashMeOpenFds — generates a crash with open file descriptors for testing
• demo-gdb-symbol-server.py — a Python script reproducing the debuginfod service for testing
• demo-windows-symbol-server.py — a Python script reproducing the Windows symsrv service for testing
• index-pdb-files — indexes PDB files into symsrv directory layout (Windows only)

Symbol Servers

Two symbol server protocols are supported:

Debuginfod (Linux/Darwin)

Specified with --gdb-symbol-servers <baseURL>. Uses the standard debuginfod URL scheme:

<baseURL>/buildid/<build-id>/debuginfo
<baseURL>/buildid/<build-id>/executable

Downloaded files are cached locally using the ELF .build-id directory layout:

<symbol-path>/.build-id/<xx>/<yyyyyyyy...>.debug
<symbol-path>/<executable-name>

where xx is the first two hex characters of the build ID and yyyyyyyy... is the remainder.

Windows Symbol Server (symsrv)

Specified with --windows-symbol-servers <baseURL>. Uses the Microsoft symsrv URL pattern:

<baseURL>/<pdbFilename>/<symsrvId>/<pdbFilename>

Downloaded PDB files are cached locally in symsrv directory layout:

<symbol-path>/<pdbName>/<symsrvId>/<pdbName>

The symsrv ID is derived by converting the LLVM-format PDB ID (as stored in crash logs) to the Microsoft GUID+age format.

Common requirements

Using either type of symbol server requires that --symbol-additional-paths (or SWIFT_SYMBOLICATE_SYMBOL_PATHS) has at least one path. The first path is used as the cache directory for downloaded symbols.

The tool sends If-Modified-Since headers when a cached file already exists, so the server can reply 304 (Not Modified) to avoid retransmitting files that haven't changed.

Known issue: bracketed IPv6 literal URLs on Windows/Linux

On Windows, swift-symbolicate's HTTP client goes through FoundationNetworking (swift-corelibs-foundation).

A bug in that layer means bracketed IPv6 literal URLs do not work with either --gdb-symbol-servers or --windows-symbol-servers. A request like:

swift-symbolicate.exe --gdb-symbol-servers https://[2001:db8::1]:8080 ...

will likely fail with NSURLErrorDomain code -1000 (NSURLErrorBadURL), even though the same URL works in curl.

(There is an outstanding issue logged against FoundationNetworking on these platforms for this issue at the time of writing.)

Workarounds:

• Use a hostname instead of an IPv6 literal. e.g. an entry to C:\Windows\System32\drivers\etc\hosts mapping a name to the IPv6 address is the easiest fix.
• Use a name resolved via DNS (AAAA record) or mDNS (my-mac.local).
• Use an IPv4 address if one is available.

This affects IPv6 hard coded addresses only, and Windows/Linux only; macOS is not affected.

Footnotes

demo-gdb-symbol-server.py

A Python script that reproduces the debuginfod service for testing swift-symbolicate.

Usage:

python3 demo-gdb-symbol-server.py symbol-server-symbols --verbose

...where your debug symbol files and executables are in the directory symbol-server-symbols.

The server serves debug symbols from buildid/{buildid}/debuginfo and executables from buildid/{buildid}/executable. Files can be laid out either flat (<BUILDID>.debug / <BUILDID>) or using the .build-id directory convention (.build-id/<XX>/<YYYYYY...>.debug).

In the simplest crude demo, you can use the same file. So, for example, if you built the demo program crashMe from this package, you could copy the executable file crashMe to the directory symbol-server-symbols twice, naming one copy based on the build id and another copy as build-id.debug.

python3 demo-gdb-symbol-server.py ./my_store --verbose

Note that when swift-symbolicate retrieves remote symbols from a symbol server, it caches them locally as described above.

An example run of swift-symbolicate on a container would be:

.build/aarch64-unknown-linux-gnu/debug/swift-symbolicate tmp2.txt --symbol-additional-paths symbol-cache --gdb-symbol-servers http://host.docker.internal:8080

This runs inside a container, pulls symbols from a demo server on the host machine, caches locally.