RFC: Support External Linkers

[adamncasey]

This RFC partially stabilises a version 0 of the rlib format, and defines a 'standard library bundle' mechanism with the aim to make this a supported workflow:

A C or C++ application built in a monorepo / hermetic-build environment can link in a library built in Rust without changes to its existing link process

Getting there requires a couple of practical steps, which would come if this RFC is accepted - or could happen in parallel if desirable:

1. Some compiler changes to support the new versions and flags - although in reality the existing rustc works without any of these flags today, they are important for the long term support for this feature.
2. rustc documents and supports a "build crate as rlib, link with a non-rustc linker" model as a first-class workflow, with clearly stated constraints that keep long-term support feasible.
3. Tests to ensure the workflow does not regress, at minimum for x86_64-unknown-linux-gnu.

I decided to pick this up after some conversations with folks where I work, and others within the Rust Community. My understanding is that this technique is already used today by a few large companies who are also active in the Rust community. Standardising this feels like a positive step forward, which makes it easier for more projects to adopt Rust in existing C++ codebases.

It's my first RFC & I probably didn't get everything right but hopefully I'm close enough. Happy to discuss here/on Zuilip/zoom/etc as needed to keep this moving.

Credit note: Although I am submitting this RFC, and stand by every word in it, a large majority of its content comes directly from @pcwalton's pre-RFC here: https://internals.rust-lang.org/t/pre-rfc-stabilize-a-version-of-the-rlib-format/17558 which was attached to rust-lang/rust#73632

Rendered

[bjorn3]

The way I was imagining this wouldn't special case the standard library. Rather we would introduce a new rstaticlib crate type to fill the following matrix:

-	Rust+C ABI	C ABI only
statically linked	rstaticlib	staticlib
dynamically linked	(rust) dylib	cdylib

So like the rust dylib crate type it would be usable as regular rust dependency. Just like the staticlib crate type it would be statically linked. And like all non-rlib crate types it can contain multiple crates and contains things like the allocator shim as appropriate.

This avoids special casing the standard library and would make it possible to use #[global_allocator] and work for no_std projects too. And in addition it would avoid the need for -Crlib-version as no ribs would need to be directly linked. And EIIs would trivially work. If necessary the compiler can insert any object files to wire up EII defaults when building the rstaticlib like would happen for (c)dylibs and staticlibs.

The way you did use this is to compile some crate as rstaticlib and have this crate depend on the standard library and have it or a dependency define #[global_allocator] if you need one. And then every other can be compiled as either rstaticlib or rlib (if you want to bundle it together with other rlibs to form another rstaticlib) with a dependency on this first rstaticlib. And then you link all rstaticlibs together as regular C static libraries.

View changes since the review

[adamncasey]

Am I right in thinking you're describing something similar to the staticlib-nobundle alternative described here:
https://github.com/rust-lang/rfcs/pull/3993/changes#diff-68e8cfbb8f0b236256ec7cb366e93d1a56e4296801ff0f97fea3cc7729217b9dR472-R480
But with this archive file also including the .rmeta data (and possibly other things) which an rlib contains, so that rustc would consume it a lot like an rlib when building a rust application?

In this world I think you're saying the supported workflow would look something like this:

1. Build at least one crate (although probably close to 1, for performance reasons since these would be large?) as an rstaticlib.
2. Build all your other rust dependencies as an rlib
3. Link your application by passing in the rlibs and rstaticlibs together.

Assuming I got the above right:

I agree this would avoid special casing the standard library. In practice, if the number of rstaticlib crates being built needs to be kept low, I imagine build systems choose to build exactly one of these at the bottom of the dependency tree?

Separately I'm not currently sure how we could support being able to link multiple rstaticlibs while having them export the standard library symbols as needed for linking against rlibs.

[bjorn3]

For step 2 you would either build all other rust dependencies as rstaticlib or if you have an EII, you build the crate that declare the EII and the one that define it as rlib and then build an rstaticlib that bundles them together (or build the one defining the EII directly as rstaticlib, bundling the crate that declares the EII).

Separately I'm not currently sure how we could support being able to link multiple rstaticlibs while having them export the standard library symbols as needed for linking against rlibs.

If one rstaticlib depends on another rstaticlib, it will assume that all crates included in that other rstaticlib don't need to be bundled into the local rstaticlib. This is the same behavior as for rust dylibs where any crates included through an upstream dylib don't get linked in locally but rather the version in the upstream dylib is used and rustc errors if two dylibs that are used as dependency together both bundle the same crate.

[teor2345]

Looks good! I'm super interested in this and would be happy to help with implementation and testing.

I have some suggestions around requiring used lang items, which it would be great for someone familiar with those details to double-check.

View changes since this review

[teor2345]

You've already said this above, but it helps to explain what doesn't work again here:

Suggested change

	RFC Note: This works today by removing the `-C emit-std-bundle=yes` and `-C rlib-version=v0` flags.
	RFC Note: This works today by removing the `-C emit-std-bundle=yes` and `-C rlib-version=v0` flags,
	but it would fail if there was a diamond dependency.

[adamncasey]

This technique actually works today even with diamond dependencies (only the standard library builds as a staticlib, the crate builds as an rlib). This note was meant to highlight that the only implementation changes required by this RFC are compiler flags/versions for a backwards compatibility strategy.

I can expand this example to include a diamond dependency to illustrate this point?

[teor2345]

There's already a diamond dependency example in the RFC, so I think it's enough to just say it works here too. (And that's an important piece of info!)

[teor2345]

Does this limitation apply to all lang items?

[teor2345]

I think there's a subtle requirement here around lang items, which is roughly:

Suggested change

	with all rlibs to be linked.
	with all rlibs to be linked.
	* All used [*language items*](https://rustc-dev-guide.rust-lang.org/lang-items.html) are supplied by exactly one rlib. Ordinarily, the standard library supplies these language items. But in `no_std` builds, some may be supplied by other rlibs.

[adamncasey]

I think this is a sensible wording, and this probably also sets the scene for tweaking this wording for a similar requirement for EIIs(same requirement?). I'll add this if no one objects for a couple of days

[teor2345]

It would be useful to test a recursive cross-language dependency:

• a Rust crate that requires symbol A from C++ and provides symbol B
• a C++ library that requires symbol B and provides A

This is currently possible but tricky to get right with some system linkers.

[adamncasey]

I agree it would be good to test at least one level of language weaving. At least the use case I have in mind would likely do this. Do you specifically want two libraries which depend on each other? In my experience that type of scenario is fairly rare/problematic (maybe I'm out of date there?).

I think I would pick to have a cross-language dependency tree more like:

        A (C++ Executable)
               |
               v
        B (Rust Library)
          |           \
          v            \
   C (C++ Library)      \
          |               \
          +---------------> D (Rust Library)

[teor2345]

I think both are good tests, and they increase test coverage in slightly different ways.

I agree recursion is rare, and we might not support it initially. But either way it would be good to know its status, and make sure it doesn't regress. Recursion might also be an opportunity to improve error messages or documentation more generally.

I've seen new interop users run into it in small projects when just starting out, too.

[teor2345]

It's not the same kind of prior art as Swift, but an example of how it's already used and how these issues are solved might be useful

Suggested change

	### Bazel-Based Rust Bootstrap
	The Rust compiler can be [bootstrapped using Bazel](https://github.com/dtolnay/buck2-rustc-bootstrap)
	instead of `x.py`. This process relies on unsupported rlib linking, but it also deals with issues such as:
	- [allocator shims](https://github.com/dtolnay/buck2-rustc-bootstrap/blob/71c6455e36b8b051efa8e5d0690c9ea680096211/allocator/allocator.rs#L29)
	- [duplicate symbols](https://github.com/dtolnay/buck2-rustc-bootstrap/blob/71c6455e36b8b051efa8e5d0690c9ea680096211/fixups/compiler_builtins/fixups.toml#L9)
	- panic handling [unwind](https://github.com/dtolnay/buck2-rustc-bootstrap/blob/71c6455e36b8b051efa8e5d0690c9ea680096211/fixups/std/fixups.toml#L19) [abort](https://github.com/dtolnay/buck2-rustc-bootstrap/blob/71c6455e36b8b051efa8e5d0690c9ea680096211/fixups/panic_abort/fixups.toml#L1)

View changes since the review

[krasimirgg]

What members does the bundle contain? One reading for it is includes everything from the sysroot, including private vendored dependencies of the standard libs, like hashbrown.

Does it include members that are only needed in some contexts, like libtest or libproc_macro?

If the rust toolchain includes the rustc-dev component (so a bunch of rustc_* packages in the sysroot), does the stdlib bundle also include them?

How does it pick between (so resolve potential symbol conflicts) libpanic_unwind and libpanic_abort?

How do we deal with no_std variants with-- and without-- alloc?

For context, in the Bazel Rust rules, when we adapt Rust standard libraries .rlibs to be consumed by the native linking infrastructure (currently unsupported by upstream), we have to deal with a bunch of cases like this to produce the standard libraries linker inputs dependency graph. So I'm imagining we may need some fine grained mechanism of controlling what exactly's in and out of the bundle.

View changes since the review

[teor2345]

How do we deal with no_std variants with-- and without-- alloc?

This is covered in the lines below:

Any desired crate-level metadata and/or compiler flags can be supplied in the process of compiling this standard library bundle, for example #![no_std] to omit the standard library (and only link the core library), or -C target-feature to enable specific CPU features.

I've also made a suggestion in the reference section, which might cover some of these choices:

All used language items are supplied by exactly one rlib. Ordinarily, the standard library supplies these language items. But in no_std builds, some may be supplied by other rlibs.

And there's this text as well:

The exact symbols that are exposed in a standard library bundle is unspecified by this RFC. In general, they are expected to change with every Rust release and may change depending on the manner in which the standard library bundle was compiled.

But I don't think that's enough, because it's important that different tools agree on how to make these choices.

So for each of the common member selection choices, it would be good to document in the RFC how it is configured. This will also help us identify any gaps.

For less common choices we could leave them for the implementation stage.

[madsmtm]

I like the RFC in general! One thing I think is important to note (don't know where it fits) is that #[used] won't work with this scheme (or, to my knowledge, any external linker scheme that only concerns itself with archives).

The expected semantics of #[used] is that once the crate is included in the crate graph, the symbol should reach the linker. This is most prominently utilized in crates such as inventory and ctor to do before-main initialization for symbols, which the rest of the crate can then (mostly) depend on.

For example:

use std::sync::atomic::{AtomicUsize, Ordering};

static FOO: AtomicUsize = AtomicUsize::new(0);

mod internal {
    use super::*;

    // Initialize `FOO` at the start of the program.
    // Roughly equivalent to what `#[ctor::ctor]` does.
    #[used]
    #[cfg_attr(target_vendor = "apple", unsafe(link_section = "__DATA,__mod_init_func,mod_init_funcs"))]
    #[cfg_attr(target_os = "windows", unsafe(link_section = ".CRT$XCU"))]
    #[cfg_attr(not(any(target_vendor = "apple", target_os = "windows")), unsafe(link_section = ".init_array"))]
    static INITIALIZER: extern "C" fn() = {
        extern "C" fn init() {
            FOO.store(42, Ordering::Relaxed);
        }
        init
    };
}

// Once the user runs `bar`, `FOO` _should_ be initialized.
#[unsafe(no_mangle)]
pub extern "C" fn bar() {
    assert_eq!(FOO.load(Ordering::Relaxed), 42);
}

rustc could then produce a mycrate.rlib with these members:

• lib.rmeta.
• foo.o which defines FOO.
• initializer.o which defines INITIALIZER, references FOO and std.
• bar.o which defines bar, references FOO and std.

Assume there's a piece of C++ code elsewhere that references and runs bar. Now, if a build system naively links against this archive, the linker will look for bar, find it via the archive index and parse bar.o, look for FOO, find it via the archive index and parse foo.o, but will completely ignore initializer.o (unless --whole-archive is used), which means that FOO will not be initialized, and bar will panic!

The way rustc currently works around this is by adding a symbols.o file to the final link that references all #[used] symbols, see rust-lang/rust#133832 and rust-lang/rust#95604 for details.

---

I can see a few ways of resolving this:

• Require build systems to use --whole-archive (might be bad for link-time performance, we would have to profile to see if it'd be acceptable. Likely acceptable in the initial implementation).
• Some kind of side-channel to the build system, so that it knows to pass -u INITIALIZER or similar to the linker.
• Add a fake reference to INITIALIZER to every object file in the archive, such that if any symbol is used from the archive, #[used] symbols are visible to the linker as well. This would help, though some use-cases such as divan would still be broken.

It may also be acceptable that certain language features are unavailable, but I feel that it should be discussed in the RFC, and should probably be linted against in dependencies if the user is compiling in this mode.

View changes since the review

[madsmtm]

I think having a good story for this is important for the ecosystem, without it, it becomes effectively a breaking change for libraries to add new #[link(...)] attributes. For example, a library may want to, in a minor version, link to the Security framework on macOS, but if nothing else in the final binary links against that, the user would get a confusing link error after updating their dependencies.

Specifying the linking in a build script with println!("cargo::rustc-link-lib=framework=Security") will likely work better with external build systems, so I fear that #[link(...)] may become a second-class citizen (which will cause us to regress on rust-lang/cargo#14948).

I won't demand that we solve this issue now, but I do think it would be useful for this RFC to discuss possible solutions to this, to give a better picture of how everything might fit together.

One partial solution might be to improve rust-lang/rust#121293.

---

Whatever we plan to do here, it should work together with link-arg kind (rust-lang/rust#99427), which IIRC also has options to specify whether a linker argument is early or late.

View changes since the review

[bjorn3]

There are already users of -Zlink-native-libraries=no like Android.

Specifying the linking in a build script with println!("cargo::rustc-link-lib=framework=Security") will likely work better with external build systems, so I fear that #[link(...)] may become a second-class citizen (which will cause us to regress on rust-lang/cargo#14948).

On the other hand you need #[link] on each extern "C" {} block that imports from a static library to correctly support building rust dylibs. Otherwise depending on what functions rustc decides to cross-crate inline or not, you might get linker errors when using the rust dylib as dependency due to functions of the static library that cross-crate inlined functions may call not getting exported from the dylib.