TL;DR: Building x86 docker images on macOS with Rust binaries is not straightforward and this blog post will only solve half of the problems encountered.

Recently, I've encountered some challenges with Rust and rules_oci on a macOS M1/M2. I was trying to build a Docker image but kept receiving errors about unresolved toolchains. If you're facing the same situation, this blog post outlines the problem I had and the steps I took to resolve it.

To make things easy for myself, I limited the scope of the problem to build a Docker arm image to start with rather than an x86 image.

The Toolchain Issue

While trying to build a Linux/Arm image using Rust and rules_oci on my macOS M1, I ran into issues with unresolved toolchains. To elaborate, I was using Bazel to compile a Rust project to a Docker image for a different platform (Linux/Arm) than the one I was currently using (macOS M1). This cross-compilation process wasn't going smoothly and was throwing an error message about unresolved toolchains.

Here's the error I received:

% bazel build //products/tailwindelements/ng-backend:image
INFO: Invocation ID: b207a0d0-9136-44b9-9bbe-ab960c25a6be
ERROR: /Users/tfr/Documents/Projects/wonop-ng/products/tailwindelements/ng-backend/BUILD:7:12: While resolving toolchains for target //products/tailwindelements/ng-backend:bin: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type, @rules_rust//rust:toolchain_type.

So I started searching for how to setup Rust toolchains and hermitic C++ toolchains.

The Toolchain Solution

After a bit of digging and troubleshooting, I managed to resolve the unresolved toolchains by adding the following to the project's WORKSPACE file:

rust_repository_set(
    name = "rust_macos_arm64_linux_tuple",
    edition = "2021",
    exec_triple = "aarch64-apple-darwin",
    extra_target_triples = ["x86_64-unknown-linux-gnu"],
    version = "1.61.0",
)

rust_repository_set(
    name = "rust_macos_x86_64_linux_tuple",
    edition = "2021",
    exec_triple = "x86_64-apple-darwin",
    extra_target_triples = ["x86_64-unknown-linux-gnu"],
    version = "1.61.0",
)

which resolves the Rust toolchain. To resolve the C++ toolchain, we add:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

HERMETIC_CC_TOOLCHAIN_VERSION = "v2.0.0"

http_archive(
    name = "hermetic_cc_toolchain",
    sha256 = "57f03a6c29793e8add7bd64186fc8066d23b5ffd06fe9cc6b0b8c499914d3a65",
    urls = [
        "https://mirror.bazel.build/github.com/uber/hermetic_cc_toolchain/releases/download/{0}/hermetic_cc_toolchain-{0}.tar.gz".format(HERMETIC_CC_TOOLCHAIN_VERSION),
        "https://github.com/uber/hermetic_cc_toolchain/releases/download/{0}/hermetic_cc_toolchain-{0}.tar.gz".format(HERMETIC_CC_TOOLCHAIN_VERSION),
    ],
)

load("@hermetic_cc_toolchain//toolchain:defs.bzl", zig_toolchains = "toolchains")

zig_toolchains()

register_toolchains(
    "@zig_sdk//toolchain:linux_amd64_gnu.2.28",
    "@zig_sdk//toolchain:linux_arm64_gnu.2.28",
    "@zig_sdk//toolchain:darwin_amd64",
    "@zig_sdk//toolchain:darwin_arm64",
    "@zig_sdk//toolchain:windows_amd64",
    "@zig_sdk//toolchain:windows_arm64",
)

This allowed me to get past the issue with the unresolved toolchains, however, it introduced a new problem: missing system libraries for the target platform.

The Next Hurdle

The issue I ran into next was that system libraries for the target platform were missing. The problem was encapsulated in this error message:

Build script process failed with exit code 101
--stderr:
thread 'main' panicked at '

Could not find directory of OpenSSL installation, and this `-sys` crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
compilation process.

It appears that OpenSSL is missing from the Linux/Arm image I was trying to build. This might be because the system libraries for Linux are not present in the macOS environment.

I'm still in the process of trying to figure out a solution to this problem. If anyone has any suggestions or managed to get past this, please do share your insights.

In conclusion, while Rust theoretically supports cross-compiling from macOS M1/M2 to Linux, in practice, it proves to be quite challenging, primarily due to the need for Linux libraries on macOS even if we were able to resolve the issues of a cross-linker for Linux on macOS. The issue with OpenSSL in this scenario is just one of the hurdles encountered in the process. The most straightforward solution may be to use Docker or dual boot (when compiling to arm) to create a Linux environment on macOS. Even so, the complexities associated with cross-compiling should be considered and potentially re-evaluated if developing software for multiple platforms.