Bitcoin Core releases binaries for multiple different operating systems (Linux, Windows & macOS),
produced using cross-compilation (on Linux).
Historically, the project has used a very “non-standard”
toolchain for producing
its macOS release binaries. This is primarily due to there not being any “official” way to build
macOS binaries, using completely open source software, on a Linux system. The only Apple-sanctioned
way to build macOS binaries is using the Xcode toolchain, running on macOS hardware.
Using a non-standard toolchain is not ideal for a number of reasons:
We are the only user. No one else is testing things, finding bugs, or making improvements.
Creating our toolchain is reliant on Apple (infrequently)
releasing source code for the tools we need.
The most promising alternative toolchain for producing macOS binaries on Linux, has been LLVMs clang + lld. However until recently, lld was unable to link large / complex programs. Much work has been done over the past few years to re-write the Mach-O backend for lld, to where it is now able to link large, complex programs, i.e Chromium, as well as self-host. The new backend has now also become the default, as of LLVM 13.
PR #21778 is a work in progress attempt to transition our macOS toolchain to clang + lld, as opposed to our current, custom-built toolchain.
<michaelfolkson> Haven't made my mind up. Presumably it is a Concept ACK as you are working on it but don't know enough to say it is a Concept ACK (yet)
<fanquake> Currently our macOS toolchain is pretty homebrew. It's constructed from pre-compiled binaries, 3rd-party sources, we compile our own llinker (ld64), and mush it all together todo macOS builds.
<fanquake> The reason LLVM is most interesting for a macOS toolchain is that Apple has also upstream a lot of work (given Clang & LLVM) is the native Apple toolchain, meaning it is "better" in a number of ways, at producing macOS binaries than GCC + friends may be.
<fanquake> In regards to macOS, GNU ld does not support producing macOS binaries, or maybe in a limited fasion?, as far as I'm aware. So a specific requirement we have is a working linker.
<fanquake> Speaking more generally, off the top of my head, I don't think there is anything particularly fancy we require in regards to compiler / linker requirements. Other than say, things more general, like c++17 support etc.
<jamesob> The only thing I can think of (and this is slightly fuzzy) is that MarcoFalke has highlighted some concerning issues with the llvm/clang development process. IIRC, basically anyone can commit to the tree and they rely on some kind of revert process if something goes afoul. But this is slightly off-topic and unsubstantiated (at least by me)
<fanquake> larryruane: that is a bit of a can of worms, probably best discussed later. However I'd say we'd be much more likely to move to clang for windows before, before thinking about moving for Linux.
<fanquake> It's also the same reason I'd like to move away from our current setup. If anything, I think there are far more eyes over the LLVM repos, as opposed to: https://github.com/tpoechtrager/cctools-port
<fanquake> We do somewhat, given that we currently used a prebuilt clang (downloaded from LLVM) for the macOS builds, but don't use any of the other tools.
<fanquake> It is not yet supported by lld. Although it's unclear if that will actually be an issue, as the reason for setting the flag may no-longer be relevant when building "modern" macOS binaries.
<fanquake> hebasto: as in targetting a more recent minimum version of macOS. As if you know that your binaries are only running on more recent versions, you can assumed certain behaviour out of the dynamic linker, which might obsolete the thing that passing the flag would achieve.
<fanquake> and we can enforce that the binaries are only run on recent versions at compile time, by passing minimum version flags to the linker (and sanity check those versions in our symbol check scripts)
<fanquake> Currently yes, Guix uses it's "own" clang, from the clang-toolchain package. We do that by setting FORCE_USE_SYSTEM_CLANG during the Guix depends build.
<fanquake> That would continue to be the case going forward, even after these changes, however as I allude to in the next Q, there is a Guix related change missing from this PR.
<danielabrozzoni> I tried running the command in the pr (`make -C depends/ HOST=x86_64-apple-darwin -j9`), but it says `error adding symbols: DSO missing from command line`. I guess I'm doing something wrong ahah
<michaelfolkson> [16:49:46] <larryruane> The PR review club notes ask, "Did you try performing a macOS cross compile?" -- is the intention to do this on the PR branch, or just on master? (I thought I'd give it a try)
<fanquake> The PR branch. It's not fully working. However a depends build should work ok (including Qt), however building bitcoin-qt or libbitcoinconsensus will not at this stage.
<fanquake> I would think we'd want to keep the clang versions in sync, so that guix builds would be using clang 14, similar to users cross-compilling on linux would be.
<fanquake> michaelfolkson: I would still consider clang a dependency, given for a macOS cross-compile, we are downloading and then compiling using clang
<fanquake> One way to make obtaining it slightly less painful could be for someone to write a tool to extract it from the macOS command line tools .pkg, which is a few hundred mb, and save having to download 12 GB of Xcode.xip.
<fanquake> hebasto: I'm not sure. I think I would like to see the version change happen together, otherwise in the interim, you'd be Guix building with Clang 10, and the CI, or developers doing cross-compiles would be using 14.
<fanquake> The discussion around whether we'd want to migrate to using Clang 14 for the Guix build, at the same time we swapped over to using Clang 14 in depends.
<fanquake> Yes, they do. Even though we use a Guix installed Clang for the Guix build, we still use the rest of the macOS toolchain ld64, cctools etc from depends.
<fanquake> So if we migrate to using the llvm binutils and lld in depends, that would also then be used by the Guix build, but it would remain using it's own installed Clang.
<fanquake> The reason is that we need to maintain a macOS toolchain, that works outside of Guix, as Guix does not run everywhere, and there shouldn't be an expectation that you would need to use it to cross-compile.
<fanquake> jamesob: I'd be surprised if it did currently, as the features making this possible are basically only emerging in LLVM 14, which was officially relased last night
<michaelfolkson> The arguments for not forcing Guix use are it is new(ish), users might not understand it, it is not industry standard outside of Bitcoin?
<fanquake> The rm was originally added in https://github.com/bitcoin/bitcoin/pull/8210, to remove any LLVM C++ ABI related objects, given at that point we were copying more files out of the lib dirs in the clang tarball. Possibly just a belt-and-suspenders thing.
<fanquake> Likely irrelevant since https://github.com/bitcoin/bitcoin/pull/19240, fbcfcf695435c9587e9f9fd2809c4d5350b2558e, where we stopped copying any c++ libs from the clang tarball.
<fanquake> The only things we copy out of lib/ are `libLTO.so` and headers from lib/clang/clang-version/include, so deleting .so files from /lib in advance of that, doesn't achieve anything.
<fanquake> Besides that, the `libc++abi.so*` objects have actually changed location inside the lib/ dir, so even if we kept the current code, it wouldn't actually remove anything anyways.
<fanquake> michaelfolkson: there are a number of arguments for not forcing anyone that wants to compile your software to use a single package manager to do so. Happy to discuss further later.
<fanquake> jamesob: somewhat. The reason we rename lld to ld, is to make things "simpler" for the build system, as most build systems, especially those using autotools, look for, and expect to use a linker called ld.
<fanquake> Given we are in full control of our build system here, we can just rename lld, and have it "pretend" to be ld, for the sake of making everything work, and the build systems expecting GNU ld should mostly be none-the-wiser.
<fanquake> So renaming some of the tools is also a bit of a convenience for autotools, and can prevent warning output like: "configure: WARNING: using cross tools not prefixed with host triplet".