Add systemtap's sys/sdt.h as depends for GUIX builds with USDT tracepoints (
build system, utils/log/libs) Dec 15, 2021
The PR branch HEAD was 0705708 at the time of this review club meeting.
Userspace, Statically Defined Tracing (USDT) allows us to hook into tracepoints during Bitcoin
Core runtime. Once a tracepoint is reached, it can pass data about process internals to a
userspace script for further processing. This is great for observability and allows for debugging,
testing, and monitoring. See
“Userspace, Statically Defined Tracing support for Bitcoin
Core” for more background information.
Initial build support for tracepoints was added in
#19866. The tracepoints (e.g.
name, arg0)) use (see
DTRACE_PROBE macros (see
from Systemtap under the hood. This requires the
header to be present during compilation. If not present,
ENABLE_TRACING will not be set to 1 in
and no DTRACE_PROBE’s are used.
On Ubuntu/Debian, for example, the
sys/sdt.h headers are included in the
package. If this package is installed, binaries with DTRACE_PROBE tracepoints will be built.
Developers wanting to use the tracepoints can build the binaries themselves.
However, tracepoints should ideally be included in release builds. This allows, for example,
hooking into the tracepoints of production deployments. No custom binaries (which might behave
differently) need to be compiled and deployed to trace Bitcoin Core.
Tracepoints are NOPs in our Bitcoin Core binaries. As we make sure
not to include any additional
expensive computations solely for the
there is only the minimal runtime overhead of the one NOP per tracepoint.
This PR adds the Systemtap package to Bitcoin Core’s depends system. GUIX builds for Linux
platforms now contain tracepoints.
We have discussed User-Space, Statically Defined Tracing (USDT) in a
Did you review the PR?
Concept ACK, approach ACK, tested ACK, or NACK?
Did you manage to build a
bitcoind binary including the tracepoints (see “
Listing available tracepoints” in doc/tracing.md)? Did you do a GUIX build?
For GUIX builds, why do we need to add the Systemtap
sys/std.h header as a dependency instead of using the header file avaliable on the GUIX build host system?
In which build step is
ENABLE_TRACING set? Under which condition is it set? What happens when it’s set to
What do we need to have consensus on before this PR is merged?
Why can we skip
make for the Systemtap dependecy? What are the problems and how are they solved?
Can you verify that the tracepoints are NOPs in the binaries? If yes, how?
1 17:00 <b10c> #startmeeting
3 17:00 <michaelfolkson> hi
5 17:00 <b10c> feel free to say hi so we know you'r here! (lurking is also fine)
7 17:01 <b10c> anyone joining the PR review club for the first time?
8 17:01 <zonemix> yep this is my first time
9 17:01 <b10c> welcome zonemix! feel free to ask questions anytime :)
11 17:01 <ragu3> first timer here. hi:)
12 17:02 <glozow> zonemix: ragu3: welcome!
13 17:02 <b10c> welcome ragu3!
16 17:03 <b10c> Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?
17 17:03 <svav> n but I read the notes
18 17:03 <michaelfolkson> Definite Concept ACK. "light conceptual agreement" undersells it :)
19 17:03 <azor> Hi everyone
21 17:05 <michaelfolkson> Not sure whether this should be fixed first "The tracepoints are currently not automatically/CI tested" before this is merged (but I think that is a later question on Approach ACK/NACK)
22 17:05 <glozow> stickes-v: yes sorry, i should have posted an announcement about the PR change
23 17:05 <glozow> (re: convo from earlier)
24 17:05 <b10c> Did you manage to build a bitcoind binary including the tracepoints? Quick no/build/build-guix
25 17:06 <svav> no did not have time
26 17:06 <michaelfolkson> No, away from my Linux machine
27 17:06 <ccdle12> build-guix
28 17:06 <merkle_noob[m]> Hi everyone
29 17:07 <b10c> svav michaelfolkson: that's fine
30 17:07 <b10c> ccdle12: did you check if the binaries include the tracepoints?
31 17:08 <ccdle12> b10c: I used gdb to check the the list of probes?
32 17:08 <b10c> ccdle12: awesome!
33 17:08 <ccdle12> I ran log_p2p_traffic on signet and that worked, but the other didnt work for me, I assumed it was my machine :(
34 17:09 <b10c> ohh, which one didn't work?
35 17:09 <ccdle12> log_utxos was getting segfault
36 17:09 <ccdle12> and the python scripts that relied on bcc
37 17:10 <ccdle12> but I think thats because I probably didn't install some of the bcc dependencies correctly
38 17:11 <b10c> maybe, happy to debug that further later if you want. Saw some reports of people having issues with old bpftrace versions (e.g. the segfault)
39 17:11 <ccdle12> ahh ok sounds great, I havne't dug too deeply into it yet so didn't want to start giving false reports :)
40 17:12 <b10c> michaelfolkson: yes, that's a valid point. I've added "not tested by CI" on purpose to maybe only merge this PR once that's setup..
41 17:12 <b10c> next question: For GUIX builds, why do we need to add the Systemtap sys/std.h header as a dependency instead of using the header file avaliable on the GUIX build host system?
42 17:14 <michaelfolkson> Because it needs to be present during compilation?!
43 17:14 <michaelfolkson> Don't know
44 17:15 <svav> Because the header needs to be a special one that contains something that makes something get built
45 17:16 <b10c> We can't really assume that a user has the systemtap header or even the right version on the GUIX build host
46 17:16 <b10c> we want the builds to be deterministic for all GUIX builders, so can't rely on host system software
47 17:16 <glozow> gotta be d e t e r m i n i s t i c
49 17:17 <b10c> GUIX builds are done in a GUIX container. this, for example, makes sure no host dependencies leak into the build
50 17:18 <b10c> now an autotools question: In which build step is ENABLE_TRACING set? Under which condition is it set? What happens when it’s set to 1?
52 17:20 <svav> Re sys/std.h dependency, as you saying in a GUIX build, when you set it as a dependency, it will look for sys/std.h within the specific GUIX container?
53 17:21 <svav> *are you saying*
54 17:21 <glozow> ehhhh when making configure from configure.ac?
55 17:22 <b10c> svav: yes! we first build the depends and then copy them as _inputs_ for our Bitcoin Core build
57 17:22 <b10c> sys/sdt.h is one of those depends
58 17:23 <b10c> (being added as one of those depends in this PR)
59 17:23 <b10c> michaelfolkson glozow: correct! what does this "AC_COMPILE_IFELSE" do?
60 17:24 <glozow> uhhhh i guess it tells autoconf something
61 17:24 <b10c> it includes the sys/sdt.h header and uses something that's named "DTRACE_PROBE", what could that be?
63 17:25 <glozow> "hi autoconf, if sys/sdt.h is included and something, set ENABLE_TRACING=1 in configure script"
64 17:25 <ccdle12> compile the input below to check if we have the depdency for sdt
65 17:25 <merkle_noob[m]> From the PR, ENABLE_TRACING is set when sys/sdt.h is found and/or when use_usdt is set to yes.
67 17:26 <b10c> we try to compile a small (2 loc) program during configure. that program includes sys/sdt.h and a simple tracepoint (the DTRACE_PROBE(context, event))
68 17:27 <michaelfolkson> Why include a simple tracepoint? To check if it is working?
69 17:28 <b10c> if this succeeds, we set ENABLE_TRACING to 1. We can we have the sys/sdt.h header and that it makes sense to it
70 17:28 <svav> DTrace provides a facility for user application developers to define customized probes in application code
72 17:31 <b10c> svav: the DTrace naming here is confusing.. DTRACE_PROBE is in-fact an alias for a Systemtap probe for compability reasons (see sys/sdt.h)
74 17:32 <svav> b10c: You are saying DTrace and Systemtap are two different things?
75 17:33 <b10c> otherwise they are undefined and no tracepoints are included in the binary
76 17:34 <b10c> svav: yes. AFAIK DTrace is older and from Sun, Systemtap was developed specifically for Linux. There exists some interoperability between the too, but I haven't tested anything in this direction
77 17:35 <b10c> feel free to ask about the configure step. I'll continue with the next question in the meantime
78 17:35 <b10c> What do we need to have consensus on before this PR is merged?
79 17:35 <b10c> What should we test before this PR is merged?
80 17:37 <michaelfolkson> Well to state the obvious that the Systemtap header is sufficient to get tracepoints working
81 17:37 <svav> I don't run a Bitcoin node yet, but I'm thinking about it ... Presumably for all this Pull Request testing, people set up separate test environments that don't affect your main node, so what sort of technology is necessary for test environments? Thanks.
82 17:37 <michaelfolkson> Including the header shouldn't have any adverse impacts on non-users of tracepoints
83 17:38 <michaelfolkson> "Merging this now could leave us with broken tracepoints in a release build." <- You wouldn't want broken tracepoints
84 17:38 <b10c> michaelfolkson: right, probably also that we don't break and platforms (for whatever reasons, I don't expect that we break anything)
85 17:40 <b10c> also right. I think we should be able to get some consensus on including the tracepoints in release builds here
86 17:41 <michaelfolkson> svav: I think it varies widely. I have a Linux laptop and Mac laptop for dev stuff. But others will use VMs, VPS, Docker etc
87 17:41 <b10c> svav: yes I run a few nodes in test setting but also for development
89 17:42 <svav> So, can you run multiple bitcoind on the same computer if you set some as test environments, or do you need to use virtual environments for each bitcoind?
90 17:43 <b10c> I also think we should have a bit more confidence that the tracepoints aren't broken by automatically testing them in the CI (or similar). michaelfolkson mentioned this earlier
91 17:44 <b10c> svav: yes, you can run multiple, even multiple on mainnet. You'll need to set different ports and data directories though
92 17:44 <b10c> The next question is a hard one:
93 17:44 <svav> b10c: ok thanks
94 17:44 <b10c> Why can we skip configure and make for the Systemtap dependecy? What are the problems and how are they solved?
95 17:45 <michaelfolkson> b10c: How do you test them in the CI? Add a tracepoint test?
97 17:49 <b10c> We can (and want to) skip configure and make for systemtap as we only need the header file, we don't need the systemtap tool for our build
98 17:49 <michaelfolkson> Ok cool, so the plan is to add them in this PR or a separate PR? I guess that gets the Approach ACK for me
99 17:50 <b10c> We don't want to build stuff we don't end up using in our build. That just wastes resources
100 17:50 <b10c> michaelfolkson: in a separate PR, that's not in-scope for GUIX builds
101 17:51 <merkle_noob[m]> Exactly the answer I wanted to give :)
102 17:51 <b10c> the problem with the sys/sdt.h header file is it includes a sdt-config.h which is only created in the configure step..
103 17:52 <b10c> sdt-config.h defines if an assembly feature can be used.
104 17:52 <b10c> This feature check was added in 2010 and we assume the feature can be used
105 17:52 <b10c> We apply a patch to the sys/sdt.h file that allows us to use it without the configure step.
106 17:53 <b10c> any questions regarding this?
107 17:54 <merkle_noob[m]> b10c: Please, could that assumption be broken?
108 17:54 <svav> Is an assembly feature referring to a specific assembly or the concept of an assembly?
109 17:55 <merkle_noob[m]> Or under what circumstances could the assumption not hold?
110 17:55 <b10c> yes, it can. I sadly haven't found any documentation on that feature being added to gcc (all documentation I saw just lists it as 'this is supported')
111 17:56 <b10c> maybe on some very old GCC version or uncommon platform/architecture?
112 17:56 <merkle_noob[m]> b10c: OK I see. Thanks.
113 17:57 <b10c> svav: a feature of the byte code assembly, does this answer your question?
114 17:57 <b10c> last question: Can you verify that the tracepoints are NOPs in the binaries? If yes, how?
115 17:58 <svav> b10c: yes thanks
116 17:59 <b10c> We can use `readelf -n bitcoind` or `info probe` in gdb to list the locations of the tracepoints in the binary.
117 17:59 <b10c> and then, for example, gdb to show us the assembly (byte code) at the tracepoint location
118 17:59 <svav> Does NOP stand for Non Op Code? Why is it important that a tracepoint is a NOP?
119 18:00 <b10c> or `objdump -d` works too
121 18:00 <b10c> svav: yes, a NOP does _nothing_
122 18:01 <michaelfolkson> no-op (no operation)
123 18:01 <b10c> the tracing uses it to hook into that exact instruction
124 18:01 <b10c> #endmeeting