These notes were written by LarryRuane to accompany the Review Club meeting for PR 22350 and GDB video tutorial. The original can be found on Larry’s gist.

Using debuggers with Bitcoin Core

Please also refer to Fabian Jahr’s excellent documentation and video. In this document, I’ll cover only some of what his document does in slightly greater detail, while trying not to duplicate too much, and focused on gdb and Linux.

Video version of (most of) this document: https://vimeo.com/576956296/df0b66fbfc NOTE if you watch the video: Near the end, I had problems attaching to a running bitcoind and entering TUI mode (TUI mode is explained below). Please see the TUI section below for the fix to this problem. (Short version: start gdb with the -tui option.)

There are two debuggers commonly used with bitcoind (and the other c++ executables), lldb and gdb. The lldb debugger is used on MacOS and on Linux with clang builds. The gdb debugger is used on Linux, and can be used with either gcc or clang builds. This document won’t discuss lldb, but it’s similar to gdb.

gdb documentation

gdb has been around since the 1980s and has excellent documentation. There’s so much there that I don’t know; I’ll present the small subset of things I use most often.

build with optimization disabled

Debugging with optimizations enabled (the default setting) is very difficult and confusing because many variables can’t be seen (they’re “optimized out”), functions are in-lined, loops unrolled, etc. Single-stepping gives the experience of “I wonder where we’ll go next?” So always build without optimizations:

$ ./configure CXXFLAGS='-O0 -g'

To tell if optimization is enabled, run grep '^CXXFLAGS src/Makefile; if you see -O0 then optimization is disabled.

Be careful not to make any performance measurements with optimization disabled.

debug build shortcut

Sometimes you need to debug only a small part of the code. Instead of rebuilding everything, you can manually change the definition of CXXFLAGS in src/Makefile so that it specifies -O0 instead of -O2, touch the files you want to debug, and run make.

running bitcoind with gdb

There are two ways of debugging (or sometimes called controlling) a bitcoind instance with gdb.

starting bitcoind from the debugger

Make sure you’re in the src directory (so the debugger can find source files). Run

gdb bitcoind
Copyright (C) 2020 Free Software Foundation, Inc.
(...)
Reading symbols from bitcoind...
(gdb) run
Starting program: /g/bitcoin/src/bitcoind 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
2021-07-09T21:53:44Z Bitcoin Core version v21.99.0-ea728a30665a (release build)
(...)

The run command can be abbreviated as r – all gdb commands can be abbreviated as long as unique. Type help to see a list of topics, and help <topic> or help <command> for detailed help on that topic or command.

If you want to pass arguments to the program, just type them on the run command line

(gdb) run -regtest -debug

Sometimes the arguments string is rather long, so instead you can specify it when starting gdb (so that way your shell history has it).

$ gdb --args bitcoind -regtest -debug

Then you only need to type run. If you want to change the arguments, you can specify them on the run command, and they’ll replace the initial ones. If you type run again, the program will be run with the most recent arguments.

When you quit out of gdb, the bitcoind process dies too. Note that bitcoind is not given a chance to shut down gracefully; it’s like sending the process a hard-kill (SIGKILL or -9). But this hasn’t ever caused me problems (and it’s very good to know if this does cause problems!).

attach to a running bitcoind

Rather than starting bitcoind from within gdb, you can attach to an existing bitcoind. Example:

gdb --pid 1234

This will interrupt the running bitcoind and attach to it. It will be suspended, so it may lose P2P connections after a minute or so, but this has never been a problem for me. It’s not possible to affect the command line arguments. When you quit, the bitcoind continues running.

command recall within gdb

Like any modern shell, gdb has command history, which isn’t preserved across gdb sessions. Initially, it’s in “emacs” mode (control-p to bring up the most recent command, control-r to search, etc.), but typing the two keys ESCAPE control-j silently puts it into “vi” mode. This setting doesn’t persist across (gdb) executions.

If you type just an Enter (return, empty line), gdb will re-execute the last command. This is very nice for single-stepping – type next or just n, then return-return- etc. to step over each line, or step or s to continue to step downward.

useful gdb commands

A few other important commands (the ones I use most often):

You can set breakpoints by function name:

(gdb) b ConnectBlock

If not unique, prepend with the object name:

(gdb) b CChainState::ConnectBlock

Or set breakpoints by filename and line number:

(gdb) b validation.cpp:1728

If you set the breakpoint by function name, or if you specify the line number of the open brace, then when gdb stops there, the function arguments may not be set up correctly (will print as garbage), so single-step (next) to the next line.

command and symbol completion

Type TAB to extend or complete the current filename, command, or symbol.

When gdb starts up, it opens and reads the file .gdbinit in your home directory if it exists. Here you can set things you’d like all the time. Here are the settings I use:

set print pretty
set logging on
set history save on

The set history save on appends your typed commands into .gdb_history on exit; set logging on appends gdb output into gdb.txt as you go.

gdb.txt

The gdb.txt file is very useful; a large data structure such as m_chainparams.GetConsensus() is almost 200 lines of output when printed; it’s almost impossible to find things in it by visual inspection. But if you open gdb.txt in an editor after printing something large, you can explore it in the editor (most editors let you find matching braces and brackets, for example, or you can just search for things). Also, you have the entire history of the state of the variables or stack traces you’ve printed out, which can be great for later analysis.

Unfortunately, when you print variables, the print request that you’ve typed does not go into gdb.txt. So if you’re looking at gdb.txt in the editor, you’ll see the contents of variables, but may not know which they are! A workaround I often use is to print a string just before (or after) printing the variable, reminding myself which variable this is. For example,

(gdb) p "m_chainparams.GetConsensus()"
$6 = "m_chainparams.GetConsensus()"
(gdb) p m_chainparams.GetConsensus()
(... large variable ...)

(You can use the command history so you don’t have to retype all of that.)

GUI (TUI) mode

One of gdb’s lesser-known features is its built-in GUI mode. Well, it’s not a real GUI, it’s a TUI (text user interface). Here’s an example. It’s best to start gdb in TUI mode by specifying -tui on the command line. Once in gdb, switch between regular and TUI mode by typing the two keys control-x a.

There’s one problem with TUI mode: If you don’t start gdb with the -tui option, but instead switch to TUI mode by typing control-x a, then nothing the debugger prints gets appended to gdb.txt. A workaround for this is to exit TUI mode (control-x a), print the variable or stack trace, then re-enter TUI mode (control-x a).

There also seems to be a problem with attaching to a running process without specifying -tui if you then enter TUI mode by typing control-x a. The terminal control becomes completely confused. If attaching to a running process (and if you want to use TUI), always specify -tui on the command line, for example:

gdb --pid 1234 -tui

It also seems to be (at least for me) that if you start gdb with -tui, then you can’t switch out of it (control-x a). Trying to do that causes gdb to crash. So you must stay in TUI mode for the entire session, but that’s probably best anyway.

If the program you’re debugging prints to the terminal, the TUI display gets completely confused (because it thinks it alone is updating the window). Luckily, it’s easy to fix, just type control-L. But, for this reason, it’s advisable to start bitcoind with the -noprinttoconsole argument, and watch what it’s doing using tail -f ~/.bitcoin/debug.log in another window. Or start bitcoind normally and attach to it with gdb from another window.

Also very helpful is single key mode (enable and disable by typing control-x s) which allows you to type just a single character to do the common things like next and step.

Debugging unit tests

It’s easy to use the debugger on unit tests, for example (still in the src directory):

$ gdb --args test/test_bitcoin --run_test=getarg_tests/logargs
(gdb) b getarg_tests::logargs::test_method
Breakpoint 1 at 0x261aa0: file test/getarg_tests.cpp, line 196.
(gdb)

Notice the nonobvious way the names of the test functions are constructed.

functional (python) tests

python breakpoints

It’s often helpful to set a breakpoint in the Python test; adding this line will set a breakpoint; the test will suspend and you will be at the python debugger prompt ((Pdb)):

import pdb; pdb.set_trace()

You must run the functional test directly, rather than using test_runner.py. It helps to specify --timeout-factor 0 on the python script command line. The python debugger is pretty basic; type help to see the list of commands.

When the python test is suspended in the debugger, there generally will be one or more bitcoind processes that the test has launched. To see these, run a command like

$ ps alx | grep bitcoind

Their command line arguments show where their data directories are; it’s often use to look at their debug.log files. You can attach to them with gdb, set breakpoints there, continue, then continue in the python debugger.

pretty printing python variables

Another thing useful in debugging python programs is to add these lines near the top of the file:

import pprint
pp = pprint.PrettyPrinter(indent=4)

To print out a complicated variable such as a dictionary during the execution of the test, add a line like:

pp.pprint(complicated_var)

This will display the variable in a much more readable format.

a “breakpoint” hack

Sometimes you’d like to attach gdb to a running bitcoind when a certain condition occurs at a particular code location. A weird trick is to add code like to the code path you’re interested in:

{ static int spin = 1; while(spin); }

(This can be conditioned on some state.) This acts as a hardcoded breakpoint. When the bitcoind reaches this code, it will infinite loop; when you notice (or guess) that this has happened, you attach to the process with gdb, type i thr to see which thread is the one you’re likely interested in, then type thr <id>, and you should be at this infinite loop. Then type set var spin=0 and then you can print variables, single-step, set breakpoints and continue, or whatever. You don’t have to have started bitcoind in the debugger ahead of time.

When I don’t know how to cause bitcoind to execute a particular code path that I’m interested in debugging or understanding, I’ve set one of these “spin” landmines and then run the entire functional test suite. When it seems to be hung, if I run top and see a bitcoind steady at 100% CPU, I attach to it, find the right thread, and then begin debugging. It’s a hack, but this has been helpful many times.