Add Fee rate Forecaster Manager (tx fees and policy)

Apr 9, 2025

https://github.com/bitcoin/bitcoin/pull/31664

Host: ismaelsadeeq - PR author: ismaelsadeeq

Notes

Motivation and Background

The current fee estimator in Bitcoin Core, used by the wallet and exposed to clients via RPC, is CBlockPolicyEstimator.

This estimator maintains a vector of fee rate buckets ranging from 1,000 sats/kvB to 10,000,000 sats/kvB. These buckets are spaced exponentially by a factor of 1.05. For example, if the first bucket is 1,000 sats/kvB, the next would be 1,050 sats/kvB, and so on. The estimator works by:

Tracking each transaction’s fee rate and assigning it to the appropriate bucket.
Monitoring transactions as they leave the mempool. If a transaction is confirmed in a block, the estimator records success in the bucket, along with the number of blocks it took to confirm (i.e., the difference between the confirmation block height and the height at which it was first seen in the mempool).
If a transaction is removed for any reason other than inclusion in a block, it is recorded as a failure.
This data is aggregated using an exponentially decaying moving average, so old data points become less relevant over time.

A naive fee rate estimate by this estimator would provide an estimate for a confirmation target n by going through these buckets from the lowest bucket to the highest bucket and returning the lowest fee rate bucket where more than 80% of the transactions first seen within these n blocks were confirmed.

However, the fee estimator is more complex than this. It maintains three sets of these buckets: Short-term, Medium-term, and Long-term, each decaying at a different rate.

These allow the estimator to provide fee estimates in two modes:

Conservative – Uses long-term data for more cautious estimates and is tailored toward users who do not plan to not fee-bump. These fee rates tend to be higher.
Economical – Relies on more recent short-term data for making fee rate estimates, potentially resulting in lower fees. For a more detailed explanation of this fee estimator’s design, see John Newbery’s Introduction to Bitcoin Core Fee Estimation and the high-level design overview by the original author.

Key advantage: This approach is highly resistant to manipulation because all transactions must be:

Relayed (ensuring visibility to miners)
Confirmed (ensuring they follow consensus rules)

Key limitations for users:

Mempool unaware: Does not consider the current mempool unconfirmed transactions when estimating fees. As a result, it remains oblivious to sudden changes, such as when high fee rate transactions are cleared and the mempool becomes sparse, leading to overpayment. Likewise, during sudden congestion, it fails to adjust, resulting in underpayment and missed confirmation targets.
Package unaware: Only considers individual transactions, ignoring parent-child relationships. As a result, it fails to account for CPFP’d transactions, which can lead to lower fee rate estimates, resulting in underpayment and missed confirmation targets.

These limitations cause practical problems for:

Bitcoin Core wallet users (many now rely on third-party providers) due to the inaccuracies mentioned above.
Lightning implementations (e.g., C-lightning) that use Bitcoin Core’s fee estimator may fail to confirm transactions before their timelocks expire if it underestimates. If it overestimates, they may construct commitment transactions that pay more than necessary.
Bitcoin clients (e.g., BTCPay Server), have switched to external fee estimators due to Bitcoin Core’s mempool unawareness.
Others (e.g., Decred DEX), continue to see high fee rate estimates more than necessary even after the recent v28 switching of estimateSmartFee default mode to “economical”.

Bitcoin Core needs to provide reliable fee rate estimates to uphold the trustlessness and self-sovereignty of node operators and the services that rely on it. Relying on external estimators undermines these principles. Bitcoin Core’s fee estimator should be as reliable and cost effective as external alternatives.

This PR is part of the Fee Estimation via Fee rate Forecasters project #30392 which aims to address these limitations. A detailed design document of the project is available: Fee Rate Forecasting Design

Implementation Details

PR #31664 introduces a framework for improved fee estimation with the following key components:

1. Core Utility Structures

Forecaster abstract class: Defines the interface for all fee rate forecasters, establishing a consistent API for different fee rate forecasting strategies (Commit a2e3326).
ForecastResult struct: Provides an output format containing fee estimates and associated metadata (Commit 1e6ce06)
ConfirmationTarget struct: Implements a flexible input format supporting block-based targets with extensibility for future time-based targets (Commit df7ffc9)
ForecastType enum: Identifies different forecaster implementations, enabling appropriate routing of fee rate requests (Commit 0745dd7)

2. MempoolForecaster Implementation

MempoolForecaster class: Inherits from Forecaster and Generates block templates and extracts the 50th and 75th percentile fee rates to produce high and low priority fee rate estimates (Commit c7cdeaf)
Performance optimization: Implements a 30-second caching mechanism to prevent excessive template generation and mitigate potential DoS vectors (Commit 5bd2220)

3. Introducing `FeeRateForecasterManager`

FeeRateForecasterManager class: Serves as the central coordinator for all fee rate forecasters, maintaining shared pointers to registered forecasters (Commit df16b70).
Node Interface: The PR updates the node context to hold a unique pointer to FeeRateForecasterManager (Commit e8f5eb5).
Backward compatibility: Exposes a raw pointer to CBlockPolicyEstimator for compatibility with existing estimateSmartFee calls and related functions

4. Integration with `CBlockPolicyEstimator`

CBlockPolicyEstimator adaptation: Refactors the existing estimator to inherit from the Forecaster base class, adapting it to the new architecture while preserving existing functionality (Commit 9355da6).
Validation Interface: Now maintains a shared pointer to CBlockPolicyEstimator.

5. Files restructuring

The PR renames fees.{h,cpp} to block_policy_estimator.{h,cpp} to better reflect component responsibility (Commit 85dce07)
It also renames fees_args.{h,cpp} to block_policy_estimator_args.{h,cpp} for consistent terminology (Commit ec92584)
The PR renames policy_fee_tests.{h,cpp} to feerounder_tests.{h,cpp} to align with tested functionality (Commit 3d9a393)

Component Relationships

The architecture establishes a clear hierarchy:

FeeRateForecasterManager sits at the top level, coordinating all fee rate forecasters.
Both CBlockPolicyEstimator and MempoolForecaster implement the Forecaster interface, providing different approaches to fee rate forecasts.

                           ┌─────────────────┐
                           │FeeRateForecaster│
                           │    Manager      │
                           └───────┬─────────┘
                                   │ holds a shared_ptr copy of
                     ┌─────────────┴───────────────┐
                     │                             │
            ┌────────▼─────────┐         ┌─────────▼────────┐
            │CBlockPolicy      │         │Mempool           │
            │Estimator         │         │Forecaster        │
            └──────────────────┘         └──────────────────┘
            (Uses historical mempool data)    (Uses current mempool data)

ForecastResult and ConfirmationTarget standardize the input and output formats across all forecasters.
The node context maintains the lifecycle of these components, with ValidationInterface ensuring they receive blockchain updates.

Design Goals of `#31664`

Creates a pluggable architecture for multiple fee estimation strategies.
Creates the pathway for making fee estimation in Bitcoin Core both mempool-aware and package-aware.
Maintains backward compatibility with existing code.

Previous work and discussions to dig deep: