Don't return incorrect replaceability status (tx fees and policy, rpc/rest/zmq)

https://github.com/bitcoin/bitcoin/pull/22665

Host: jnewbery  -  PR author: darosior

The PR branch HEAD was 2bb244ea95 at the time of this review club meeting.

Notes

  • Opt-in Replace By Fee was implemented in Bitcoin Core in PR 6871 in 2015. The policy was described in a bitcoin-dev mailing list post.

  • After the implementation was merged, the mailing list description was written up as a BIP, and merged into the BIPs repository as BIP125.

  • The RBF policy in Bitcoin Core uses the nSequence field to signal opt-in for replaceability. According to the BIP:

    This policy specifies two ways a transaction can signal that it is replaceable.

    • Explicit signaling: A transaction is considered to have opted in to allowing replacement of itself if any of its inputs have an nSequence number less than (0xffffffff - 1).

    • Inherited signaling: Transactions that don’t explicitly signal replaceability are replaceable under this policy for as long as any one of their ancestors signals replaceability and remains unconfirmed.

  • It was recently discovered that the Bitcoin Core implementation does not treat the descendants of unconfirmed, opt-in RBF-signaling transactions as implicitly signaling for replaceability.

  • PR 21946 added code comments to document the discrepancy between BIP125 and the implementation.

  • PR 22665 replaces the bip125_replaceable field in the mempool and wallet RPC commands with a replaceable field. The bip125_replaceable field returned whether a transaction was replaceable according to the BIP125 rules (including through inherited signaling), while the new replaceable field returns whether the Bitcoin Core mempool considers the transaction replaceable.

Questions

  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?

  2. There’s an alternative PR 22698 which implements the inherited signaling as documented in BIP125. Which of the two approaches do you prefer? Why?

  3. Prior to this PR, there was a Chain interface member function isRBFOptIn(). This PR changes the caller to use SignalsOptInRBF() instead. Why is SignalsOptInRBF() a free function and not a member of the Chain interface class?

  4. Before this PR, the bip125-replaceable field could be yes, no, or unknown. The new replaceable field can only be true or false. Why is it never set to unknown?

  5. What is the IsDeprecatedRPCEnabled() function used for? Why does this PR move that function from rpc/server to rpc/util? Describe the process for deprecating RPC methods and fields. Why do we deprecate in this way?

  6. Do you agree that we should use a deprecation process to change the name from bip125-replaceable to replaceable? Why don’t we just update the value that is returned in bip125-replaceable?

Meeting Log

  117:00 <jnewbery> #startmeeting
  217:00 <michaelfolkson> hi
  317:01 <jnewbery> hi everyone! Feel free to say hi to let people know you're here (or not - lurking is also fine!)
  417:01 <theStack> hi
  517:01 <schmidty> howdy!
  617:01 <glozow> yeehaw
  717:01 <jnewbery> Today we're going to be looking at RBF and specifically PR 22665. Notes and questions are here: https://bitcoincore.reviews/22665
  817:02 <emzy> hi
  917:02 <jnewbery> Is anyone here for the first time?
 1017:02 <Azorcode> Hello Everyone
 1117:02 <merkle_noob[m]> Hi everyone.
 1217:03 <jnewbery> Alright, let's start with an easy question. Who had time to review the PR and notes/questions? (y/n)
 1317:03 <raj> y
 1417:03 <glozow> y
 1517:03 <michaelfolkson> ~0.5
 1617:03 <emzy> n only read the notes.
 1717:03 <theStack> 0.5y
 1817:04 <b10c> notes only
 1917:04 <schmidty> y
 2017:04 <jnewbery> any initial thoughts? Concept ACK/NACK?
 2117:05 <glozow> concept ACK because it’s unhelpful to report bip125-replacability when actually replacability is different
 2217:05 <glozow> rn you could be getting bip125-replaceable when actually it’s not
 2317:05 <raj> Seems like backward to me. Instead of fixing rpc reporting we should fix the behavior? But I guess we would get to the question.
 2417:05 <michaelfolkson> I guess doing something about the "CVE" is the Concept ACK. And then the Approach ACK is which of the two PRs. If so it is a Concept ACK
 2517:05 <glozow> i disagree with that, although it seems counterintuitive
 2617:06 <glozow> in this case, bip125 was written to document the code, and it had this inaccuracy
 2717:06 <glozow> (afaik)
 2817:06 <raj> glozow, it is counterintuitive, would love to know more..
 2917:07 <michaelfolkson> There are a few discussions to be had on this one. If BIPs are being used by other implementations and Core code doesn't follow the BIP...
 3017:07 <jnewbery> this ties nicely with the next question: There’s an alternative PR 22698 which implements the inherited signaling as documented in BIP125. Which of the two approaches do you prefer? Why?
 3117:07 <jnewbery> (https://github.com/bitcoin/bitcoin/pull/22698)
 3217:09 <raj> I would prefer to fix the behavior and follow BIP125, but thats just me.
 3317:09 <jnewbery> it sounds like glozow prefers the approach in 22665. Anyone agree/disagree?
 3417:09 <jnewbery> raj: why do you think a BIP is important in this case?
 3517:09 <glozow> i don't think "following" the BIP that simply misdocumented the code should be a goal
 3617:10 <glozow> if we determine inherited signaling is a better policy, then that would make sense
 3717:10 <raj> jnewbery, Its not that the BIP is specifically important. But the ancestor inheritance seems like a logical thing to have.
 3817:11 <glozow> would love to discuss this, why is inherited signaling better?
 3917:11 <jnewbery> raj: can you explain why inherited signaling is a logical thing to have?
 4017:11 <raj> Also there can be downstream wallets that depends on the BIP described behavior for their operation. Either directly on core, or via their own implementation.
 4117:12 <raj> jnewbery, because if I have an ancestor replaceable by fee, I would expect replacing it would also replace the descendants. Thus descendants naturally inherits the replacement.
 4217:14 <michaelfolkson> ^ Personally agree
 4317:14 <darius27> if BIP 125 was merged, doesn't it mean it was decided at that point that that was the more desired behavior? And it seems like it was unintentional that bitcoin core did not implement this behavior. So yeah i would have thought it would make sense to implement BIP 125 instead
 4417:14 <glozow> that's a good point, thinking of replaceability as "this transaction could be evicted due to RBF"
 4517:15 <raj> Ya and this PR would tag such descendant replaceablity as "false". Which could be very confusing.
 4617:16 <jnewbery> darius27: no, the wording in BIP125 was intended to be documentation of the behavior in Bitcoin Core. If anything, this is a docs bug
 4717:16 <darius27> jnewbery ah i see. Separately though, I agree with the points raj made
 4817:17 <michaelfolkson> jnewbery: Was that intention documented? Or have the BIP authors said that?
 4917:17 <jnewbery> michaelfolkson: yes
 5017:17 <jnewbery> https://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-12-03-18.59.log.html#l-147
 5117:18 <raj> jnewbery, but doesn't the fact that replaceability inheritance documented in the BIP irrespective of code behavior, makes that it a more intuitive behavior? One could equally argue that the code missed a behavior which is logical?
 5217:19 <glozow> you can also think of replaceability as "can these transaction's inputs be re-spent in an RBF transaction"
 5317:19 <jnewbery> I think it's worth reviewing the discussion in https://github.com/bitcoin/bitcoin/pull/7222, which added the bip125-replaceable field to the wallet RPCs. It's a little subtle, but I think there are definitely arguments against saying that the descendant of a bip125-replaceable is itself replaceable
 5417:20 <theStack> side-question: are there any other cases of bitcoin improvements where the BIP was following/documenting the implementation, rather than the other way round?
 5517:20 <glozow> the question of "can this transaction be evicted from the mempool" is always yes
 5617:20 <raj> glozow, in that case doesn't this PR breaks that definition too? I would get "false" for transaction whose inputs are indeed RBF replaceable.
 5717:20 <glozow> raj: you do? :O
 5817:20 <michaelfolkson> From scanning that IRC conversation log I think the BIP authors just agree to write a BIP after the code has been written rather than state they will write a BIP to document what the code does
 5917:21 <jnewbery> if the descendant's bip125-signalling ancestor is confirmed, then the descendant suddenly becomes non-replaceable
 6017:21 <glozow> michaelfolkson: uhhh but what else would they be writing the BIP about?
 6117:22 <michaelfolkson> Well in a world with multiple implementations that is the point of the BIPs. It isn't documentation of Core entirely (although it is hopefully that as well)
 6217:23 <jnewbery> theStack: I think often the BIP will be written in parallel with the implementation. There's often an "implementation" section of the BIP that links to an (unmerged) branch
 6317:23 <raj> glozow, let me know if i have it wrong, if I have a descendant with RBF disables, but have one of its ancestor RBF enables, replacing that ancestor would remove the descendant from mempool. Yet when I query its RBF status, I would get false. Isn't that correct?
 6417:24 <jnewbery> raj: that's true, but the ancestor could also be conflicted and removed from the mempool, along with all of its descendants, regardless of whether they're signaling replaceability
 6517:24 <glozow> raj: so you have transactions A -> B where A is parent, B is child
 6617:24 <glozow> you're saying A is signaling RBF, B isn't
 6717:24 <glozow> but A is replaced -> B is evicted
 6817:24 <michaelfolkson> If another implementation comes along and implements it according to the BIP, it is a little awkward to say "Oh sorry you should have ignored the BIP and just copied what Core did"
 6917:24 <glozow> is that right?
 7017:24 <glozow> and B's RBF status is false, even though it could get evicted due to A being replaced
 7117:25 <jnewbery> remember that replaceability is simply a local policy. Miners or other nodes can decide what policy they have for replaceability, which may be BIP125, or something completely different
 7217:25 <glozow> right. and there's another world where A is mined without B. then B is no longer inheriting RBF from anybody, so it goes back to being false
 7317:26 <glozow> i don't think "maybe evicted due to RBF" and "may be replaced using RBF" are the same thing
 7417:26 <raj> On BIPS: I don't thinks BIPs are (or should) be written to document code behavior. BIPS are suppose to be standards that other parts of the industry can follow knowing that it will not make any breaking behavior change. Which is violated in this case.
 7517:26 <jnewbery> I think it's unusual that a BIP was written for a local mempool policy. I agree that it should be documented, but a BIP doesn't seem like the right place for it. Implementers should be free to update their policy at any time, without reference to BIPs (again, those policies should be well documented)
 7617:26 <theStack> jnewbery: yes, i can remember seeing an implementation section some BIPs. i just thought that there is a rather strict rule that the BIP has to be merged first
 7717:27 <sipa> BIPs are simply means of publishing ideas
 7817:27 <jnewbery> theStack: For consensus changes or P2P protocol changes, I'd agree - the specification (BIP) should be finalized before the code is merged into Bitcoin Core
 7917:27 <sipa> having a BIP merged helps, because it gives an unambiguous way of referring to that idea
 8017:28 <sipa> but that is all it does
 8117:28 <sipa> it's not an indication of quality
 8217:28 <sipa> or community acceptance
 8317:28 <raj> glozow, yes, that was what I was referring.
 8417:28 <darius27> jnewbery - RE "I think it's unusual that a BIP was written for a local mempool policy". Isn't this similar to having BIPs for standardness rules? Or am i missing something
 8517:28 <MarcoFalke> all of this will go away with full-rbf anyway (hides)
 8617:29 <glozow> sipa: do you think we should try to make any parts of mempool policy universal across the network?
 8717:29 <sipa> glozow: can we? :)
 8817:30 <michaelfolkson> sipa: I think that kind of perspective cripples the value of the BIPs personally. Why bother with them? Alternative implementations should just look at Core code as the BIPs are unreliable
 8917:30 <jnewbery> sipa: I think "an unambigous way of referring to that idea" is more useful when the idea is probably going to be mostly fixed (ie consensus or p2p protocol). For a policy, it's much more likely that implementers will make future changes in the implementation, in which case the BIP can potentially become more harmful than useful.
 9017:30 <sipa> michaelfolkson: i completely disagree
 9117:30 <sipa> michaelfolkson: other implementations should do what they think is right
 9217:30 <jnewbery> darius27: I'm not aware of any other BIPs for standardness/policy
 9317:30 <michaelfolkson> sipa: For BIPs on consensus and policy?
 9417:31 <glozow> sipa: e.g. RBF signaling, since wallets might take that into consideration when choosing nSequence numbers
 9517:31 <sipa> in particular, i don't think there is any problem with other implementations implementing BIP125 as-is, if they feel that's the right approach
 9617:31 <sipa> michaelfolkson: there are plenty of other BIPs i think are bad ideas
 9717:32 <sipa> the issue here is bitcoin core claiming it implementing bip125, not the fact that bip125 exists
 9817:32 <sipa> (i have no opinion on whether bitcoin core's policy or bip125 is a better one)
 9917:32 <raj> sipa, if a BIP is accepted, it should not be considered in general a bad idea right? Otherwise the BIP process is unreliable.
10017:32 <sipa> raj: *wrong*
10117:32 <sipa> BIPs are for publishing ideas
10217:33 <sipa> not for approving them
10317:33 <sipa> sorry, there is no authority who can decide thos
10417:33 <michaelfolkson> Lightning is a very different world (not one dominant implementation) but they seem to take greater care with their BOLTs and have a different perspective on what they are attempting to achieve
10517:34 <sipa> that works with a small ecosystem with just 3 implementations :)
10617:34 <raj> sipa, But there are BIPs at different status. Some are accepted some are drafts, some are rejected. Can a accepted BIP (an implemented) become a bad idea?
10717:34 <michaelfolkson> sipa: 6 last time I checked ;)
10817:34 <MarcoFalke> raj: This is explained in BIP 2
10917:34 <sipa> raj: accepted just means it's in use in multiple implementations afaik
11017:34 <MarcoFalke> Any idea (that is loosely related to Bitcoin and is technically sound) can be written down as a BIP
11117:35 <jnewbery> raj: there's nothing official about any of the BIPs. You can open a PR against the BIP respository and as long as it meets a minimum quality standard (in terms of formatting, etc, not whether it's a good idea), then it should get merged eventually: https://github.com/bitcoin/bips/blob/master/bip-0002.mediawiki#bip-editor-responsibilities--workflow
11217:35 <darius27> jnewbery interesting, i didn't realize standardness/policy rules did not have BIPs. Thanks!
11317:35 <jnewbery> This is very interesting discussion, but I'm going to suggest we move on to some more technical details of the PR :)
11417:35 <michaelfolkson> "technically sound" needs an authority. I think this is a discussion for another day. This PR is only partly about BIPs
11517:35 <jnewbery> Prior to this PR, there was a Chain interface member function isRBFOptIn(). This PR changes the caller to use SignalsOptInRBF() instead. Why is SignalsOptInRBF() a free function and not a member of the Chain interface class?
11617:36 <jnewbery> and more generally, why is there a Chain interface class?
11717:36 <raj> jnewbery, yes and there are BIPS that are marked FINAL. BIP125 is one such. What does that mean in this case?
11817:37 <sipa> i'm not all that happy personally about the status of BIPs; apart from consensus bips for which there is an objective determination of "in use", it seems the bip status mostly reflects whether the author took the time to update ot
11917:37 <michaelfolkson> I'm trying to arrange a couple of meetings on the BIP process raj. Whenever I manage it come along ;)
12017:38 <glozow> `isRBFOptIn` needs mempool, `SignalsOptInRBF` ust needs to look at the transaction
12117:38 <michaelfolkson> But we should park it for now I think
12217:38 <raj> michaelfolkson, agreed.
12317:38 <jnewbery> sipa: glozow: that's right. So what does that have to do with the Chain interface?
12417:38 <jnewbery> sorry, not sipa, just glozow!
12517:41 <jnewbery> ok, so the Chain interface is used by the wallet to access the blockchain and mempool state: https://github.com/bitcoin/bitcoin/blob/4fc15d15667d9d9c4fb5515ce73c05b4596298ec/src/interfaces/README.md#L5
12617:41 <jnewbery> if we're considering a transaction's unconfirmed ancestors, then we need mempool state, so the wallet needs to go through the Chain interface
12717:42 <jnewbery> if we're _just_ looking at the nSequence fields in the transaction itself, then we don't need any context, so we can just call a utility function
12817:42 <jnewbery> Does that make sense to everybody?
12917:42 <glozow> jnewbery: ya
13017:42 <michaelfolkson> Yup
13117:43 <jnewbery> ok, next question. Before this PR, the bip125-replaceable field could be yes, no, or unknown. The new replaceable field can only be true or false. Why is it never set to unknown?
13217:43 <glozow> that reminds me i had question, what's rpc/server? is it server as in, for node stuff?
13317:43 <glozow> i assumed stuff was moving so that wallet rpc could use it
13417:43 <raj> jnewbery, yes. Can't we then have that function as a member of CTransaction? Since every transaction will have such an attribute?
13517:44 <jnewbery> raj: I think the idea is to keep CTransaction as mostly a struct. You can use an external function to interpret the meaning of the struct
13617:44 <glozow> unknown was when we couldn't look at mempool ancestors, but now it doesn't matter
13717:44 <raj> jnewbery, because we are not checking mempool any more. So the attribute is either true or false.
13817:45 <jnewbery> especially for something like bip125 replacebility, it's really not a property of the CTransaction, but rather how we decide to interpret the CTransaction
13917:46 <jnewbery> glozow: I think rpc/server is code for running on the node. rpc/util may also be run in the bitcoin-cli client. I'm a bit rusty on that though
14017:47 <raj> jnewbery, hmm, makes sense.
14117:47 <jnewbery> I think maybe rpc/server is not accessible to the wallet, which is why the deprecated function needed to be moved to rpc/util
14217:48 <jnewbery> Which leads us nicely to the next question. What is the IsDeprecatedRPCEnabled() function used for? Why does this PR move that function from rpc/server to rpc/util? Describe the process for deprecating RPC methods and fields. Why do we deprecate in this way?
14317:50 <glozow> makes sense
14417:50 <raj> jnewbery, it checks a list of strings to find deprecated methods. If a method is found in the list, then only corresponding functions are called and results are displayed (with an warning that it has been deprecated). Which gives a nice non breaking interface for downstream users to adopt the breaking changes.
14517:51 <jnewbery> raj: yes. So why don't we just remove the fields immediately? Whay benefits are there to deprecating over multiple releases?
14617:51 <raj> jnewbery, to reduce catastrophe? :D
14717:52 <michaelfolkson> So third party applications have time to migrate off relying on them
14817:52 <jnewbery> michaelfolkson: right
14917:53 <jnewbery> ok, final question. Do you agree that we should use a deprecation process to change the name from bip125-replaceable to replaceable? Why don’t we just update the value that is returned in bip125-replaceable?
15017:53 <raj> downstream folks would get time to upgrade, and still have old methods available to use for time.
15117:53 <raj> jnewbery, as its not following BIP125 anyway, so the name change makes sense to me.
15217:55 <darius27> would it also be bad/confusing to keep the name of the field the same while changing its behavior?
15317:55 <jnewbery> I think we may also want to add a note to https://github.com/bitcoin/bitcoin/blob/4fc15d15667d9d9c4fb5515ce73c05b4596298ec/doc/bips.md#L35 that we don't implement BIP125 exactly according to the spec
15417:55 <jnewbery> darius27: yes. I agree!
15517:55 <jnewbery> ok, 5 minutes left. Any final questions before we wrap up?
15617:56 <michaelfolkson> We didn't (really) discuss the alternative PR but I think long term we go with the superior solution (assuming we can get consensus on what the superior solution is). And if that means changing the code and/or the BIP we do that
15717:56 <michaelfolkson> I'm in the raj camp and think the BIP actually outlines the superior solution. If it didn't we should change the BIP
15817:57 <jnewbery> michaelfolkson: we discussed the two different approaches for the first half hour
15917:57 <raj> jnewbery, is this PR gets merged, would that mean BIP125 also needs to be updated (going with the logic that its code behavior documentation)?
16017:57 <michaelfolkson> jnewbery: Right but some of the answers to these questions are dependent on which approach/PR you go with
16117:58 <michaelfolkson> For both a rename seems necessary
16217:58 <jnewbery> raj: I'm not sure. As sipa says, BIPs are just there to document ideas. BIP125 remains unchanged as an idea whatever we decide to implement in Bitcoin Core.
16317:58 <sipa> BIP125 is a name given to an idea; bitcoin core does not implement that idea; the solution is either documenting that bitcoin core does not implement bip125, or perhaps writing another BIP that does (if people feel that idea is BIP-worthy). Under no circumstances can BIP125 be changed to suddenly mean something else
16417:58 <sipa> there are many other BIPs that Bitcoin Core doesn't implement
16517:58 <michaelfolkson> Ok that's fine. So if in raj camp there should be a new BIP
16617:59 <sipa> if it were in draft status, and still subject to change, that'd be different, but changing BIP125 now is both impossible (per BIP2) and would be extremely confusing
16718:00 <michaelfolkson> If in glozow camp should there be new BIP?
16818:00 <michaelfolkson> For what Core has implemented?
16918:00 <sipa> i'm not convinced this sort of policies need a BIP in the first place
17018:00 <jnewbery> I don't think a new BIP is necessary - as long as Bitcoin Core clearly documents its policy, that's enough. There's no reason it should be in the BIPs repository.
17118:00 <michaelfolkson> Ok
17218:01 <jnewbery> michaelfolkson: please stop trying to put people in "camps"
17318:01 <raj> Is there any reason why we don't wanna have RBF inheritance?
17418:01 <jnewbery> ok, that's time. Thanks everyone!
17518:01 <emzy> Thank you jnewbery
17618:01 <jnewbery> #endmeeting