Calculate UTXO set hash using Muhash (
utils/log/libs) May 27, 2020
The PR branch HEAD was 4438aed09 at the time of this review club meeting.
The idea to use Muhash in Bitcoin Core was initially introduced by Pieter
Wuille in this
Pieter proposed an implementation in
#10434, which still comprises
most of the code in this week’s PR.
Fabian Jahr then picked up the proposal and made further research in 2019. A snapshot
of the work can be seen in
Based on further insights and feedback, the idea evolved into implementing
an index for all coin statistics and not only for the hash of the UTXO
set. This was implemented in
This week’s PR is the first in a series of PRs to incrementally implement
This PR modifies how the hash of the UTXO set is calculated. That hash, as well
as other coin statistics, can be accessed through the
RPC. It uses the
algorithm which allows for
The truncated hash
SHA512/256 is used
to prepare data for use in Muhash. Questions
What does “incremental hashing” mean? Which of its properties are interesting
for our use case?
What were the use cases for RPC
gettxoutsetinfo described in the resource
links? Can you think of others? Which use cases are the most relevant to you?
What do you think could be the biggest downside to introducing Muhash for the
UTXO set hash?
This PR not only adds Muhash but also
Why is the Muhash class implemented in Bitcoin Core and not in
Did you look into the other proposed hashing algorithms in some of the
resources? What were their drawbacks? Do you think Muhash is the right
What are your thoughts on the Muhash benchmarks, e.g. (a) the benchmarking
code, and (b) your observations if you tested the code locally?
Considering the trade-offs, should the old hashing algorithm be kept around
and accessible (using a flag, for example)?
1 13:00 <fjahr> #startmeeting
12 13:00 <fjahr> Hi everyone, welcome to this weeks pr review club. I am happy to share this one with you, since I have been working on this for a while.
14 13:00 <fjahr> Feel free to ask questions any time, there can be multiple topics discussed at the same time.
15 13:00 <fjahr> Who had a chance to review the PR? (y/n)
16 13:01 <michaelfolkson> hi
23 13:01 <raj_149> what does bogosize mean?
24 13:02 <sipa> it's a vague metric for size of the utxo set, but it doesn't have any actual meaning
25 13:02 <sipa> it's arbitrarily chosen constants
26 13:02 <fjahr> "A meaningless metric for UTXO set size" says the help :p
28 13:03 <sipa> it's inspired by "bogomips" in the linux /proc/cpuinfo which is a meaningless indicator for processing speed
29 13:03 <raj_149> sipa: fjahr oh.. :p
30 13:03 <fjahr> but good question, definitely something to consider removing :)
31 13:03 <sipa> all it means is larger number -> larger utxo set, somewhat
33 13:03 <sipa> it's useful for comparison
34 13:03 <sipa> the number just doesn't have any physical meaning
35 13:04 <sipa> there is also a field for the on-disk size, but that's nondeterministic
36 13:04 <fjahr> Let's start with the questions I came up with, they are focussed on conceptual understanding, but definitely throw in more technical questions! What does “incremental hashing” mean? Which of its properties are interesting for our use case?
37 13:05 <fjahr> sipa: but is it used by anyone? I could not figure that out tbh
38 13:05 <michaelfolkson> Updating the hash value rather than calculating a new hash from scratch
39 13:06 <raj_149> fjahr: adding new items into the hash input set without redoing the full hash. Easy to add and remove items from the set seems disreable for this purpose.
40 13:06 <uproar> incremental hashing: change in work to recompute message digest is a function of how much the input message has changed
41 13:06 <uproar> is my attempted definition
42 13:07 <jnewbery> a hash function that takes a set of items and returns a digest, with operations to add and remove items from the set
43 13:07 <uproar> and that that change is proportional to input message delta
44 13:07 <gzhao408> particularly useful here, since the UTXO set is very large and unordered - we frequently add and remove items in no particular order
45 13:07 <fjahr> raj_149: not sure if you mean to say that but it does need to be a set in the strict definition :)
46 13:08 <sipa> i think a better term is homomorphic set hashing
47 13:08 <uproar> unpacking that phrase would help
48 13:08 <sipa> incremental just means you can easily add items, but doesn't imply anything about removal or the order in which you can do so
49 13:09 <fjahr> true, that is the better phrase that includes the properties I was hinting at in the second question
50 13:09 <sipa> probably not worth the semantics discussion :)
51 13:09 <willcl_ark> that you can recompute the hash based on the old hash, after adding/removing items
52 13:10 <theStack> does a merkle tree also count as incremental hash structure? it's quite easy to add a new leaf item and propagate up to the root hash
53 13:10 <fjahr> I think these answer were all good, lets move on: What were the use cases for RPC gettxoutsetinfo described in the resource links? Can you think of others? Which use cases are the most relevant to you?
54 13:10 <sipa> theStack: if you treat the entire merkle tree as the "hash", i'd say yes - but it's not a compact one
55 13:11 <ecurrencyhodler> Can be used to validate snapshots for AssumeUTXO and also used for BTCPayServer's FastSync.
56 13:11 <willcl_ark> I think it might be nice for an SPV wallet, along with checking valid headers (with most work) were provided, to also query multiple nodes' UTXO set hashes
57 13:11 <raj_149> sipa: does that then also imply collisons are much common in case of incremental hashing? or its collison is not related to incremeental property atall?
58 13:11 <uproar> Use case, hmm maybe: produce new soundness proofs based on the UTXO set?
59 13:12 <sipa> uproar: muhash/ecmh/... don't permit compact proofs of inclusion or exclusion, so no - all you can do is check easily whether two sets you already have are identical
60 13:12 <nehan> it can also serve as a checksum to check for corruption of the utxoset
61 13:12 <sipa> raj_149: i don't know how to answer that
62 13:12 <fjahr> How about total_amount? What can that be used for? :)
63 13:13 <sipa> raj_149: it turns out that it's much harder to construct a homomorphic set hash and have it still be collision resistant
64 13:13 <fjahr> total_amount is one of the values returned from gettxoutsetinfo
65 13:13 <uproar> fast auditing on total supply
66 13:14 <raj_149> fjahr: isn't that the total circulating supply?
67 13:14 <sipa> willcl_ark: i don't think this is useful for SPV wallets (unless they also maintain a full UTXO set, which would be kind of pointless...)
68 13:14 <uproar> raj_149 it's the total sum of UTXO maybe not circulating nor circulatable
69 13:14 <fjahr> uproar: raj_149: yes! and that was one of the motivations that auditing could actually be really fast using the index :)
70 13:15 <raj_149> fjahr: i think its very useful then to get normies get into Bitcoin. Thats the golden number to show. :D
71 13:16 <uproar> fjahr fast amount/supply auditing is something I'd like but on the other hand I question to what end does it serve nodes (rather than users of those nodes), is MuHash decreasing of node-operation costs when the command is run?
72 13:16 <fjahr> Next question: did you come across downsides from implementing muhash for gettxoutset?
73 13:17 <raj_149> uproar: yes right, thanks for pointing that.
74 13:17 <willcl_ark> sipa: hmmm yes, I guess you get nothing more from this as an SPV node than querying multiple full nodes for headers
75 13:17 <sipa> uproar: muhash itself is slower than the current hash, but the advantage is that it can be incrementally computed and maintained in the background (updating the hash at every block, rather than at RPC time)... making the RPC instantaneous
76 13:17 <fjahr> raj_149: yeah, and it's definitely much cooler if it takes a second rather than minutes :)
77 13:17 <uproar> the UX benefit is: look no surprising inflation! but from the node operation perspective is: There shouldn't have been unless some pretty fundamental aspect about validation are broken
78 13:18 <adiabat> how much slower is it if you're only running once after full IBD?
79 13:18 <uproar> sipa that's a valuable datum
80 13:18 <sipa> adiabat: my very rough guess is 5x or so
81 13:18 <sipa> (of course, if you're I/O bottlenecked it's much less)
82 13:18 <adiabat> oh that's not too bad. Once you run it more than a few times you're ahead
83 13:18 <uproar> gettxoutsetinfo is painfully slow even on high resource machines
84 13:19 <uproar> currently, that is >:]
85 13:19 <sipa> with this PR it becomes a lot worse
86 13:20 <sipa> fjahr: i guess you have more recent performance numbers
87 13:20 <raj_149> sipa: i dont understand, how is it worse?
88 13:20 <sipa> raj_149: because MuHash is many times slower than SHA256
89 13:20 <fjahr> it definitely exceedes the standard rpc timeout on lower powered machines, which was the main driver to get concerned for me.
90 13:20 <sipa> (and the current hash uses SHA256)
91 13:21 <willcl_ark> Ok so it's slower, but happens in the background, at each new block?
92 13:21 <raj_149> sipa: but the node will do that hashing internally for each block right? so getutxosetinfo will be very fast, right?
93 13:21 <sipa> willcl_ark: not with this PR
94 13:21 <nehan> raj_149: this change doesn't actually implement a rolling UTXOset hash. It just uses MuHash to calculate it when requested
95 13:21 <willcl_ark> ah ok
96 13:21 <sipa> willcl_ark: this just swaps out the SHA256 based hash for MuHash
97 13:21 <sipa> but it paves the way for a background-updated utxo set hash
98 13:21 <uproar> sipa is it that the first gettxoutsetinfo is going to get substantially slower but the incremental calls on gettxoutsetinfo will get faster relative to old gettxoutsetinfo subsequent calls?
99 13:21 <willcl_ark> 1 step back...
100 13:21 <sipa> uproar: no, all of them
101 13:22 <raj_149> nehan: ah ok.. thats true.
102 13:22 <adiabat> I guess for future PRs there are several options: increment at each block, or update a cache every time gettxouset is called and increment then
103 13:22 <fjahr> I don't have them at hand right now but I think I posted them in an older pr version
104 13:22 <sipa> uproar: again, this PR doesn't do any background hashing
105 13:22 <willcl_ark> we need the next PR to make it rolling
106 13:22 <adiabat> sipa: is later intent to make it happen every block or only update when requested? (or something else?_
107 13:23 <sipa> adiabat: ask fjahr :)
108 13:23 <jnewbery> it also adds a flag so you can still get the legacy SHA256 hash from gettxoutset if you want
109 13:23 <willcl_ark> Would it not be wise to add a commit for the rolling hash as part of this PR? Why would you split this way (with a performance degredation)?
110 13:23 <raj_149> basic question: utxo set is updated currently at each new block? or by some other trigger?
111 13:23 <michaelfolkson> There's another downside in terms of time taken to brute force the pre-image right? Once the pre-image is known from the original hash an attacker can brute force the pre-image for the new hash in a shorter time than normal?
112 13:23 <fjahr> will_clark: yes, that was 18000 which had everything, now i have split it up and this is the first part.
113 13:23 <adiabat> fjahr: is the intent.. :)
114 13:23 <willcl_ark> fjahr: I see
115 13:24 <sipa> michaelfolkson: it is supposed to have 128-bit collision resistance security, just like SHA256
116 13:24 <fjahr> adiabat: The plan is to update with every block/reorg and have an index
117 13:24 <sipa> adiabat: updating only when requested requires iterating over the whole utxo still, so that wouldn't give any gain
118 13:24 <fjahr> so older states can be queried quickly as well
119 13:25 <michaelfolkson> When you say "supposed to" sipa what does that mean? Makes it sound like you doubt that claim ;)
120 13:26 <fjahr> Keep those questions coming :) But here is one of mine as well: This PR not only adds Muhash but also TruncatedSHA256Writer. Why?
121 13:26 <adiabat> I don't see why updating when requested needs to look at the whole utxo set? Couldn't it replay blocks since last call and perform the additions / deletions that way?
122 13:26 <fjahr> TruncatedSHA512Writer actually :D
123 13:26 <jnewbery> michaelfolkson: that's because it's a hash function. We assume that it has that much security until someone breaks it
124 13:27 <adiabat> (though I guess that could be even slower if it's very infrequently called)
125 13:27 <raj_149> fjahr: wild guess. to add extra layer of security?
126 13:27 <sipa> michaelfolkson: assuming the discrete logarithm in GF(2^3072 - 1103717) takes is 128-bit secure, that ChaCha20 and truncated SHA512 have their intended security, ...
127 13:27 <jnewbery> adiabat: we don't have those utxos any more after the block has been connected
128 13:27 <raj_149> and may be added collison resistance..
129 13:27 <michaelfolkson> Ok gotcha thanks sipa jnewbery
130 13:27 <sipa> adiabat: oh sure, it could, but that'd be incompatible with pruning
131 13:28 <uproar> maybe I'm missing something but if currently SHA256 hasing in gettxoutsetinfo makes calls take ~5 minutes, and MuHash increases the time, what's the benefit? being able to do it in the background or that the total sum of updates costs less because of the "homomorphic set" properties?
132 13:28 <adiabat> ah right that makes sense
133 13:28 <sipa> uproar: updating is super fast
134 13:28 <nehan> sipa: can your replay blocks to update the rolling utxoset hash without the utxoset at the point you start replaying?
135 13:28 <sipa> nehan: sure
136 13:28 <jnewbery> right, you could look up each utxo in the previous block files, but that would be painfully slow, and incompatible with pruning
137 13:28 <sipa> if you still have the block
138 13:29 <sipa> and the undo data
139 13:29 <uproar> sipa ok, thanks I think i mistook "Muhash is slower" to mean "the entire update process for post PR will be slower" which I take be not the case
140 13:29 <nehan> sipa: are you suggesting you'd have to reconstruct the UTXOset at that point in the past using the undo data?
141 13:29 <sipa> uproar: yes it will be much slower, but it paves the way for incrementally computing the hash, which will make the RPC instant
142 13:29 <jnewbery> oh sorry I'm wrong - of couse you can use the undo data
143 13:29 <sipa> uproar: however this PR doesn't do any of the incrementality
144 13:29 <uproar> now I understand the bigger picture, thanks
145 13:30 <sipa> nehan: no
146 13:30 <fjahr> ray_149: so the truncated hash does not decrease security but i don't think it increases it either. It is an efficient way to turn the raw data of utxos into a 256bit number.
147 13:30 <nehan> sipa: k, thanks
148 13:31 <raj_149> fjahr: but why trucncated 512? why not sha256 it directly?
149 13:31 <sipa> nehan: blocks are "patches" to the UTXO set; if you have the hash state as of block N, you you roll that state forward by applying the blocks after it to it
150 13:31 <sipa> you don't need the actual UTXO set anymore at that point
151 13:31 <fjahr> well, sipa made the choice but from the research papers I saw it is faster
152 13:32 <willcl_ark> faster to SH512 and then truncate?
153 13:32 <sipa> yeah, it's because most UTXOs are of a size that would require 2 SHA256 compressions, but only one SHA512 compression
154 13:32 <sipa> so SHA512 may be faster for such inputs
155 13:32 <sipa> it's probably worth benchmarking that again, now that we have AVX2 and SHA-NI optimized SHA256
156 13:32 <fjahr> There is a link to a SHA512/256 research paper somewhere in a #18000 comment
157 13:33 <fjahr> I think that was the best resource I found if you want to dig deeper
159 13:34 <fjahr> found it :)
160 13:34 <willcl_ark> StackExchange is tellign me the max message size for SHA256 is 2 million terabytes
161 13:34 <willcl_ark> thanks fjahr
162 13:34 <fjahr> sipa: correct me if that is not the right resource
163 13:34 <sipa> willcl_ark: that's correct, but i don't think that's relevant?
164 13:34 <sipa> fjahr: i think the right resource is benchmarking yourself :)
166 13:35 <sipa> UTXOs are limited to 10000 bytes ish
167 13:35 <willcl_ark> sipa what is " most UTXOs are of a size that would require 2 SHA256 compressions" in reference to then?
168 13:35 <uproar> I think it's indicating the mechanics of how 256 blocks are compressed
169 13:35 <sipa> willcl_ark: SHA256 transforms an internal 32-byte state by consuming 64 byte blocks at a time; the runtime is proportional to have many of those 64-byte blocks you need to consume
170 13:36 <uproar> max message size and how many blocks you have to compress aren't proportional
171 13:36 <fjahr> Ok, I hope it's not too obvious for everyone, but: Why is the Muhash class implemented in Bitcoin Core and not in libsecp256k1?
172 13:36 <willcl_ark> hmmm ok. that's definitely a good one for me to read up on :)
173 13:36 <sipa> SHA512 transforms an internal 64-byte state by consuming 128 byte blocks at a time; the runtime per consumption is longer than SHA256's, but typically faster per byte
174 13:36 <sipa> willcl_ark: as UTXOs are often larger than 64 bytes, but less than 128 bytes, they'd need 2 SHA256 consumptions, or 1 SHA512 consumption
175 13:36 <sipa> so the question is which of those two is faster
176 13:37 <raj_149> sipa: correct me if i am wrong, but its are not hashing the utxos directly, its hashing the finalised muhash, which is 3072 bit.
177 13:37 <willcl_ark> sipa: ah ok. that makes it clear now!
178 13:37 <fjahr> or why would ECMH be in lipsecp and not Muhash/
180 13:37 <sipa> raj_149: no, the individual UTXOs are hashed, to produce a 256-bit key; that key is than expanded into a 3072-bit number using ChaCha20
181 13:38 <sipa> the 3072-bit numbers are then multiplied modulo 2^3072-1103717
182 13:38 <sipa> and then the final 3072-bit output is again SHA512 hashed to produce a 256-bit output
185 13:39 <fjahr> hi jonatack
186 13:39 <jonatack> great PR, but separated from 18000 it's a bit neither here nor there ;)
187 13:39 <jonatack> i'll explain:
188 13:39 <jnewbery> the final 3072-bit output could be hashed using SHA256, but I suppose you use 512 for consistency?
189 13:39 <sipa> jnewbery: yeah
190 13:39 <sipa> that one doesn't matter
191 13:40 <raj_149> sipa: ok got it. i thought we were talking about the final one here.
192 13:40 <jonatack> fjahr: it is good that you added the legacy_hash bool to the RPC, but i would suggest making MuHash opt-in rather than the default until the index is merged
193 13:40 <fjahr> well, I hope most know libsecp does eliptic curve crypto and there is not EC used in muhash but ECMH does use it :)
194 13:40 <jnewbery> currently SHA512 is only used for encrypting the wallet and seeding the PRNG as far as I can tell
195 13:40 <sipa> fjahr: if you're going to cache the hash for every block... that's actually an argument in favor of ECMH, as the minimal "state" to keep for ECMH is 33 bytes only, while for MuHash it's 384 bytes
196 13:41 <jonatack> because otherwise gettxoutsetinfo is essentially broken for me on testnet and mainnet until then
198 13:41 <fjahr> sipa; I am only writing the finalized hash into the index otherwise there is one muhash that is update with every block
199 13:42 <willcl_ark> thanks uproar, I will take a look at that later
200 13:42 <sipa> fjahr: ah, gotcha
201 13:42 <sipa> that makes sense
202 13:42 <jnewbery> jonatack: by broken do you mean 'i have to pass a flag'?
203 13:42 <fjahr> jonatack: you mean it's too slow, right?
204 13:42 <thomasb06> jonatack: as expected, the archwiki admins are not interested to make a new page on how to compile and test the Bitcoin core. Your page will remain the reference for Linux builds for another while...
205 13:43 <jonatack> jnewbery: yes, without the flag it times out and raises... i haven't found the counterflag for that yet :)
206 13:43 <jnewbery> thomasb06: that's off-topic. Perhaps you can message jonatack after the meeting
207 13:43 <jonatack> fjahr: right
208 13:43 <willcl_ark> jonatack: I think I'd agree that defaulting to the faster impl. makes sense, with a flag for the new.
209 13:43 <fjahr> yeah, I am still a bit undecided what the right way to go is concerning the old way, do we want it temporary or permanently
210 13:44 <jnewbery> jonatack: I agree that the default should be switched to maintain existing behaviour
211 13:44 <jonatack> i'm building the index now... ~100k blocks/hour to build so far
212 13:44 <jonatack> (with #18000)
213 13:45 <jonatack> jnewbery: yes
214 13:45 <willcl_ark> I didn't try the PR on my server, but current performance is 60 seconds exactly for gettxoutset so at ~5x slower that will be an issue (that a flag can solve, but still...)
215 13:45 <nehan> fjahr: at the very least don't enable it by default until you have the index, since this change just makes gettxoutsetinfo strictly worse
216 13:45 <jonatack> nehan: ^
217 13:46 <fjahr> nehan: for the people who don't run the index it will always be worse, so I guess it would need to stay the default then :)
218 13:46 <uproar> nehan is your comment about optimal migration or something else I didn't understand?
219 13:47 <fjahr> noted that default should be the old one
220 13:47 <nehan> uproar: i think migration? not sure what you mean. but yeah, whether to turn something on by default and in what order to do so
221 13:47 <uproar> e.g. "keep using the old method until the new method has caught up" or something else
222 13:47 <nehan> fjahr: hmm that's tricky. and if it's off by default you've got two hashes for the utxoset in the world.
223 13:47 <jnewbery> fjahr: I don't agree. Once there's an optional index, then I think it's possible we might want to remove the SHA256 version
225 13:48 <sipa> maybe not immediately, but certainly with a deprecation window that should be ok
227 13:48 <nehan> jnewbery: sipa: how did you reach that conclusion? cause the rpc is not used very much?
228 13:48 <jnewbery> you have two options: run an index and have fast access to the utxo hash, or don't and be a little bit patient
229 13:48 <fjahr> hm, ok, I will have to think about it and test more on slower machines
230 13:49 <sipa> nehan: i suspect that once the index code exist, everyone who more than very exceptionally uses gettxoutsetinfo will enable the index (as it's pretty much free)
231 13:49 <willcl_ark> is the RPC timeout 2 minutes?
232 13:49 <jnewbery> if you're someone who wants to query the utxo set hash frequently, then build the index. If you're not, a one-off hit on the rare ocassion you do run it is acceptable
233 13:49 <fjahr> 15 is the default i think
234 13:49 <willcl_ark> oh, right
235 13:50 <nehan> jonatack: ah. i see i just restated what you said earlier with the comment on the default flag
236 13:51 <fjahr> Ok, I have another question on hashing algos but I would skip that unless someone has thoughts on it.
237 13:51 <thomasb06> (do you have the doc page for RPC under the arm by the way?)
238 13:51 <fjahr> For those that tested the code already: What are your thoughts on the Muhash benchmarks, e.g. (a) the benchmarking code, and (b) your observations if you tested the code locally?
239 13:51 <jonatack> nehan: i was seconding your comment :)
240 13:52 <nehan> fjahr: depends onif anyone is relying on it. how do people usually use this rpc?
241 13:52 <sipa> tbh, i mostly use it for statistics like number of UTXOs and circulating supply
243 13:53 <sipa> but once there is a fast way to compute a utxo set hash (or access it), i think that aspect of it will become much more useful
244 13:53 <nehan> if no one is relying on the old hash format it seems quite reasonable to get rid of it!
245 13:54 <sipa> we generally have a policy of not breaking RPC compatibility without deprecation cycle
246 13:54 <jnewbery> fjahr: my high-level thought about the PR is that it's good that you've split this from #18000, but there's still a lot of code there to review. It's a mix of python cryptography, C++ crypto, ASM, RPC code,... If you could split off smaller parts to review, it'd be easier to make progress
247 13:54 <jonatack> same, and #18000 is very welcome to me because it's that RPC is nigh unuseable at the moment
248 13:54 <fjahr> The two use cases I met mostly were: a) "for fun" to check if the node was running and to se the stats and b) checking circulating supply regularly (Bitmex publishes the numbers for example)
249 13:54 <jnewbery> (that assumes that there is consensus ACK for the overall concept and approach)
250 13:54 <fjahr> jnewbery: yeah, it has grown the last two days, agree
251 13:55 <sipa> fjahr: one thing that could be done if you have the index is make the startup consistency check roll back the utxo hash a few blocks, and compare it with the index
252 13:55 <sipa> which would prove correctness of the block and undo data on disk
253 13:56 <jonatack> jnewbery: fjahr: i agree, it's a long and hard review because it's so wide ... multi-day if you get into the crypto algo implementation and the assembly optimising
254 13:56 <fjahr> sipa: interesting, will conisder adding it when adding the index
256 13:57 <fjahr> ok, three minutes to go, it went way too fast as always. Any last comments?
257 13:57 <fjahr> or questions?
258 13:58 <fjahr> I am always open for DMs if you have questions on this btw
259 13:58 <sipa> fjahr: thanks for taking this up, and helping push it forward :)
260 13:58 <willcl_ark> yes thanks fjahr, it's a nice PR IMO, once it's fully-formed
261 13:58 <uproar> much thanks fjahr !
262 13:59 <jnewbery> If there's a split PR on the python implementation, it'd be great to have a review club on that where we can really get into the weeds on the crypto :)
263 13:59 <nehan> thanks fjahr!
264 13:59 <raj_149> thanks fjahr for hosting the review sesion too
265 13:59 <troygiorshev> thanks fjahr
266 13:59 <jonatack> thanks fjahr and sipa!
267 13:59 <raj_149> jnewbery: that sounds very fun.
268 13:59 <uproar> jnewbery +!
270 14:00 <fjahr> Thanks everyone for attending, especially sipa :)
271 14:00 <jnewbery> thanks fjahr!
272 14:00 <fjahr> #endmeeting
273 14:00 <thomasb06> thanks fjahr
274 14:00 <theStack> thanks for hosting fjahr, that was an interesting read today