CoinJoin - Fix derivation paths, transaction detection

[HashEngineering]

The CoinJoin derivation path is specified in 3 places in this project, but none match DIP9.

Additionally, the transaction classifier does not correctly identify all CoinJoin transactions.

[HashEngineering]

Another issue was found.

During the blockchain sync process, some coinjoin transactions are skipped.

This may be due to the gap limits of the coinjoin (and other) address pools not being updated soon enough.

I believe the process is as follows:

1. sync headers
2. sync compact filters (might be done in batches and in parallel)
3. sync filters/blocks -- here the gap limits are maintained for the address pools of the wallet

In step 2, some since the gap limits are not maintained, a transaction may not be found as relevant to the wallet.

If we increase the gap limit to 100 or 500, we will catch more transactions, but eventually we miss transactions in step 2.

In step 3, when blocks are processed, each transaction is processed and the gap limits are maintained.

with Bloom Filters, what is done is that 500 filtered blocks are downloaded at time and the gap limits are maintained after each transaction or block. If it is found that a number of keys needs to be added to maintain the gap limit, then bloom filters are resubmitted and blocks are requested from that point. Any other previously downloaded filtered blocks are discarded as they may not have relevant transactions based on the new keys. It could be that something similar happens with the compact filters system.

Maybe there are two paths:

1. maintain the gap limit while processing the Compact Filters (2).
2. maintain the gap limit while processing the compact blocks (3). If we need to increase the size of our address pools, then we request compact filters again from that point.

@PastaPastaPasta Does this explanation make sense? My understanding of the system may not be correct or it may be out of date.

[PastaPastaPasta]

I do think there is an issue here actually.

When we process filters, I think we do so in parallel, and then we request whatever blocks we get hits for and find the matches. This creates an issue, where if we download filters 1-10, and process them async, if we hit in 1, our gap will change, but this isn't reflected when we checked match for 2-10.

I guess what we would want to do, to still retain the performance is to try to process filters in parallel, but as soon as we get a match, stall until we fully process the block, OR since there are false positives, continue in parallel and then re-wind if the match turns out to be a true positive. and recheck the filters based on our new set of addresses / scripts.

[PastaPastaPasta]

#204 may resolve this

[HashEngineering]

@PastaPastaPasta I agree that your approach might work. I am going to look at how another open source software (neutrino) works. Neutrino uses BIP157.

And I will see the #204