huge memory leak right after reconnect

[gzivdo]

The code just like in example
only with subscribe_to_exchange_deltas

Output

2018-05-06 18:55:35 - bittrex_websocket.websocket_client - ERROR - Connection is already closed..
2018-05-06 18:55:35 - bittrex_websocket.websocket_client - ERROR - Initiating reconnection procedure
2018-05-06 18:55:35 - bittrex_websocket.websocket_client - INFO - Establishing connection to Bittrex through https://beta.bittrex.com/signalr.
2018-05-06 18:55:39 - bittrex_websocket.websocket_client - INFO - Successfully subscribed to [QueryExchangeState] for [BTC-ZCL].
2018-05-06 18:55:39 - bittrex_websocket.websocket_client - INFO - Successfully subscribed to [SubscribeToExchangeDeltas] for [BTC-ZCL].
2018-05-06 18:55:39 - bittrex_websocket.websocket_client - INFO - Successfully subscribed to [QueryExchangeState] for [BTC-RDD].
2018-05-06 18:55:39 - bittrex_websocket.websocket_client - INFO - Successfully subscribed to [SubscribeToExchangeDeltas] for [BTC-RDD].
2018-05-06 18:55:39 - bittrex_websocket.websocket_client - INFO - Successfully subscribed to [QueryExchangeState] for [BTC-TRX].
2018-05-06 18:55:39 - bittrex_websocket.websocket_client - INFO - Successfully subscribed to [SubscribeToExchangeDeltas] for [BTC-TRX].
2018-05-06 18:56:20.775091 15593
2018-05-06 18:57:20.835102 15659
2018-05-06 18:58:21.108694 15691
2018-05-06 18:59:30.381624 15736
2018-05-06 19:00:06 - bittrex_websocket.websocket_client - ERROR - Undocumented error message received with code -1 and payload: 32. Report to https://github.com/slazarov/python-bittrex-websocket
2018-05-06 19:00:06 - bittrex_websocket.websocket_client - ERROR - Initiating reconnection procedure
2018-05-06 19:00:06 - bittrex_websocket.websocket_client - INFO - Establishing connection to Bittrex through https://beta.bittrex.com/signalr.

After first reconnect app starts to eat all available memory and swap.

[slazarov]

Error 32 should refer to socket.error: [Errno 32] Broken pipe and is due to connection drop. If your connection is working fine, then I suspect that the error is coming from Bittrex's side since recently they have been pushing constant updates in order to get the main endpoint stable. Note that we are using the beta endpoint currently in order to avoid bypassing cloudflare. You can try:

ws = MySocket(url='https://socket.bittrex.com/signalr')

Just make sure you have installed cfscrape and node.js.

The reconnection memory leak is due to the fact that the client tries to reconnect and since it's denied immediately it goes into an infinite loop which causes high CPU usage. This is on my side and I will implement a fix. I am thinking of maximum connection retries solution.

[gzivdo]

cpu usage not high at all, but memory usage rising quite fast.

[slazarov]

Doesn't change the fact.

Also when posting make sure you specify what Python version you are using and also provide the code you are executing. In this case you said you are using the example but you are also invoking QueryExchangeState and I have no idea whether you are executing it in a loop.

[gzivdo]

Python 2.7.x

from bittrex_websocket.websocket_client import BittrexSocket
import cPickle, time, datetime

tickers = {
    'ZCL':  (35, 5),
    'RDD': (20000, 2000),
    'TRX': (5000, 2000),
}


class BittrexSocket1(BittrexSocket):
        def __init__(self, tickers, url=None):
            super(BittrexSocket1, self).__init__(url)
            self.tickers = tickers
            self.cnt = 0
            for ticker in self.tickers:
                time.sleep(0.01)
                self.last[ticker] = time.time()
                self.query_exchange_state(['BTC-'+ticker])
                self.subscribe_to_exchange_deltas(['BTC-'+ticker])


        def on_public(self, msg):
            ticker = msg['M'].split('-')[-1]
            self.cnt += 1
            return

def main():
    ws = BittrexSocket1(tickers)
    ws.enable_log()
    try:
        while True:
            sleep(60)
            print datetime.datetime.now(), ws.cnt
    except KeyboardInterrupt:
        pass
    print 'exiting'
    ws.disconnect()
    sleep(10)

if __name__ == "__main__":
    main()

[slazarov]

I will test it now.

1.) How often does the issue happens?

2.) Is there any pattern in the average time between starting the code and the occurrence of the error? E.g every 10mins or so.

[gzivdo]

No, it happens just after network error. Sometimes its 2h, sometimes 6-8h

[slazarov]

I see, I will work on a fix and will push it to github in due time.
Thank you for reporting the bug.

[hamkline]

I'm having issues with reconnection too. Mine lasts quite a bit, but at least once a day it breaks. I'm running on a server with 1GB RAM only though.
I don't run connections on a loop, it happens once and then I call a function to take care of the data + rest requests from the same library you use.
Also, I don't have the other errors, but I think I got a different one last time even though it was from reconnection also.
I can take a print next time if it helps. (python 3)

[slazarov]

Its a temporary fix but do the following: update to the latest version from github and replace line 140-143:

            elif invoke == BittrexMethods.GET_AUTH_CONTENT:
                self.connection.corehub.server.invoke(invoke, payload[0])
                self.invokes.append({'invoke': invoke, 'ticker': payload[0]})
                logger.info('Retrieving authentication challenge.')

with

            elif invoke == BittrexMethods.GET_AUTH_CONTENT:
                self.connection.corehub.server.invoke(invoke, self.credentials['api_key'])
                self.invokes.append({'invoke': invoke, 'ticker': self.credentials['api_key']})
                logger.info('Retrieving authentication challenge.')

[ayy1337]

I'm also having troubles on reconnection.. Seems like it blew through by VPS' available memory and was killed by linux oom-killer.

[ayy1337]

Also, I notice that you have in the websocket_client.py put a variable 'self.threads = []', but when a thread dies or is finished it is not removed from this list, preventing it from being garbage collected (as there are still references to it)
Honestly I can't even tell why the list is there at all, since it doesn't seem to be accessed anywhere..

[ayy1337]

Managed to catch it in htop. What are all these threads doing?

[slazarov]

Hi, self.threads = [] is a leftover from v<1.0, so thanks for noticing it. I am aware of what might be causing the thread spam and will fix in due time. As always thanks for raising an issue but next time provide additional information such as the whole log, code snippet, python version and a few word explanation since its difficult to debug from a picture.

Is node.js installed together with cfscrape?

[slazarov]

Update the package to v1.0.4 through:
pip install git+https://github.com/slazarov/python-bittrex-websocket.git@1.0.4

Theoretically this should fix it, waiting for your feedback after testing.