Snapshots not backing up

[dfredell]

I have started up a 2 mysql node system connected via 3 autopilotpattern/consuls running in a private Triton environment backed up to minio. I would have expected when I came in on Monday to see 2 snapshots stored in minio from the weekend. I logged into the docker of the primary-mysql and ran a few commands (below) to see if I could get the snapshot to run. Note the code running is that of https://github.com/certusoft/mysql

root@1a5d9a1a4ff6:/usr/local/bin# python manage.py snapshot_task
Traceback (most recent call last):
File "manage.py", line 437, in
main()
File "manage.py", line 434, in main
cmd(node)
File "/usr/local/bin/manager/utils.py", line 65, in wrapper
out = apply(fn, args, kwargs)
File "manage.py", line 249, in snapshot_task
if not node.is_snapshot_node() or not node.consul.lock_snapshot(node.name):
File "/usr/local/bin/manager/utils.py", line 65, in wrapper
out = apply(fn, args, kwargs)
File "/usr/local/bin/manager/discovery.py", line 239, in lock_snapshot
return self.lock(BACKUP_LOCK_KEY, hostname, session_id)
File "/usr/local/bin/manager/utils.py", line 65, in wrapper
out = apply(fn, args, kwargs)
File "/usr/local/bin/manager/discovery.py", line 94, in lock
return self.client.kv.put(key, value, acquire=session_id)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 534, in put
CB.json(), '/v1/kv/%s' % key, params=params, data=value)
File "/usr/local/lib/python2.7/dist-packages/consul/std.py", line 40, in put
self.session.put(uri, data=data, verify=self.verify)))
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 186, in cb
CB.__status(response, allow_404=allow_404)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 143, in __status
raise ConsulException("%d %s" % (response.code, response.body))
consul.base.ConsulException: 500 rpc error: invalid session "9e196856-5447-f942-a606-d9cf9445713f"
root@1a5d9a1a4ff6:/usr/local/bin# ls /tmp/
backup/ backup.tar mysql-backup-running mysql-session percona-version-check
root@1a5d9a1a4ff6:/usr/local/bin# rm /tmp/mysql-backup-running
root@1a5d9a1a4ff6:/usr/local/bin# python manage.py snapshot_task
171009 13:22:46 innobackupex: Starting the backup operation
....

[dfredell]

My theory is that discovery.lock_snapshot() needs pyconsul.ConsulException added to the except clause.

[dfredell]

So I read the code some more and did some testing. Autopilot mysql will only do a snap shot when there is another bin log created. So I put a bunch of inserts and drop in a loop to make the /var/lib/mysql/mysql-bin.000002 fill up (1.1G) then the snapshot took.

It would be nice if there was a flag to tell autopilot mysql to always do a backup every 24hr, that way if all mysql nodes go down we only loose a day of data. Or even better if it could do an incremental back up every hour.