-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Add GTID support for binlog streaming #1584
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds GTID (Global Transaction Identifier) support to gh-ost for streaming the binary log, which is necessary for implementing migration checkpoints and resumability. The main changes introduce a new --gtid flag that requires gtid_mode=ON and enforce_gtid_consistency=ON.
- Replaced file-based binlog coordinates with a
BinlogCoordinatesinterface supporting both file and GTID coordinate types - Updated reconnection logic to use last complete transaction coordinates for better reliability
- Added GTID validation and configuration checks
Reviewed Changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| go/mysql/binlog.go | Converted from struct to interface for coordinate abstraction |
| go/mysql/binlog_file.go | New file implementing file-based binlog coordinates |
| go/mysql/binlog_gtid.go | New file implementing GTID-based binlog coordinates |
| go/binlog/gomysql_reader.go | Updated to handle both coordinate types and track complete transactions |
| go/logic/streamer.go | Enhanced reconnection logic using last transaction coordinates |
| go/logic/inspect.go | Added GTID configuration validation |
| go/cmd/gh-ost/main.go | Added --gtid command line flag |
| localtests/test.sh | Added GTID test support and configuration |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Co-authored-by: Copilot <[email protected]>
Description
This PR is a continuation of @timvaillancourt's PR #950 to add GTID support for streaming the binlog. Enable it with the
--gtidflag. It requiresgtid_mode=ONandenforce_gtid_consistency=ON, which is available in all MySQL versions supported by gh-ost.Replacing file-based coordinates with GTID set coordinates is necessary for implementing migrations checkpoints and resumability in the future.
Details
mysql.BinlogCoordinatesinterface, implemented byFileBinogCoordinatesandGTIDBinlogCoordinates. GTID set operations are provided by go-mysql'sreplicationpackage.GoMySQLReader.currentCoordinatestracks the currently streaming transaction, whileGoMySQLReader.LastTrxCoordstracks the last read transaction (similar toRetrieved_Gtid_Set).EventsStreamer.StreamEventsreconnection logic to use theLastTrxCoords. Surprisingly, the retry logic is currently unused because the go-mysqlreplication.BinlogSyncerhas its own retry mechanism. My testing showed no problems with using the new reconnection logic, but I will not enable it now to keep this PR low risk.Testing
All localtests pass with gtid mode (
./script/docker-gh-ost-replica-tests run -g). Additionally, internal replica tests on production tables have been running for a day and revealed no data integrity issues.script/cibuildreturns with no formatting errors, build errors or unit test errors.