Skip to content

Conversation

@meiji163
Copy link
Contributor

@meiji163 meiji163 commented Sep 23, 2025

Description

This PR is a continuation of @timvaillancourt's PR #950 to add GTID support for streaming the binlog. Enable it with the --gtid flag. It requires gtid_mode=ON and enforce_gtid_consistency=ON, which is available in all MySQL versions supported by gh-ost.
Replacing file-based coordinates with GTID set coordinates is necessary for implementing migrations checkpoints and resumability in the future.

Details

  • Add mysql.BinlogCoordinates interface, implemented by FileBinogCoordinates and GTIDBinlogCoordinates. GTID set operations are provided by go-mysql's replication package.
  • In GTID mode, GoMySQLReader.currentCoordinates tracks the currently streaming transaction, while GoMySQLReader.LastTrxCoords tracks the last read transaction (similar to Retrieved_Gtid_Set).
  • Update EventsStreamer.StreamEvents reconnection logic to use the LastTrxCoords. Surprisingly, the retry logic is currently unused because the go-mysql replication.BinlogSyncer has its own retry mechanism. My testing showed no problems with using the new reconnection logic, but I will not enable it now to keep this PR low risk.

Testing

All localtests pass with gtid mode (./script/docker-gh-ost-replica-tests run -g). Additionally, internal replica tests on production tables have been running for a day and revealed no data integrity issues.

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

@meiji163 meiji163 mentioned this pull request Oct 8, 2025
2 tasks
@meiji163 meiji163 mentioned this pull request Oct 10, 2025
2 tasks
@meiji163 meiji163 marked this pull request as ready for review October 10, 2025 17:25
Copilot AI review requested due to automatic review settings October 10, 2025 17:25
@meiji163 meiji163 changed the title [WIP] Add GTID Add GTID support for binlog streaming Oct 10, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds GTID (Global Transaction Identifier) support to gh-ost for streaming the binary log, which is necessary for implementing migration checkpoints and resumability. The main changes introduce a new --gtid flag that requires gtid_mode=ON and enforce_gtid_consistency=ON.

  • Replaced file-based binlog coordinates with a BinlogCoordinates interface supporting both file and GTID coordinate types
  • Updated reconnection logic to use last complete transaction coordinates for better reliability
  • Added GTID validation and configuration checks

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
go/mysql/binlog.go Converted from struct to interface for coordinate abstraction
go/mysql/binlog_file.go New file implementing file-based binlog coordinates
go/mysql/binlog_gtid.go New file implementing GTID-based binlog coordinates
go/binlog/gomysql_reader.go Updated to handle both coordinate types and track complete transactions
go/logic/streamer.go Enhanced reconnection logic using last transaction coordinates
go/logic/inspect.go Added GTID configuration validation
go/cmd/gh-ost/main.go Added --gtid command line flag
localtests/test.sh Added GTID test support and configuration

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@meiji163 meiji163 merged commit 005043d into master Oct 10, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants