Merge branch 'master' into meiji163/bit-key-bug

Author: meiji163

.github/workflows/ci.yml

@@ -11,7 +11,7 @@ jobs:
     - uses: actions/checkout@v4
 
     - name: Set up Go
-      uses: actions/setup-go@v5
+      uses: actions/setup-go@v6
       with:
         go-version-file: go.mod
 

.github/workflows/codeql.yml

@@ -17,6 +17,8 @@ jobs:
 
     strategy:
       fail-fast: false
+      matrix:
+        language: [ 'actions', 'go' ]
 
     runs-on: ubuntu-latest # windows-latest and ubuntu-latest are supported. macos-latest is not supported at this time. 
 
@@ -26,6 +28,8 @@ jobs:
 
     - name: Initialize CodeQL 
       uses: github/codeql-action/init@v3
+      with:
+          languages: ${{ matrix.language }}
 
     - name: Perform CodeQL Analysis
       uses: github/codeql-action/analyze@v3

.github/workflows/golangci-lint.yml

@@ -14,10 +14,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-go@v5
+      - uses: actions/setup-go@v6
         with:
           go-version-file: go.mod
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v6
         with:
-          version: v1.54.2
+          version: v1.61.0

.github/workflows/replica-tests.yml

@@ -3,22 +3,29 @@ name: migration tests
 on: [pull_request]
 
 jobs:
-  build:
-
-    runs-on: ubuntu-20.04
+  docker-tests:
+    runs-on: ubuntu-22.04
     strategy:
+      fail-fast: false
       matrix:
-        version: [mysql-5.7.25,mysql-8.0.16,PerconaServer-8.0.21]
+        image: ['mysql/mysql-server:5.7.41','mysql:8.0.41','mysql:8.4.3','percona/percona-server:8.0.41-32']
+    env:
+      TEST_MYSQL_IMAGE: ${{ matrix.image }}
 
     steps:
-    - uses: actions/checkout@v4
+      - uses: actions/checkout@v4
+
+      - name: Install sysbench
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y sysbench
+
+      - name: Setup environment
+        run: script/docker-gh-ost-replica-tests up
 
-    - name: Set up Go
-      uses: actions/setup-go@v5
-      with:
-        go-version-file: go.mod
+      - name: Run tests
+        run: script/docker-gh-ost-replica-tests run
 
-    - name: migration tests
-      env:
-        TEST_MYSQL_VERSION: ${{ matrix.version }}
-      run:  script/cibuild-gh-ost-replica-tests
+      - name: Teardown environment
+        if: always()
+        run: script/docker-gh-ost-replica-tests down

.golangci.yml

@@ -11,9 +11,7 @@ linters:
     - durationcheck
     - errname
     - errorlint
-    - execinquery
     - gofmt
-    - ifshort
     - misspell
     - nilerr
     - nilnil

Dockerfile.packaging

@@ -1,4 +1,4 @@
-FROM golang:1.22-bullseye
+FROM golang:1.23-bullseye
 
 RUN apt-get update
 RUN apt-get install -y ruby ruby-dev rubygems build-essential

Dockerfile.test

@@ -1,4 +1,4 @@
-FROM golang:1.22-bullseye
+FROM golang:1.23-bullseye
 LABEL maintainer="[email protected]"
 
 RUN apt-get update

doc/command-line-flags.md

@@ -64,6 +64,15 @@ It is not reliable to parse the `ALTER` statement to determine if it is instant
 ### binlogsyncer-max-reconnect-attempts
 `--binlogsyncer-max-reconnect-attempts=0`, the maximum number of attempts to re-establish a broken inspector connection for sync binlog. `0` or `negative number` means infinite retry, default `0`
 
+### checkpoint
+
+`--checkpoint` enables periodic checkpoints of the gh-ost's state so that gh-ost can resume a migration from the checkpoint with `--resume`. Checkpoints are written to a separate table named `_${original_table_name}_ghk`. It is recommended to use with `--gtid` for checkpoints.
+See also: [`resuming-migrations`](resume.md)
+
+### checkpoint-seconds
+
+`--checkpoint-seconds` specifies the seconds between checkpoints. Default is 300.
+
 ### conf
 
 `--conf=/path/to/my.cnf`: file where credentials are specified. Should be in (or contain) the following format:
@@ -119,7 +128,7 @@ See also: [`skip-foreign-key-checks`](#skip-foreign-key-checks)
 
 `gh-ost` reads event from the binary log and applies them onto the _ghost_ table. It does so in batched writes: grouping multiple events to apply in a single transaction. This gives better write throughput as we don't need to sync the transaction log to disk for each event.
 
-The `--dml-batch-size` flag controls the size of the batched write. Allowed values are `1 - 100`, where `1` means no batching (every event from the binary log is applied onto the _ghost_ table on its own transaction). Default value is `10`.
+The `--dml-batch-size` flag controls the size of the batched write. Allowed values are `1 - 1000`, where `1` means no batching (every event from the binary log is applied onto the _ghost_ table on its own transaction). Default value is `10`.
 
 Why is this behavior configurable? Different workloads have different characteristics. Some workloads have very large writes, such that aggregating even `50` writes into a transaction makes for a significant transaction size. On other workloads write rate is high such that one just can't allow for a hundred more syncs to disk per second. The default value of `10` is a modest compromise that should probably work very well for most workloads. Your mileage may vary.
 
@@ -160,6 +169,10 @@ Table name prefix to be used on the temporary tables.
 
 Add this flag when executing on a 1st generation Google Cloud Platform (GCP).
 
+### gtid
+
+Add this flag to enable support for [MySQL replication GTIDs](https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-concepts.html) for replication positioning. This requires `gtid_mode` and `enforce_gtid_consistency` to be set to `ON`.
+
 ### heartbeat-interval-millis
 
 Default 100. See [`subsecond-lag`](subsecond-lag.md) for details.
@@ -202,6 +215,12 @@ List of metrics and threshold values; topping the threshold of any will cause th
 
 Typically `gh-ost` is used to migrate tables on a master. If you wish to only perform the migration in full on a replica, connect `gh-ost` to said replica and pass `--migrate-on-replica`. `gh-ost` will briefly connect to the master but otherwise will make no changes on the master. Migration will be fully executed on the replica, while making sure to maintain a small replication lag.
 
+### panic-on-warnings
+
+When this flag is set, `gh-ost` will panic when SQL warnings indicating data loss are encountered when copying data. This flag helps prevent data loss scenarios with migrations touching unique keys, column collation and types, as well as `NOT NULL` constraints, where `MySQL` will silently drop inserted rows that no longer satisfy the updated constraint (also dependent on the configured `sql_mode`).
+
+While `panic-on-warnings` is currently disabled by defaults, it will default to `true` in a future version of `gh-ost`.
+
 ### postpone-cut-over-flag-file
 
 Indicate a file name, such that the final [cut-over](cut-over.md) step does not take place as long as the file exists.
@@ -216,6 +235,11 @@ Optionally involve the process ID, for example: `--replica-server-id=$((10000000
 It's on you to choose a number that does not collide with another `gh-ost` or another running replica.
 See also: [`concurrent-migrations`](cheatsheet.md#concurrent-migrations) on the cheatsheet.
 
+### resume
+
+`--resume` attempts to resume a migration that was previously interrupted from the last checkpoint. The first `gh-ost` invocation must run with `--checkpoint` and have successfully written a checkpoint in order for `--resume` to work.
+See also: [`resuming-migrations`](resume.md)
+
 ### serve-socket-file
 
 Defaults to an auto-determined and advertised upon startup file. Defines Unix socket file to serve on.

doc/interactive-commands.md

@@ -41,6 +41,7 @@ Both interfaces may serve at the same time. Both respond to simple text command,
 - `throttle-control-replicas='replica1,replica2'`: change list of throttle-control replicas, these are replicas `gh-ost` will check. This takes a comma separated list of replica's to check and replaces the previous list.
 - `throttle`: force migration suspend
 - `no-throttle`: cancel forced suspension (though other throttling reasons may still apply)
+- `postpone-cut-over-flag-file=<path>`: Postpone the [cut-over](cut-over.md) phase, writing a cut over flag file to the given path
 - `unpostpone`: at a time where `gh-ost` is postponing the [cut-over](cut-over.md) phase, instruct `gh-ost` to stop postponing and proceed immediately to cut-over.
 - `panic`: immediately panic and abort operation
 
@@ -60,7 +61,7 @@ $ echo status | nc -U /tmp/gh-ost.test.sample_data_0.sock
 # Throttle additional flag file: /tmp/gh-ost.throttle
 # Serving on unix socket: /tmp/gh-ost.test.sample_data_0.sock
 # Serving on TCP port: 10001
-Copy: 0/2915 0.0%; Applied: 0; Backlog: 0/100; Elapsed: 40s(copy), 41s(total); streamer: mysql-bin.000550:49942; ETA: throttled, flag-file
+Copy: 0/2915 0.0%; Applied: 0; Backlog: 0/100; Time: 41s(total), 40s(copy); streamer: mysql-bin.000550:49942; Lag: 0.01s, HeartbeatLag: 0.01s, State: throttled, flag-file; ETA: N/A
 ```
 
 ```shell
@@ -88,5 +89,5 @@ $ echo status | nc -U /tmp/gh-ost.test.sample_data_0.sock
 # Throttle additional flag file: /tmp/gh-ost.throttle
 # Serving on unix socket: /tmp/gh-ost.test.sample_data_0.sock
 # Serving on TCP port: 10001
-Copy: 0/2915 0.0%; Applied: 0; Backlog: 0/100; Elapsed: 59s(copy), 59s(total); streamer: mysql-bin.000551:68067; ETA: throttled, commanded by user
+Copy: 0/2915 0.0%; Applied: 0; Backlog: 0/100; Time: 59s(total), 59s(copy); streamer: mysql-bin.000551:68067; Lag: 0.01s, HeartbeatLag: 0.01s, State: throttled, commanded by user; ETA: N/A
 ```

doc/local-tests.md

@@ -2,7 +2,7 @@
 
 `gh-ost` is continuously tested in production via `--test-on-replica alter='engine=innodb'`. These tests check the GitHub workload and usage, but not necessarily the general case.
 
-Local tests are an additional layer of tests. They will eventually be part of continuous integration tests.
+Local tests are an additional layer of tests used for continuous integration tests and local development.
 
 Local tests test explicit use cases, such as column renames, mix of time zones, special types and alters. Traits of a single test:
 
@@ -19,4 +19,26 @@ Tests are found under [localtests](https://github.com/github/gh-ost/tree/master/
 
 New data-integrity, synchronization issues or otherwise concerns are expected to be tested by new test cases.
 
-While this is merged work is still ongoing.
+## Run with docker compose
+
+Local tests can be run locally with docker compose using the helper script [script/docker-gh-ost-replica-tests](https://github.com/github/gh-ost/tree/master/script/docker-gh-ost-replica-tests). 
+
+Example usage:
+```shell
+# create primary-replica containers with specified mysql image
+TEST_MYSQL_IMAGE="mysql-server:8.0.16" ./script/docker-gh-ost-replica-tests up
+
+# run all tests
+./script/docker-gh-ost-replica-tests run
+
+# cleanup containers
+./script/docker-gh-ost-replica-tests down
+```
+
+Pass the `-t` flag to run the tests with a toxiproxy between gh-ost and the MySQL replica. This simulates network conditions where MySQL connections are closed unexpectedly.
+
+```shell
+# run tests with toxiproxy
+./script/docker-gh-ost-replica-tests up -t
+./script/docker-gh-ost-replica-tests run -t
+```