database: Add vitess + mysql 8.0 to our development environment

[beautifulentropy]

The original plan for getting the Vitess infrastructure running was to use vttestserver as a starting point to reach a minimum viable setup. However, vttestserver didn’t work out because some of its defaults conflicted with how I clean up rows and the level of resources (threads) I need.

Fortunately, vttestserver is just a wrapper around vtcombo that generates a vttest protobuf describing the configuration for an in-memory topology server started by vtcombo, encoded in JSON. By modifying vttestserver’s run.sh, I was able to interact with vtcombo directly, passing the JSON configuration along with other vttestserver defaults reverse-engineered from run.sh and vtprocess.go.

I built and uploaded a boulder-vtcomboserver image on top of Oracle’s MySQL 8.0 image, which provides native arm64 support. The accompanying tag-and-upload shell script defaults to amd64 for CI. Since Vitess doesn’t provide a vtcomboserver image, I would have needed to build my own regardless.

As an aside, Vitess’s official Dockerfiles are only published for amd64, and modifying them to build for arm64 proved difficult because Oracle doesn’t publish MySQL arm64 binaries in its Debian apt repository.

With boulder-vtcomboserver up and running I was able to find/validate the following issues and provide workarounds:

• Problem: db-migrate, the tool used to apply database migrations, must be configured to talk to MariaDB through ProxySQL and to MySQL through Vitess (vtgate + vttablet).
  Solution: Use test.sh to symlink the appropriate dbconfig.yml file depending on whether MariaDB or MySQL is in use.

• Problem: Vitess does not allow database CREATE statements and any DDL containing them will be rejected by vtgate.
  Solution: These databases are already created by vtcombo since they’re defined as KEYSPACES. Skip database creation in test/create_db.sh.

• Problem: Vitess does not allow user creation or grants (CREATE USER, GRANT), and any DDL containing these commands will be blocked by vtgate.
  Solution: Skip user creation and grant steps in test/create_db.sh. Set % for --vschema_ddl_authorized_users as vttestserver does, and revisit this later for a more complete approach.

• Problem: Vitess does not allow table partition creation and any DDL containing partition commands will be blocked by vtgate.
  Solution: Remove partitions from all schemas under sa/db*.

• Problem: Two database system variables — max_statement_time and long_query_time — are injected by augmenting the DSN passed to Boulder. Both are derived from the SA JSON’s configured ReadTimeout. max_statement_time was last supported in MySQL 5.7 and replaced by max_execution_time, which uses milliseconds instead of seconds.
  Solution: Stop injecting these variables in Go at runtime, instead pass both via different DSNs, symlinked by test.sh depending on whether MariaDB or MySQL is in use.

• Problem: vttablet default for maximum number of rows returned from a (non-streaming) query (10,000) is too low for Boulder’s needs, causing queries to fail due to vttablet rejecting them.
  Solution: Increase --queryserver-config-max-result-size to 1,000,000 and --queryserver-config-warn-result-size to 1,000,000.

• Problem: vttablet default for connection pool size (16) and maximum number of concurrent transactions (20) are too low for Boulder’s needs, causing queries to fail due to vttablet being overloaded.
  Solution: Increase --queryserver-config-pool-size to 64 and --queryserver-config-transaction-cap to 80.

• Problem: Vitess does not allow TRIGGER statements and any DDL containing them will be rejected by vtgate. Without TRIGGER statements TestIssuanceCertStorageFailed, an integration test, will fail.
  Soluton: Run these TRIGGER statements in an entrypoint scripttest/vtcomboserver/install_trigger.sh, bypassing vtgate entirely.

Fixes #7736