[Ignore] Workflow experiments

[achamayou]

Target setup

Subject to change as tests are run.

• gha-vmss-d16av5-ci upgraded to d16ads to use much faster local storage [DONE]
• License checks collapsed in Misc Checks job to shorten time-to-start [DONE, Merge CI checks into VMSS A job #7429]
• Shrink C-ACI tests on Genoa and Milan (SNP-relevant tests) [DONE, Focus SNP CI and make it identical across archs #7426]
• Big SKU Genoa pool [DONE, Align Genoa pool size on Milan #7424]
• Packing pass over the end to end tests [WIP]

Key Findings:

• d8as (8 cores) is roughly twice as slow at building as d16as (16 cores)
• d8as runs tests in about the same time as d16as
• d8ads (local temp disk) runs tests 20% faster than d16as (no local temp disk)
• when local temp disk is enabled, symlinks are not necessary, the agent will use /mnt
• FIO benchmarks show temp local disk is has at least an order of magnitude more throughput than the OS disk
• N-core GitHub Codespaces are really N/2 cores with SMT

FIO comparison

fio --name=cpp-build --ioengine=libaio --direct=0 --rw=randrw --rwmixread=60 --bs=4k-256k --numjobs=18 --size=100M --nrfiles=50 --runtime=60 --time_based --group_reporting --directory=...

Temp disk (d16ads)

Run status group 0 (all jobs):
   READ: bw=41.4MiB/s (43.4MB/s), 41.4MiB/s-41.4MiB/s (43.4MB/s-43.4MB/s), io=2486MiB (2607MB), run=60002-60002msec
  WRITE: bw=1564MiB/s (1640MB/s), 1564MiB/s-1564MiB/s (1640MB/s-1640MB/s), io=91.6GiB (98.4GB), run=60002-60002msec

Disk stats (read/write):
  sdb: ios=107270/131208, sectors=858160/190591216, merge=0/12140, ticks=986944/4243494, in_queue=5230438, util=84.57%

Default OS Disk (d16as)

Run status group 0 (all jobs):
   READ: bw=2349KiB/s (2405kB/s), 2349KiB/s-2349KiB/s (2405kB/s-2405kB/s), io=138MiB (145MB), run=60132-60132msec
  WRITE: bw=87.4MiB/s (91.7MB/s), 87.4MiB/s-87.4MiB/s (91.7MB/s-91.7MB/s), io=5257MiB (5512MB), run=60132-60132msec

Disk stats (read/write):
  sda: ios=11231/8145, sectors=89848/9082784, merge=0/958, ticks=1034452/6677418, in_queue=7730930, util=95.11%

Details:

Configure a d8adsv5 with local temp disk, which is exposed as /mnt.

This may help our I/O issues, at it offers high IOPS caps: 9000 - 450000 IOPS (RR)

Build with clang-tidy takes 17 minutes with this configuration, versus 9 with the d16as. Is this because of CPU? Memory? Disk?

Oddly, on a 16 core machine where a ninja without arguments completes in 9 minutes:

# time ninja -j8

real    11m10.751s
user    81m34.624s
sys     2m19.969s

Local build with clang-tidy seems to peak around 28Gb of memory usage, less than the 32Gb available to a d8adsv5, so it's probably not memory. Even so, and with the CI checks added, the job finishes in 30 minutes, faster than the main CI.

The debug build time also has doubled, from 3 to 6 minutes, which is surprising and different from local experiments with different -j values.

While the build is slower, the tests seem to run quite a bit faster (22 vs 28 minutes), and so Virtual CI is about 4 minutes quicker end to end that way than with the d16as.

-j8/10(default)/12 does not change the build time substantially on d8adsv5.

clang-tidy speedups

There is a caching wrapper called ctcache and it is published to pypi. Will update after results.

The default configuration for cltcache attempts to use c++ (gcc) to pre-process the files, rather than clang, which results in failure because we use -fcolor-diagnostic, which is only recognised by clang. With some tweaks, it's possible to get it to work, but there is no benefit on a single run, only on repeat runs.

container pre-building

We could publish base containers to GHCR, and pull them from there to avoid the dependency set up phase. This means maintaining an extra job with write tokens however.

ACI in virtual too

With 1ES pools, there is no option other than local disk, which is very slow on C-ACI. Temp disks on VMSS offer much better throughput and are a preferable option for the majority of our end to end tests.

[achamayou]

Sliced off into #7429, #7426, #7424, all merged. Test packing continues in #7448.