make eval script also handle performance measurement

[vkuzo]

Summary:

1. refactors the eval script to also handle performance measurement in
   vllm
2. adds a simple vllm bench latency script to bench in vllm for prefill and decode

Also, add convenience flags to skip model creation, lm_eval, vllm as
needed to enable running just a single model + single step.

Test Plan:

with-proxy ./benchmarks/quantization/measure_accuracy_and_performance.sh h100
// full output: https://www.internalfb.com/phabricator/paste/view/P2094641791

Results (on H100):

Library Versions:
================================================================================
torch.__version__: 2.9.0+cu128
torch.cuda.get_device_name(): NVIDIA H100
torchao.__version__: 0.14.0+git5c8a14207
vllm.__version__: 0.13.0

Quantization Recipe Results:
================================================================================
+--------------------------+--------------+--------------+--------------+--------------+-----------+----------+-----------+-----------+
| Recipe                   |   Checkpoint |     Wikitext |   Winogrande |   Winogrande |   Prefill |   Decode |   Speedup |   Speedup |
|                          |         (GB) |   Perplexity |          Acc |       Stderr |    toks/s |   toks/s |   Prefill |    Decode |
+==========================+==============+==============+==============+==============+===========+==========+===========+===========+
| None                     |        16.08 |       7.5435 |       0.7419 |       0.0123 |   30946.5 |  6612    |     1     |     1     |
+--------------------------+--------------+--------------+--------------+--------------+-----------+----------+-----------+-----------+
| float8_rowwise           |         9.1  |       7.5919 |       0.7348 |       0.0124 |   45312.5 |  8025.95 |     1.464 |     1.214 |
+--------------------------+--------------+--------------+--------------+--------------+-----------+----------+-----------+-----------+
| int8_rowwise_weight_only |         9.11 |       7.5561 |       0.7427 |       0.0123 |   28231.9 |  4309.8  |     0.912 |     0.652 |
+--------------------------+--------------+--------------+--------------+--------------+-----------+----------+-----------+-----------+
| int8_rowwise             |         9.1  |       7.6567 |       0.738  |       0.0124 |           |          |           |           |
+--------------------------+--------------+--------------+--------------+--------------+-----------+----------+-----------+-----------+

Reviewers:

Subscribers:

Tasks:

Tags:

[vkuzo]

Stack from ghstack (oldest at bottom):

• -> make eval script also handle performance measurement #3473

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3473

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 6896527 with merge base 486fe0d ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Run Regression Tests / test-nightly (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://downloa... / linux-job (gh) (trunk failure)
  test/test_low_bit_optim.py::TestFSDP2::test_fsdp2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[jainapurva]

LGTM, thanks!