Why evaluation suits renting
Benchmarks, regression suites and A/B comparisons run in bursts: start a job, wait for it to finish, read the numbers. Paying by the minute means a two-hour run costs two hours, with no idle machine to justify.
A clean setup
- Launch a fresh instance, so there is nothing left over from earlier runs.
- Install exact versions of your serving software and evaluation harness, and write the versions down.
- Fix random seeds, sampling settings and prompts.
- Run every model variant on the same instance in one session, so hardware conditions match.
- Copy the raw outputs and logs out, then delete the instance.
What to record
- Model name, precision and the software and its version.
- Batch size, context length and sampling settings.
- Both prefill speed and decode speed, not just one of them.
- The quality metric for your task alongside the speed numbers.
Published speed figures for this hardware vary with framework and version, so compare your own runs against each other rather than against numbers from elsewhere.
When one machine is not enough
Models too large for 128GB can use the dual-node size, which pools 256GB across two machines.
Frequently asked questions
Are results on a rented machine comparable between runs?
Each rental is a dedicated machine with nobody else on it, which helps consistency. Still record software versions and settings, since those change results more than the machine does.