Gemma4 Benchmarking

Use scripts/bench-gemma4-suite.sh when you want a repeatable Gemma-focused benchmark run without changing active manifests, runtime code, or controller behavior.

The suite wraps scripts/bench-model.sh and adds four things that were awkward to do by hand:

• fast vs long profile comparison
• warm vs cold run comparison
• prompt-length matrix runs across short, medium, and long
• one suite-level JSON artifact that links every child benchmark report

What it runs

By default the suite targets:

• model: gemma4-e4b-turboquant
• profiles: fast,long
• states: warm,cold
• phases: short,medium,long

Profile defaults:

Profile	Iterations	Warmup	Max tokens	Intent
fast	2	1	128	quick regression checks during iteration
long	4	1	512	slower but more stable comparison runs

State behavior:

• warm uses the profile warmup count.
• cold forces WARMUP=0.
• For a true cold run, pass a COLD_HOOK that unloads or restarts the target model between cases.

Common commands

Quick warm-only matrix:

./scripts/bench-gemma4-suite.sh --profiles fast --states warm

Default full comparison:

./scripts/bench-gemma4-suite.sh

Include multiturn and stream phases:

./scripts/bench-gemma4-suite.sh --extra-phases multiturn,stream

Run against a directly port-forwarded model pod:

DIRECT=1 \
./scripts/bench-gemma4-suite.sh \
  --endpoint http://127.0.0.1:8000 \
  --profiles fast \
  --states warm

Approximate true cold runs by injecting your own reset hook:

COLD_HOOK='kubectl -n flexinfer-system rollout restart deploy/flexinfer-runtime-gfx1100' \
./scripts/bench-gemma4-suite.sh --states cold

That hook is intentionally external to the script. The suite owns orchestration and artifact capture, not cluster mutation policy.

Output layout

Every suite creates a dedicated artifact directory under REPORT_DIR:

/tmp/<model>-gemma4-suite-<timestamp>-<rand>/
  suite.json
  cases.jsonl
  fast-warm-short/
    stdout.log
    bench-model-<model>-<run-id>.json
  ...

The top-level suite.json contains:

• suite metadata (suite_run_id, git_sha, model, endpoint)
• requested profiles, states, and phases
• per-case config and artifact paths
• embedded child benchmark reports
• comparison sections for:
  • warm_vs_cold
  • fast_vs_long

This makes it easy to keep one machine-readable artifact for later analysis, while still preserving the raw child reports generated by bench-model.sh.

Tuning

Override profile defaults with environment variables:

FAST_ITERATIONS=1 \
LONG_ITERATIONS=5 \
FAST_MAX_TOKENS=64 \
LONG_MAX_TOKENS=768 \
./scripts/bench-gemma4-suite.sh

Change the prompt matrix entirely:

./scripts/bench-gemma4-suite.sh --phases short,long

Dry-run the plan before spending GPU time:

./scripts/bench-gemma4-suite.sh --dry-run

Notes

• The suite does not try to infer what "cold" means for your deployment. If you care about cold-start behavior, use COLD_HOOK.
• The suite is intentionally additive. It does not replace existing benchmark scripts or runtime probes.
• If a child benchmark fails, the suite continues and records the failure in suite.json so partially successful runs are still usable.