Evaluate AI generation cost and quality

The right AI generation choice is not the model with the lowest single request price or the prettiest demo. It is the route that reaches an accepted asset with a defensible total cost. Total cost includes provider usage, failed attempts, retries, review time, repair, storage, and handoff. Quality must be defined for the job: readable packaging for a product image, stable identity for a character, or a clear action for a video.

Short answer: Hold the brief, source pack, format, and acceptance rubric constant. Run the same task through each model, record every request and failure, measure review and repair minutes, and calculate cost per accepted asset. Keep the branches and evidence together so a later decision can distinguish a stable result from a lucky one.

Answer in practice: Use three units: cost per request, cost per accepted asset, and editor minutes per accepted asset. Add the provider's documented price only with the model, account, date, and billing unit in hand. Never invent a price from a plan name. A model that is inexpensive per request can be expensive per accepted result if it needs more retries or manual repair.

The controlled evaluation protocol

1. Write the quality gate first

For a product image, define geometry, label readability, color, and framing. For a short video, define subject continuity, action, duration, and endpoint. For a style test, define palette, lighting, and texture. Mark each requirement hard or soft. A hard failure rejects the asset even when the image is attractive.

2. Freeze the input pack

Use one brief, one source pack, one output format, and one review rubric for every model. Record model, provider, input mode, resolution, aspect ratio, duration, reference count, and any seed or control exposed by the product. If a model cannot accept the same input, record the difference instead of pretending the test is identical.

3. Run a small repeated sample

Run three candidates per route for a pilot. The number is a test design choice, not a universal statistical guarantee; it exposes obvious variance without spending a full production budget. Keep every candidate, including failures, and label the branch with the route and settings. If the pilot is inconclusive, expand the sample deliberately and record why.

4. Score quality separately from preference

Use a rubric with pass/fail hard requirements and 0–2 scores for soft requirements such as composition, motion smoothness, or stylistic fit. Have the same reviewer score the pilot where possible. Record the reason for rejection: label drift, unstable face, broken action, unreadable text, or missing handoff. “Looks bad” cannot guide a repair.

5. Measure the full effort

Record request count, failed requests, retry count, provider time, review minutes, repair minutes, export minutes, and storage or model-download time where relevant. The useful quantity is accepted-asset effort, not raw generation latency. Note when a human had to recreate an input in another application.

Units and formulas

Use the provider's current invoice or official pricing documentation for request cost. Then calculate:

• Provider spend: sum of documented request or credit charges for the test.
• Cost per accepted asset: provider spend divided by the number of assets passing the hard gate.
• Request success rate: successful requests divided by total requests.
• Acceptance rate: accepted assets divided by total candidates.
• Editor effort: review, repair, and export minutes divided by accepted assets.
• Total comparison effort: provider spend plus separately reported human minutes and storage, without inventing a dollar conversion.

Keep monetary and time units separate unless the team has an agreed internal labor rate. This avoids hiding a subjective salary assumption inside a supposedly objective model comparison.

A worked image and video test

For an image test, use one approved product reference and request three 4:5 catalog images. Hard gates are geometry and label readability; soft scores are composition and lighting. For a video test, use the accepted still as the input and request three five-second demonstrations with the same action. Hard gates are subject continuity and a clear start/end; soft scores are motion and framing.

For each route, record the exact model, input mode, dimensions, duration, request result, reviewer score, and repair needed. Suppose one route creates one accepted image after three attempts while another creates two accepted images after five attempts. Do not declare a winner from the request count alone. Compare cost per accepted image, review minutes, and whether the second route produced a reusable source for video.

Infiknit as the comparison record

Infiknit keeps the brief, Style/Product/Character references, model branches, generated media, failed candidates, and approval notes on one canvas. A branch can preserve the input and output relationship rather than flattening a result into an unlabeled folder. The internal agent can create and connect supported nodes and queue eligible work through validated tools. It cannot directly edit pixels, operate Audio nodes, or decide that a failed hard gate is acceptable.

Make the result decision-ready

At the end of a pilot, write a one-paragraph decision record: route tested, fixed inputs, quality gate, number of candidates, provider spend from the dated source, accepted count, request success rate, reviewer minutes, repair minutes, and the next action. If the evidence is inconclusive, say which variable needs another test. Do not convert uncertainty into a ranking.

Separate model fit from workflow fit. A model can pass the image rubric but fail the video handoff. Another can produce a slightly weaker first frame while preserving a reference better across a sequence. Keep separate scores for output quality, repeatability, source preservation, and handoff. The final choice may be different for a product catalog and a cinematic concept.

When comparing providers, capture the official pricing or usage page with the date and unit used. A credit-based plan may not map directly to a request, while a per-second video charge may depend on duration or resolution. Report the provider's unit as published and calculate only the test total you can support. Never extrapolate a monthly production budget from three samples without labeling the assumption.

Use the canvas as an audit trail rather than a scoreboard. Keep a failed branch when it shows a capability boundary, attach the acceptance note to the winning branch, and preserve the source settings. A later team can rerun the same test or understand why the earlier winner was selected.

Primary sources

• Infiknit product documentation
• OpenAI API pricing
• OpenAI image generation guide
• Google Vertex AI generative AI pricing
• Runway help center

Common interpretation errors

Comparing unlike inputs: If one model receives a clean reference and another receives a compressed crop, the test measures inputs as well as models.

Counting only successful outputs: Failed requests and rejected candidates are part of the route's effort.

Using a single hero sample: One lucky output does not show normal variance.

Converting every minute into invented dollars: Keep provider spend and human minutes separate unless an agreed internal rate exists.

Ignoring downstream reuse: An accepted still that becomes a stable video reference may be more valuable than an isolated image.