Published about 6 hours ago 4 min read

Building a Reference Manifest for Multi-Input AI Video Workflows

When I first added reference files to an AI video workflow, I treated them like ordinary uploads.

The request looked roughly like this:

{
  "prompt": "Create a slow product reveal",
  "images": [
    "product.png",
    "lighting.jpg"
  ],
  "video": "camera-motion.mp4"
}

This works for a prototype.

It becomes much harder to maintain once a project has several generations, multiple versions of the same asset, and different people reviewing the output.

A few questions appear quickly:

Which image defines the subject?
Which file is only a style reference?
Did generation v12 use the original product image or the compressed copy?
Can I reproduce a useful result two weeks later?
How do I detect that somebody replaced a reference without changing its filename?

I ended up solving this outside the prompt itself by introducing a small reference manifest.

This post describes that implementation.

The Problem With a Flat File List

Consider a generation request containing four files:

product.png
studio.jpg
motion.mp4
music.wav

The filenames tell us almost nothing about intent.

A human may remember that studio.jpg controls the environment and motion.mp4 is only a camera-motion reference, but the application does not know this.

Instead of passing anonymous files around, I wanted every reference to answer three questions:

What is this file?
What role does it have?
Is it still the same file that was originally submitted?

That led to the following structure.

{
  "shotId": "product-intro-03",
  "prompt": "Slow push-in toward the product",
  "references": [
    {
      "path": "refs/product.png",
      "type": "image",
      "role": "subject"
    },
    {
      "path": "refs/studio.jpg",
      "type": "image",
      "role": "environment"
    },
    {
      "path": "refs/motion.mp4",
      "type": "video",
      "role": "camera_motion"
    },
    {
      "path": "refs/music.wav",
      "type": "audio",
      "role": "pacing"
    }
  ]
}

This is already easier to inspect, but it still doesn't guarantee reproducibility.

Defining the Manifest in TypeScript

I kept the schema deliberately small.

type ReferenceType = "image" | "video" | "audio";

type ReferenceRole =
  | "subject"
  | "environment"
  | "style"
  | "camera_motion"
  | "pacing";

interface ReferenceInput {
  path: string;
  type: ReferenceType;
  role: ReferenceRole;
  sha256?: string;
}

interface ShotManifest {
  shotId: string;
  prompt: string;
  references: ReferenceInput[];
}

The important field here is role.

type tells the application how to process the file.

role tells the application why the file exists.

Those are different concerns and keeping them separate turned out to be useful.

For example, two references can both be images while serving completely different purposes:

{
  "path": "product.png",
  "type": "image",
  "role": "subject"
}

and:

{
  "path": "warm-light.jpg",
  "type": "image",
  "role": "style"
}

Validate Before Sending Anything to a Model

The next step was validation.

I didn't want malformed manifests reaching the generation layer.

A minimal validator looks like this:

const allowedTypes = new Set([
  "image",
  "video",
  "audio"
]);

const allowedRoles = new Set([
  "subject",
  "environment",
  "style",
  "camera_motion",
  "pacing"
]);

function validateManifest(manifest: ShotManifest): string[] {
  const errors: string[] = [];

  if (!manifest.shotId) {
    errors.push("shotId is required");
  }

  if (!manifest.prompt.trim()) {
    errors.push("prompt is required");
  }

  for (const [index, ref] of manifest.references.entries()) {
    if (!allowedTypes.has(ref.type)) {
      errors.push(
        `references[${index}]: unsupported type ${ref.type}`
      );
    }

    if (!allowedRoles.has(ref.role)) {
      errors.push(
        `references[${index}]: unsupported role ${ref.role}`
      );
    }
  }

  return errors;
}

Now the generation pipeline becomes:

manifest
   ↓
schema validation
   ↓
file validation
   ↓
hash references
   ↓
build provider request
   ↓
generate
   ↓
store result + manifest

The model provider is deliberately at the end of the pipeline.

Everything before that is application logic we can test independently.

Hash the References

File names are not reliable identifiers.

Someone can overwrite:

product.png

with another file while keeping exactly the same name.

To detect this, I calculate a SHA-256 hash before generation.

import { createHash } from "crypto";
import { createReadStream } from "fs";

function sha256(filePath: string): Promise<string> {
  return new Promise((resolve, reject) => {
    const hash = createHash("sha256");
    const stream = createReadStream(filePath);

    stream.on("data", chunk => hash.update(chunk));
    stream.on("error", reject);

    stream.on("end", () => {
      resolve(hash.digest("hex"));
    });
  });
}

Then references are normalized before they enter the generation layer:

async function prepareReferences(
  refs: ReferenceInput[]
): Promise<ReferenceInput[]> {

  return Promise.all(
    refs.map(async ref => ({
      ...ref,
      sha256: await sha256(ref.path)
    }))
  );
}

The stored manifest now contains something like:

{
  "path": "refs/product.png",
  "type": "image",
  "role": "subject",
  "sha256": "81b4e2..."
}

If the file changes later, the mismatch becomes detectable.

Separate the Manifest From Provider-Specific Payloads

Another mistake I made early was storing model-specific parameters directly in the project data.

That creates unnecessary coupling.

Instead, the internal manifest stays provider-neutral:

interface GenerationAdapter {
  generate(manifest: ShotManifest): Promise<string>;
}

A provider adapter is responsible for translating it:

class VideoProviderAdapter implements GenerationAdapter {

  async generate(manifest: ShotManifest): Promise<string> {

    const subjectRefs = manifest.references.filter(
      ref => ref.role === "subject"
    );

    const styleRefs = manifest.references.filter(
      ref => ref.role === "style"
    );

    const request = {
      prompt: manifest.prompt,
      subjectReferences: subjectRefs,
      styleReferences: styleRefs
    };

    return submitGeneration(request);
  }
}

The exact request format will obviously depend on the service being integrated.

The useful part is the boundary:

Project data
     |
     v
Reference Manifest
     |
     v
Generation Adapter
     |
     +---- Provider A
     |
     +---- Provider B

If a provider changes its API, the project representation does not need to change with it.

Store the Manifest With the Result

After generation, I store the output and its manifest together.

A simple directory structure works:

generations/
  product-intro-03/
    v001/
      manifest.json
      output.mp4

    v002/
      manifest.json
      output.mp4

    v003/
      manifest.json
      output.mp4

Now each video has an answer to:

How was this generated?

I can diff two manifests instead of trying to remember what changed.

For example:

- "role": "camera_motion",
- "path": "motion-fast.mp4"
+ "role": "camera_motion",
+ "path": "motion-slow.mp4"

Everything else remained identical.

That is much more useful during debugging than a note saying:

v3 looked better

A Small CLI Helper

I eventually added a tiny command for checking manifests before submitting a generation.

async function check(manifest: ShotManifest) {
  const errors = validateManifest(manifest);

  if (errors.length > 0) {
    console.error("Invalid manifest:");

    for (const error of errors) {
      console.error(`- ${error}`);
    }

    process.exit(1);
  }

  const references =
    await prepareReferences(manifest.references);

  console.log(
    JSON.stringify(
      { ...manifest, references },
      null,
      2
    )
  );
}

This makes it possible to validate a shot locally or as part of CI before spending time or generation credits on it.

A production version could also check:

MIME type
maximum file size
image dimensions
video duration
duplicate hashes
missing files
unsupported combinations

Where I Tested the Pattern

I started exploring this structure while working with reference-heavy AI video workflows, including XMK Seedance

The specific tool isn't essential to the architecture.

What matters is that newer video workflows can involve several input types rather than a single prompt. Once that happens, treating references as structured application data becomes more useful than treating them as attachments.

The same manifest pattern could sit in front of another video generation API without changing the project model.

One Important Boundary: Don't Put Sensitive Assets Into the Manifest

A manifest improves reproducibility, but it doesn't make an asset appropriate for an external AI workflow.

Before a reference enters this pipeline, the application should separately determine whether the asset is permitted to leave the organization's environment.

I would avoid using:

private customer data
internal screenshots without upload permission
credentials or API keys
confidential product designs
identifiable personal information
assets without appropriate usage rights

Sanitizing a filename or removing a person's name is not the same as having permission to send the underlying asset to an external service.

For internal interfaces, a safer option can be recreating the necessary state with fictional data and generic components.

That validation belongs before the generation manifest.

What Changed After Adding the Manifest

The main benefit wasn't better generation quality.

It was debuggability.

Instead of asking:

Why does this output look different?

I can ask:

What changed between these two generation manifests?

That's a much easier engineering question.

For quick experiments, a prompt and a folder of images may still be enough. But once AI video generation becomes part of a repeatable application workflow, references start behaving like dependencies.

And dependencies are easier to manage when they are explicit, validated, versioned, and reproducible.