AI Gateway Motion Control Video Generation

Transfer motion from a reference video to a character in an image. The model analyzes the movements in your reference video and applies them to your character, creating a video where the character performs those same actions.

Every model here also runs as a background job instead of one long-lived request. See asynchronous generation below.

KlingAI's motion control models transfer motion from a reference video to a character image. The character image accepts buffers, URLs, or base64. The reference video must be a URL (use Vercel Blob for local files).

Browse the latest KlingAI video models on the AI Gateway Models page.

The character image (prompt.image) must meet these requirements:

• Formats: .jpg, .jpeg, .png
• File size: 10MB or less
• Dimensions: 300px to 65536px
• Aspect ratio: Between 1:2.5 and 2.5:1

For best results:

• Character proportions should match the reference motion. Avoid driving half-body characters with full-body motions.
• Show clear upper body or full body including limbs and head. Avoid occlusion.
• Avoid extreme orientations (upside down, lying flat). Character should occupy sufficient screen area.
• Supports realistic and stylized characters, including humans, humanoid animals, some pure animals, and humanoid body proportion characters.

When using base64 encoding, submit only the raw base64 string without any prefix:

// Correct
const image = 'iVBORw0KGgoAAAANSUhEUgAAAAUA...';
 
// Incorrect - do not include data: prefix
const dataUrl = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA...';

The reference video (providerOptions.klingai.videoUrl) must meet these requirements:

• Formats: .mp4, .mov
• File size: 100MB or less
• Dimensions: 340px to 3850px
• Duration: Minimum 3 seconds. Maximum depends on characterOrientation:
  • 'image': Maximum 10 seconds
  • 'video': Maximum 30 seconds

For best results:

• Character should show clear upper body or full body including all limbs and head. Avoid occlusion.
• Use single-person action video. For multiple people, actions are taken from the character with the largest screen proportion.
• Use real person actions. Some stylized characters with humanoid body proportions may work.
• Video should be a single continuous shot with character always visible. Avoid cuts or camera movements.
• Avoid overly fast actions. Relatively stable actions produce better results.

For complex or fast motions, results may be shorter than the uploaded video duration. The model can only extract valid motion segments and requires a minimum of 3 seconds of usable continuous motion.

import { experimental_generateVideo as generateVideo } from 'ai';
import fs from 'node:fs';
 
const result = await generateVideo({
  model: 'klingai/kling-v3.0-motion-control',
  prompt: {
    image: fs.readFileSync('./character.png'),
  },
  providerOptions: {
    klingai: {
      videoUrl: 'https://example.com/dance-reference.mp4',
      characterOrientation: 'video',
      mode: 'std',
    },
  },
});
 
fs.writeFileSync('output.mp4', result.videos[0].uint8Array);

If you have a local video file, upload it to Vercel Blob first:

import { experimental_generateVideo as generateVideo } from 'ai';
import { put } from '@vercel/blob';
import fs from 'node:fs';
 
const referenceVideo = fs.readFileSync('./dance.mp4');
const { url: videoUrl } = await put('dance.mp4', referenceVideo, {
  access: 'public',
});
 
const result = await generateVideo({
  model: 'klingai/kling-v3.0-motion-control',
  prompt: {
    image: fs.readFileSync('./character.png'),
  },
  providerOptions: {
    klingai: {
      videoUrl: videoUrl,
      characterOrientation: 'video',
      mode: 'std',
    },
  },
});
 
fs.writeFileSync('output.mp4', result.videos[0].uint8Array);

Pass a webhook factory and the SDK registers your URL with the job instead of polling for it. The factory returns that URL and a received promise it waits on, then it fetches the videos itself.

const result = await generateVideo({
  model: gateway.videoModel('klingai/kling-v3.0-motion-control'),
  prompt: {
    // Hosted URL rather than `fs.readFileSync`: the start request is capped at
    // 300 KiB, which inline image bytes pass immediately.
    image: 'https://example.com/character.png',
  },
  providerOptions: {
    klingai: {
      videoUrl: 'https://example.com/dance-reference.mp4',
      characterOrientation: 'video',
      mode: 'std',
    },
  },
  webhook: async () => ({
    url: callbackUrl,
    received: waitForDelivery(token),
  }),
  poll: { timeoutMs: 10 * 60 * 1000 },
});

The start request AI Gateway persists is capped at 300 KiB, so pass your character image as a hosted URL on this flow rather than inline data.

token, callbackUrl, and waitForDelivery come from webhook-driven completion, which also covers the receiver, signature verification, and the payload shape.