Skip to content

Building community moderation agents using AI SDK

Learn how to build an AI-powered community moderation agent with Next.js, the AI SDK, and AI Gateway.

Knowledge Base/AI
19 min read
Last updated May 29, 2026

Maintaining a thriving social space has been one of the most tedious tasks for moderators due to the ever-growing problem of spam and hate speech. AI agents not only solve this problem but do it without being affected by the scale of the platform.

In this guide, you’ll learn how to build a moderation agent in a community platform. The stack includes Next.js as the full-stack framework, AI SDK to build the agent, and AI Gateway to call models from 40+ providers with built-in observability.

Before you begin, make sure you have:

Create a new Next.js App Router project:

pnpm create next-app@latest review-desk --yes --typescript --tailwind --eslint --app --no-src-dir --import-alias "@/*"

Change directory into the project root:

cd review-desk

Install dependencies:

pnpm add ai @upstash/redis zod

To get started, link your app to a Vercel project by running:

vercel link

For this project, you’ll use AI Gateway to access AI models and Redis to persist post and moderation data. To set up Redis for this project, run the following command:

vercel integration add upstash

When prompted, select Upstash for Redis.

Now, pull the development environment variables by running the following command:

vercel env pull

This command creates a .env.local file with the following environment variables:

KV_REST_API_READ_ONLY_TOKEN=....
KV_REST_API_TOKEN=....
KV_REST_API_URL=....
KV_URL=....
REDIS_URL=....
VERCEL_OIDC_TOKEN=....

Vercel lets you use AI Gateway with an OIDC token, so you don’t need to create an API key.

To keep all inputs and schema consistent across the application, start by defining the moderation model. Create lib/types.ts :

import { z } from "zod";


export const ACTION_IDS = [
  "restore",
  "hide",
  "warn",
  "hide_and_warn",
  "ban",
  "dismiss",
] as const;


export const ActionIdSchema = z.enum(ACTION_IDS);
export type ActionId = z.infer<typeof ActionIdSchema>;


const NAMESPACE = "mod_decision";


export function hookTokenForPost(postId: string): string {
  return `${NAMESPACE}:${postId}`;
}


export function actionIdFor(action: ActionId): string {
  return `${NAMESPACE}:${action}`;
}


export const ALL_MOD_ACTION_IDS: readonly string[] = ACTION_IDS.map(actionIdFor);


export const POST_STATUSES = [
  "live",
  "under_review",
  "hidden",
  "warned",
  "hidden_and_warned",
] as const;


export type PostStatus = (typeof POST_STATUSES)[number];


export type User = {
  id: string;
  name: string;
  banned: boolean;
};


export type Post = {
  id: string;
  authorId: string;
  body: string;
  createdAt: string;
  status: PostStatus;
  warning?: string;
  runId?: string;
};


export type AuditEntry = {
  id: string;
  postId: string;
  at: string;
  action: ActionId | "auto_classified" | "escalated";
  actorId: string;
  note?: string;
};

Also, add the TriageSchema using Zod for validation and type safety across the app.

lib/types.ts
export const TriageSchema = z
  .object({
    category: z
      .enum(["spam", "harassment", "hate_speech", "self_harm", "off_topic", "benign", "other"])
      .describe("Best-fit category for the post."),
    severity: z
      .enum(["none", "low", "medium", "high"])
      .describe("How severe the violation is, if any."),
    confidence: z
      .number()
      .min(0)
      .max(1)
      .describe("How confident the agent is in its decision (0-1)."),
    decision: z
      .enum(["safe", "auto_hide", "auto_warn", "request_ban", "second_opinion"])
      .describe(
        "Routing decision. 'safe' leaves the post live. 'auto_hide' / 'auto_warn' are taken without a human. 'request_ban' / 'second_opinion' escalate to a moderator.",
      ),
    reasoning: z
      .string()
      .describe("Short explanation of the decision, surfaced in the mod card and audit log."),
    draftedWarning: z
      .string()
      .optional()
      .describe(
        "When decision involves warning the user (auto_warn, or any escalation that includes warn / hide_and_warn options), the message to DM them. REQUIRED when decision is auto_warn.",
      ),
    humanRequest: z
      .string()
      .optional()
      .describe(
        "When escalating, the prose question to ask the moderator. REQUIRED when decision is request_ban or second_opinion.",
      ),
    actionOptions: z
      .array(
        z.object({
          label: z.string().describe("Button label shown in Slack."),
          action: ActionIdSchema.describe("Action to apply if the moderator clicks this button."),
        }),
      )
      .optional()
      .describe(
        "Buttons to render on the mod card. The agent picks a relevant subset of the action vocabulary. REQUIRED when escalating.",
      ),
  })
  // Cross-field invariants the model must respect. The workflow only routes
  // after a schema-valid `submitTriage` call, so invalid tool input should be
  // handled explicitly if you adapt this schema for production.
  .superRefine((v, ctx) => {
    if (v.decision === "auto_warn" && !v.draftedWarning?.trim()) {
      ctx.addIssue({
        code: z.ZodIssueCode.custom,
        path: ["draftedWarning"],
        message: "draftedWarning is required when decision is 'auto_warn'.",
      });
    }
    if (
      (v.decision === "request_ban" || v.decision === "second_opinion") &&
      !v.humanRequest?.trim()
    ) {
      ctx.addIssue({
        code: z.ZodIssueCode.custom,
        path: ["humanRequest"],
        message:
          "humanRequest is required when decision is 'request_ban' or 'second_opinion'.",
      });
    }
  });


export type TriageOutput = z.infer<typeof TriageSchema>;

This project uses Uptash Redis to store the posts, users, and audit information. The app stores each user and post as Redis hashes. That lets independent workflow steps update single fields, such as status and runId, without overwriting each other.

Create the data model in the lib/db.ts:

import { Redis } from "@upstash/redis";


import type { AuditEntry, Post, PostStatus, User } from "./types";


const SEED_USERS: User[] = [
  { id: "u_alice", name: "Alice", banned: false },
  { id: "u_bob", name: "Bob", banned: false },
  { id: "u_carol", name: "Carol", banned: false },
];


const KEY_USERS_INDEX = "users:index";
const KEY_POSTS_INDEX = "posts:index";
const userKey = (id: string) => `user:${id}`;
const postKey = (id: string) => `post:${id}`;
const auditKey = (postId: string) => `audit:${postId}`;


let _redis: Redis | null = null;
let _seeded = false;


function redis(): Redis {
  if (!_redis) _redis = Redis.fromEnv();
  return _redis;
}


async function ensureSeed(): Promise<void> {
  if (_seeded) return;
  const r = redis();
  const exists = await r.scard(KEY_USERS_INDEX);
  if (exists === 0) {
    const p = r.pipeline();
    for (const u of SEED_USERS) {
      p.hset(userKey(u.id), {
        id: u.id,
        name: u.name,
        banned: u.banned ? "1" : "0",
      });
      p.sadd(KEY_USERS_INDEX, u.id);
    }
    await p.exec();
  }
  _seeded = true;
}


function uid(prefix: string) {
  return `${prefix}_${Math.random().toString(36).slice(2, 10)}`;
}

Add database helpers for the users model:

lib/db.ts
function deserializeUser(
  raw: Record<string, unknown> | null | undefined,
): User | undefined {
  if (!raw || Object.keys(raw).length === 0) return undefined;
  return {
    id: String(raw.id),
    name: String(raw.name),
    banned: raw.banned === "1" || raw.banned === 1 || raw.banned === true,
  };
}


export async function listUsers(): Promise<User[]> {
  await ensureSeed();
  const r = redis();
  const ids = await r.smembers(KEY_USERS_INDEX);
  if (ids.length === 0) return [];
  const p = r.pipeline();
  for (const id of ids) p.hgetall(userKey(id));
  const results = (await p.exec()) as (Record<string, unknown> | null)[];
  return results
    .map(deserializeUser)
    .filter((u): u is User => Boolean(u))
    .sort((a, b) => a.id.localeCompare(b.id));
}


export async function getUser(id: string): Promise<User | undefined> {
  const raw = await redis().hgetall<Record<string, unknown>>(userKey(id));
  return deserializeUser(raw);
}


export async function ensureUser(id: string, name?: string): Promise<User> {
  const existing = await getUser(id);
  if (existing) return existing;
  const u: User = { id, name: name ?? id, banned: false };
  const r = redis();
  const p = r.pipeline();
  p.hset(userKey(id), { id: u.id, name: u.name, banned: "0" });
  p.sadd(KEY_USERS_INDEX, id);
  await p.exec();
  return u;
}


export async function setBanned(
  userId: string,
  banned: boolean,
): Promise<void> {
  // Atomic single-field write , no read-modify-write race.
  await redis().hset(userKey(userId), { banned: banned ? "1" : "0" });
}

Add database helpers for the posts model:

lib/db.ts
function deserializePost(
  raw: Record<string, unknown> | null | undefined,
): Post | undefined {
  if (!raw || Object.keys(raw).length === 0) return undefined;
  const post: Post = {
    id: String(raw.id),
    authorId: String(raw.authorId),
    body: String(raw.body),
    createdAt: String(raw.createdAt),
    status: raw.status as PostStatus,
  };
  if (raw.warning) post.warning = String(raw.warning);
  if (raw.runId) post.runId = String(raw.runId);
  return post;
}


export async function createPost(input: {
  authorId: string;
  body: string;
}): Promise<Post> {
  // New posts start `under_review` so <AgentStream> mounts on the very first
  // render and subscribes to the run's stream , otherwise we'd race the
  // workflow's first step and miss the early tool calls.
  const post: Post = {
    id: uid("p"),
    authorId: input.authorId,
    body: input.body,
    createdAt: new Date().toISOString(),
    status: "under_review",
  };
  const r = redis();
  const p = r.pipeline();
  p.hset(postKey(post.id), {
    id: post.id,
    authorId: post.authorId,
    body: post.body,
    createdAt: post.createdAt,
    status: post.status,
  });
  p.sadd(KEY_POSTS_INDEX, post.id);
  await p.exec();
  return post;
}


export async function getPost(id: string): Promise<Post | undefined> {
  const raw = await redis().hgetall<Record<string, unknown>>(postKey(id));
  return deserializePost(raw);
}


export async function listPosts(): Promise<Post[]> {
  const r = redis();
  const ids = await r.smembers(KEY_POSTS_INDEX);
  if (ids.length === 0) return [];
  const p = r.pipeline();
  for (const id of ids) p.hgetall(postKey(id));
  const results = (await p.exec()) as (Record<string, unknown> | null)[];
  return results
    .map(deserializePost)
    .filter((post): post is Post => Boolean(post))
    .sort((a, b) => b.createdAt.localeCompare(a.createdAt));
}


export async function setPostStatus(
  id: string,
  status: PostStatus,
  warning?: string,
): Promise<void> {
  // Atomic single-field write. setPostStatus and setPostRunId target
  // different fields in the same hash, so they no longer race.
  const fields: Record<string, string> = { status };
  if (warning !== undefined) fields.warning = warning;
  await redis().hset(postKey(id), fields);
}


export async function setPostRunId(id: string, runId: string): Promise<void> {
  await redis().hset(postKey(id), { runId });
}

Add audit trail and clearAllPosts() helpers:

lib/db.ts


export async function appendAudit(
  entry: Omit<AuditEntry, "id" | "at">,
): Promise<AuditEntry> {
  const e: AuditEntry = {
    ...entry,
    id: uid("a"),
    at: new Date().toISOString(),
  };
  await redis().lpush(auditKey(entry.postId), e);
  return e;
}


export async function getAuditForPost(postId: string): Promise<AuditEntry[]> {
  const entries = await redis().lrange<AuditEntry>(auditKey(postId), 0, -1);
  // LPUSH stores newest first; reverse so the UI reads oldest → newest.
  return entries.reverse();
}


export async function clearAllPosts(): Promise<{ cleared: number }> {
  const r = redis();
  const ids = await r.smembers(KEY_POSTS_INDEX);
  if (ids.length > 0) {
    const keys = [...ids.map(postKey), ...ids.map(auditKey)];
    const p = r.pipeline();
    p.del(...(keys as [string, ...string[]]));
    p.del(KEY_POSTS_INDEX);
    await p.exec();
  }
  // Reset bans so the demo starts clean. In-flight workflows targeting
  // cleared posts will short-circuit (loadPost returns null,
  // executeAction no-ops).
  const userIds = await r.smembers(KEY_USERS_INDEX);
  if (userIds.length > 0) {
    const p = r.pipeline();
    for (const id of userIds) p.hset(userKey(id), { banned: "0" });
    await p.exec();
  }
  return { cleared: ids.length };
}

The AI agent should be autonomous in researching and classifying the incoming posts. For this, we need to give it the tools to access information about posts, authors, and moderation policy.

This project uses a fast, low-cost model to generate diverse seed data for users' post history and policy. You can replace this data fetching with your production systems.

Create lib/agent-tools.ts with the tool schema and implementation:

import { generateText, Output } from "ai";
import { z } from "zod";


const FAST_MODEL = "google/gemini-3-flash";


const AuthorHistorySchema = z.object({
  accountAgeDays: z.number().int().min(0).max(3000),
  totalPosts: z.number().int().min(0).max(5000),
  strikes: z.number().int().min(0).max(5),
  tone: z.enum(["constructive", "mixed", "frequently-hostile"]),
  priorActions: z.array(
    z.object({
      action: z.enum(["warn", "hide", "hide_and_warn", "ban", "restore"]),
      date: z.string().describe("ISO date, within the last 18 months."),
      reason: z.string().describe("One-line reason."),
    }),
  ),
  notes: z
    .string()
    .describe("One-sentence summary a moderator would find useful."),
});


export async function getAuthorHistory({
  authorId,
}: {
  authorId: string;
}): Promise<z.infer<typeof AuthorHistorySchema>> {
  const { output } = await generateText({
    model: FAST_MODEL,
    output: Output.object({
      schema: AuthorHistorySchema,
    }),
    system:
      "You are a stub that simulates a community-moderation database. " +
      "Generate plausible, realistic histories. Vary the data , most users are clean, " +
      "some have one or two strikes, a small number are repeat offenders. Dates should " +
      "fall within the last 18 months relative to today.",
    prompt: `Generate a moderation history for forum user "${authorId}".`,
  });
  return output;
}


const SimilarReportsSchema = z.object({
  results: z
    .array(
      z.object({
        postId: z.string().describe("Fake post ID like p_8421."),
        similarity: z.number().min(0).max(1),
        excerpt: z.string().describe("Short excerpt of the similar post."),
        outcome: z.enum([
          "restore",
          "hide",
          "warn",
          "hide_and_warn",
          "ban",
          "dismiss",
        ]),
        date: z.string().describe("ISO date within the last 6 months."),
      }),
    )
    .max(3),
});


export async function findSimilarReports({
  text,
}: {
  text: string;
}): Promise<z.infer<typeof SimilarReportsSchema>> {
  const { output } = await generateText({
    model: FAST_MODEL,
    output: Output.object({
      schema: SimilarReportsSchema,
    }),
    system:
      "You are a stub that simulates a vector search across recent moderation reports. " +
      "Return 0-3 plausibly-similar prior cases with their outcomes. Sometimes return zero " +
      "results to simulate a novel case.",
    prompt: `Find prior moderation cases similar to: """${text}"""`,
  });
  return output;
}


const PolicySchema = z.object({
  category: z.string(),
  citation: z
    .string()
    .describe("Short citation like 'Community Guidelines §3.1'."),
  text: z.string().describe("2-3 sentence excerpt of the policy."),
});


export async function lookupPolicy({
  category,
}: {
  category: string;
}): Promise<z.infer<typeof PolicySchema>> {
  const { output } = await generateText({
    model: FAST_MODEL,
    output: Output.object({
      schema: PolicySchema,
    }),
    system:
      "You are a stub that simulates lookup against a community guidelines document. " +
      "Return a plausible policy excerpt for the given category.",
    prompt: `Look up the community guideline that covers: ${category}`,
  });
  return output;
}

Define the actions to call based on the agent’s reasoning. For example, sending a warning to the user or banning them from the community.

Create moderation actions in lib/actions.ts:

import {
  appendAudit,
  ensureUser,
  getPost,
  setBanned,
  setPostStatus,
} from "./db";
import type { ActionId } from "./types";


// Stub for the cross-platform user-DM integration. In production this would
// route to your transactional email / Slack / push notification provider.
export async function sendWarningToCommunityUser(
  userId: string,
  message: string,
): Promise<void> {
  await ensureUser(userId);
  console.log(`[warn-dm] -> ${userId}: ${message}`);
}


export async function executeAction(
  postId: string,
  action: ActionId,
  draftedWarning: string | undefined,
  actorId: string,
): Promise<void> {
  const post = await getPost(postId);
  if (!post) return;


  switch (action) {
    case "restore":
      await setPostStatus(postId, "live");
      break;
    case "hide":
      await setPostStatus(postId, "hidden");
      break;
    case "warn":
      await setPostStatus(postId, "warned", draftedWarning);
      if (draftedWarning) {
        await sendWarningToCommunityUser(post.authorId, draftedWarning);
      }
      break;
    case "hide_and_warn":
      await setPostStatus(postId, "hidden_and_warned", draftedWarning);
      if (draftedWarning) {
        await sendWarningToCommunityUser(post.authorId, draftedWarning);
      }
      break;
    case "ban":
      await setPostStatus(postId, "hidden");
      await setBanned(post.authorId, true);
      break;
    case "dismiss":
      await setPostStatus(postId, "live");
      break;
  }


  await appendAudit({
    postId,
    action,
    actorId,
    note: draftedWarning,
  });
}

sendWarningToCommunityUser is a placeholder function to send a warning, and executeAction() is where moderation decisions are applied. Automatic agent decisions and human decisions both flow through this function.

You have all the building blocks ready. Assemble the tools, instructions, and schema in a ToolLoopAgent. Create lib/triage-agent.ts:

import { ToolLoopAgent, tool, hasToolCall } from "ai";
import { z } from "zod";


import {
  findSimilarReports,
  getAuthorHistory,
  lookupPolicy,
} from "./agent-tools";
import { TriageSchema } from "./types";
import type { TriageOutput } from "./types";


const TRIAGE_MODEL = "anthropic/claude-haiku-4.5";


export const TRIAGE_INSTRUCTIONS = `You are a moderation triage agent for a small online community forum.


You communicate ONLY by calling tools. Never reply in plain text. Every response
must be a tool call. Your final tool call MUST be \`submitTriage\` , the workflow
ignores any text you emit and only reads the \`submitTriage\` arguments.


For every post you see, decide what should happen to it. You have three resolution paths:


1. SAFE , the post is fine, leave it live.
2. AUTO-RESOLVE , clear, unambiguous violation that you can act on without a human:
   - "auto_hide" hides the post.
   - "auto_warn" hides the post and DMs the user a warning.
3. ESCALATE , borderline, ban-worthy, or otherwise needs a human moderator:
   - "second_opinion" , you're unsure or the case is borderline.
   - "request_ban" , you believe a ban is warranted; the human confirms.


Action vocabulary the moderator can choose from:
- restore: post stays live, clears any "under review" badge
- hide: post is hidden, no warning sent
- warn: post stays live, user receives a warning DM
- hide_and_warn: post is hidden AND user receives a warning DM
- ban: user is banned (cosmetically marked across the forum); their posts are dimmed
- dismiss: take no action; close the case


Tools available to you:
- getAuthorHistory(authorId): prior strikes, account age, tone, recent moderation actions on this user
- findSimilarReports(text): up to 3 similar prior cases with their outcomes
- lookupPolicy(category): the relevant community-guideline excerpt


Use the context tools when they meaningfully change your decision. You don't have
to use all of them. A clean account posting obvious spam doesn't need a policy lookup.


When you escalate, you must:
- Set "humanRequest" to a tight, specific question for the moderator (not a generic
  "please review"). Include the key signal that made you uncertain.
- Set "actionOptions" to the relevant subset of the action vocabulary, with human-readable
  labels. For a ban request, that's typically [Approve ban, Just warn, Restore]. For a
  second opinion on a borderline case, [Confirm violation, Restore] is enough. Keep it
  short , 2 to 4 buttons. Always include "dismiss" only if relevant.


When the decision involves a warning ("auto_warn", or any escalation where one of the
options is "warn" / "hide_and_warn"), draft the warning message in "draftedWarning".
Address the user directly, cite which guideline, keep it under 3 sentences.


Always include a one-or-two-sentence "reasoning" explaining your decision.


FINALIZATION: As soon as you have enough context , even on the very first turn for
clearly safe posts like a greeting , call \`submitTriage\` with the complete decision.
Do not narrate, do not summarize, do not output text. The submitTriage call is your
ONLY way to finish; the workflow stops listening once it lands.`;

The runTriage() function creates a new ToolLoopAgent and returns the triaged output. It runs a tool-calling loop with three context tools:

  • getAuthorHistory: fetches author’s post history to see gauge their behaviour.
  • findSimilarReports: observe previous actions on similar posts.
  • lookupPolicy: understand category specific moderation policies.

And one terminator tool: submitTriage, which handles decision routing.

The agent can call context tools, but it only routes on the final submitTriage tool call. This keeps the workflow deterministic by using structured data validated by TriageSchema rather than prose parsing.

export async function runTriage({
  authorId,
  body,
}: {
  authorId: string;
  body: string;
}): Promise<TriageOutput | undefined> {
  const agent = new ToolLoopAgent({
    model: TRIAGE_MODEL,
    instructions: TRIAGE_INSTRUCTIONS,
    tools: {
      getAuthorHistory: tool({
        description:
          "Look up the author's moderation history: account age, prior strikes, recent actions, overall tone.",
        inputSchema: z.object({
          authorId: z
            .string()
            .describe("Internal user ID of the post's author."),
        }),
        execute: getAuthorHistory,
      }),
      findSimilarReports: tool({
        description:
          "Find up to 3 prior moderation cases similar to the post, with their outcomes.",
        inputSchema: z.object({
          text: z.string().describe("The post body to find similar cases for."),
        }),
        execute: findSimilarReports,
      }),
      lookupPolicy: tool({
        description: "Look up the community-guidelines section for a category.",
        inputSchema: z.object({
          category: z
            .string()
            .describe(
              "Category like 'harassment', 'spam', 'hate_speech', 'self_harm', 'off_topic'.",
            ),
        }),
        execute: lookupPolicy,
      }),
      submitTriage: tool({
        description:
          "Submit your final triage decision. This is your last action, call it " +
          "exactly once, with the complete triage object. After this call the app " +
          "will route the post (auto-resolve or escalate to a human moderator).",
        inputSchema: TriageSchema,
        execute: async (triage: TriageOutput) => {
          // Identity tool. The model's input is the triage decision.
          return triage;
        },
      }),
    },
    stopWhen: hasToolCall("submitTriage"),
  });


  const result = await agent.generate({
    prompt:
      `Triage this forum post.\\n\\n` +
      `Author ID: ${authorId}\\n` +
      `Body:\\n"""\\n${body}\\n"""`,
  });


  const submitCall = result.steps
    .flatMap((step) => step.toolCalls)
    .find((call) => call.toolName === "submitTriage");


  return submitCall?.input as TriageOutput | undefined;
}

To connect the app’s frontend with the triage agent, use Next.js Server Functions to submit the post and triage. Create app/actions.ts:

"use server";


import { revalidatePath } from "next/cache";
import { executeAction } from "@/lib/actions";
import {
  appendAudit,
  clearAllPosts,
  createPost,
  ensureUser,
  setPostStatus,
} from "@/lib/db";
import { runTriage } from "@/lib/triage-agent";
import type { TriageOutput } from "@/lib/types";


export async function submitPost(formData: FormData): Promise<void> {
  const authorId = String(formData.get("authorId") ?? "").trim();
  const body = String(formData.get("body") ?? "").trim();
  if (!authorId || !body) return;


  await ensureUser(authorId);
  const post = await createPost({ authorId, body });
  const triage = await runTriage({ authorId, body });


  if (!triage) {
    await setPostStatus(post.id, "live");
    await appendAudit({
      postId: post.id,
      action: "auto_classified",
      actorId: "agent",
      note: "agent did not call submitTriage",
    });
    revalidatePath("/");
    return;
  }


  await recordClassification(post.id, triage);


  if (triage.decision === "safe") {
    await setPostStatus(post.id, "live");
  } else if (triage.decision === "auto_hide") {
    await executeAction(post.id, "hide", undefined, "agent");
  } else if (triage.decision === "auto_warn") {
    const warning = triage.draftedWarning?.trim();
    await executeAction(
      post.id,
      warning ? "hide_and_warn" : "hide",
      warning,
      "agent",
    );
  } else {
    await setPostStatus(post.id, "under_review");
    await appendAudit({
      postId: post.id,
      action: "escalated",
      actorId: "agent",
      note: triage.humanRequest ?? "needs human review",
    });
  }


  revalidatePath("/");
}


export async function clearForum(): Promise<void> {
  await clearAllPosts();
  revalidatePath("/");
}


async function recordClassification(postId: string, triage: TriageOutput) {
  await appendAudit({
    postId,
    action: "auto_classified",
    actorId: "agent",
    note:
      `decision=${triage.decision}; ` +
      `category=${triage.category}; ` +
      `severity=${triage.severity}; ` +
      `confidence=${triage.confidence.toFixed(2)}; ` +
      `reasoning=${triage.reasoning}`,
  });
}

Build the forum UI with a form to submit new posts and a posts timeline that shows the audit trail with a live status of triaging. Update the app/page.tsx with the following content:

import { getAuditForPost, listPosts, listUsers } from "@/lib/db";


import { clearForum, submitPost } from "./actions";


export const dynamic = "force-dynamic";


const statusTone = {
  live: "border-emerald-200 bg-emerald-50 text-emerald-700",
  under_review: "border-blue-200 bg-blue-50 text-blue-700",
  hidden: "border-red-200 bg-red-50 text-red-700",
  warned: "border-amber-200 bg-amber-50 text-amber-700",
  hidden_and_warned: "border-red-200 bg-red-50 text-red-700",
} as const;


export default async function Home() {
  const [users, posts] = await Promise.all([listUsers(), listPosts()]);
  const auditByPost = Object.fromEntries(
    await Promise.all(
      posts.map(async (post) => [post.id, await getAuditForPost(post.id)] as const),
    ),
  );


  return (
    <main className="min-h-screen bg-neutral-50 px-6 py-10 text-neutral-950">
      <div className="mx-auto grid max-w-3xl gap-6">
        <header className="space-y-2">
          <p className="text-sm font-medium text-neutral-500">Review Desk</p>
          <h1 className="text-3xl font-semibold tracking-tight">
            Moderate forum posts with the AI SDK
          </h1>
          <p className="max-w-2xl text-sm leading-6 text-neutral-600">
            Submit a post, run AI SDK triage, and persist the moderation state
            in Redis.
          </p>
        </header>


        <form
          action={submitPost}
          className="rounded-lg border border-neutral-200 bg-white p-5 shadow-sm"
        >
          <div className="grid gap-4">
            <label className="grid gap-2 text-sm font-medium text-neutral-700">
              Author
              <select
                name="authorId"
                className="h-10 rounded-md border border-neutral-300 bg-white px-3 text-sm text-neutral-950 outline-none transition focus:border-neutral-950 focus:ring-2 focus:ring-neutral-950/10"
              >
                {users.map((user) => (
                  <option key={user.id} value={user.id}>
                    {user.name} ({user.id})
                  </option>
                ))}
              </select>
            </label>
            <label className="grid gap-2 text-sm font-medium text-neutral-700">
              Post
              <textarea
                name="body"
                rows={4}
                className="resize-none rounded-md border border-neutral-300 bg-white px-3 py-2 text-sm leading-6 text-neutral-950 outline-none transition focus:border-neutral-950 focus:ring-2 focus:ring-neutral-950/10"
              />
            </label>
            <button
              type="submit"
              className="inline-flex h-10 w-fit items-center rounded-md bg-neutral-950 px-4 text-sm font-medium text-white transition hover:bg-neutral-800"
            >
              Submit for moderation
            </button>
          </div>
        </form>


        <section className="rounded-lg border border-neutral-200 bg-white shadow-sm">
          <div className="flex items-center justify-between border-b border-neutral-200 px-5 py-4">
            <div>
              <h2 className="text-lg font-semibold">Posts</h2>
              <p className="mt-1 text-sm text-neutral-500">
                {posts.length} {posts.length === 1 ? "post" : "posts"}
              </p>
            </div>
            <form action={clearForum}>
              <button
                type="submit"
                className="rounded-md border border-neutral-300 px-3 py-2 text-sm font-medium text-neutral-700 transition hover:bg-neutral-100 disabled:opacity-50"
                disabled={posts.length === 0}
              >
                Clear forum
              </button>
            </form>
          </div>


          {posts.length === 0 ? (
            <div className="px-5 py-12 text-center">
              <p className="text-sm font-medium text-neutral-700">
                No posts yet
              </p>
              <p className="mt-1 text-sm text-neutral-500">
                Submit a safe, obvious, or borderline post to test the agent.
              </p>
            </div>
          ) : (
            <ol className="divide-y divide-neutral-200">
              {posts.map((post) => {
                const audit = auditByPost[post.id] ?? [];


                return (
                  <li key={post.id} className="px-5 py-5">
                    <div className="flex flex-wrap items-start justify-between gap-3">
                      <p className="max-w-2xl text-sm leading-6 text-neutral-800">
                        {post.body}
                      </p>
                      <span
                        className={`rounded-full border px-3 py-1 text-xs font-medium ${
                          statusTone[post.status]
                        }`}
                      >
                        {post.status.replaceAll("_", " ")}
                      </span>
                    </div>


                    {post.status === "under_review" ? (
                      <div className="mt-4 rounded-md border border-blue-200 bg-blue-50 p-4">
                        <p className="text-sm font-medium text-blue-950">
                          Waiting for human review
                        </p>
                        <p className="mt-1 text-sm text-blue-800">
                          The next guide connects this state to Slack and Vercel
                          Workflow.
                        </p>
                      </div>
                    ) : null}


                    <details className="mt-4">
                      <summary className="cursor-pointer text-sm font-medium text-neutral-700">
                        Audit log
                      </summary>
                      {audit.length === 0 ? (
                        <p className="mt-2 text-sm text-neutral-500">
                          No audit entries yet.
                        </p>
                      ) : (
                        <ol className="mt-3 grid gap-2">
                          {audit.map((entry) => (
                            <li
                              key={entry.id}
                              className="rounded-md bg-neutral-50 px-3 py-2 text-sm text-neutral-700"
                            >
                              <span className="font-medium">{entry.action}</span>
                              <span className="text-neutral-500"> by </span>
                              <span>{entry.actorId}</span>
                              {entry.note ? (
                                <span className="text-neutral-500">
                                  {" "}
                                  · {entry.note}
                                </span>
                              ) : null}
                            </li>
                          ))}
                        </ol>
                      )}
                    </details>
                  </li>
                );
              })}
            </ol>
          )}
        </section>
      </div>
    </main>
  );
}

The form submits a new post using the submitPost() server action to triage, and the audit log UI renders the latest status for the post moderation along with an audit trail.

To start the application, run the following command:

pnpm dev

Open http://localhost:3000 and submit a harmless post, then submit an obvious policy violation. The page should render an agent decision card with the decision, severity, category, reasoning, and any moderator options.

At this point, you have a working moderation agent UI. It is small, but the agent, tool schemas, context tools, and model configuration work together.

Review desk forum UI with post submission and moderation

In production systems, you need reliability and durability so moderation can scale with your user base. A preview can finish within a single request, but a production moderation flow can’t assume the reviewer will stay present for the entire request. It must also handle failures and retries while ensuring the application repeats only the failed steps. Borderline posts may also require escalation and human review.

Was this helpful?

supported.

Read related documentation

Explore more AI SDK guides