Maintaining a thriving social space has been one of the most tedious tasks for moderators due to the ever-growing problem of bots, spam, and hate speech. Large language models are well suited to this problem and never need a break or experience fatigue, meaning they can run moderate at huge scale.
In this guide, you’ll learn how to build a moderation agent in a community platform. The stack includes Next.js as the full-stack framework, AI SDK to build the agent, and AI Gateway to call models from 40+ providers with built-in observability.
Copy link to headingPrerequisites
Before you begin, make sure you have:
- Node.js 22 or later
- A Vercel account
- Vercel CLI installed
Copy link to headingCreate the project
Create a new Next.js App Router project:
Change directory into the project root:
Install dependencies:
Copy link to headingConfigure local environment variables
To get started, link your app to a Vercel project by running:
For this project, you’ll use AI Gateway to access AI models and Redis to persist post and moderation data. To set up Redis for this project, run the following command:
When prompted, select Upstash for Redis.
Now, pull the development environment variables by running the following command:
This command creates a .env.local file with the following environment variables:
Vercel lets you use AI Gateway with an OIDC token, so you don’t need to create an API key.
Copy link to headingDefine the shared types
To keep all inputs and schema consistent across the application, start by defining the moderation model. Create lib/types.ts :
Also, add the TriageSchema using Zod for validation and type safety across the app.
Copy link to headingAdd Redis for persistence
This project uses Upstash Redis to store the posts, users, and audit information. The app stores each user and post as Redis hashes. That lets independent workflow steps update single fields, such as status and runId, without overwriting each other.
Create the data model in the lib/db.ts:
Add database helpers for the users model:
Add database helpers for the posts model:
Add audit trail and clearAllPosts() helpers:
Copy link to headingAdd the agent tools
The AI agent should be autonomous in researching and classifying the incoming posts. For this, we need to give it the tools to access information about posts, authors, and moderation policy.
This project uses a fast, low-cost model to generate diverse seed data for users' post history and policy. You can replace this data fetching with your production systems.
Create lib/agent-tools.ts with the tool schema and implementation:
Copy link to headingAdd moderation actions
Define the actions to call based on the agent’s reasoning. For example, sending a warning to the user or banning them from the community.
Create moderation actions in lib/actions.ts:
sendWarningToCommunityUser is a placeholder function to send a warning, and executeAction() is where moderation decisions are applied. Automatic agent decisions and human decisions both flow through this function.
Copy link to headingCreate the triage agent
You have all the building blocks ready. Assemble the tools, instructions, and schema in a ToolLoopAgent. Create lib/triage-agent.ts:
The runTriage() function creates a new ToolLoopAgent and returns the triaged output. It runs a tool-calling loop with three context tools:
getAuthorHistory: fetches author’s post history to see gauge their behaviour.findSimilarReports: observe previous actions on similar posts.lookupPolicy: understand category specific moderation policies.
And one terminator tool: submitTriage, which handles decision routing.
The agent can call context tools, but it only routes on the final submitTriage tool call. This keeps the workflow deterministic by using structured data validated by TriageSchema rather than prose parsing.
Copy link to headingCreate moderation Server Functions
To connect the app’s frontend with the triage agent, use Next.js Server Functions to submit the post and triage. Create app/actions.ts:
Copy link to headingCreate a minimal triage UI
Build the forum UI with a form to submit new posts and a posts timeline that shows the audit trail with a live status of triaging. Update the app/page.tsx with the following content:
The form submits a new post using the submitPost() server action to triage, and the audit log UI renders the latest status for the post moderation along with an audit trail.
Copy link to headingRun your moderator
To start the application, run the following command:
Open http://localhost:3000 and submit a harmless post, then submit an obvious policy violation. The page should render an agent decision card with the decision, severity, category, reasoning, and any moderator options.
At this point, you have a working moderation agent UI. It is small, but the agent, tool schemas, context tools, and model configuration work together.

Copy link to headingScale moderation with Vercel Workflows
In production systems, you need reliability and durability so moderation can scale with your user base. A preview can finish within a single request, but a production moderation flow can’t assume the reviewer will stay present for the entire request. It must also handle failures and retries while ensuring the application repeats only the failed steps. Borderline posts may also require escalation and human review.