Reinforcement learning environments are systems in which AI agents take actions, observe the results, and receive rewards. An environment defines the task, the actions available, and how the world changes after each action. Training uses this experience to improve the policy that determines the agent's actions.
For a coding agent, the environment might contain a repository with a bug and tests that check the repair. Vercel Sandbox can provide isolated compute for that work. The task definition and reward function determine success.
Copy link to headingHow does an RL environment work?
An agent interacts with an environment through an observation and action loop. It receives an observation, chooses an action, and gets a new observation and a reward after the environment processes that action. The reward may arrive after each step or only when the task ends.
An observation contains the information available to the agent at that point. It can be text, a screenshot, or a tool response without revealing the environment's complete state. For example, a browser agent may see a confirmation page without being able to inspect the database record behind it.
The following example maps the loop to a coding task:
Gymnasium's environment API expresses this pattern through reset() and step(action). Reset starts an episode. Each step returns an observation and reward, plus separate flags for termination and truncation.
Reaching a terminal state and hitting an external time limit are different events. Gymnasium distinguishes them as terminated and truncated. The distinction lets a training algorithm handle unfinished attempts differently from episodes that reach a terminal state, which can represent success or failure.
Copy link to headingHow do environments relate to training, evaluation, and sandboxes?
An environment supplies interactions. Training uses those interactions to change how an agent chooses actions. In Q-learning, for example, the learner updates estimates of the future reward associated with actions. Other algorithms update a parameterized policy, which may be a neural network.
Evaluation runs an agent against tasks to measure its behavior. You can compare two model versions or tool configurations while keeping their parameters fixed. Collecting more trajectories, the recorded sequences of observations and actions, doesn't by itself perform a learning update.
An evaluation environment can also support training if its interface and feedback work with the chosen learning algorithm. Keep training tasks separate from held-out evaluation tasks so a reported improvement measures performance beyond the examples used to optimize the agent.
Copy link to headingHow does experience from an environment become training data?
During an attempt, a collector records the observations available to the agent and the actions it chooses. It associates rewards with that sequence and records how the episode ends. Collecting this experience is often called a rollout. For a coding agent, the record includes tool calls and their results, followed by the submitted change and its grading outcome.
An outcome reward may arrive after many decisions. Successful repairs can follow failed test commands that helped the agent locate a bug. The learning algorithm must relate that delayed feedback to earlier actions, a problem called credit assignment. Failed commands within a successful attempt therefore carry different information from an unfinished task.
The trainer converts collected experience into the inputs its algorithm requires. For example, Hugging Face's GRPO trainer generates multiple responses to a prompt and uses their relative rewards to calculate a model update. The updated policy then generates further attempts, so the experience changes as training progresses.
Choose the training interface before designing the export format. Readable tool-call logs may omit information a trainer needs, such as generated token IDs and their log probabilities. Preserve the distinction between model-generated actions and observations supplied by tools. The integration must tell the trainer which outputs came from the policy being optimized.
Copy link to headingWhat kinds of tasks can AI agents practice?
Coding environments give agents files and execution tools for tasks such as bug fixes, dependency migrations, or implementing a specification. Supply enough project context for realistic work. The verifier also needs checks that distinguish correct changes from solutions tailored to the visible tests.
Browser environments expose websites through navigation and interaction tools. Success can mean creating a record or finding a particular piece of information. WebArena, for example, pairs functioning websites with tasks evaluated for functional correctness. Following a click sequence doesn't prove the requested change reached the application.
Tool environments can expose a smaller set of operations, such as searching a catalog and updating a test inventory. Here, the action space is the tool interface, and the environment owns the state those tools modify. This can make individual errors easier to inspect than a task with unrestricted shell access.
The right task depends on what the agent must learn. If production work requires recovering from a failed API call, an environment that always returns a successful response leaves that behavior untested. Add realistic failure cases while preserving a way to reproduce them.
Copy link to headingWhat does a coding-task reward look like?
Consider a task that asks an agent to repair clamp(value, lower, upper), a function that keeps an integer within inclusive bounds. The starting implementation handles values below the lower bound but fails to cap values above the upper bound.
The episode starts from the broken repository. The agent can inspect the implementation, edit it, and run visible tests. At submission, an independent runner executes the function in isolation. The grader compares its return values with expected results outside the agent's workspace.
The following Python example shows only the scoring step. Save it as score_clamp.py and run it with Python 3.9 or newer. The lists at the end stand in for return values collected by that runner; the example doesn't execute submitted code or train an agent.
# Inputs supplied by the runner, with bounds 0 and 10:# -2, 0, 4, 10, 15EXPECTED = [0, 0, 4, 10, 10]
def score_clamp(observed: list[object]) -> float: if len(observed) != len(EXPECTED): return 0.0
passed = sum( type(actual) is int and actual == expected for actual, expected in zip(observed, EXPECTED) ) return passed / len(EXPECTED)
print(score_clamp([0, 0, 4, 10, 15])) # 0.8: upper bound still brokenprint(score_clamp([0, 0, 4, 10, 10])) # 1.0: all five cases passThis reward gives partial credit for passing cases. An alternative is a binary reward that returns 1 only when every required check passes. That choice changes the feedback available to the learner. Partial credit identifies progress, but a high score can still conceal a required behavior that fails.
Five fixed cases are enough to illustrate scoring. Real tasks need broader checks, including different bounds and inputs the agent hasn't seen. The runner should collect actual function outputs and distinguish crashes or timeouts from valid responses. Never accept an agent-written score: 1 as the grading result.
Copy link to headingWhat makes an RL environment useful and trustworthy?
Repeatable starting conditions let you compare attempts. Pin the repository revision and dependencies, restore the initial data, and reset state outside the working directory too. Recreating a process won't reset a database or a remote service it already modified. Seeds help control randomness within the environment, but they don't make changing external services deterministic.
The grader also needs a boundary the agent cannot edit. If the agent can replace a test or overwrite a result file, a passing score may reflect changes to the measurement itself. Harbor supports separate verifier environments, with submitted artifacts passed into a fresh grading environment. When those artifacts include executable code, that code still needs isolation from trusted scoring logic.
Even a protected grader can reward the wrong behavior. This is a form of specification gaming: an agent satisfies the measured objective without completing the intended task. Rewarding the number of tests run, for example, could favor repeated execution over a correct repair. Prefer checks tied to the requested result, and inspect successful trajectories for shortcuts the metric misses.
Record enough context to explain a score. Keep the task version, model configuration, and action budget alongside the trajectory and grader result. Report infrastructure failures separately so a package registry outage doesn't silently become evidence that the agent cannot solve the task.
Copy link to headingHow do you run RL environments at scale?
Run independent attempts in parallel, giving each its own writable task state. Multiple coding agents can start from the same repository revision, but their edits must stay in separate working copies. Parallel environments can increase the rate of experience collection. The training algorithm must support the resulting batches; increasing the number of workers alone doesn't guarantee faster learning.
Set concurrency around the slowest shared dependency. You may have capacity for more sandboxes than your model endpoint can serve, or collect completed attempts faster than the grader can process them. Queue tasks before provisioning environments and cap the number of active workers.
Measure model-call time separately from command execution and grading. These timings show where work is waiting and which capacity limit needs attention.
Budget for the whole attempt. Code repairs can consume model tokens while a sandbox remains allocated, then require additional compute for verification. Set an episode time limit and a tool-call or token budget, with a separate allowance for grading. Preserve the result before releasing resources, and make cleanup run when an attempt fails or is canceled.
Track completed, graded attempts alongside cost per attempt. More concurrent workers are useful only if the system can finish and retain their results. When comparing configurations, keep task difficulty and execution budgets consistent so a throughput increase doesn't conceal shorter or less demanding attempts.
Copy link to headingHow does Vercel Sandbox support agent environments?
Vercel Sandbox provides an execution boundary for agents that run commands or generate code. Each sandbox runs in a Firecracker microVM with its own kernel. You can place a task's working files and processes inside that boundary while keeping orchestration and trusted scoring outside the agent's reach.
Choose network access explicitly. The Sandbox firewall supports a deny-all policy that blocks outbound access, including DNS. Offline coding tasks can use it after dependencies are available. For tasks that call external services, allow the required destinations and scope credentials to test data.
Start by configuring Sandbox authentication and running a command. An RL environment adds task state and a reward mechanism around that execution. Your training system remains responsible for learning from the resulting experience.
Copy link to headingRun an AgentEnv evaluation on Vercel Sandbox
Scale AI's AgentEnv integration runs the agent and its task environment in separate Sandboxes. Its Vercel provider is available in AgentEnv 0.9.1334 and later.
Complete the AgentEnv demo setup to clone the example branch and install its dependencies. It requires Git, Python 3.11.4 or newer, and uv, plus Node.js and Vercel CLI 62.5.0 or newer.
Your team and project need permission and capacity for two sandboxes with 4 vCPUs and 8 GB of memory each, plus one with 2 vCPUs and 4 GB. Sign in through the CLI, then run this command from the cloned repository:
./vercel-eval --scope YOUR_TEAM --project YOUR_PROJECTReplace YOUR_TEAM with your team slug and YOUR_PROJECT with an existing project name. The example uses a deterministic agent to add an item through Model Context Protocol tools, then grades the resulting state. Inspect the output directory for the score and recorded tool calls.
This demo checks the evaluation integration. To measure model quality, supply a model-calling agent and tasks suited to the behavior being tested. Training also needs a learning algorithm that uses the experience to update the policy.
Copy link to headingFrequently asked questions
Copy link to headingDoes running an agent repeatedly in a sandbox train it?
No. Repeated runs produce attempts and results. Reinforcement learning requires reward feedback to update the policy that selects actions. Revising an answer during a run can leave the model's parameters unchanged.
Copy link to headingCan an RL environment use a simulator?
Yes. Simulators let agents interact with a model of a system, and software tasks can also use test applications or isolated repositories. Choose a setup that reproduces the behavior you need to study and lets you restore the starting state without changing production data.
Copy link to headingDoes an agent need a reward after every action?
No. Some tasks award a score only when the agent submits its work. Intermediate rewards can provide earlier feedback, but they must reward progress toward the task rather than actions that inflate the score.
Copy link to headingWhen does an RL environment need a code sandbox?
Code sandboxes help contain operations when agents can write programs or run shell commands. Environments limited to predefined actions may not execute agent-authored code at all. In either case, task design and grading determine the feedback the agent receives.