Use Jev when your application needs to interpret information and choose from a defined set of answers, while keeping explicit business rules in code and using a generative model for written responses. Before changing your workflow, test this division of work on representative cases to assess the results.
Copy link to headingWhat should each part of the workflow produce?
Start with the output your application needs.
Queue assignments and an email draft require different evaluation criteria, even when both begin with the same customer message.
One useful selection question is whether you can describe a satisfactory result before seeing it. For a routing decision, you can name the destinations and explain when each applies. For a reply, you can specify requirements, but the wording remains part of the work.
This distinction does not establish which model performs better on your data. Valid output format also does not establish that a classification is semantically correct.
Copy link to headingWhere would a decision model help?
Consider a hypothetical appointment service. The customer writes, "I can't make Tuesday. Is there anything after lunch next week?" The service must interpret the request before it can search the calendar.
Jev could classify the message as a rescheduling request, allowing application code to retrieve available appointments. Those results could then inform a reply written by a generative model.
TypeSafe's intent-routing pattern allows dispatch to deterministic handlers, specialist LLMs, or people. Use that pattern as a design starting point. Each category should tell your application which handler should receive the request.
If the customer adds, “Also, your confirmation has the wrong address,” classifying the message only as a scheduling request could cause the application to overlook the address correction. Before evaluating a model, decide whether to send mixed requests for review or route them to a handler that can address both needs, so you can distinguish classification errors from workflow gaps.
Write down ambiguous cases early. Include messages that mention an appointment without asking to change it, quoted messages, and requests with missing dates. Those cases tell you more about workload fit than another clean example.
Copy link to headingWhich other tasks are worth testing?
These hypothetical examples suggest tasks to test with Jev and show how to define their scope, without making claims about model performance.
Once you’ve defined the possible answers, check that the supplied context gives the model enough evidence to choose between them.
If reviewers need information from another system to label an example, your application may need to retrieve it first so the evaluation tests the intended judgment without forcing the model to guess at missing facts.
Copy link to headingWhich responsibilities should stay in code?
TypeSafe recommends keeping rules, control flow, and side effects in code while supplying current context for narrow model questions.
For the hypothetical appointment service, treat an account's active status as a database fact. Do not ask a model to infer it from a polite message. Keep appointment availability in the scheduling system, and require the application to check it before making a booking.
The application owns execution and policy. A predicted scheduling category should not itself authorize changing an appointment. Your code must establish what the customer may change and what confirmation the workflow requires.
This separation also gives you a useful debugging question. Did the interpretation fail, did the source data become stale, or did the handler apply the wrong rule? Record enough information to distinguish those failures without collecting unnecessary customer content.
Copy link to headingWhat does Jev leave to another component?
Jev does not generate prose or reasoning explanations. Use a generative model for the customer reply and assess that reply separately.
For the appointment example, choosing the right handler would not prove that an email contains the right time. Check the chosen appointment against the booking record. If your product requires an explanation for a decision, design that requirement explicitly; a category alone cannot satisfy it.
Copy link to headingHow does eve use Jev to select a response model?
eve's auto from eve/models selects from an authored model allowlist before language-model inference. Its evaluator defaults to typesafe-ai/jev.
The evaluator selects a model for each turn using option keys, descriptions, and recent text, without receiving credentials or serialized model instances. Its choice stays fixed throughout that turn, separating model selection from response generation without guaranteeing better quality or lower cost.
For the hypothetical appointment service, you might offer a response model for routine scheduling and another for messages requiring more involved explanation. Write descriptions that distinguish their intended use. Avoid vague descriptions like "best model" that make routing choices hard to evaluate.
Test the selection and the resulting reply together. A plausible model choice can still produce an unsuitable response. Compare against using your existing response model for every case before deciding that another selection step earns its place.
Copy link to headingHow should you test workload fit?
Define the decision by specifying allowed outcomes and the consequences of errors, keeping cancellation requests separate from scheduling questions when confusion could change a booking.
Label representative cases from routine traffic and ambiguous messages, resolving reviewer disagreements before using those labels to judge results.
Compare the proposed workflow with your existing implementation on the same cases, measuring incorrect routing, unnecessary review, and any relevant cost or response-time differences.
Investigate disagreements to distinguish unclear labels and missing context from model mistakes, then address the underlying issue before expanding automation.
Set release criteria that identify unacceptable errors and cases requiring human review, with a way to restore the previous workflow if needed.
For the hypothetical appointment service, write an acceptance scorecard before looking at model results. Specify how the application should respond to each test case and what evidence would show that it behaved correctly.
Use it to check whether the workflow handles the request correctly.
Choose acceptance criteria that reflect the consequences in your application. A mistaken queue assignment that a person can correct differs from an unauthorized cancellation. Review those outcomes separately so an acceptable average does not conceal the failure you most need to prevent.
For implementation through AI Gateway, follow the evaluation documentation.
Copy link to headingFrequently asked questions
Copy link to headingWhich tasks are a good fit for Jev?
Consider Jev for a bounded interpretation that selects an application path. A useful candidate has defined outcomes and labeled cases you can use to test the choice.
Copy link to headingShould deterministic rules call a model?
No, a fully specified rule should run in code. Add a model evaluation when the workflow requires interpretation that the rule does not resolve.
Copy link to headingCan Jev work alongside an existing chat model?
Yes. Evaluate a bounded interpretation separately, then pass the relevant facts to the model responsible for the reply. Test the classification and the written response as distinct outputs.
Copy link to headingHow should you evaluate whether Jev fits an existing workflow?
Compare a candidate workflow with your current implementation on the same labeled examples. Track consequential mistakes and review effort, then inspect cases where the implementations disagree.
Copy link to headingRelated resources
Classify, route, and score with Jev
Use TypeSafe AI's Jev with the AI SDK experimental_evaluate API to get typed, probabilistic decisions your code can branch on.
Read the guide