Evaluate issues with Jev using ReqLLM and a plain Elixir GenServer

Evaluate issues with Jev and ReqLLM

Section

This notebook teaches the new ReqLLM.evaluate/4 API. It starts with one model call. It then uses a plain GenServer to manage work. It does not use Jido or a Jev.Server package.

Jev is an evaluation model. It takes one text or JSON state and a map of named questions. It returns answers, not chat text. ReqLLM sends these questions to TypeSafe and returns a normal %ReqLLM.Response{}. Our Elixir code then turns the answers into issue labels.

The notebook has two paths:

• The local examples use fixed answers. You can run them without a key or network call.
• The live examples call Jev. They run only after you add a Livebook secret. Each live cell can make a paid API call.

For learning, not security enforcement. The limits in this notebook are sample policy values. Review security results with a person before you use them to make real decisions.

1. Install the released package

This notebook uses ReqLLM from Hex. It does not use the local ReqLLM source tree.

Mix.install([{:req_llm, "~> 1.24.0"}])

2. Add your TypeSafe key here

In Livebook, open Secrets in the top bar. Add a secret with the exact name TYPESAFE_API_KEY. Paste the key into the secret's Value field. Livebook makes it available to this notebook as LB_TYPESAFE_API_KEY. Do not paste the key into a code cell or save it in this file.

After you add the secret, run the next cell again. It reports only whether the key is ready. It does not display the key.

api_key = System.get_env("LB_TYPESAFE_API_KEY")

if is_binary(api_key) and api_key != "" do
  "TypeSafe key is ready. You can run the live cells."
else
  "No key yet. Add TYPESAFE_API_KEY in Livebook Secrets, then run this cell again."
end

If Livebook asks for permission to use the secret, allow it for this notebook. A reader who opens this file from a gist must add their own key.

3. Define the state and questions

Here, the state is an issue. It is a JSON-compatible map. The questions name the facts we want Jev to evaluate. :choice selects one kind. :boolean asks for the chance that the answer is yes.

defmodule Triage.Questions do
  def questions do
    %{
      kind: %{
        type: :choice,
        instructions: "Which kind best describes this software issue?",
        criteria: %{
          bug: "Existing behavior is broken or gives an error",
          feature: "A request for new behavior or a new option",
          other: "Neither a bug nor a feature request"
        }
      },
      security: %{
        type: :boolean,
        instructions: "Does the issue report a possible security vulnerability?",
        criteria: %{
          true: "It describes a possible exploit, data leak, or access-control failure",
          false: "It does not describe a possible vulnerability"
        }
      }
    }
  end
end

issue = %{
  id: "ISSUE-101",
  title: "Login fails after password reset",
  body: "The reset link works, but sign-in returns HTTP 500."
}

Triage.Questions.questions()

The model spec is "typesafe:jev-latest". The typesafe: part selects the ReqLLM provider. ReqLLM sends jev-latest to TypeSafe. The latest alias can move, so use a versioned Jev model when you need repeatable results.

4. Make one live evaluation

This cell does nothing until the secret exists. It keeps the full result in live_result, but it displays only a small, safe view. The request has a time limit and one retry for a transient failure.

live_result =
  if is_binary(api_key) and api_key != "" do
    ReqLLM.evaluate(
      "typesafe:jev-latest",
      issue,
      Triage.Questions.questions(),
      api_key: api_key,
      receive_timeout: 30_000,
      total_timeout: 60_000,
      max_retries: 1
    )
  else
    {:skipped, :missing_key}
  end

case live_result do
  {:ok, response} ->
    %{
      response_type: response.__struct__,
      model: response.model,
      kind: response.object["kind"],
      security: response.object["security"],
      usage: response.usage
    }

  {:error, %ReqLLM.Error.API.Request{status: status}} ->
    {:error, {:http_status, status}}

  {:error, reason} ->
    kind = if is_map(reason), do: Map.get(reason, :__struct__, :other), else: :other
    {:error, {:evaluation_failed, kind}}

  {:skipped, :missing_key} ->
    "No network call. Add the Livebook secret, then run this cell again."
end

The result is {:ok, %ReqLLM.Response{}} or {:error, reason}. It is not a separate EvaluationResponse. Answers are in response.object, with string keys even though our input question names are atoms. The choice answer has "choice", "probabilities", and "confidence". The yes/no answer has "probability" from 0 to 1, not a Boolean value.

The next cell shows the field paths. It displays no real data until you make a live call.

case live_result do
  {:ok, response} ->
    %{
      chosen_kind: response.object["kind"]["choice"],
      kind_probabilities: response.object["kind"]["probabilities"],
      kind_confidence: response.object["kind"]["confidence"],
      security_probability: response.object["security"]["probability"],
      input_tokens: response.usage.input_tokens,
      raw_security_answer:
        response.provider_meta.raw_response["answers"]["security"]
    }

  _ ->
    "Run the live evaluation after you add the key."
end

TypeSafe calls its yes/no type noul. ReqLLM accepts :boolean and returns "type" => "boolean" with a "probability" field. The original noul answer remains in provider_meta.raw_response. The raw response can contain data you do not want to share. Use toy issues in a public notebook.

Jev can also score an ordered rubric. This cell builds a :score question, but makes no network call:

score_questions = %{
  impact: %{
    type: :score,
    instructions: "How much does this issue affect normal use?",
    criteria: ["Small inconvenience", "Important function is impaired", "Service is unusable"]
  }
}

score_questions

ReqLLM.generate_object/4 is a different operation. It asks a text model to generate an object that follows a schema. Jev answers named evaluation questions; it does not use a chat or arbitrary object-generation endpoint. Later evaluation models can use ReqLLM.evaluate/4 too, but their question and answer types can differ.

5. Keep the decision rule in Elixir

The model supplies answers. This pure function owns the routing rule. It checks the security answer first. It maps only known kind strings to atoms; it never creates an atom from model output.

defmodule Triage.Policy do
  @kinds %{"bug" => :bug, "feature" => :feature, "other" => :other}

  def decide(answers) when is_map(answers) do
    kind = answer(answers, "kind")
    security = answer(answers, "security")
    security_probability = Map.get(security, "probability")

    cond do
      valid_probability?(security_probability) and security_probability > 0.5 ->
        [:security]

      true ->
        case Map.fetch(@kinds, Map.get(kind, "choice")) do
          {:ok, label} ->
            confidence = Map.get(kind, "confidence")

            if valid_probability?(confidence) and confidence > 0.85 do
              [label]
            else
              [label, :"needs-triage"]
            end

          :error ->
            [:"needs-triage"]
        end
    end
  end

  def decide(_), do: [:"needs-triage"]

  defp answer(answers, name) do
    case Map.get(answers, name) do
      %{} = value -> value
      _ -> %{}
    end
  end

  defp valid_probability?(value),
    do: is_number(value) and value >= 0 and value <= 1
end

These answers are fixed examples, not Jev output. They let us test the rule without an API key:

examples = %{
  security: %{
    "security" => %{"probability" => 0.91},
    "kind" => %{"choice" => "bug", "confidence" => 0.98}
  },
  clear_bug: %{
    "security" => %{"probability" => 0.03},
    "kind" => %{"choice" => "bug", "confidence" => 0.93}
  },
  uncertain_feature: %{
    "security" => %{"probability" => 0.08},
    "kind" => %{"choice" => "feature", "confidence" => 0.62}
  },
  missing_kind: %{"security" => %{"probability" => 0.07}}
}

Map.new(examples, fn {name, answers} -> {name, Triage.Policy.decide(answers)} end)

The 0.5 and 0.85 limits are only examples. Test them with labeled issues before use. A model probability is evidence, not a guarantee. Keep the model ID, answers, and the rule version if you must audit a decision.

6. Put the call in a GenServer

A direct call to ReqLLM.evaluate/4 inside handle_call/3 is easy to read, but it blocks that GenServer while HTTP runs. For a shared server, start one task per issue. The server holds pending callers and sends each reply once the task ends.

The key GenServer rule is: {:reply, value, state} replies now. For a later reply, return {:noreply, state} and then call GenServer.reply(from, value) once. The proposed use Jev.Server and handle_answer/3 syntax is a useful design idea, but it is not part of plain GenServer or ReqLLM.

defmodule Triage.Server do
  use GenServer

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts)

  def triage(server, issue, timeout \\ 70_000),
    do: GenServer.call(server, {:triage, issue}, timeout)

  @impl true
  def init(opts) do
    {:ok,
     %{
       task_supervisor: Keyword.fetch!(opts, :task_supervisor),
       evaluate: Keyword.fetch!(opts, :evaluate),
       max_in_flight: Keyword.get(opts, :max_in_flight, 4),
       pending: %{}
     }}
  end

  @impl true
  def handle_call({:triage, issue}, from, state) do
    if map_size(state.pending) >= state.max_in_flight do
      {:reply, {:error, :busy}, state}
    else
      evaluate = state.evaluate
      task = Task.Supervisor.async_nolink(state.task_supervisor, fn -> evaluate.(issue) end)
      {:noreply, put_in(state.pending[task.ref], from)}
    end
  end

  @impl true
  def handle_info({ref, result}, state) when is_reference(ref) do
    case Map.pop(state.pending, ref) do
      {nil, _pending} ->
        {:noreply, state}

      {from, pending} ->
        Process.demonitor(ref, [:flush])
        GenServer.reply(from, format_result(result))
        {:noreply, %{state | pending: pending}}
    end
  end

  def handle_info({:DOWN, ref, :process, _pid, reason}, state) do
    case Map.pop(state.pending, ref) do
      {nil, _pending} ->
        {:noreply, state}

      {from, pending} ->
        GenServer.reply(from, {:error, {:task_exit, reason}})
        {:noreply, %{state | pending: pending}}
    end
  end

  def handle_info(_message, state), do: {:noreply, state}

  defp format_result({:ok, %ReqLLM.Response{} = response}) do
    {:ok,
     %{
       decision: Triage.Policy.decide(response.object),
       answers: response.object,
       model: response.model,
       usage: response.usage
     }}
  end

  defp format_result({:error, reason}), do: {:error, reason}
  defp format_result(other), do: {:error, {:unexpected_result, other}}
end

The task result arrives as {ref, result} in handle_info/2. A task crash arrives as {:DOWN, ref, :process, pid, reason}. Both paths remove the pending caller. The normal result path also removes the task monitor, so the caller does not get two replies. The max_in_flight limit makes overload visible as {:error, :busy}.

The next cell tests the full GenServer path with a fixed response. It does not call TypeSafe.

fake_evaluate = fn _issue ->
  {:ok,
   %ReqLLM.Response{
     id: "local-test",
     model: "local-fixed-answer",
     context: ReqLLM.Context.new(),
     object: examples.clear_bug,
     usage: %{input_tokens: 0, output_tokens: 0, total_tokens: 0},
     provider_meta: %{operation: :evaluate}
   }}
end

{:ok, task_supervisor} = Task.Supervisor.start_link()

{:ok, server} =
  Triage.Server.start_link(
    task_supervisor: task_supervisor,
    evaluate: fake_evaluate,
    max_in_flight: 4
  )

Triage.Server.triage(server, issue)

When you have added the key, this last cell uses the same server with the real ReqLLM.evaluate/4 call. It is a second paid call. Leave it unrun if the first live call gave you enough data.

if is_binary(api_key) and api_key != "" do
  {:ok, live_task_supervisor} = Task.Supervisor.start_link()

  evaluate = fn current_issue ->
    ReqLLM.evaluate(
      "typesafe:jev-latest",
      current_issue,
      Triage.Questions.questions(),
      api_key: api_key,
      receive_timeout: 30_000,
      total_timeout: 60_000,
      max_retries: 1
    )
  end

  {:ok, live_server} =
    Triage.Server.start_link(
      task_supervisor: live_task_supervisor,
      evaluate: evaluate,
      max_in_flight: 4
    )

  case Triage.Server.triage(live_server, issue) do
    {:ok, %{decision: decision, answers: answers, model: model, usage: usage}} ->
      %{decision: decision, answers: answers, model: model, usage: usage}

    {:error, %ReqLLM.Error.API.Request{status: status}} ->
      {:error, {:http_status, status}}

    {:error, reason} ->
      kind = if is_map(reason), do: Map.get(reason, :__struct__, :other), else: :other
      {:error, {:evaluation_failed, kind}}
  end
else
  "No network call. Add the Livebook secret, then run this cell again."
end

ReqLLM.evaluate/4 returns errors as {:error, reason}. ReqLLM already retries transient transport errors and HTTP 429 and 529 up to max_retries; it can use a Retry-After value when present. Do not match %Jev.Error{} in this code. For a service, set a request time limit, a GenServer.call/3 limit that is longer, and a clear response for failure. If the caller times out, the server still needs a cleanup rule; this small example does not cancel the task for you.

7. Other ways to use a GenServer

The deferred reply above fits a normal triage(issue) call. These other designs may fit different systems:

A future wrapper could make the named questions and answer callback look like handle_answer/3. Keep it small: let ReqLLM make the model call, let a pure function make the decision, and let GenServer own work and replies. No Jido is needed.

What to test next

Try toy bug, feature, and security reports. Compare the answers with labels supplied by people. In particular, check security false negatives. Add a :score question only when you need an ordered scale. For repeatable tests, choose a versioned Jev model instead of jev-latest.

This file is safe to share only after you review saved outputs. Do not share a TypeSafe key, a .env file, real issue text, or raw provider data from a private issue.

Further reading: ReqLLM, TypeSafe API, Task.Supervisor, Livebook secrets, and Livebook dev endpoints.