Quickstart: Train a support agent

Train your first LoRA with Serverless RL by teaching a customer support agent to search a knowledge base and answer product questions.

This quickstart shows you how to train your first LoRA adapter with Serverless Training. You’ll use Serverless RL to train a customer support agent that searches a product knowledge base and answers customer questions, then you’ll send inference requests to the model you trained.

The agent in this quickstart supports a fictional smart thermostat. The same pattern applies to any multi-turn agentic task where the model uses tools to gather context before answering: customer support, agentic RAG, deep research, or internal help desks. You don’t need labeled answers or a hand-written reward function. Instead, RULER (Relative Universal LLM-Elicited Rewards) uses an LLM judge to rank the agent’s attempts against each other, and Serverless RL uses those rankings to update a LoRA adapter for the base model.

What you’ll learn

This quickstart shows you how to:

• Register a trainable model with the W&B training cluster.
• Define a tool-using rollout for a support agent.
• Score the agent’s attempts with RULER instead of a hand-written reward function.
• Run an RL training loop that produces LoRA checkpoints.
• Send inference requests to your trained model.

Serverless Training provisions and autoscales the GPUs for you: inference requests during rollouts run on Serverless Inference, and training steps run on managed training GPUs. Training is free during the public preview; you pay only for inference usage and artifact storage. For details, see Usage information and limits.

Prerequisites

Before you begin, complete the Serverless Training prerequisites: a W&B account, an API key, and a project.

You also need the following:

• Python 3.10 or later, running in an environment that supports async/await at the top level, such as a Jupyter notebook. To run the code as a script instead, wrap it in a function and call it with asyncio.run().
• An OpenAI API key. RULER uses an LLM as a judge to rank trajectories; this example uses an OpenAI model as the judge. To use a different judge, pass any LiteLLM-supported model to ruler_score_group.

Install the OpenPipe ART framework, the open source RL training client that Serverless Training uses:

pip install openpipe-art

Then set your credentials as environment variables. Replace [YOUR-WANDB-API-KEY] and [YOUR-OPENAI-API-KEY] with your keys:

import os

os.environ["WANDB_API_KEY"] = "[YOUR-WANDB-API-KEY]"    # Used for training, inference, and logging
os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]"  # Used by the RULER judge

Register a trainable model

Declare a trainable model and register it with the ServerlessBackend. The backend sends inference and training requests to the W&B training cluster, which autoscales to match your job’s demand. Registration also sets up metric logging to your W&B project.

import art
from art.serverless.backend import ServerlessBackend

model = art.TrainableModel(
    name="support-agent-001",
    project="support-agent",
    base_model="OpenPipe/Qwen3-14B-Instruct",
)

backend = ServerlessBackend()
await model.register(backend)

base_model must be one of the available models. This example uses a 14B model optimized for building agents with fine-tuning.

Create the agent’s environment

The agent answers questions using a small product knowledge base and a single search tool. This example defines the knowledge base inline so the quickstart is self-contained; in a real application, the tool would query your documentation index, help desk, or vector store.

KNOWLEDGE_BASE = [
    {
        "id": "kb-01",
        "title": "Pairing the thermostat with the mobile app",
        "content": "To pair the thermostat, open the mobile app, tap Add Device, and hold the thermostat's center button for five seconds until the display shows a pairing code. Enter the code in the app. Pairing requires Bluetooth and a 2.4 GHz Wi-Fi network.",
    },
    {
        "id": "kb-02",
        "title": "Blank screen after installation",
        "content": "A blank screen usually means the thermostat isn't receiving power. Confirm the C-wire (common wire) is connected at both the thermostat base and the HVAC control board. If your system has no C-wire, install the included power adapter.",
    },
    {
        "id": "kb-03",
        "title": "Wi-Fi disconnects and offline status",
        "content": "If the thermostat shows as offline, confirm your router broadcasts a 2.4 GHz network; the thermostat doesn't support 5 GHz-only networks. Move the router within 10 meters if the signal indicator shows one bar, then restart the thermostat from Settings > Restart.",
    },
    {
        "id": "kb-04",
        "title": "Battery and power adapter",
        "content": "The thermostat has a backup battery that keeps settings for 48 hours during a power outage. The battery charges from the C-wire or power adapter. A battery icon on the display means the thermostat is running on backup power and will shut down soon.",
    },
    {
        "id": "kb-05",
        "title": "Creating temperature schedules",
        "content": "Create schedules in the app under Schedule > New. Each schedule supports up to eight temperature changes per day. Schedules sync to the thermostat within one minute. Manual temperature changes override the schedule until the next scheduled change.",
    },
    {
        "id": "kb-06",
        "title": "Temperature reading seems wrong",
        "content": "If the displayed temperature differs from the room temperature, check that the thermostat isn't in direct sunlight or near a heat source. You can apply a calibration offset of up to ±5 degrees in Settings > Temperature Offset.",
    },
    {
        "id": "kb-07",
        "title": "Warranty and returns",
        "content": "The thermostat includes a two-year limited warranty covering manufacturing defects. Returns are accepted within 60 days of purchase with proof of purchase. Warranty claims are handled through the app under Support > Start a Claim.",
    },
    {
        "id": "kb-08",
        "title": "Updating firmware",
        "content": "Firmware updates install automatically overnight when the thermostat is connected to Wi-Fi. To update manually, go to Settings > About > Check for Updates. The display shows a progress bar during updates; don't cut power while an update is in progress.",
    },
    {
        "id": "kb-09",
        "title": "Vacation mode",
        "content": "Vacation mode holds an energy-saving temperature while you're away and resumes your normal schedule when you return. Enable it in the app under Modes > Vacation and set a start and end date. Geofencing can end vacation mode automatically when your phone arrives home.",
    },
    {
        "id": "kb-10",
        "title": "Compatible HVAC systems",
        "content": "The thermostat supports most 24V heating and cooling systems, including gas, oil, electric, forced air, and heat pumps with auxiliary heat. It doesn't support high-voltage (120/240V) baseboard heaters. Use the compatibility checker in the app before installing.",
    },
]


def search_kb(keywords: list[str]) -> list[dict]:
    """Return knowledge base articles that match any of the given keywords."""
    results = []
    for article in KNOWLEDGE_BASE:
        text = f"{article['title']} {article['content']}".lower()
        if any(keyword.lower() in text for keyword in keywords):
            results.append(article)
    return results


SEARCH_TOOL = {
    "type": "function",
    "function": {
        "name": "search_kb",
        "description": "Search the product knowledge base for articles matching any of the given keywords.",
        "parameters": {
            "type": "object",
            "properties": {
                "keywords": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "Keywords to search for.",
                }
            },
            "required": ["keywords"],
        },
    },
}

Next, define the questions the agent trains on. RULER compares the agent’s attempts against each other, so the questions don’t need labeled answers:

TRAINING_QUESTIONS = [
    "My thermostat's screen is completely blank after I installed it. What should I check?",
    "How do I pair the thermostat with my phone?",
    "The app says my thermostat is offline but my other devices are fine on Wi-Fi.",
    "Can I make the thermostat follow a different temperature at night?",
    "The thermostat says it's 75 degrees but my room thermometer says 71. Can I fix that?",
    "How long does the battery last if the power goes out?",
    "I'm going out of town for two weeks. How do I keep my energy bill down?",
    "Does this work with my baseboard heaters?",
    "How do I update the thermostat's software?",
    "My thermostat is six months old and the display flickers. Is this covered?",
    "I changed the temperature by hand. Why did it change back an hour later?",
    "Do I need a C-wire to install the thermostat?",
]

HELD_OUT_QUESTION = "I set up vacation mode but I'm coming home early. Will the house be cold when I arrive?"

Define a rollout

A rollout is one complete attempt by the agent to handle a scenario. The rollout function sends the question to the model, executes any tool calls it makes, and returns the full interaction as an art.Trajectory. During training, Serverless RL runs many rollouts in parallel against your model’s current LoRA weights.

import json

from openai import AsyncOpenAI

MAX_TURNS = 5

SYSTEM_PROMPT = (
    "You are a customer support agent for a smart thermostat company. "
    "Use the search_kb tool to find relevant knowledge base articles before answering. "
    "Answer the customer's question accurately and concisely based on the articles you find, "
    "and include the steps the customer should take. If the knowledge base doesn't cover the "
    "question, say so instead of guessing."
)


async def rollout(model: art.Model, question: str) -> art.Trajectory:
    trajectory = art.Trajectory(
        messages_and_choices=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
        tools=[SEARCH_TOOL],
        reward=0.0,
    )

    client = AsyncOpenAI(
        base_url=model.inference_base_url,
        api_key=model.inference_api_key,
    )

    for _ in range(MAX_TURNS):
        response = await client.chat.completions.create(
            model=model.get_inference_name(),
            messages=trajectory.messages(),
            tools=trajectory.tools,
            temperature=1,
        )
        choice = response.choices[0]
        trajectory.messages_and_choices.append(choice)

        # No tool calls means the agent gave its final answer
        if not choice.message.tool_calls:
            break

        for tool_call in choice.message.tool_calls:
            arguments = json.loads(tool_call.function.arguments)
            results = search_kb(**arguments)
            trajectory.messages_and_choices.append(
                {
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "name": tool_call.function.name,
                    "content": json.dumps(results),
                }
            )

    return trajectory

A few details matter here:

• The AsyncOpenAI client points at your model’s Serverless Training inference endpoint (model.inference_base_url), so every rollout uses the latest LoRA weights as training progresses.
• temperature=1 keeps the agent’s attempts diverse. RULER needs variation within a group of attempts to produce a useful ranking.
• The trajectory records the entire interaction, including tool calls and tool results, so training can credit the behavior that led to a good answer.

Score trajectories with RULER

Instead of writing a reward function, pass each group of trajectories to RULER. An LLM judge compares the attempts for the same question against each other and scores them from 0 to 1 based on how well each one achieves the goal described in the system prompt. Because the RL algorithm normalizes scores within each group, only the relative ranking matters.

Verify that RULER works before you start the training loop:

from art.rewards import ruler_score_group

test_group = art.TrajectoryGroup(
    [await rollout(model, TRAINING_QUESTIONS[0]) for _ in range(3)]
)

judged_group = await ruler_score_group(test_group, "openai/gpt-5.4", debug=True)

for trajectory in sorted(judged_group.trajectories, key=lambda t: t.reward, reverse=True):
    print(f"Score: {trajectory.reward:.3f}")
    print(f"Answer: {trajectory.messages()[-1]['content'][:120]}\n")

The debug=True flag prints the judge’s reasoning so you can confirm the rankings make sense for your task before you train on them.

Run the training loop

Each training step gathers a batch of trajectory groups, scores them with RULER, and sends the scored trajectories to the W&B training cluster, which updates your LoRA adapter and saves a checkpoint. Metrics for each step log to your W&B project automatically.

from art.rewards import ruler_score_group
from art.utils import iterate_dataset

GROUPS_PER_STEP = 4        # Questions per training step
TRAJECTORIES_PER_GROUP = 4 # Attempts per question, compared by RULER
NUM_EPOCHS = 2             # Passes over the training questions
LEARNING_RATE = 1e-5

training_iterator = iterate_dataset(
    TRAINING_QUESTIONS,
    groups_per_step=GROUPS_PER_STEP,
    num_epochs=NUM_EPOCHS,
    initial_step=await model.get_step(),
)

for batch in training_iterator:
    print(f"Training step {batch.step} (epoch {batch.epoch})")

    groups = [
        art.TrajectoryGroup(
            rollout(model, question) for _ in range(TRAJECTORIES_PER_GROUP)
        )
        for question in batch.items
    ]

    finished_groups = await art.gather_trajectory_groups(
        groups,
        pbar_desc="gather",
        max_exceptions=TRAJECTORIES_PER_GROUP * len(batch.items),
    )

    judged_groups = [
        await ruler_score_group(group, "openai/gpt-5.4")
        for group in finished_groups
    ]

    result = await backend.train(
        model,
        judged_groups,
        learning_rate=LEARNING_RATE,
    )

    await model.log(
        judged_groups,
        metrics=result.metrics,
        step=result.step,
        split="train",
    )

    print(f"Completed training step {result.step}")

This configuration runs six training steps. While the loop runs, open your support-agent project at wandb.ai to watch reward and training metrics update in real time. Each step also saves a LoRA checkpoint as an artifact in your project.

To trace every message in each rollout, including the judge’s rankings, add Weave to your project: call weave.init(model.project) before the training loop and decorate rollout with @weave.op. Weave records each step of the training loop so you can inspect exactly how your agent’s behavior changes during training.

Use your trained model

Compare the agent’s behavior on the held-out question. Your model’s endpoint continues to serve the latest trained weights, so the same rollout function now runs against your trained LoRA:

result = await rollout(model, HELD_OUT_QUESTION)
print(result.messages()[-1]["content"])

Every checkpoint is also deployed automatically for inference, so you can call your trained model from any OpenAI-compatible client using a wandb-artifact model reference and the Serverless Training API:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.training.wandb.ai/v1",
    api_key=os.environ["WANDB_API_KEY"],
)

response = client.chat.completions.create(
    model="wandb-artifact:///[YOUR-ENTITY]/support-agent/support-agent-001:step6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": HELD_OUT_QUESTION},
    ],
)
print(response.choices[0].message.content)

Replace [YOUR-ENTITY] with your W&B entity (team) name, and choose the checkpoint step you want to serve. To see the checkpoints your training run produced, open the Artifacts tab of your project. For details on constructing endpoints for trained models, see Use your trained models.

Next steps

You trained a LoRA adapter with Serverless RL and served it without managing any infrastructure. To keep going:

• Scale up the task: Point the search tool at your real documentation or help desk data, expand the question set, and increase the number of epochs and validation checks. The ART·E notebook shows a larger version of this pattern, including periodic validation and checkpoint cleanup.
• Warm up with SFT: If you have curated example conversations, fine-tune on them first with Serverless SFT, then apply RL for further refinement.
• Manage storage: Each checkpoint counts toward your artifact storage. Delete low-performing checkpoints with the ART SDK.
• Go deeper on ART: See the ART documentation for advanced training configuration, validation splits, and additional examples.