Record experiment variants in Agent Analytics

The experiments option records which A/B test or feature flag variant an agent ran under. Pass the assignment your flag client already returned, and every [Agent] event in the session carries [Agent] Experiments and, for a single experiment, [Agent] Experiment Variant. Group any chart by that property to compare quality, cost, or abandonment across the arms of a prompt, model, router, or tool test.

Use experiments for the variant that changes agent behavior. Use [Agent] Context for other segmentation dimensions, such as the UI surface, and agentVersion (Node) or agent_version (Python) for releases that aren't randomized.

The experiments option and the helpers on this page require Node @amplitude/ai 0.25.0 or later, or Python amplitude-ai 1.22.0 or later. Amplitude doesn't assign variants: keep your existing flag or experimentation SDK, such as Amplitude Experiment, Statsig, or LaunchDarkly, and pass its result. For the three shapes an agent experiment usually takes, in one agent loop, refer to Run an agent experiment.

Pass the assignment with the AI SDK

Set experiments on the agent, the session, or a single tracking call. Each entry takes a key and a variant, plus an optional provider (the flag system's name) and unit.

typescript
const variant = await flags.getVariant('planner'); // your existing flag client
const agent = ai.agent('support-bot', { userId });

await agent
  .session({ sessionId, experiments: [{ key: 'planner', variant, provider: 'statsig' }] })
  .run(async (s) => {
    // Provider wrappers, tools, and tracking calls in the session all carry the experiment.
  });

The SDK applies the entries in this order:

  • Agent. Applies to every session of the agent. Child agents inherit the parent's experiments and can add or override entries.
  • Session. Overrides an agent entry with the same key.
  • Call. A single agent or session tracking call overrides both, for that event only.

An override replaces the whole entry, including its provider and unit. An entry keeps the position where its key first appeared.

Use the helpers for your flag tool

Three helpers build experiments entries from the flag client the application already calls. None of them sends an exposure event; the flag tool keeps that job.

  • Amplitude Experiment. experimentsFromAmplitude(variants, { keys }) (Node) and experiments_from_amplitude(variants, keys=) (Python) turn the result of fetchV2(), evaluateV2(), fetch_v2(), evaluate_v2(), or all() into entries with provider: "amplitude", skipping fallback variants. keys names the experiment flags, so configuration flags in the same result stay out of the charts. Without keys, the helper records every flag and logs a warning when more than 10 would apply. Numeric flag values become decimal labels such as 3 or 0.25.
  • Any OpenFeature provider. AmplitudeAIHook from @amplitude/ai/openfeature (Node) or amplitude_ai.openfeature (Python) is an OpenFeature hook. Register it once with OpenFeature.addHooks([hook]) or openfeature.api.add_hooks([hook]), and every flag the application evaluates inside a session becomes an experiment on the events that follow, with provider set from the OpenFeature provider's name. flags limits the hook to the flags you list, unit sets the unit, and variantOf (Node) or variant_of (Python) names the arm for object-valued flags whose provider returns no variant, such as Statsig experiments and dynamic configs.
  • An assignment learned mid-session. session.addExperiments(entries) (Node) and session.add_experiments(entries) (Python) record an arm after the session started, for example a router decision. Events tracked from then on carry it, through provider wrappers and tools inside the session and across runAs() or run_as() delegation. An entry with an existing key replaces that entry.
typescript
import { experimentsFromAmplitude } from '@amplitude/ai';
import { AmplitudeAIHook } from '@amplitude/ai/openfeature';

// Amplitude Experiment: name the experiment flags.
const variants = await experiment.fetchV2({ user_id: userId });
const session = agent.session({ sessionId, experiments: experimentsFromAmplitude(variants, { keys: ['planner'] }) });

// OpenFeature: register once at startup.
OpenFeature.addHooks([new AmplitudeAIHook({ flags: ['planner'] })]);

// Learned mid-session.
session.addExperiments([{ key: 'router', variant: 'fast', unit: 'turn' }]);

For a flag tool without a helper, build the entry from the value it returned: { key: 'planner', variant, provider: 'launchdarkly' }. experimentFromEvaluation() (Node) and experiment_from_evaluation() (Python) are the mapping behind the OpenFeature hook, for an SDK that returns evaluation details in a different shape.

Properties on each event

With two or more experiments, the event carries only the array. Every privacy mode sends these properties.

Choose the unit

unit records what your flag client randomized, so analysis compares like with like:

  • agent_session (default). One assignment per session, such as a prompt or model chosen when the conversation starts.
  • turn. A new assignment per user message, such as a router evaluated on each turn.
  • span. A new assignment per operation inside a turn, such as a tool or retrieval strategy. OpenTelemetry evaluations default to span.

Validation and limits

The SDKs, the OpenTelemetry receiver, and the warehouse import templates apply the same rules. Invalid input never throws an error.

  • key, variant, and provider are 1 to 256 characters after trimming ASCII whitespace.
  • A key can't contain :. A variant can, for example openai:gpt-4o.
  • Amplitude drops an entry without a valid key or variant. The SDKs log each dropped entry at debug level.
  • An unknown unit falls back to the default, and an invalid provider is left out.
  • At most 10 experiments apply to an event, counted after the agent, session, and call entries merge.

Keys, variants, and providers aren't redacted. Use flag and variant names, never user data.

Send assignments from OpenTelemetry

Agent Analytics reads OpenTelemetry feature flag evaluations from GenAI spans. Each feature_flag.evaluation span event, and the span's own feature_flag.* attributes, become one experiment on every event the span produces. For the attribute names, refer to Feature flag attributes in the OTLP reference.

Send assignments from the HTTP API or a warehouse import

Events you send yourself carry the same five properties. Set [Agent] Experiments as an array of key:variant strings, and the single-experiment properties only when one experiment applies. Refer to Send agent events without the AI SDK.

The warehouse import templates read optional experiment_key and experiment_variant columns, one experiment per row, with unit agent_session. Refer to Import agent conversations from your data warehouse.

Split a chart by variant

  1. Open any chart built on [Agent] events, such as session cost or abandonment.
  2. Add a group-by on [Agent] Experiments. Each key:variant value of the array becomes its own group, so the chart splits by arm whether one experiment ran or several. When exactly one experiment runs, [Agent] Experiment Variant gives the same split with shorter labels.
  3. Filter on [Agent] Experiments contains planner: to keep one test in view, or on [Agent] Experiment Key when only one experiment ran.

Migrate from [Agent] Context

Earlier guidance put the variant in context, for example context: { experiment_variant: 'treatment' }. Those values keep working in [Agent] Context filters. For new code, pass the assignment as experiments and remove it from context, so each event records the assignment once and charts can group by a dedicated property.

Was this helpful?

For AI agents: a documentation index is available at /docs/llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.

On this page