Skip to main content

Compacting conversation context

A conversation grows until the raw messages approach a provider's input limit. Compaction replaces an older message prefix with one compaction.Summary and may keep a recent suffix available for replay, so the conversation can keep going without losing what happened.

Compaction is stateless. The host application owns both the request and the returned context. It can happen in three ways:

  • the Agent maintains context automatically when the configured threshold is crossed.
  • the Agent recovers after the provider rejects a request for exceeding the context window.
  • the application compacts on demand at any time.

Configuring a Compactor​

A Compactor needs a summarizer and a policy for how much recent conversation to keep.

summarizer := compaction.GeneratorSummarizer{
	Generator: generator,
	Model:     summaryModel,
	Options: generation.Options{
		MaxOutputTokens: new(4000),
	},
}

compactor, err := compaction.New(compaction.Config{
	KeepRecentTokens: 2000,
	TriggerRatio:     0.8,
	Summarizer:       summarizer,
})
if err != nil {
	return err
}

KeepRecentTokens cannot be negative. TriggerRatio defaults to 1.0, must be positive, and may exceed 1.0 to delay maintenance. GeneratorSummarizer.Model must be non-blank. Its default system instruction asks for a structured summary, and a non-blank SystemPrompt replaces it. Summary requests contain no tools and force ToolChoiceNone.

An Agent accepts the compactor through agent.Config.Compactor.

assistant, err := agent.New(agent.Config{
	Generator:           generator,
	Model:               model,
	Compactor:           compactor,
	ContextWindowTokens:  contextWindowTokens,
	ReserveTokens:       4000,
	Estimator:           token.Heuristic{},
})

ContextWindowTokens should be the model's documented context size. The estimator is a planning aid, not a provider contract. The default token.Heuristic is dependency-free, and an application can supply a tokenizer-backed token.Estimator instead.

Maintaining context when the threshold is crossed​

With a positive ContextWindowTokens and an attached Compactor, the Agent plans proactive maintenance before accepting a prompt and after the complete tool loop succeeds. It does not compact between ordinary follow-up turns or in the middle of a tool batch.

estimate request
  → choose a tool-safe cut
  → CompactionStartEvent
  → summarize selected messages
  → replace Summary and retained Messages
  → CompactionEndEvent with Record

The budget comes from a local estimate. The Agent reserves the larger of ReserveTokens and a per-run MaxOutputTokens value, and the estimated input budget is:

max(0, ContextWindowTokens - max(ReserveTokens, MaxOutputTokens))

When a run carries skill invocations, the Agent first withholds their estimated tokens from that budget. TriggerRatio multiplies the budget, and an estimate strictly above the result plans maintenance. An equal estimate does not. KeepRecentTokens selects recent raw messages to retain and does not cap summary length.

The estimate includes the system prompt, projected summary, raw messages, and tool definitions. It guides maintenance, but it never vetoes a request. The provider decides whether the actual request fits.

Maintenance is optional. Estimate failures, planning errors, and no-op plans emit no events. Once a rewrite starts, it emits one CompactionStartEvent and a matching CompactionEndEvent. A non-cancellation rewrite failure leaves the current context unchanged. Before the first request, that request continues, and after a successful tool loop, the answer is preserved. Cancellation after the rewrite starts ends the run.

Recovering when the provider rejects the request​

When a Compactor is attached, a provider error matching provider.ErrContextWindowExceeded makes the Agent compact, require a strict estimate reduction, and retry the request. It permits two overflow recoveries, and a third provider rejection ends the run. No other provider error starts this recovery.

Recovery is required, so its failure behavior differs from maintenance. A planning or summarizer failure ends the run. The overflow attempt that failed stays in NewMessages with an error or abort stop reason, while Final becomes the later successful retry, and the compacted context excludes it. The Storing and continuing conversation state chapter covers what survives a failed run.

A successful rewrite emits one start/end pair and adds one compaction.Record to agent.Result.Compactions. Retries inside one summary generation stay inside that pair, and a rewrite committed before cancellation remains in the returned context.

Compacting on demand​

When the application owns the conversation update, it calls Compact directly. compaction.Request describes the state being compacted and how strictly to compact it: SystemPrompt, Summary, and Messages are the current conversation state, Tools are the definitions used for the token estimate, InputTokenBudget feeds threshold planning, Estimator drives the estimate for that check and for the RequireReduction decrease, and RequireReduction enforces a strict decrease.

result, err := compactor.Compact(ctx, compaction.Request{
	Estimator:        token.Heuristic{},
	SystemPrompt:     state.SystemPrompt,
	Summary:          state.Summary,
	Messages:         state.Messages,
	Tools:            definitions,
	RequireReduction: true,
})
if err != nil {
	return err
}
state.Summary = result.Summary
state.Messages = result.Messages

RequireReduction: true bypasses the configured threshold and requires a strict estimate decrease. The host application stores the returned summary and messages as its updated conversation context. Without it, Compact can return a no-op with the same summary, a cloned message slice, and a nil compaction.Record.

Plan and Apply split the operation into a decision and its execution. Plan is pure decision: it estimates the request, checks the trigger, and picks a safe cut, all without calling the summarizer. Apply executes the decision: it summarizes the selected messages, replaces the summary, trims Messages, and writes the record.

plan, err := compactor.Plan(request)
if err != nil {
	return err
}
// Inspect plan.MessagesToSummarize, plan.TurnPrefixMessages, and plan.FirstKeptIndex here.
result, err := compactor.Apply(ctx, request, plan)
if err != nil {
	return err
}
state.Summary = result.Summary
state.Messages = result.Messages

The split exists because the first step is cheap and side-effect free, while the second calls the model and can fail. A non-nil plan shows exactly what would change: MessagesToSummarize is the history before the cut, TurnPrefixMessages is the beginning of a split turn, and FirstKeptIndex is where the retained suffix starts. Without RequireReduction, Plan returns nil when the threshold is not passed or no raw prefix can be removed. With it, Plan bypasses the threshold and reports compaction.ErrNoReduction when no reducible prefix exists. Apply accepts a nil plan as an unconditional no-op, even with RequireReduction: true, and trusts a non-nil plan rather than planning again. FirstKeptIndex may equal len(request.Messages), which summarizes all raw messages. A valid cut never separates a known tool call from its result, and when it falls inside a turn, the beginning of that turn is summarized separately so the retained suffix can be understood. The Agent itself uses this same sequence internally and only emits CompactionStartEvent once the plan is confirmed.

Apply passes its context to the summarizer but does not check cancellation before or after that callback. A custom summarizer that returns successfully after cancellation can therefore still be committed by a direct Apply call.

Understanding the summary​

An empty compaction.Summary projects no message. A non-empty value projects exactly one user message:

The conversation history before this point was compacted into the following summary:

<summary>
{summary}
</summary>

GeneratorSummarizer receives compact categorical records rather than raw provider JSON. User and assistant visible text stays complete. Images become counts. Failed and aborted assistant attempts are omitted. Thinking, tool arguments, and tool results are truncated to 100 lines or 2 KiB. Thinking keeps the tail, tool arguments keep the head, and tool results keep the tail for bash and errors and the head otherwise. Tool names and result status remain. Tool call IDs, provider metadata, signatures, and image payloads do not. A custom Summarizer receives typed message slices in SummarizeRequest and controls its summary text instead of receiving these serialized records.

After generation, Golato trims whitespace, removes model-produced <read-files> and <modified-files> blocks, and adds its own file lists. It accumulates paths from read, write, and edit calls in the summarized messages and previous summary. Modified paths are listed as modified rather than read, both lists are sorted, and other tools such as bash are ignored. The lists describe requested calls, not successful side effects.

A cut inside a turn adds **Turn Context (split turn):** after the history summary. A successful compaction.Record contains the new summary, FirstKeptIndex, comparable estimates before and after, and optional summarizer usage. Split-turn usage combines both summary calls.

Retrying and handling errors​

GeneratorSummarizer attempts each summary call once and retries up to two times, waiting two seconds and then four. It retries transport errors, every provider *StreamError, and HTTP 408, 409, 429, or 5xx responses. It does not retry cancellation, deadlines, context-window rejection, filtered content, invalid requests or responses, protocol errors, empty summaries, or unknown failures.

When both history and a split turn prefix require model calls, their independent retry budgets can make up to six attempts. If there is no complete history, the built-in summarizer reuses the previous summary or a placeholder and makes only the split-prefix call. An application Summarizer is called once, and any retry policy is its responsibility.

ErrorMeaning
compaction.ErrInvalidConfigNew rejected the configuration
compaction.ErrInvalidRequestrequest, messages, or plan was invalid
compaction.ErrSummarizationa summarizer failed or panicked
compaction.ErrEmptySummarytext was empty after trimming and file-list removal
compaction.ErrNoReductionno removable prefix or required reduction failed

A failed compaction returns no replacement state and no record. The underlying provider or callback error stays classifiable with errors.Is and errors.As.

API reference​