Skip to main content

Using a Generator

A generation.Generator is the direct path to a model. It returns one streamed response for one request, and nothing else happens automatically. We already met it in Getting started with the greeting and the weather tool. A Generator never executes tools and never sends a follow-up request. When the model returns tool calls, the application owns the next step.

Starting one request​

generator.Stream opens one request. generation.Request carries the model, the system prompt, the message history, optional tool definitions, and generation.Options.

stream, err := generator.Stream(ctx, generation.Request{
	Model:        model,
	SystemPrompt: "You are a concise technical assistant. Golato is a small Go library for generation, embeddings, and agent runs.",
	Messages: []message.Message{
		message.NewUserMessage("What does a Golato Generator return?"),
	},
	Options: generation.Options{
		Temperature:     new(0.2),
		MaxOutputTokens: new(300),
	},
})
if err != nil {
	return err
}

An error from Stream means that no stream was established. Once you hold a stream, choose one of two ways to consume it.

Taking the finished answer​

Result waits for the complete response and returns it directly. Any queued events are released in this result mode.

answer, err := stream.Result(ctx)
if err != nil {
	fmt.Fprintf(os.Stderr, "partial answer: %s\n", answer.Text())
	return err
}
fmt.Println(answer.Text())

Watching the answer arrive​

Call Next first to read events live. The loop ends at io.EOF once the terminal event has been read, and Result still returns that same terminal message afterward.

for {
	event, err := stream.Next(ctx)
	if errors.Is(err, io.EOF) {
		break
	}
	if err != nil {
		return err
	}
	switch event := event.(type) {
	case generation.TextDeltaEvent:
		fmt.Fprint(os.Stderr, event.Delta)
	case generation.ThinkingDeltaEvent:
		fmt.Fprint(os.Stderr, event.Delta)
	case generation.ToolCallEndEvent:
		fmt.Fprintf(os.Stderr, "\ntool call: %s %s\n", event.ToolCall.Name, event.ToolCall.Arguments)
	}
}

answer, err := stream.Result(ctx)
if err != nil {
	return err
}
fmt.Println(answer.Text())

Deltas are for display only. A provider can deliver complete text in a start or end event without any matching delta, so concatenating deltas does not reliably rebuild the answer. The authoritative answer is what Result returns. Treat deltas as progress for the screen, and treat answer as the answer for everything else, including storage and the next request.

Reading the final message​

Result returns a message.AssistantMessage with the text, tool calls, stop reason, usage, and provider facts.

answer.Text()        // visible text blocks, in order
answer.ToolCalls()   // calls requested by the model, in order
answer.StopReason    // why the response ended
answer.Usage         // complete provider counts, when reported

Building messages and content explains how to read the message and its content blocks.

Handling a failed attempt​

A terminal ErrorEvent keeps the content accepted before the failure. AssistantMessage.Error holds the exact text projection of the live error, so the message and the error never disagree. Classify the live error with errors.Is or errors.As and do not parse the Error string.

A Result wait canceled before a terminal event returns a zero message and the context error. That wait still claims result-only mode and releases the queue, so a later Next returns io.EOF. A producer protocol violation returns generation.ErrInvalidStreamEvent, and a close before a terminal event returns generation.ErrStreamClosedBeforeTerminal. Provider failures keep their own classification. The Configuring the OpenAI-compatible adapter chapter lists the available sentinels and typed errors.

Returning tool calls to the application​

Definitions in generation.Request.Tools tell the model which operations it may request, and a Generator never invokes them. The application reads the requested calls and decides what to do.

for _, call := range answer.ToolCalls() {
	// Validate call.Arguments, execute the operation, and create a result.
}

Calls run only when Result returned err == nil. A failed or aborted response can still contain partial tool call content, and a response with message.StopReasonLength can contain truncated arguments. The loop that executes calls and feeds results back is the host loop from Getting started. See Defining tools for the schema and validation rules.

Configuring the request​

generation.Options holds the small set of generation controls. Pointer fields keep the difference between “not set” and an explicit zero, as the example above shows with Temperature and MaxOutputTokens.

  • MaxOutputTokens, when present, must be positive.
  • ThinkingLevel accepts off, minimal, low, medium, high, xhigh, and max, and an unknown value is an error. off explicitly disables reasoning instead of using the provider default.
  • ToolChoice modes are auto, none, required, and a named function choice.

Canceling safely​

A request has two independent stop switches, and they control different work.

The context passed to generator.Stream controls the provider side. When it is canceled or expires, the HTTP request and the stream reading stop. This context decides how long one model call may live.

ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
stream, err := generator.Stream(ctx, request)

The contexts passed to Result and Next control only the current wait. Canceling one makes that call return a context error, while the stream stays alive and a later Result or Next can still read from it. The two contexts are separate because how long the provider works and how long we are willing to wait are different concerns. A network call can still be running while the user interface has already moved on, and giving up one wait must not kill the stream.

waitCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
answer, err := stream.Result(waitCtx)
if err != nil {
	// The stream is still alive. Call stream.Result(ctx) again when ready.
}

A Next call whose context is already done claims nothing, so a later Next or Result can still choose the consumption mode. A deadline or a cancellable context on every provider request is the safe default, and the shorter wait contexts above are only for the moments where a long wait is the problem.

API reference​