Task logs

A task's log comes from one of two places: a copy core stored when the task finished, or the dispatcher still holding the task. This page is for operators setting up both, and Loki as an optional live source.

Before you start

• A dispatcher connected to core. See Run the dispatcher.
• For stored logs, the artifact store turned on. See Connect the artifact store.
• To change settings, a platform admin who can open Admin › Settings.

Where does a task's log come from?

Every read goes through core, GET /api/v2/taskrun/{id}/log, which is what a task's View Log in the webapp calls. Core checks that the reader may see the run, and scrubs the run's secret values from the log as it relays it.

While a task runs, the live route answers even when a copy is stored. A copy stored during this attempt answers instead only when the dispatcher registered no live route, or its route fails before answering. A requeued task never shows its earlier attempt's log.

Store finished logs

When a task ends, Flow's Kubernetes dispatcher reads the pod's log before the Job or TaskRun can be deleted, and uploads it to core. Core keeps it in the artifact store as part of the run. It outlives the pod, needs no dispatcher to read it, and is purged with the run.

A stored log isn't listed with the run's artifacts, can't be read through the artifact routes, and doesn't count toward artifact storage. Settings take effect without a restart.

The dispatcher logs TaskRun (<id>) log stored (<n> bytes). for each upload. When an upload fails, the task's outcome is unchanged:

Follow a running task's log

The dispatcher serves a running task's log at /api/v1/logs/stream, on port 7702. At start, it registers that route's address with core, with a bearer token it generates. The route accepts only that token, so only core can read it: a pod log isn't scrubbed until core relays it.

The earlier properties flow.agent.service.host and flow.agent.logstream.url are gone from core: nothing on core names a dispatcher. Don't route /api/v1/logs from your ingress; core reaches the dispatcher directly.

• The log streams as the task writes it, for up to two hours.
• A quiet task: after 10 minutes with nothing from the dispatcher, core stops following. Open the log again to see anything written since.
• Before the container starts, the log shows Waiting for the task's container to start., then follows the log once it starts. After five minutes it says the container hasn't started yet.
• Several replicas of one dispatcher behind one address can refuse each other's reads. See Run several dispatchers.

When a read fails, the log shows one of these messages instead of an error:

Read live logs from Loki

The dispatcher's live route reads the task's pod by default. Set dispatcher.logging.type=loki and it queries Loki instead. Use it when task pods are removed quickly and no artifact store keeps their logs, since a finished task's stored copy wins over the live route.

dispatcher.logging.type=loki
dispatcher.logging.host=loki.loki.svc.cluster.local
dispatcher.logging.port=3100

The dispatcher calls GET http://<host>:<port>/loki/api/v1/query_range, forward from the start, with this stream selector:

{boomerang_io_taskrun_ref="<task run id>",boomerang_io_workflow_ref="<workflow id>",container=~"task|step-task"}

The upload of a finished log still reads the pod, not Loki.

Map the pod labels in your log shipper

The dispatcher stamps each task pod with boomerang.io/* labels, but Loki doesn't index pod labels by itself. Your log shipper must write each stream with these labels:

The names are what a promtail or Grafana Alloy labelmap on the pod's labels writes, since Kubernetes service discovery turns ., / and - into _. Keep both pod labels and the container name on every task pod's stream.

When no stream matches, the log reads Unable to retrieve logs. This could be due to data retention policy, activity cleanup, or unavailable in log store. Compare the labels on a task's stream in Loki with the table.

The Loki client has limits:

• Plain HTTP, with no authentication and no tenant header. Loki must accept unauthenticated reads from the dispatcher.
• Its own HTTP client, which ignores the proxy and trust store settings. See Loki task logs ignore the HTTP client settings.
• elastic, a v3 value, now answers only Unable to retrieve logs. The log option configured has been deprecated and no longer works.

Retire the v3 promtail configuration

The promtail configuration published for v3 relabels v3's pod labels into bmrg_* names, and drops any pod without them. v5 task pods carry none of them, so it drops every v5 task pod, and the v5 query asks for different names. Remove the v3 scrape job and map the labels in the table above instead. Upgrading lists this step for a v3 upgrade.

Next steps

• Run the dispatcher: executors, the Deletion Policy and task pods.
• Configuring: the artifact store and core's other properties.
• Runs: how long runs, artifacts and logs are kept.