Transform events with pipelines

Pipelines are channels through which events enter for processing before being stored in Imply Lumi. Use pipelines to transform incoming events.

When you create a pipeline, you set the conditions to filter events that enter the pipeline. Events that satisfy the condition go through the pipeline, and events that don't satisfy the condition get stored without transformation.

In the context of the event model, pipelines operate on an event's user attributes. User attributes are metadata that provide more information to events, such as its source or error code. You can use a pipeline to extract text from the event message and store it as a user attribute. You can also create or remove user attributes.

In this topic, you'll learn about the components of a pipeline and how to create and manage pipelines.

Processors

Pipelines contain processors, which perform the work to transform events. Processors can perform field extractions using regular expressions as well as add or remove attributes. Each processor is unique to a pipeline. That is, you can't reuse a processor in multiple pipelines.

Processors operate on events in numerical order. If you add an attribute in one processor, a subsequent processor in the same pipeline can refer to it.

The following diagram shows an example pipeline that does the following:

• Extracts text from the event message (_log) using a regular expression and assigns it to a user attribute named realm.
• Creates a user attribute http_status from the value of the status attribute.
• Deletes the environment and status event attributes.

To follow along interactively with this example, see the pipelines tutorial.

For reference and examples of all processor types, see Processors.

Create and update pipelines

This section walks you through the process of creating and managing pipelines.

Prerequisites

To create and manage pipelines in Imply Lumi, you need the Data manager role or higher. For information on roles and permissions, see Manage roles.

Create a pipeline

To process events in a pipeline, you create the pipeline then add processors to the pipeline. Once you create new pipelines and processors, Imply Lumi enables them by default. Disabled elements appear dimmed in the pipelines list.

To create a pipeline:

1. Click Pipelines from the navigation menu.
2. Click + Create pipeline.
3. Enter pipeline details:
   • Position: Place in the sequential order.
   • Name: Name to identify the pipeline.
   • Expression: Query that qualifies events for the pipeline. For examples, see Event conditions.
4. Click Create.

Create a processor

Add functionality to a pipeline by creating one or more processors. To create a processor in a pipeline:

1. Click Pipelines from the navigation menu.
2. Select a pipeline, then click Create processor.
3. Select the position and processor type, and enter a name for the processor.
4. Fill in the processing rules. The available fields depend on the processor type.
   For examples, see Processors. For guidance on how to specify attributes, see Attribute names.
5. Click Create.

Update a pipeline

You can update a pipeline's position, name, and expression. To update a pipeline:

1. Click Pipelines from the navigation menu.
2. For the pipeline you want to update, click the ellipsis and select Edit.
3. Edit the pipeline's properties as desired and click Save.

From the pipeline ellipsis menu, you can also reorder its position, enable or disable it, add processors, or delete it.

Update a processor

You can update the position, type, and name of processors in a pipeline. To update a processor:

1. Click Pipelines from the navigation menu.
2. For the processor you want to update, click the ellipsis and select Edit.
3. Edit the processor's properties as desired and click Save.

From the processor ellipsis menu, you can also reorder its position, enable or disable it, or delete it.

Syntax for pipelines

This section describes the syntax for pipeline eligibility criteria and for referring to attributes.

Event conditions

For each pipeline, you set the conditions to determine what events enter the pipeline. You define the condition using Imply Lumi query syntax. Use AND to join multiple search criteria.

The following examples are valid event conditions:

• index=main
• environment=production AND source=otel

For events that satisfy multiple pipelines, Imply Lumi processes the events by each pipeline in numerical order. If you add an attribute in one pipeline, a subsequent pipeline can use that attribute.

Attribute names

Use the following guidelines when defining input and output attributes on a processor:

• Refer to event attributes by name, such as clientip.
• Refer to the event message itself using _log.
• You can use system attributes as input attributes, but you can't use them as output attributes.
• If you have a system attribute with the same name as a user attribute, the user attribute takes precedence.

Learn more

See the following topics for more information:

• To learn how to create an example pipeline and compare user attributes with and without pipeline processing, see the pipelines tutorial.
• For a list of processors you can use in pipelines, see Processors.
• For integrations to send events, see Send events.