Skip to main content

Transform events with pipelines

Pipelines are channels through which events enter for processing before being stored in Imply Lumi. Use pipelines to transform incoming events.

When you create a pipeline, you set the conditions to filter events that enter the pipeline. Events that satisfy the condition go through the pipeline, and events that don't satisfy the condition get stored without transformation.

In the context of the event model, pipelines operate on an event's user attributes. User attributes are metadata that provide more information to events, such as its source or error code. You can use a pipeline to extract text from the event message and store it as a user attribute. You can also create or remove user attributes.

In this topic, you'll learn about the components of a pipeline and how to create and manage pipelines.

Processors​

Pipelines contain processors, which perform the work to transform events. Processors can perform field extractions using regular expressions as well as add or remove attributes.

The following behavior applies to processors:

  • Each processor is unique to a pipeline. That is, you can't reuse a processor in multiple pipelines.
  • Processors operate on an event in numerical order. When one processor adds an attribute to an event, a subsequent processor can refer to the attribute.
  • If processing fails for any reason, the event continues without changes to the next processor if applicable. Otherwise Lumi stores the event as is.

The following diagram shows an example pipeline that does the following:

  1. Extracts text from the event message using a regular expression and assigns it to a user attribute named realm.
  2. Renames the incoming metadata field named status to http_status.
  3. Deletes the userid and status event metadata.

Example pipeline

To follow along interactively with this example, see the pipelines tutorial.

For reference and examples of all processor types, see Processors.

Create and update pipelines​

This section walks you through the process of creating and managing pipelines.

Prerequisites​

To create and manage pipelines in Imply Lumi, you need the Data manager role or higher. For information on roles and permissions, see Manage roles.

Create a pipeline​

To process events in a pipeline, you create the pipeline then add processors to the pipeline. Once you create new pipelines and processors, Imply Lumi enables them by default. Disabled elements appear dimmed in the pipelines list.

To create a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Click + Create pipeline.
  3. Enter pipeline details:
    • Position: Place in the sequential order.
    • Name: Name to identify the pipeline.
    • Expression: Query that qualifies events for the pipeline. For examples, see Pipeline conditions.
  4. Click Create.

Create a processor​

Add functionality to a pipeline by creating one or more processors. To create a processor in a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Select a pipeline, then click Create processor.
  3. Select the position and processor type, and enter a name for the processor.
  4. Fill in the processing rules. The available fields depend on the processor type.
    For examples, see Processors. For guidance on how to specify attributes, see Attribute names.
  5. Click Create.

Update a pipeline​

You can update a pipeline's position, name, and expression. To update a pipeline:

  1. Click Pipelines from the navigation menu.
  2. For the pipeline you want to update, click the ellipsis and select Edit.
  3. Edit the pipeline's properties as desired and click Save.

From the pipeline ellipsis menu, you can also reorder its position, enable or disable it, add processors, or delete it.

Update a processor​

You can update the position, type, and name of processors in a pipeline. To update a processor:

  1. Click Pipelines from the navigation menu.
  2. For the processor you want to update, click the ellipsis and select Edit.
  3. Edit the processor's properties as desired and click Save.

From the processor ellipsis menu, you can also reorder its position, enable or disable it, or delete it.

Syntax for pipelines and processors​

This section describes the syntax for pipeline filters and referring to attributes in processors.

Pipeline conditions​

Each pipeline requires a filter expression to identify events to process. If an event doesn't satisfy the conditions for any pipelines, Imply Lumi stores it without processing. Use Imply Lumi query syntax to define filters on incoming event metadata or system attributes.

The following examples are valid event conditions:

# search events destined for the main index
index=main

# search events coming from an OTel collector with a specific source type
source=otel AND sourcetype=access_combined

For an event that satisfies multiple pipelines, Imply Lumi processes the event through each pipeline in numerical order. If you add an attribute in one pipeline, a subsequent pipeline can use that attribute.

The following syntax behavior applies to event conditions:

  • Use AND or OR to join multiple search criteria.
  • A standalone string searches for event messages that contain the string.
  • For system attributes such as env, preface the name with #.
  • You can't use the equality operator = with the event message or timestamp.

For example, the following expression filters events containing hello world in the prod environment:

"hello world" AND #env=prod

Attribute names​

Use the following guidelines when defining input and output attributes on a processor:

  • Refer to incoming event metadata by name, such as clientip.
  • The regex parser uses the event message as the default input. Leave the input field blank to use the default. Only the regex parser can refer to the event message.
  • You can't use system attributes as input or output attributes.

Learn more​

See the following topics for more information: