Skip to main content

How to transform events with pipelines

This tutorial shows you how to transform events coming into Imply Lumi using a pipeline. A pipeline is an ordered list of processors that operate on events that meet specific search criteria. A processor performs a distinct task based on its type and processing rules.

In the tutorial, you learn how to create a pipeline and add processors to the pipeline. With the processors, you apply a regular expression to extract attribute values, map one attribute to another, and remove attributes.

After you create the pipeline, you send events to Imply Lumi and compare user attributes with and without pipeline processing.

The following diagram summarizes the end-to-end process of creating pipelines and sending events to Imply Lumi. Click any box in the diagram to jump to that step.

Prerequisites​

To complete the tutorial, you need the following:

  • Access to Imply Lumi with the Data manager or higher role. For more information on roles and permissions, see Manage roles.

  • An application that can send HTTP requests, such as a Unix terminal or Postman. This tutorial shows example requests using the command-line tool curl.

1. Create a pipeline​

Create a pipeline that will process all events that meet the condition of the pipeline.

  1. In the Imply Lumi navigation menu, click Pipelines. Pipelines page
  2. Click + Create pipeline.
  3. Enter pipeline details:
    • Position: 1
    • Name: Authentication events
    • Expression: source=curl AND status=4* Create pipeline
  4. Click Create.

2. Add processors​

Add processors, which are components of a pipeline that perform individual processing tasks.

Regex parser​

First, add a regex parser to extract the name of the realm from the event message.

  1. In the Pipelines page, click the ellipsis next to the pipeline. Enable pipeline
  2. Select Create processor.
  3. Enter processor details:
    • Position: 1
    • Type: Regex parser
    • Name: Extract realm
    • Input attribute: Leave this field blank to parse the event message.
    • Regular expression: realm\s\[(\w+)\]
      This regular expression looks for the format realm [NAME] and extracts the value NAME.
    • Output attributes: realm
      The processor stores the value in a user attribute named realm. Create regex parser
  4. Scroll down to try out the regular expression. In Sample, enter the following event: 
    Service token grant failed for realm [demo], status [401], response [Invalid API key]
  5. Press Tab to move to the next field. The Expected output now shows the result demo. This value populates the realm user attribute when the event message is as listed.

Attribute mapper​

Next, add an attribute mapper to rename status to http_status.

  1. In the Pipelines page, click the ellipsis next to the pipeline.
  2. Select Create processor.
  3. Enter processor details:
    • Position: 2
    • Type: Attribute mapper
    • Name: Rename status
    • Input attribute: status
    • Output attribute: http_status
  4. Leave the override toggle unselected.
  5. Click Create.

Attribute remover​

Finally, add an attribute remover to remove userid and status. In this scenario, you remove userid to redact personal information associated with the event, and you remove status since you now store its information in http_status.

  1. In the Pipelines page, click the ellipsis next to the pipeline.
  2. Select Create processor.
  3. Enter processor details:
    • Position: 3
    • Type: Attribute remover
    • Name: Redact and pare back
    • Input attribute: userid, status
  4. Click Create.

3. Send an event​

Send an event to the HTTP event collector that will get processed by the pipeline.

  1. In the Imply Lumi navigation menu, click Integrations > HEC.
  2. Click Add HEC integration.
  3. Click Create. If there are existing IAM keys, open the IAM key drop-down menu to select Create.
  4. For the IAM key, enter the name tutorial-pipelines. Leave all other fields empty.
  5. Click Create.
  6. In the pane for Configure event forwarding, select the curl tab. curl pane for HEC
  7. Copy and paste the command into a text editor.
  8. Replace the example JSON data with the following object: 
    {"event": "Service token grant failed for realm [demo], status [401], response [Invalid API key]", "source": "curl", "fields": {"userid": "wilma", "status": 401}}
    Your command should look similar to the following: 
    curl -X POST \
         -H "Content-Type: application/json" \
         -H "Authorization: Splunk IAM_KEY_TOKEN" \
         -d '{ "event": "Service token grant failed for realm [demo], status [401], response [Invalid API key]", "source": "curl", "fields": {"userid": "wilma", "status": 401}}' \
     LUMI_ENDPOINT
  9. Copy and paste the command into a terminal and submit the request.

4. View the processed event​

  1. On the HEC integration page, confirm that you see the new event in the Preview incoming data pane.
  2. Click Go to explore view.
  3. Adjust the time filter if needed, and click the event you sent.
  4. Notice the user attributes for realm and http_status. Event with processing If you disable the pipeline, the user attributes won't contain realm but will have userid and status. Event without processing

Explore further​

To explore more with pipelines, try the following activities:

  • Send an event that already includes the http_status attribute. In the attribute mapper, view how the result changes when you toggle the override setting on or off.
  • Disable a processor in the pipeline and compare the results.
  • Create a second pipeline with the same conditions and note the order of operations.

Learn more​

See the following topics for more information: