Send events using the OpenTelemetry collector

Imagine that you have a business for which you want to analyze web traffic. This analysis would help you better serve customers and manage inventory. You have a system that collects web logs, but you want to improve the process for more efficient storage and analysis of the data.

This tutorial shows you how to send data to Imply Lumi using the Splunk® distribution of the OpenTelemetry (OTel) collector. You'll learn how to create an IAM key with HEC-specific attributes and how to configure an OTel collector to forward logs to Imply Lumi.

The following diagram summarizes the end-to-end process of sending events to Imply Lumi using the OTel collector. Shaded boxes represent steps taken within Imply Lumi, and unshaded boxes represent steps taken outside Imply Lumi. Click any box in the diagram to jump to that step.

Prerequisites

To complete the tutorial, you need the following:

Access to Imply Lumi with the Data manager or higher role. For more information on roles and permissions, see Manage roles.
OTel collector distribution with the Splunk extension. You can use one of the following:
- Splunk distribution of the OTel collector. For installation instructions, refer to the Splunk documentation.
- Contributor distribution of the OTel collector. Download the distribution from the OpenTelemetry collector release page. Click the link to show all assets to find the distribution for your operating system. Be sure to download the contributor distribution, prefixed by otelcol-contrib. Note that the core distribution doesn't include the filelog receiver nor the splunk_hec exporter.

If you're already using the OTel collector to send data to Splunk, you can follow along using your existing setup. You may need to restart the collector after making modifications for Imply Lumi.

1. Create an IAM key

In this section, you create an IAM key and set global and HEC-specific attributes.

In the Imply Lumi navigation menu, click Integrations > HEC.
On the Splunk HEC integration page, click View instructions.
Click Create. If there are existing IAM keys, open the IAM key drop-down menu to select Create.
Enter the IAM key information.
Note that only the Name field is required to create the IAM key.
- Name: tutorial-otel
- Description: IAM key used for the OTel tutorial
Enter the global attributes.
You can view the global attributes env and team on an event, but these are system attributes and can't be filtered on.
- Environment: tutorial
- Team: learning
Enter the HEC attributes.
Imply Lumi enriches events sent through HEC with the HEC attributes. You can view and search events using HEC attributes. See Set HEC attributes for the default values.
- Source: tutorial-otel-file
- Source type: access_combined
- Default index: demo
- Allowed indexes: demo
Click Create.
Leave the HEC page open. You'll copy the IAM key token and endpoint as well as preview incoming data on this page.

2. Configure the OTel collector

In this section, you configure the OTel collector to forward logs to Imply Lumi.

In a text editor, open a file named lumiconfig.yaml. You'll add the following sections in the OTel configuration file:
- receivers: Collects and parses telemetry data from the specified sources.
- processors: Modifies or transforms the collected data, such as by adding metadata or renaming fields.
- exporters: Sends the processed data to the specified destinations.
- service: Enables the previously configured sections.
If you want to skip the details, you can use the full configuration file in the expander at the end of this section.
Copy and paste the following section to add the receiver. The filelog receiver tracks the lumina_visitors.log file, which you'll generate in a later step.
```
receivers:
    filelog:
        include: [ logs/lumina_visitors.log ]
```

Copy and paste the following section to add to the filelog receiver. The example data follows the standardized pattern of the Apache server log combined format. The receiver parses each log using the specified regular expression. For details on the fields that will get parsed, see Example log.

        operators:
          - type: "regex_parser"
            regex: '(?m)^(?P<clientip>[^ ]*) (?P<identd>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<uri>[^\"]*?)(?: +(?P<protocol>\S+))?)?" (?P<status>[^ ]*) (?P<bytes>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<useragent>[^\"]*)")?$'

Copy and paste the following section to add the processors. The collector compresses data into batches and adds a metadata field for region. You also add an attribute myhost which won't be sent with the event but will be mapped to the host attribute by the exporter.

processors:
    batch:
    attributes/add_metadata:
        actions:
            - key: region
              value: "eu-west-1"
              action: insert
            - key: myhost
              value: "192.0.2.1"
              action: insert

Copy and paste the following section to add the exporter. The splunk_hec exporter named logs_lumi takes HEC connection details for Imply Lumi.

The HEC attributes—source, sourcetype, and index—sent with the exporter overwrite the HEC attributes on the IAM key.

The exporter finds the host value from the metadata field myhost, which you added in the processors section.
```
exporters:
    splunk_hec/logs_lumi:
        endpoint: "LUMI_ENDPOINT"
        token: "IAM_KEY_TOKEN"
        source: "logs/lumina_visitors.log"
        sourcetype: "access_combined"
        index: "demo"
        otel_attrs_to_hec_metadata:
          host: "myhost"
```
1. Replace LUMI_ENDPOINT with your Imply Lumi endpoint. Access the endpoint and your token from the HEC integration page.
2. Replace IAM_KEY_TOKEN with the token of your IAM key. For the purposes of this tutorial, you can include the token directly in the configuration file. However, for production deployments, consider more secure ways of storing and reading the token, such as an environment variable.
Copy and paste the following section to add the service section. It adds the preceding components to a logs pipeline to collect, process, and export events. Note that the previously configured components aren't enabled until they're specified in service.
```
service:
    pipelines:
        logs:
            receivers: [ filelog ]
            processors: [ batch, attributes/add_metadata ]
            exporters: [ splunk_hec/logs_lumi ]
```

Your configuration should resemble the following. Save and close the file.

Full configuration file for the OTel collector

receivers:
    filelog:
        include: [ logs/lumina_visitors.log ]
        operators:
          - type: "regex_parser"
            regex: '(?m)^(?P<clientip>[^ ]*) (?P<identd>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<uri>[^\"]*?)(?: +(?P<protocol>\S+))?)?" (?P<status>[^ ]*) (?P<bytes>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<useragent>[^\"]*)")?$'

processors:
    batch:
    attributes/add_metadata:
        actions:
            - key: region
              value: "eu-west-1"
              action: insert
            - key: myhost
              value: "192.0.2.1"
              action: insert

exporters:
    splunk_hec/logs_lumi:
        endpoint: "LUMI_ENDPOINT"
        token: "IAM_KEY_TOKEN"
        source: "logs/lumina_visitors.log"
        sourcetype: "access_combined"
        index: "demo"
        otel_attrs_to_hec_metadata:
          host: "myhost"

service:
    pipelines:
        logs:
            receivers: [ filelog ]
            processors: [ batch, attributes/add_metadata ]
            exporters: [ splunk_hec/logs_lumi ]

Start the OTel collector and provide the custom configuration lumiconfig.yaml.

The way you start the collector depends on your distribution. See the OpenTelemetry docs for details for each operating system and architecture. For example, on MacOS, run the collector in a terminal with the following command:
```
./otelcol-contrib --config=lumiconfig.yaml
```
If you're not able to start the collector successfully, see the troubleshooting section.

3. Send events

In this section, you update the file monitored by the OTel collector to export events in the file to Imply Lumi.

In the same directory that contains lumiconfig.yaml, create a folder called logs.
Download the example data, lumina_visitors.log. Save the file in the logs folder in the location you have the OTel collector and configuration file. For details on events in the log file, see Example data for Imply Lumi tutorials.
Revisit where you have the OTel collector running. It should print information about watching the file:
OTel collector log for successful file monitoring
2025-03-21T16:21:20.087-0700 info fileconsumer/file.go:265 Started watching file {"otelcol.component.id": "filelog", "otelcol.component.kind": "Receiver", "otelcol.signal": "logs", "component": "fileconsumer", "path": "logs/lumina_visitors.log"}
Since you have the collector running, the file that you downloaded should already be received and exported by the collector.

4. Preview data

Return to the HEC integration page in Imply Lumi. In the section to Preview incoming data, you should be able to see the newly added events.
Select Go to explore view. Imply Lumi takes you to the Explore events page and applies search filters for your IAM key and the HEC integration.
Select an event to view the event and its attributes.

The Message section shows the raw event message, a line in the log file.

The System attributes section shows some fields determined by Imply Lumi as well as the global attributes you set on the IAM key.

The User attributes section shows three categories of attributes: HEC attributes such as source, fields the OTel collector parsed such as method, and the metadata attribute the OTel collector added (region).

For information on searching events and filtering on event attributes, see Search for events with Imply Lumi.

Troubleshoot the OTel collector

If your operating system can't verify the identity and security of the OTel collector, it may prevent you from starting the application. Go to your system's security settings and allow use of the collector.

If the collector fails and raises the error error decoding 'exporters': unknown type: "splunk_hec", you may have installed the core distribution, which doesn't have the Splunk-specific components. Revisit the prerequisites to install the contributor distribution or a distribution that includes the Splunk component.

If you receive the warning no files match the configured criteria, that may appear because the file doesn't exist. However, the collector will be able to recognize and monitor the file when you create it.

If you receive the error regex pattern does not match, open and confirm that the downloaded log file isn't corrupt. It should be a text file that contains Apache HTTP server access logs with a single log per line.

Learn more

For more information, see the following topics:

Prerequisites​

1. Create an IAM key​

2. Configure the OTel collector​

3. Send events​

4. Preview data​

Troubleshoot the OTel collector​

Learn more​