How to send events with S3

This tutorial shows you how to send data to Imply Lumi using Splunk® and its S3 ingest action.

In the tutorial, you set up the S3 ingest action in Imply Lumi, configure your Splunk instance to connect with the S3 ingest endpoint, apply parsing settings, and forward a sample log file for ingestion. You then verify that Imply Lumi successfully receives and parses data, and review how data appears after ingestion with parsing rules applied.

See Tutorial data for a description of the sample log file.

The following diagram summarizes the end-to-end process of sending events to Imply Lumi using Splunk S3. Shaded boxes represent steps taken within Imply Lumi, and unshaded boxes represent steps taken outside Imply Lumi. Click any box in the diagram to jump to that step.

For more information on the S3 integration in Imply Lumi, see Send events with S3.

Prerequisites

To send events to Splunk and Imply Lumi using S3 ingest actions, you need the following:

• An Imply Lumi user with the Data manager role or higher. For information on roles and permissions, see Manage roles.
• A Splunk heavy forwarder configured in your environment for routing events via S3 ingest actions. Download a trial and learn more in the Splunk forwarder documentation. For more information on Splunk forwarders, refer to the Splunk documentation.

1. Create an IAM key

In this section, you create an IAM key and set event parsing attributes on the key.

1. From the Imply Lumi navigation menu, click Integrations > S3 ingest actions > View instructions.

   

2. In the top section, click or select Create in the dropdown.

   

3. Enter the following information in the Create IAM key dialog:

   • Name: tutorial-s3
     Note that only the Name field is required to create the IAM key.
   • Description: IAM key for S3 tutorial
   • Environment: tutorial
   • Team: learning

   When you provide Environment and Team, events sent to Imply Lumi contain env and team as system attributes. Note that these values don't overwrite env and team if they are already present in raw events. See system attributes for more information.

   

4. Click Create.

5. Keep the Instructions page open, as you'll need it to configure the S3 destination in the Create an S3 destination.

   

2. Create an S3 destination

In this section, you create a remote S3 destination in Splunk Enterprise to forward events to Imply Lumi.

1. In the Splunk Enterprise UI, go to Settings > Ingest actions > Destinations > New destination > S3.

2. Enter the following details and leave all other fields unchanged:

   • S3 destination title: Name the destination as lumi-s3-tutorial.
   • S3 bucket name: Use the bucket name generated by Imply Lumi in the Create an IAM key section.
   • S3 endpoint: Use the endpoint provided by Imply Lumi in the Create an IAM key section.

3. Click Next.

4. Configure authentication with the following details and leave all other fields unchanged:

   • Authentication method: Select Access key and secret key.
   • Access key ID: Enter the IAM key ID from Imply Lumi that you created in the Create an IAM key step.
   • Secret access key: Enter the IAM key token from Imply Lumi that you created in the Create an IAM key step.

5. Click Test connection. A valid connection returns the message "Successfully connected to the server."

6. Click Save to finish creating the destination.

7. The newly created destination is now listed in your S3 destinations table.

3. Create an S3 ruleset

In this section, you create a ruleset in Splunk to route events to Imply Lumi. The ruleset uses the S3 destination you created earlier.

1. In Splunk, go to Settings > Ingest actions > Rulesets > New ruleset.

2. Enter the following details:

   • Name: lumi-s3-tutorial
   • Description: set ruleset to send events to lumi through s3

3. Under Event stream, set the sourcetype to access_combined. This is the source type for Apache logs you will upload in the next section. Leave all other fields unchanged.

4. Click Add rule > Route to destination.

5. In the Immediately send to field, select the name of the S3 destination you created in the Create an S3 destination step.

6. Click Apply and then Save.

7. The newly created ruleset is now listed in your S3 ruleset table.

4. Send the data

In this section, you upload a sample log file to Splunk Enterprise to test event routing to Imply Lumi.

1. Download and save the example data file lumina_visitors.log.

2. In Splunk, go to Settings > Add Data > Upload.

3. Click Select File, then choose the lumina_visitors.log file you downloaded.

4. Click Next.

5. On the source type screen, set Source type to access_combined. This matches the source type defined in your ingest action ruleset.

6. Leave all other settings unchanged and click Next.

7. On the Review screen confirm the Title is lumina_visitors.log and Source type is access_combined.

8. Click Submit.

9. Successful upload returns the message: “File has been uploaded successfully.”

5. Preview the data

In this section, you preview the data you sent to Imply Lumi from your S3 integration and view the events in the Explore view.

1. Return to the S3 integration page in Imply Lumi: click Integrations > S3 ingest actions > View instructions. In the Preview incoming data section, view the newly added events.

   

2. Select Explore events. Imply Lumi takes you to the Explore view and applies search filters for your IAM key and the S3 integration. You may need to adjust the time range selector and refresh the page to see the events.

   

3. Select an event to view its attributes. For details on the attributes, see Event model.

   

For information on searching events and filtering on event attributes, see Search for events with Imply Lumi.

Learn more

For more information, see the following topics:

• Send events with S3 for details on the S3 ingest action integration.
• Send events to Imply Lumi for other options to send events.
• How to search for events with Imply Lumi to walk through a set of example queries using Imply Lumi search syntax.