Skip to main content

Send events to Splunk & Imply Lumi using S3

You can send events to Imply Lumi and Splunk® using an ingest action for routing to S3 in Splunk. This protocol routes events to an S3-compatible Imply Lumi endpoint. Use this integration when you send events to Splunk using a heavy forwarder in your own environment (Splunk Enterprise).

For other approaches to send events to Imply Lumi, see Send events.

This topic provides details on configuring event forwarding using S3 ingest actions.

Prerequisites​

To send events to Splunk and Imply Lumi using S3 ingest actions, you need the following:

  • An Imply Lumi user with the Data manager role or higher. For information on roles and permissions, see Manage roles.
  • An Imply Lumi IAM key. See Create an IAM key for details.

When sending events with S3 ingest actions, the IAM key authenticates connections to send events to Imply Lumi. If you assign any global attributes to the key, Imply Lumi enriches incoming events with those values.

For reference information on IAM key attributes, see IAM keys.

Configure event forwarding​

To forward events to Imply Lumi, access your Splunk instance to create an S3 remote storage destination and a ruleset to route to the destination.

Before configuring event forwarding, access the S3 ingest actions integration page in Imply Lumi. Select your IAM key. The page populates with the required to authenticate the connection. Specifically, you need the following details from Imply Lumi:

  • S3 bucket name: Suggested bucket name resembling imply-f590a036-3590-4a6f-b5a5-da8d37c53463
  • S3 endpoint: Imply Lumi endpoint resembling https://splunk-s3.us1.api.lumi.imply.io
  • Access key ID: IAM key ID in UUID format
  • Secret access key: IAM key token in UUID format
Unique bucket name

Splunk may require you to have unique bucket names for S3 destinations. Imply Lumi generates a new bucket name with the format imply- followed by a randomly generated UUID. The S3 bucket name changes every time you access the S3 ingest actions integration page. You're not required to use the suggested bucket name, but ensure that your S3 bucket name is unique.

This section describes how to set up S3 ingest actions either using the Splunk configuration file outputs.conf or in the Splunk Enterprise UI.

Splunk configuration file​

Configure ingest actions directly on the heavy forwarder by editing outputs.conf. You create both the destination and the ruleset in the forwarder settings. If you have multiple heavy forwarders, configure the ingest actions on each one.

  1. Update or add the remote file system (RFS) stanza in outputs.conf to include the following settings:
[rfs:logs_lumi]
path = s3://LUMI_BUCKET/
remote.s3.endpoint = LUMI_ENDPOINT
remote.s3.access_key = IAM_KEY_ID
remote.s3.secret_key = IAM_KEY_TOKEN

  1. Replace the following values provided by Lumi:

    • LUMI_BUCKET: S3 bucket name suggested by Lumi
    • LUMI_ENDPOINT: Lumi S3 endpoint
    • IAM_KEY_ID: IAM key ID
    • IAM_KEY_TOKEN: IAM key token

  2. Configure any other output RFS properties based on your specific setup and requirements.

  3. Restart your forwarder for the changes to take effect.

To view Splunk recommendations and requirements, see Heavy forwarders managed through a deployment server.

Splunk Enterprise UI​

In the Splunk Enterprise UI, follow the steps to create an S3 destination and create a ruleset to route to the destination.

Create an S3 destination​

Create a remote storage destination for S3 in Splunk. After you create or select an IAM key, Imply Lumi populates the details in Configure event forwarding with your specific IAM key ID and token.

In Splunk, go to Settings > Ingest actions > Destinations > New destination > S3.

  1. Enter the following fields:
    • S3 bucket name: S3 bucket name suggested by Imply Lumi
    • S3 endpoint: Endpoint provided by Imply Lumi
  2. Complete all other fields based on your specific setup and requirements.
  3. Click Next.
  4. Enter the authentication details for Imply Lumi:
    • Authentication method: Access key and secret key
    • Access key ID: IAM key ID
    • Secret access key: IAM key token
  5. Click Test connection to validate the connection.
    A valid connection returns the message, "Successfully connected to the server."
  6. Click Save.
Splunk Cloud

If you don't see options for "S3 endpoint" or "Authentication method," you may be on Splunk Cloud. Please contact Imply support for assistance.

Route events using a ruleset​

Create a ruleset in Splunk that routes events to Imply Lumi at ingest time. The ruleset references the S3 destination you created in the previous section.

  1. While still on the Ingest actions page in Splunk, go to Ruleset > New ruleset.
  2. Under Event stream, set sourcetype to the type that matches your data.
    For all other fields, use the default settings.
  3. Click Add rule > Route to destination.
  4. In the field for Immediately send to, enter the name of the S3 destination created in the previous step.
  5. Click Apply.
  6. Click Save.
  7. Restart your heavy forwarder for the changes to take effect.

Check Imply Lumi for events​

Once you configure event forwarding and send events, you can preview the incoming data in Imply Lumi:

  1. Click Integrations in the navigation menu.

  2. Click S3 ingest actions.

  3. Click View instructions.

  4. In Select or create an IAM key, select your key.

  5. In Preview incoming data, you'll see the events coming in to Imply Lumi. Imply Lumi automatically refreshes the preview pane to display the latest events.

  6. Click Go to explore view to see more events associated with the IAM key. The Explore view populates the search bar with your IAM key ID and the S3 receiver type. For example:

    #iamKeyId=3e99daf3-8266-4017-9323-1aa2b41d62ba #receiver=splunk.s3

    Adjust the time filter to choose the range of data displayed.

Once events start flowing into Imply Lumi, you can search them. See Search for events with Imply Lumi for details and information on supported search syntax.

Learn more​

See the following topics for more information:

For information on ingest actions in Splunk, refer to the following Splunk documentation: