How to send events with Splunk S2S

Imagine that you work for a company that runs a web store that sells to customers and businesses. You want to analyze web traffic to your store to better serve customers and manage inventory. Your system collects web logs, but you want to improve how you store and analyze data.

To improve your existing process, you incorporate Imply Lumi as the storage and compute layer of your observability ecosystem, which will reduce data storage costs and improve data accessibility with faster searches.

In this tutorial, you learn how to send data to Imply Lumi using Splunk® and its Splunk-to-Splunk (S2S) protocol, a TCP-based data transmission protocol. You learn how to configure your Splunk instance to access the TCP endpoint in Imply Lumi, apply settings to parse your events, and send a log file to Imply Lumi. Having a basic understanding of event forwarding in Splunk is helpful but not required to complete this tutorial.

For more information on the S2S integration in Imply Lumi, see Send events with S2S. For a description of the sample log file, see Example data for tutorials.

The following diagram summarizes the end-to-end process of sending events to Imply Lumi using Splunk S2S. Shaded boxes represent steps taken within Imply Lumi, and unshaded boxes represent steps taken outside Imply Lumi. Click any box in the diagram to jump to that step.

For more information on the S2S integration in Imply Lumi, see Send events with Splunk S2S.

Prerequisites

To complete the tutorial, you need the following:

• Access to Imply Lumi with the Data manager role or higher.
  For more information on roles and permissions, see Manage roles.
• Port 9997 available as an outbound TCP port on the machine forwarding the events.
• A Splunk universal or heavy forwarder.
  If you don't already have access to one, see the Splunk website for a free trial installation of the universal forwarder or heavy forwarder (a full Splunk Enterprise instance). For more information on the Splunk forwarders, refer to the Splunk documentation.

1. Create an IAM key

In this section, you create an IAM key and set event parsing attributes on the key.

1. From the Imply Lumi navigation menu, click Integrations > S2S. Then, click View instructions at the top.

   

2. Open the dialog to create an IAM key. If there aren't any IAM keys, click Create. Otherwise, open the IAM key drop-down menu to select Create.

3. Enter the following information in the Create IAM key dialog:

   • Name: tutorial-s2s
     Only the Name field is required to create the IAM key.
   • Description: IAM key for S2S tutorial
   • Environment: tutorial
   • Team: learning

   Events sent to Imply Lumi contain env and team as system attributes.

   

4. For the S2S attributes section, enter the following details:

   • Time prefix: [\w\.:]*\s[\w-]*\s[\w-]*\s\[
     Regex pattern representing the string that precedes the timestamp.
   • Max timestamp lookahead: 20
     Number of characters into the line in which to search for the timestamp.
   • Time format: %d/%b/%Y:%H:%M:%S
     Format of the timestamp.

   These settings apply to the sample events for this tutorial. The S2S attributes aren't stored with the events. For more details on the attributes and these example values, see Event parsing for S2S events.

   

5. Click Create.

6. In the Configure event forwarding section, view the configuration for Splunk outputs.conf. This configuration is specific to your IAM key and Imply Lumi environment. You copy this configuration into your Splunk file in the next section.

   

2. Add the Imply Lumi receiver in Splunk outputs

In this section, you add Imply Lumi as a receiver on the Splunk forwarder. You edit the Splunk configuration file outputs.conf to create a tcpout processor with Imply Lumi details. See the Splunk documentation for additional information on configuring receivers in Splunk.

1. In your terminal, navigate to the directory that contains the CLI of your Splunk installation: $SPLUNK_HOME on Unix-based machines or %SPLUNK_HOME% on Windows.

   info

   The SPLUNK_HOME variable is set at installation time and specifies the path where Splunk is installed. For example, /Applications/SplunkForwarder on a Mac OS or C:\Program Files\Splunk on Windows.

2. In a text editor, open a new file called outputs.conf inside the directory etc/system/local/.

3. Copy and paste the Imply Lumi receiver configuration into the file. Your file should resemble the following:

   [tcpout]
   defaultGroup = logs_lumi
   
   [tcpout:logs_lumi]
   server = LUMI_ENDPOINT
   token = IAM_KEY_TOKEN
   useSSL = true
   useClientSSLCompression = false

   Ensure your values for LUMI_ENDPOINT and IAM_KEY_TOKEN represent your specific endpoint and token. For details on these settings, see Send events with Splunk S2S.

4. Save and close the file. You restart Splunk in a later step, and Splunk encrypts the token value upon restart.

3. Define processing rules in Splunk props

In this section, you define processing rules for the Splunk forwarder in the configuration file props.conf. See the Splunk documentation for additional information on processing rules in Splunk.

1. In a text editor, open a new file called props.conf inside the directory etc/system/local/.

2. Copy and paste the following content into the file:

   TIME_PREFIX = [\w\.:]*\s[\w-]*\s[\w-]*\s\[
   MAX_TIMESTAMP_LOOKAHEAD = 20
   TIME_FORMAT = %d/%b/%Y:%H:%M:%S
   SHOULD_LINEMERGE = false

   The first three properties describe timestamp extraction. These values match the timestamp parsing properties you assigned on the IAM key.

   The rule SHOULD_LINEMERGE = false directs Splunk to reach each line of data as its own event. Otherwise, by default, Splunk comes several lines of data into a single multi-line event.

3. Save and close the file.

4. Specify the source of events in Splunk inputs

In this section, you direct the forwarder to an input source of log data. You edit the Splunk configuration file inputs.conf to create a batch input with the log file details. See the Splunk documentation for additional information on input sources in Splunk.

1. In a text editor, open a new file called inputs.conf inside the directory etc/system/local/.

2. Copy and paste the following content into the file:

   sourcetype = access_comed
   
   [batch:///Applications/SplunkForwarder/lumina_visitors.log]
   move_policy = sinkhole
   
   [blacklist:/Applications/SplunkForwarder/var]

   Note the following stanzas in this configuration:

   • You set the sourcetype of the file to the recognized type access_comed. Refer to the Splunk documentation for a list of source types that Splunk automatically recognizes.

   • You use the batch input to effectively upload a file through the forwarder. This is a destructive method to send events; Splunk deletes the file after reading it. If you have a file you want to continuously monitor, use the monitor input instead.

   • With the default forwarder settings, the forwarder sends internal Splunk log files such as metrics.log. The blacklist stanza in this example prevents those log files from being sent. If you prefer to monitor these files, remove the blacklist stanza. You can filter specific source files in the Explore view.

3. Start the Splunk forwarder:

   .//splunk start

   If the forwarder is already running, be sure to restart it: .//splunk restart.

4. To send the data, download the example data, lumina_visitors.log and save it in your SPLUNK_HOME directory. Splunk automatically reads the data and removes the file.

5. Preview data

In this section, you preview the data you sent to Imply Lumi and view the events in the Explore view.

1. In Imply Lumi, return to the Splunk S2S integration page. In the Preview incoming data section, view the newly added events.

   

2. Select Explore events. Imply Lumi takes you to the Explore view and applies search filters for your IAM key and the S2S integration. You may need to edit the time range or refresh the page to see the events.

   

3. Select an event to view the event and its attributes. For details on the attributes, see Event model.

   

For information on searching events and filtering on event attributes, see Search for events with Imply Lumi.

Learn more

For more information, see the following topics:

• Send events with Splunk S2S for details on the S2S integration.
• Event parsing for Splunk S2S events for reference on parsing S2S events.
• Send events to Imply Lumi for other options to send events.