Splunk Universal Forwarder Explained: A Comprehensive Guide

For individuals new to Splunk, a common question arises: "How do I get my data into Splunk?" The Splunk Universal Forwarder offers a streamlined solution. This article provides a comprehensive overview of Splunk Universal Forwarders, covering their purpose, benefits, functionality, and configuration.

Introduction to Splunk Forwarders

The story of universal forwarders starts with a simple purpose: getting data into Splunk. Splunk is a powerful data analytics and monitoring platform used to harness valuable insights from machine-generated data. The Splunk platform, whether Splunk Cloud Platform or Splunk Enterprise on-premises or cloud-deployed, can index and monitor all IT data, including streaming, machine, and historical data.

To get data into Splunk, forwarders are used. Forwarders are lightweight installations of Splunk that live on your production servers and forward data in your Splunk instance for indexing. Actually, there are four ways to get data in: Universal Forwarders, Heavy Forwarders, Light Forwarders, and direct input.

Understanding the Universal Forwarder

The Universal Forwarder is a Splunk instance designed for installation on nearly any operating system (OS). Once installed, it collects systems data and forwards it to Splunk Indexers. The Universal Forwarder can also be configured to send data to other forwarders or third-party systems.

A universal forwarder contains only the components required for forwarding data, nothing more, nothing less. A heavy forwarder is a full Splunk Enterprise instance that can index, search, change and forward data. A light forwarder is also a full Splunk Enterprise instance, with even more features disabled to achieve as small a resource footprint as possible. A universal forwarder collects data from a variety of places - whether data sources or other forwarders - and then sends it to a forwarder or a Splunk deployment.

Benefits of Using Universal Forwarders

The universal forwarder is the best option when it comes to forwarding data to Indexers. The universal forwarder is a dedicated, streamlined version of Splunk Enterprise that contains only the essential components needed to forward data. The primary benefits of universal forwarders include reliability, security and broad platform support.

Scalability and Resource Efficiency

Perhaps the biggest benefit is the scalability of our universal forwarders. Universal Forwarders use significantly fewer resources than other Splunk products. You can install literally thousands of them without impacting network performance and cost.

Data Forwarding

For many use cases, plenty of data comes directly from files and directories. It’s best to use universal and heavy forwarders to monitor and send those to Splunk Cloud Platform. With Splunk Enterprise, you’re on-premises, so you can get your data into the instance directly or you can opt for universal and heavy forwarders.

Limitations

The universal forwarder does have one significant limitation: it forwards only unparsed data.

Universal Forwarder vs. Heavy Forwarder

Universal Forwarders are more commonly utilized in most environments; Heavy Forwarders are used for specific use cases. There are some instances where the format of the data might not be very pretty or even readable, or the data contains Personally Identifiable Information (PII), credit card information, etc. which needs to be masked or omitted-this is where the Heavy Forwarder comes in. A Heavy Forwarder can be configured to parse and perform transformative changes on the data BEFORE it is forwarded to Splunk indexers or another destination.

Downloading the Universal Forwarder

The Universal Forwarder can be downloaded two ways, and both involve logging into Splunk.com.

Once logged into Splunk.com, hover over the Products tab at the top of the page and click on “Free Trials & Downloads”.
From the downloads page, scroll down toward the bottom until you see the “Download Now” link for the Splunk Universal Forwarder and click it.

It is on this page that you will be presented with a variety of choices for which OS you wish to install your Universal Forwarder package on. The Universal Forwarder can be installed on a wide variety of platforms such as Windows, Linux, Mac OS, Free BSD, Solaris, and AIX. This is where you get to choose how you want to download your Universal Forwarder package. Clicking the download button also loads a new page, it is here where you will have the option to copy a wget command (my preferred method) and download the install package directly to your system or any other system that has wget installed.

Installation on Linux

Since I chose the tarball download, there is an additional step that needs to be done before installing and that is creating the Splunk user. Now that we have downloaded the Universal Forwarder, we need to extract the archive file to the “/opt” directory. Next, we will start Splunk for the very first time and accept the license agreement so that we don’t get bombarded with a wall of text. NOTE there is a message that you will get about an invalid stanza, this is currently a known issue with the 9.0.1 Universal Forwarder.

Installation on Windows

Install a Windows universal forwarder using an installer or the command line. Use the installer for larger deployments and the command line for smaller deployments. Before you begin, see the universal forwarder deployment prerequisites.

Download the Splunk universal forwarder from splunk.com. On the Certificate Information page, select Next as a best practice. By default the universal forwarder is installed with a least-privileged user. Create credentials for your administrator account. The default username is "Admin" and you can check Generate a password to automatically create a password. Select Install. The installer runs and displays the Installation Completed dialog box.

Read also: HBCU Academic Scholarship

When installing version 9.1.0 and higher of the universal forwarder with the command line, the default account on the domain controllers is the local system. If the USEVIRTUALACCOUNT or LOGON_USERNAME flags is enabled, then the GROUPPERFORMANCEMONITORUSERS flag must be 0, otherwise the installation fails.

Follow the prompts on screen to complete the installation. Always create a password for the Splunk admin user.

Post-Installation Considerations

If you have enabled Windows to automatically run scripts, Splunk installation might fail if the autorun script fails. In some circumstances, the Microsoft installer might present a reboot prompt during the uninstall process.

Configuring the Universal Forwarder

Now that we have the Universal Forwarder installed, it’s time to configure it. The Universal Forwarder does not have a graphical user interface, but you can interact with it through the command line or REST endpoints. The Universal Forwarder has two main files that need to be configured for it to collect and forward data, inputs.conf and outputs.conf. If you will be utilizing a Deployment Server to manage your Universal Forwarders, you will also need to configure a deploymentclient.conf file that tells the Universal Forwarder where to ‘phone home’ to retreive the appropriate Splunk apps and any other configuration information. We will assume this is a simple, no Deployment Server installation for this article.

Inputs.conf: Configuring Data Collection

Inputs.conf is where you configure the Universal Forwarder to collect data. For this tutorial, we will add a monitor stanza for /var/log/messages. But first, we will need to create an inputs.conf within /opt/splunkforwarder/etc/system/local. Now you may be wondering why we are creating the file in this location and why one already exists in /opt/splunkforwarder/etc/system/default. As a rule of thumb and best practice, you should never modify files within the default directory. These files exist to provide default settings and can help you identify certain settings that you may need to adjust, and this directory will be overwritten during upgrades so any changes you might make in the .conf files in the default directory would be lost. So now we will create an inputs.conf in the local directory and add the monitor stanza. You can use your text editor of choice for this task. Underneath the stanza we will apply two additional settings, one for the index our events will be sent to and one for enabling the input.

Outputs.conf: Configuring Data Forwarding

Next up we will create an outputs.conf in the same directory and configure our forwarder to forward data to two indexers. Here you will specify the IP address of the Indexers that you want to forward data to and the port that you want that data to be forwarded over.

With our settings applied, we now must restart the forwarder for our changes to be committed to disk.

Configuring a Receiving Indexer

The first step is to set up our receiver or indexer. We login using an admin account on the Splunk Enterprise instance that will be indexing the data. From the Settings menu, we select “Forwarding and receiving”. Next, we select “Add New” next to “Configure receiving”. Here we specify which TCP port the indexer should listen on. In this example, we are going to use port 9997. On the following screen, we see our settings were successfully saved. We could also use the “Splunk enable listen” comment via the CLI or edit the configuration file manually to enable the port.

Now that we have our receiving indexer configured, we will set up a universal forwarder to send data.

Configuring the Universal Forwarder to Send Data

We will be installing a universal forwarder on an Apache web server, but you can forward almost any type of machine data to Splunk.

To download the universal forwarder, we go to the Splunk.com home page and select “Free Splunk”. In the top right corner, we are asked to create an account, or login to an existing one. From the download page, we select the “Download Now” button for the Splunk Universal Forwarder. We select the OS and version for our web server and click “Download Now”

Splunk also provides an option to use wget if you choose. For this demo, we’ve already logged in and uploaded the files to our web server.

Splunk can live in any accessible directory. In this demo we’ll install in the “opt” directory, so let’s untar the archive there. We then navigate to the “bin” directory inside the “Splunk forwarder” folder. The bin directory is where we can run the executable command “Splunk”. We start the forwarder for the first time using “Splunk Start” command. We want to automatically accept the license, so we add the optional “accept-license” argument. We’re prompted to set up an admin account with a password and confirm it. To keep data from being lost, a universal forwarder should start whenever the server reboots, so we use the “enable boot-start” command next. And give “exit” command.

Now we configure the universal forwarder to send data to our receiving indexer. We use the “add forward-server” command with arguments of the address and port of our receiving indexer. We are prompted for the username and password of the admin account we created during installation. Next, we need to tell the universal forwarder what data to send to the indexer. The “add monitor” command is used to tell the universal forwarder what machine data you’d like to send. Here, we are telling the forwarder to send all logs located in the www1 folder to Splunk for indexing. Note: If you want to send mule log here you just send to the mule log folder through this command.

Returning to the receiving indexer from the Search and Reporting app, we click on “Data Summary” to verify the data is being indexed on our Splunk Enterprise instance. Now that we have the data indexed, we can start building reports, visualizing our data, and so much more.

Security Considerations

Avoid running the Universal Forwarder as a Local System account or domain user unless absolutely necessary, as these accounts grant elevated permissions that may exceed what's required for log collection. Where possible, use a dedicated service account with the minimum necessary privileges. Since local user groups are not available on the domain controller, the GROUPPERFORMANCEMONITORUSERS flag is unavailable, which might affect Windows Management Instrumentation and performance monitor inputs.

Least privilege mode is enabled to read any file on Windows version 9.1.0 and later. Since the universal forwarder is able to read far more files than before, the forwarder consumes more resources such as CPU, memory, and disk input/output. After installation, you can remove the SeBackupPrivilege capability from the Windows local security policy. If you do not want your least-privileged user to be able to modify Security Event Logs, do not grant the SeSecurityPrivilege privilege. Instead, update the EventlogReaders group with a user that has permissions to run the universal forwarder. If a universal forwarder is running on Domain Controllers, the "EventLogReaders" group is not available by Windows design because there is no local user or group on the domain controller.

Since the universal forwarder user is not added to the local admin group by default, you might experience permission issues, particularly if you have installed any custom add-ons that require additional permissions. Add to specific groups based on the required permission. Add the user to some local or global user groups.

Troubleshooting

Note: Version 9.1.0 and higher does not work with version 3 of the Splunk-to-Splunk protocol. Upgrade all of your instances if possible, but if you must use the old version of the Splunk-to-Splunk protocol, see the Troubleshooting guide.

NOTE there is a message that you will get about an invalid stanza, this is currently a known issue with the 9.0.1 Universal Forwarder.

tags: #splunk #universal #forwarder #explained