How can I install Splunk Enterprise and Splunk Forwarder into Bright 8.2?

How can I install Splunk Enterprise and Splunk Forwarder on Bright 8.2?

By following the procedure outlined here:

Installing Splunk-Forwarder and building a new software image:

The method used here is to use a working regular node, with splunk-forwarder installed and configured on it. This will then be used to create a new software image using the grabimage command.

The following Splunk documentation is used as a guide for creating the image:

https://docs.splunk.com/Documentation/Forwarder/7.2.5/Forwarder/Makeauniversalforwarderpartofahostimage

The steps:

1. Use a working/clean node -- node001 in our case -- to install and configure Splunk-forwarder.
   
   

2. Download the Splunk-Forwarder from:
   https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/linux
   
   

3. Install it on the regular node:
   [root@node001 ~]# yum localinstall splunkforwarder-7.2.5.1-962d9a8e1586-linux-2.6-x86_64.rpm

Start the fowarder for the first time, accept the license and create an admin account:
[root@node001 ~]# cd /opt/splunkforwarder/bin/
[root@node001 bin]# ./splunk start --accept-license
This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.

Create credentials for the administrator account.

Characters do not appear on the screen when you type in credentials.

Please enter an administrator username:

WARN: You entered nothing, using the default 'admin' username.

Password must contain at least:

  * 8 total printable ASCII character(s).

Please enter a new password:

Please confirm new password:

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...

Checking mgmt port [8089]: open

Creating: /opt/splunkforwarder/var/lib/splunk

Creating: /opt/splunkforwarder/var/run/splunk

Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n

Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css

Creating: /opt/splunkforwarder/var/run/splunk/upload

Creating: /opt/splunkforwarder/var/spool/splunk

Creating: /opt/splunkforwarder/var/spool/dirmoncache

Creating: /opt/splunkforwarder/var/lib/splunk/authDb

Creating: /opt/splunkforwarder/var/lib/splunk/hashDb

New certs have been generated in '/opt/splunkforwarder/etc/auth'.

Checking conf files for problems...

Done

Checking default conf files for edits...

Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.2.5.1-962d9a8e1586-linux-2.6-x86_64-manifest'

All installed files intact.

Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...  

Done

                                                  [ OK ]

4. Enable auto-start
   [root@node001 bin]# ./splunk enable boot-start
   Init script installed at /etc/systemd/system/.
   Init script is configured to run at boot.
   
   

5. Configure the forwarder to send the logs to your Splunk indexer:
   [root@node001 bin]# ./splunk add forward-server 10.141.255.254:9997 -auth admin:<YOUR SPLUNK-FORWARDER PASSWORD CONFIGURED IN SETP 2>
   Added forwarding to: 10.141.255.254:9997.
   
   

6. Add a monitor(s) that you want. These are monitored by the forwarder, and forwarded to the indexer:
   [root@node001 bin]# ./splunk add monitor /var/log/maillog
   Added monitor of '/var/log/maillog'.
   
   

7. Make sure everything is configured correctly
   [root@node001 bin]# ./splunk list monitor

Monitored Directories:

$SPLUNK_HOME/var/log/splunk

/opt/splunkforwarder/var/log/splunk/audit.log

[...]

Monitored Files:

$SPLUNK_HOME/etc/splunk.version

/var/log/maillog

[root@node001 bin]# ./splunk list forward-server

Active forwards:

None

Configured but inactive forwards:

10.141.255.254:9997

8. Stop the forwarder:
   [root@node001 bin]# ./splunk stop

Stopping splunkd...

Shutting down.  Please wait, as this may take a few minutes.

...........................................................[  OK ]....................................................

Stopping splunk helpers...

                                                          [ OK ]

Done.

9. Clear the node specific configuration - preparing for grabimage:
   [root@node001 bin]# ./splunk clone-prep-clear-config

Erased key "serverName" from server.conf; contained "node001"

Erased key "guid" from instance.cfg; contained "EB49B792-EF31-4E4E-8D49-C8CBFF12A9AC"

Erased key "host" from inputs.conf; contained "node001"

10. We will clone the original image used for node001, and then use grabimage to rsync the changes only to the new image. This will save us time and network bandwidth, instead of transferring the full image from node001:
    [test14->softwareimage]% clone default-image splunk-image

[test14->softwareimage*[splunk-image*]]% commit

[test14->softwareimage[splunk-image]]%

Tue Apr 16 10:52:15 2019 [notice] test14: Started to copy: /cm/images/default-image -> /cm/images/splunk-image (184)

[test14->softwareimage[splunk-image]]%

Tue Apr 16 10:55:54 2019 [notice] test14: Copied: /cm/images/default-image -> /cm/images/splunk-image (196)

[test14->softwareimage[splunk-image]]%

Tue Apr 16 10:55:54 2019 [notice] test14: Initial ramdisk for image splunk-image is being generated

[test14->softwareimage[splunk-image]]%

Tue Apr 16 10:56:21 2019 [notice] test14: Initial ramdisk for image splunk-image was generated successfully

11. Grab the changes to the new image “splunk-image”
    [test14->softwareimage]% device

[test14->device]% use node001

[test14->device[node001]]% grabimage -w -i splunk-image

[test14->device[node001]]%

Tue Apr 16 10:58:49 2019 [notice] test14: Provisioning started: sending node001:/ to test14:/cm/images/splunk-image, mode GRABNEW, dry run = no

[test14->device[node001]]%

Tue Apr 16 10:59:20 2019 [notice] test14: Provisioning completed: sent node001:/ to test14:/cm/images/splunk-image, mode GRABNEW, dry run = no

grabimage -w -i splunk-image [ COMPLETED ]

12. We will need to set our exclude lists correctly to avoid overwriting node-specific configuration every time it is rebooted. You can set the exclude lists at category level or at node level. We are going to use category level here:
    [test14->softwareimage[splunk-image]]% category

[test14->category]% use default

[test14->category[default]]% set excludelistsyncinstall
[...]
#Splunk forwarder

- /opt/splunkforwarder/var/*

- /opt/splunkforwarder/etc/system/local/*

- /opt/splunkforwarder/etc/*.cfg

- /opt/splunkforwarder/etc/*.conf

- /opt/splunkforwarder/etc/passwd

- /opt/splunkforwarder/etc/auth/*

- /opt/splunkforwarder/etc/myinstall/*

- /opt/splunkforwarder/etc/system/local/*

no-new-files: - /opt/splunkforwarder/ftr

no-new-files: - /usr/share/*splunk*

no-new-files: - /etc/systemd/system/SplunkForwarder.service

no-new-files: - /etc/systemd/system/multi-user.target.wants/SplunkForwarder.service

13. Do the same for excludelistupdate and commit:
    [test14->category[default]]% set excludelistupdate
    [test14->category*[default*]]% commit
    
    
    

14. Set the new software image to be used for your nodes, category level or node level:
    [test14]% category use default

[test14->category[default]]% set softwareimage splunk-image

[test14->category*[default*]]% commit

15. Reboot the nodes to use the new software image with splunk-forwarder installed and configured
    
    

Install Splunk Enterprise on your Bright cluster, if it is not already available in your environment:

We are installing it on the head node, just for testing/demo purposes. Usually you should not install any additional 3rd party software on the head node.

The following Splunk documents can be looked up for details on how to install and configure Splunk:
https://docs.splunk.com/Documentation

1. Register and download Splunk, install on your preferred node:
   [root@test14 ~]# yum localinstall splunk-7.2.5.1-962d9a8e1586-linux-2.6-x86_64.rpm

Start splunk server, accept the license and create the admin account as we did with the forwarder:

[root@test14 ~]# /opt/splunk/bin/splunk start
This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.

Create credentials for the administrator account.

Characters do not appear on the screen when you type in credentials.

Please enter an administrator username:

WARN: You entered nothing, using the default 'admin' username.

Password must contain at least:

  * 8 total printable ASCII character(s).

Please enter a new password:

Please confirm new password:

Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.

Generating RSA private key, 2048 bit long modulus

..........+++++

.......+++++

e is 65537 (0x10001)

writing RSA key

Generating RSA private key, 2048 bit long modulus

....................+++++

..........+++++

e is 65537 (0x10001)

writing RSA key

Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...

Checking http port [8000]: open

Checking mgmt port [8089]: open

Checking appserver port [127.0.0.1:8065]: open

Checking kvstore port [8191]: open

Checking configuration...  Done.

Creating: /opt/splunk/var/lib/splunk

Creating: /opt/splunk/var/run/splunk

Creating: /opt/splunk/var/run/splunk/appserver/i18n

Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css

Creating: /opt/splunk/var/run/splunk/upload

Creating: /opt/splunk/var/spool/splunk

Creating: /opt/splunk/var/spool/dirmoncache

Creating: /opt/splunk/var/lib/splunk/authDb

Creating: /opt/splunk/var/lib/splunk/hashDb

New certs have been generated in '/opt/splunk/etc/auth'.

Checking critical directories...Done

Checking indexes...

Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary

Done

Checking filesystem compatibility...  Done

Checking conf files for problems...

Done

Checking default conf files for edits...

Validating installed files against hashes from '/opt/splunk/splunk-7.2.5.1-962d9a8e1586-linux-2.6-x86_64-manifest'

All installed files intact.

Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...  

Generating a 2048 bit RSA private key

.+++++

.......................+++++

writing new private key to 'privKeySecure.pem'

-----

Signature ok

subject=/CN=test14/O=SplunkUser

Getting CA Private Key

writing RSA key

Done

                                                          [ OK ]

Waiting for web server at http://127.0.0.1:8000 to be available.... Done

If you get stuck, we're here to help.  

Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://test14:8000

2. The admin interface is on port 8000. To allow it through the Shorewall firewall, the following rule is added in “/etc/shorewall/rules” before the last line.
   
   

   ACCEPT   net      fw tcp     8000

    

#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

    The shorewall service is then restarted:

[root@test14 ~]# systemctl restart shorewall

3. Now, Splunk needs to accept forwarded logs on the port that was specified earlier on, when the forwarder “9997” was configured.
   This can be done under the path: Settings > Forwarding and receiving:
   
   

   
   
   
   
   
   

4. The receiving configuration is found under Configure receiving > Add new:
   
   

   
   
   
   
   
   

5. Port 9997 can then be added:
   
   

   
   
   
   
   
   

6. The data forwarded from the node(s) are then seen to be indexed, under Search & Reporting > Data summary: