Advanced Nutanix: Splunk Forwarder Deployment

advNutanix_640

Recently I’ve become a huge fan of Splunk and it’s capabilities and have put together the Splunk on Nutanix Reference Architecture.  In this post I’ll cover how to automatically deploy the Splunk Universal Forwarder on the Nutanix Controller VM to allow Splunk to index and search Nutanix logs.

By deploying Splunk universal forwarders on my Nutanix CVMs I can forward my Nutanix logs over to Splunk and allow myself to do some interesting log and data analysis.  You can also correlate these to release versions, outages and performance runs and get some very interesting data.

Here we show an example of searching Nutanix log data in Splunk:

Splunk_SearchStargate

Here are the steps to deploy the universal forwarder on the CVM:  (NOTE: You can put all of this together into a single script, however, I’ve chosen to keep apart to show the various steps of deployment and to keep things clear)

  1. Download Splunk universal forwarder rpm
  2. SCP forwarder rpm to ~/tmp on a Nutanix CVM where you’ll be running the commands
  3. Copy bits to other CVMs
    • for i in svmips;do scp ~/tmp/<Rpm Name> $i:/home/nutanix/tmp/;done
    • Example: for i in svmips;do scp ~/tmp/splunkforwarder-6.0-178852-linux-2.6-x86_64.rpm $i:/home/nutanix/tmp/;done
  4. Install package (NOTE: This will install to /opt/splunkforwarder)
    • for i in svmips;do ssh $i sudo rpm -i ~/tmp/<Rpm Name>;done
    • Example: for i in svmips;do ssh $i sudo rpm -i ~/tmp/splunkforwarder-6.0-178852-linux-2.6-x86_64.rpm;done
  5. Start Splunk forwarder
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk start –answer-yes –no-prompt –accept-license;done
  6. Enable forwarder to start on boot
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk enable boot-start;done
  7. Add the forward server
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add forward-server <Forward Server>:9997 -auth admin:<Local Password>;done
    • Example: for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add forward-server 10.2.133.196:9997 -auth admin:changeme;done
  8. Add monitors
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/<Log to Monitor> -sourcetype <Log Type>;done
    • Example: for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/stargate.FATAL -sourcetype stargate.FATAL;done

Here are some interesting log files to monitor and feed into Splunk (includes the commands to add the monitor):

  • stargate.FATAL
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/stargate.FATAL -sourcetype stargate.FATAL;done
  • stargate.ERROR
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/stargate.ERROR -sourcetype stargate.ERROR;done
  • stargate.WARNING
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/stargate.WARNING -sourcetype stargate.WARNING;done
  • stargate.INFO
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/stargate.INFO -sourcetype stargate.INFO;done
  • genesis.out
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/genesis.out -sourcetype genesis.out;done
  • zookeper.out
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/zookeeper.out -sourcetype zookeeper.out;done
  • cassandra system log
    • for i in svmips;do ssh $i sudo /opt/splunkforwarder/bin/splunk add monitor /home/nutanix/data/logs/cassandra/system.log -sourcetype cassandra_system;done

Happy Splunking! :)

Comments are closed.

Legal Mumbo Jumbo

Copyright © Steven Poitras, The Nutanix Bible and StevenPoitras.com, 2014. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Steven Poitras and StevenPoitras.com with appropriate and specific direction to the original content.