Tuesday, November 25, 2008

Splunk on FreeBSD 7.0

Although there is not a version of Splunk compiled natively for FreeBSD 7.0, I was told to try using Splunk 3.4.1 on FreeBSD 7.0 via FreeBSD's compat6x libraries.

I did the following:

freebsd70:/usr/local/src# pkg_add -v splunk-3.4.1-45588-freebsd-6.1-intel.tgz
Requested space: 106458852 bytes, free space: 1565927424 bytes in
Running pre-install for splunk-3.4.1-45588-freebsd-6.1-intel..
extract: Package name is splunk-3.4.1-45588-freebsd-6.1-intel
extract: CWD to /opt
extract: /opt/splunk/README.txt
extract: /opt/splunk/bin/btool
extract: /opt/splunk/bin/bunzip2
extract: /opt/splunk/splunk-3.4.1-45588-FreeBSD-i386-manifest
extract: CWD to .
Running post-install for splunk-3.4.1-45588-freebsd-6.1-intel..
Splunk has been installed in:

To start Splunk, run the command:
/opt/splunk/bin/splunk start

To use the Splunk Web interface, point your browser at:

Complete documentation is at http://www.splunk.com/r/docs
Attempting to record package into /var/db/pkg/splunk-3.4.1-45588-freebsd-6.1-intel..
Package splunk-3.4.1-45588-freebsd-6.1-intel registered in

If you try to start Splunk at this point you'll get an error like the following:

freebsd70:/usr/local/src# /opt/splunk/bin/splunk start
/libexec/ld-elf.so.1: Shared object "libc.so.6" not found, required by "splunk"

To fix the problem I installed compat6:

freebsd70:/usr/local/src# pkg_add -vr ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/
scheme: [ftp]
user: []
password: []
host: [ftp.freebsd.org]
port: [0]
document: [/pub/FreeBSD/ports/i386/packages-7.0-release/misc/
---> ftp.freebsd.org:21
looking up ftp.freebsd.org
connecting to ftp.freebsd.org:21
<<< 220 ftp.FreeBSD.org NcFTPd Server (licensed copy) ready.
>>> USER anonymous
<<< 331 Guest login ok, send your complete e-mail address as password.
>>> PASS analyst@freebsd70.localdomain
<<< 230-You are user #147 of 800 simultaneous users allowed.
<<< 230-
<<< 230 Logged in anonymously.
>>> PWD
<<< 257 "/" is cwd.
>>> CWD pub/FreeBSD/ports/i386/packages-7.0-release/misc
<<< 250 "/pub/FreeBSD/ports/i386/packages-7.0-release/misc" is new cwd.
>>> MODE S
<<< 200 Mode okay.
>>> TYPE I
<<< 200 Type okay.
setting passive mode
>>> PASV
<<< 227 Entering Passive Mode (62,243,72,50,214,227)
opening data connection
initiating transfer
>>> RETR compat6x-i386-6.3.602114.200711.tbz
<<< 150 Data connection accepted from; transfer starting for compat6x-
i386-6.3.602114.200711.tbz (3164256 bytes).
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.0-release/misc/compat6x-
i386-6.3.602114.200711.tbz...x +CONTENTS
extract: CWD to /usr/local
extract: /usr/local/libdata/ldconfig/compat6x
extract: CWD to .
Running mtree for compat6x-i386-6.3.602114.200711..
mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null
Attempting to record package into /var/db/pkg/compat6x-i386-6.3.602114.200711..
Package compat6x-i386-6.3.602114.200711 registered in

* *
* Do not forget to add COMPAT_FREEBSD6 into *
* your kernel configuration (enabled by default). *
* *
* To configure and recompile your kernel see: *
* http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig.html *
* *

Then I could start Splunk:

freebsd70:/usr/local/src# /opt/splunk/bin/splunk start
Splunk Free Software License Agreement
Do you agree with this license? [y/n]: y
Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default'
to '/opt/splunk/etc/myinstall/splunkd.xml'.
Copying '/opt/splunk/etc/openldap/ldap.conf.default'
to '/opt/splunk/etc/openldap/ldap.conf'.
Copying '/opt/splunk/etc/modules/distributedSearch/config.xml.default'
to '/opt/splunk/etc/modules/distributedSearch/config.xml'.
/opt/splunk/etc/auth/audit/private.pem generated.
/opt/splunk/etc/auth/audit/public.pem generated.

/opt/splunk/etc/auth/audit/private.pem generated.
/opt/splunk/etc/auth/audit/public.pem generated.

This appears to be your first time running this version of Splunk.
Validating databases...
Creating /opt/splunk/var/lib/splunk/audit/thaweddb
Creating /opt/splunk/var/lib/splunk/blockSignature/thaweddb
Creating /opt/splunk/var/lib/splunk/_internaldb/thaweddb
Creating /opt/splunk/var/lib/splunk/fishbucket/thaweddb
Creating /opt/splunk/var/lib/splunk/historydb/thaweddb
Creating /opt/splunk/var/lib/splunk/defaultdb/thaweddb
Creating /opt/splunk/var/lib/splunk/sampledata/thaweddb
Creating /opt/splunk/var/lib/splunk/splunkloggerdb/thaweddb
Creating /opt/splunk/var/lib/splunk/summarydb/thaweddb
Validated databases: _audit, _blocksignature, _internal, _thefishbucket,
history, main, sampledata, splunklogger, summary

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Verifying configuration. This may take a while...
Finished verifying configuration.
Checking index directory...
Verifying databases...
Verified databases: _audit, _blocksignature, _internal, _thefishbucket,
history, main, sampledata, splunklogger, summary

Checking index files
All index checks passed.
All preliminary checks passed.
Starting splunkd...
Starting splunkweb.../opt/splunk/share/splunk/certs does not exist. Will create
Generating certs for splunkweb server
Generating a 1024 bit RSA private key
writing new private key to 'privkeySecure.pem'
Signature ok
Getting CA Private Key
writing RSA key

Splunk Server started.

The Splunk web interface is at http://freebsd70.localdomain:8000

I was then able to connect to the Splunk Web interface, add a directory (/var/log) to monitor, and access results.

Documentation for FreeBSD installation is also available. Thanks Splunk!

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.


Anonymous said...

Hi. Sorry for writing the comment in this article:

Are you going to publish the rankings of the best books you read this year?

Please, please, please do so!

Richard Bejtlich said...

Hi Anonymous,

Yes, I will publish my Best Books Bejtlich Read in 2008 at the end of December. Thank you for your interest.

Anonymous said...

Thanks, Richard! I will be checking every day to see if you have published the list!

Jared Evans said...

Sorry for contacting you via the comments on this blog. I couldn't find an email address where I could contact you directly.

I'm following up on FreeBSD running as transparent bridge with snort in-line. It wasn't possible earlier this year but I'm wondering if development has progressed far enough where it is now possible to do this?

Thanks for your reply.
You can also contact me at jnevans@gmail.com

Richard Bejtlich said...

Jared, I haven't looked at inline Snort for FreeBSD any time recently.

Anonymous said...

Splunk on FreeBSD 7.0-release

This is a decent walk through. Unfortunately I'm running into the following error when trying to start splunk after your example.

Undefined symbol "__malloc_lock"

Have you run into this?

Richard Bejtlich said...

I am not encountering that problem.

Anonymous said...

If you get the "__malloc_lock" error you will need to install the glib1.x package (pkg_add -r glib)and relink/overwrite the existing "libc.so.6" file from the one in the compat6x that Richard installed.

cp /lib/libc.so.6 /lib/libc.so.6.ORIG
cp /usr/local/lib/compat/libc.so.6 /lib/

This can also happen if you have glib2.x installed and not glib1.x.

now splunk should start
/opt/splunk/bin/splunk start

James Pearce said...

Thanks for this article Richard.

This worked perfectly for me and allowed me to transition our Splunk server from Debian to FreeBSD without too many troubles. Thankfully I did the research before setting up the FreeBSD box, so I was glad to know the solution to this problem as soon as it arose!