Normalize: Touch-up/rename fields, etc.
Enrich: Enrich and part of the message.
Check: Checks parts of message (now enriched) against known bad stuff.
Archive: Push document into elasticsearch. Can also log to file.

Installation on FreeBSD

Configure Kafka topics (run on one kafka node)

doc/kafka_topics.sh

Initialize elasticsearch

curl -X PUT 'http://<elasticsearch>:9200/threatline' -d@doc/es_mapping.json

Install service file

cp doc/threatline /usr/local/etc/rc.d/threatline

Enable threatline

sysrc threatline_enable=YES
sysrc threatline_agents="normalize enrich check archive"

Start threatline

service threatline start

Monitor logs

tail -f /tmp/tl_worker.log