initial commit

README

@@ -0,0 +1,26 @@
+Configure Kafka topics (run on one kafka node)
+doc/kafka_topics.sh
+
+Initialize elasticsearch:
+curl -X PUT 'http://<elasticsearch>:9200/threatline' -d@doc/es_mapping.json
+
+Install service file (FreeBSD):
+cp doc/threatline /usr/local/etc/rc.d/threatline
+
+Enable threatline:
+sysrc threatline_enable=YES
+sysrc threatline_agents="normalize enrich check archive"
+
+Start threatline:
+service threatline start
+
+Monitor logs:
+tail -f /tmp/tl_worker.log
+
+
+Stages:
+Normalize: Touch-up/rename fields, etc.
+Enrich: Enrich and part of the message.
+Check: Checks parts of message (now enriched) against known bad stuff.
+Archive: Push document into elasticsearch. Can also log to file.
+