[BBLISA] Re: Large scale log processing

[Mike Sprague]

Mike Devlin wrote:
> How many log lines do you think you would be collecting?
> 
> We ended up using syslog-ng to receive all the data, and syslog-ng hands
> it off to splunk along with writing it to files. For the most part
> splunk is great, but sometimes its just easier to go through the logs
> from the command line. Splunk can get bogged down at times and can also
> get somewhat pricey. We have about 5 million log lines a day going into
> our logging setup (damn mail servers), and the most usable way to use
> splunk was to split the load up between multiple splunk servers so the
> query load is somewhat distributed.

Very rough guess, about 100 million lines/day from both mail and web.
Though they would be broken up into various 'classes'.  For example, I
would expect about 10 million lines/day from our outgoing mail servers
and I would want them to be considered separate from our incoming servers.

Thanks for your input!

mikeS

-- 
Michael F. Sprague
mfs at komerex.com