How many log lines do you think you would be collecting?<br><br>We ended up using syslog-ng to receive all the data, and syslog-ng hands it off to splunk along with writing it to files. For the most part splunk is great, but sometimes its just easier to go through the logs from the command line. Splunk can get bogged down at times and can also get somewhat pricey. We have about 5 million log lines a day going into our logging setup (damn mail servers), and the most usable way to use splunk was to split the load up between multiple splunk servers so the query load is somewhat distributed.<br>
<br><br>Mike Devlin<br>Manager of Operations<br>boston..com<br><br><br><br><br><div class="gmail_quote">On Fri, May 15, 2009 at 10:13 AM, Sean Lutner <span dir="ltr"><<a href="mailto:sean@rentul.net">sean@rentul.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I'd second the recommendation for splunk. It's a fantastic product, is easy to setup and would provide you with a way to aggregate and then easily search over all your data. Aggregation is the easy part, the searching, correlation, etc is not easy. I've implemented splunk at three different places and am in the middle of a deployment currently.<div>
<div></div><div class="h5"><br>
<br>
On May 15, 2009, at 9:49 AM, seph wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Mike Sprague <<a href="mailto:mfs@komerex.com" target="_blank">mfs@komerex.com</a>> writes:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I work for a web hosting company with about a thousand linux servers.<br>
We're discussing options on how to process the logs mainly from our mail<br>
and web servers to make troubleshooting easier. We're not really<br>
looking for long term storage; just a better way to be able to search<br>
the logs to diagnose either specific customer issues, broad system<br>
attacks, issues across a pool of servers or issues with a specific server.<br>
</blockquote>
<br>
splunk would be the obvious commercial product for this space.<br>
<br>
<a href="http://www.opensyslog.com" target="_blank">http://www.opensyslog.com</a> is a online hosted log consolidation tool. In<br>
beta.<br>
<br>
I've seen a variety of open source things in the log processing world. I<br>
don't think they're generally very flashy, and I'm not sure what the<br>
current favorites are. googling around syslog processing finds some<br>
suggestions.<br>
<br>
seph<br>
<br>
_______________________________________________<br>
bblisa mailing list<br>
<a href="mailto:bblisa@bblisa.org" target="_blank">bblisa@bblisa.org</a><br>
<a href="http://www.bblisa.org/mailman/listinfo/bblisa" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>
<br>
</blockquote>
<br>
<br>
_______________________________________________<br>
bblisa mailing list<br>
<a href="mailto:bblisa@bblisa.org" target="_blank">bblisa@bblisa.org</a><br>
<a href="http://www.bblisa.org/mailman/listinfo/bblisa" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>
</div></div></blockquote></div><br>