<div dir="ltr"><div>We're straddling two tools and doing on call with pager duty. </div><div><br></div>- AWS systems - Cloudwatch + custom metrics -> pager duty for alerting<div>- Traditional datacenters - check_mk (based on nagios see: <a href="http://omdistro.org">omdistro.org</a>) does servers and networking equip and sends some emails -> pager duty for alerting </div><div>- Softlayer - Zabbix, because their dev group loves it for some reason -> pager duty for alerting</div><div><br></div><div>We also looked at datadog and really liked it, but it had a minimal story for networking equipment. Stack driver also looked great as a "statsd + graphite as a service" service, but like datadog, not the greatest story for networking equipment. <br></div><div><br></div><div>It seems like if you're doing datacenter compute, datacenter networking, and cloud monitoring all together youre in the wastelands of multiple tools or having visibility gaps. There's another saas called <a href="http://flapjack.io">flapjack.io</a> that does event roll up. I think victorops is in that space too. We just punted and configure multiple tools to roll alerting into pager duty. It all feels bad and lacking cohesion, but it works. </div><div><br></div><div>I've also done some playing with the ELK stack + topbeat and done logging / metrics in the same place. Feels weird putting my system metrics in the same hole as my system logs, but it works surprisingly well. Watcher (elasticsearch commercial offering) does a decent job with a alerting, but is a bear to config right (a mess of json and lucene search queries). </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 4, 2016 at 8:38 AM, Antony Rudie <span dir="ltr"><<a href="mailto:antony.rudie@gmail.com" target="_blank">antony.rudie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That last sentence is god's own truth. <div><br></div><div>One more thing you need: A solid plan for what happens to the alerts. Where do they go? who deals with them? How do you know if they've been dealt with? It's not rocket science, but it's really important. I worked in a place where that piece was missing, and frankly, I'm not sure the whole monitoring setup added any value.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 3, 2016 at 7:52 PM, John Stoffel <span dir="ltr"><<a href="mailto:john@stoffel.org" target="_blank">john@stoffel.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
One of the things that people seem to miss, or overlook in my opinion<br>
is the cost of doing all this monitoring, and the steep learning curve<br>
you have for all of it. It's going to suck in a bunch of time at<br>
first, way more than people think, and getting it tuned so that it's<br>
not sending out false alarms is a huge task.<br>
<br>
I've played with Nagios, and we have Solarwinds at $WORK, but neither<br>
is well done or really used outside of silos. I also played around<br>
with collectd and graphite, but found it too simplistic in terms of<br>
access control for what I/we wanted. And we have an old instance of<br>
WhatsUp running as well for another group. It's all hodge podge. We<br>
really should dedicate someone to doing this work, but we all keep<br>
getting pulled in new directions all the time.<br>
<br>
It might be easier if you're just upgrading from something and you<br>
know what you want to monitor, etc. But it's not a simple drop-in<br>
tool like some people make out. It requires commitment and discipline<br>
to use effectively.<br>
<br>
John<br>
<br>
_______________________________________________<br>
bblisa mailing list<br>
<a href="mailto:bblisa@bblisa.org" target="_blank">bblisa@bblisa.org</a><br>
<a href="http://www.bblisa.org/mailman/listinfo/bblisa" rel="noreferrer" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
bblisa mailing list<br>
<a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a><br>
<a href="http://www.bblisa.org/mailman/listinfo/bblisa" rel="noreferrer" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div><span style="font-family:arial,helvetica,sans-serif;font-size:20px"><span style="color:rgb(64,64,65)"><strong><span style="font-size:14px">Patrick </span></strong></span><span style="font-size:14px"><span style="color:rgb(64,64,65)"><strong>Flaherty </strong></span><span style="color:rgb(105,105,105)"><span style="font-size:16px">|</span></span></span></span><span style="font-family:arial,helvetica,sans-serif;font-size:14px"><span style="color:rgb(64,64,65);margin-left:5px">Systems Architect</span></span></div><span style="font-family:arial,helvetica,sans-serif"><div style="color:rgb(64,64,65);font-size:14px"><span style="color:rgb(105,105,105);font-size:12px"><b>@platformpatrick</b></span><span style="font-weight:normal;color:rgb(105,105,105);font-size:12px"> <strong>e:</strong> <a href="mailto:patrick.flaherty@weather.com" target="_blank">patrick.flaherty@weather.com</a></span></div></span></div></div></div>
</div>