Wednesday, June 12, 2013

Continuous Integration of Infrastructure

Nick Cammorato

Stability is the name of the game for systems and networks operators. It's what we expect and what our end-users expect, but it can be incredibly difficult to achieve - a constant stream of security patches, dead-end releases, broken software, broken hardware, and demands from other internal stakeholders force change. Constant, never ending change.

Complicating this is the fact that these changes are rarely isolated. A change on one system (particularly a lower-layer system like a router) can ripple through entire services. We have, in effect, an integration problem with every software update or configuration change we apply. So to solve this, we look to the software development world to see how they deal with it, and the answer is continuous testing and continuous integration.

At TERC, we're currently combining configuration management (puppet), hierarchical configs (hiera), monitoring (nagios), test software (rspec/cucumber), and a software development CI stack (jenkins/rake/vagrant) in order to address this. This talk will focus on my experiences in rolling all of this out, what the limitations of current software are, and touch on what my plans are to improve it.

MIT E-51, Room 145
7:00 - Announcements & Introductions
7:30 - Formal presentation



Future Events Past Events

Wednesday, July 10, 2013

Cloud storage options and DFS

Jeff Darcy (Red Hat)

This talk will be compare some of the cloud-storage options that are out there, with a particular focus on distributed filesystems. GlusterFS and Ceph will be compared in detail. Other options including HDFS, object stores, and NoSQL document stores will also be discussed.
Jeff Darcy has been working on distributed storage since DECnet and NFS version 2 in the early 1990s. Later he was one of the original developers for MPFS while at EMC, and is currently an architect for GlusterFS at Red Hat.

Wednesday, August 14, 2013

TBD

Wednesday, May 8, 2013
Asterisk and VOIP
K.M. Peterson
The near-universal provision of voice services and their terminals (called "telephones") predates the Internet. While on some level, voice traffic via TCP/IP is just another protocol, there are challenges in making it "just work" like the traditional phones that we are all used to. There are the technical issues of the nature of the data, interfacing with the still robust telephone network, and of course the UI expectations and experience.

That means that the protocols involved - SIP and the related suite - were developed in the setting of a preexisting, mature, and complex switched network. I found that from the perspective of a systems administrator or network engineer there are complications, terminology, and conventions that aren't necessarily obvious.

This talk will provide insight into the these technologies from that perspective to allow you to grasp the protocols and the context in which they interoperate, using an example implementation of Asterisk.
Wednesday, April 10, 2013
Lightning Talks
Matt Simmons et al.
Wednesday, March 13, 2013
Mastering Human Communication Patterns for Techies
Dan Hermes
Missed human connections in the software industry account for most of our project failures. Improving communication can dramatically improve individual and team performance. Typical frustrations:
  • They just don't get it
  • They talked at you for an hour and made no sense
  • That meeting was like a construction committee for the Tower of Babel

In the same way that programming languages have interfaces and design patterns, so do human beings. Problems that appear technical in nature can usually be traced back to failed interactions between people. We'll explore:
  • The human interface between system administrators, customers, and managers
  • Key patterns of communication including negotiation and resolving miscommunication
  • Examine techniques for how to listen to and understand others
  • How to be heard and understood yourself.

Mr. Hermes, principal consultant of Lexicon Systems, has over twenty five years experience as a software management consultant, .NET architect and developer. From start-ups to blue chips, Mr. Hermes has served dozens of software companies striving to develop successful, lasting enterprise systems. He has taught software architecture and development at Northeastern University, Microsoft User Groups, and Microsoft Certification classes at corporate training facilities. Cited on National Public Radio, Forbes, and Reuters, Mr. Hermes has had articles published by Media-N and MIT Press. He has served on the board of the Institute of Management Consultants New England Chapter and is currently director of Art Technology New England(ATNE).

Wednesday, February 13, 2013
Centrify From a Sysadmin Perspective
Tony Rudié
"Centrify" is a commercial product that facilitates Unix and Linux machines joining Microsoft Active Directory and using that for authentication and directory. "directory", in this context, means the information that was traditionally culled from NIS, NIS+ or LDAP, such as automount maps. This talk will take a quick look at the problem, give an overview of what Centrify does and how it does it, and offer a few cautionary tales about implementation, based on the presenter's experience.
Wednesday, January 9, 2013
LISA Recap
Adam Moskowitz et al.
A review of the events of LISA as seen by LISA attendees. Come and share your experiences at LISA 2012.
Wednesday, December 12, 2012
No Meeting
Wednesday, November 14, 2012
Internet buffer bloat
Jim Gettys
Bufferbloat: Problem, Migitation, and Solution
VOIP and teleconferencing often perform much more poorly on today's Internet than the Internet of a decade ago, despite great gains in bandwidth. Lots of fiber, cheap memory, smart hardware, variability of wireless goodput, changes in web browser behaviour, changes in TCP implementations, and a focus on benchmarking Internet performance solely by bandwidth, and engineer's natural reluctance to drop packets have conspired to encourage papering over problems by adding buffers; each of which may introduce latency when filled.
The mistaken quest to never drop packets has destroyed interactivity under load, and often results in actual higher packet loss, as TCP's congestion avoidance algorithms have been defeated by these buffers. The lessons of the "RED manifesto" of 1997 have been forgotten or never learned by a new generation of engineers.
Bufferbloat mitigation by tuning queue length is beginning to be deployed in cable broadband systems. But solving bufferbloat requires careful queue management that must be present anywhere a queue may form. With the publication of the new CoDel AQM algorithm by Nichols and Jacobson (and inclusion in Linux 3.5) we now have the opportunity to solve rather than mitigate bufferbloat. I will also touch on fq_codel, which combines stochastic fair queuing and CoDel, and why we like the combination so much, and the remaining challenges.
Jim Gettys is at Alcatel-Lucent Bell Labs, USA. Today he works on bufferbloat in all of its forms, including helping establish bufferbloat.net to serve as a rallying point in its solution.
He was the Vice President of Software at the One Laptop per Child project, one of the original developers of the X Window System, and the editor of the HTTP/1.1 specification in the IETF. In 1997 he won Bob Metcalfe's Internet Plumber of the Year award on behalf of the group who worked on HTTP/1.1.
slides (PDF, 2MB)
Wednesday, October10, 2012
The State of ZFS
Peter Baer Galvin
ZFS has taken the world by storm, and is still advancing. This talk will summarize the state of ZFS, including its availability, feature set, and recent changes.

Peter Baer Galvin is a seasoned tech writer, columnist, consultant, teacher and author. He is the CTO for systems integrator and VAR, Corporate Technologies (www.cptech.com). He's a Lecturer at Boston University and co-author of the Operating Systems Concepts textbooks. He's given talks and tutorials at USENIX conferences and other venues.
slides (PDF, 65MB)
Wednesday, September 12, 2012
Solid State Drives: Use, Performance, Caching, and More
Dan Noé, VeloBit
Solid State Drives can deliver high performance, but their prices still break the budget. SSD caching is a lower-cost method to improve application performance by taking advantage of fewer Solid State Drives to improve I/O. Dan will discuss SSD performance characteristics, best practices and risks of SSD deployment, as well as how SSD caching works and whether it can improve your performance.

Dan Noé is a Senior Software Engineer at VeloBit; previously, Dan was an engineer at IBM/Netezza,where he worked on database storage layer technology for the massively parallel Netezza Database Appliance. Dan holds a B.S. in Computer Science from University of New Hampshire, is an avid pilot and maintains Linux servers in his spare time.
slides (PDF, 1.5MB)
Wednesday, August 8, 2012
(Vacation)
Wednesday, July 11, 2012
Solaris Dynamic Tracing - DTrace
Jim Mauro
DTrace is a revolutionary software framework that enables unprecedented observability up and down the entire software stack. DTrace was first introduced in Solaris 10, and continues to ship with Solaris, with ports to Mac OS X and FreeBSD. Others are underway. This talk will provide an overview of the DTrace framework and key components, as well as a tour of using DTrace to measure and observe system behavior.

Jim Mauro is a Principal Software Engineer for Oracle Corporation. Jim's focus for the last several years has been systems performance, doing both internal performance-related engineering projects, as well as engaging in real customer production performance issues. Jim's most recent work involved performance and benchmarking of Oracle's ZFS Storage Appliance. Jim is the co-author of Solaris Internals (1st and 2nd Ed), Solaris Performance and Tools, and recently published DTrace.
slides (PDF, 1.2MB)
Wednesday, June 13, 2012
Keeping up with Systems Management across Windows, Linux and Mac platforms.
Harold Moore
How do you keep all of your systems (Windows, Mac, Linux)? Do you find yourself having to use multiple toolsets? Enterprises keep adding new types of system s (a resurgence in the Mac community and new Linux applications being developed every day) that only increase the demands on the systems administration team. On top of that, there are growing regulatory and compliance (software licensing) demands placed on the team. How can you streamline the systems management processes (software distribution, asset management, patching, O/S deployment, compliance reporting, etc.)? The Dell Kace systems management appliance was designed to make it easy for you to manage all of your systems from one console. We'll discuss common challenges that systems administrators face today and how best to address them.

Harold Moore has worked in the systems management field for over 15 years. Harold worked at Novell from 2000-2007. He worked on the System Engineering team that was responsible for systems management and the Suse Linux/Open Enterprise Server. Harold then joined Altiris/Symantec working in systems management group supporting management of MAC, Linux and Windows. Harold Joined Dell in March 2011 and works in the Dell-Kace division. Harold has a BA from the University at Albany in Communications and Computer Science. He also hold a MS in Computer Science from Long Island University, CW Post.
Wednesday, May 9, 2012
What does a CIO do anyway?
Martin Leach
The average job life of a CIO is about 2 1/2 years, it even comes with the pleasant acronym of '*C*areer *I*s *O*ver'. The Broad Institute just hired their first CIO, and will try to justify his existence at this presentation and discussion.

Martin Leach is chief information officer at the Broad Institute He came to the Broad from Merck & Co., where he led IT for Discovery and Pre-Clinical Sciences across all the North American research sites. Over his career he has provided support and strategic vision for IT, informatics, and data-mining activities at a range of life sciences organizations. Martin received his B.Sc. in cell and molecular sciences from Anglia Polytechnic University and his Ph.D. in pharmacology from Boston University School of Medicine.
Wednesday, April 11, 2012
A fresh look at SELinux and what it is complaining about.
Daniel Walsh
The Four main causes of SELinux problems.
  1. Labeling Problems
  2. SELinux has to know how you configured your processes
  3. Bug in Policy or an Application.
  4. Your machine has been compromised.

Daniel Walsh has worked in the computer security field for almost 30 years. Dan joined Red Hat in August 2001. He has led the SELinux project, concentrating on the application space and policy development. Dan helped developed sVirt, Secure Vitrualization. He also created the SELinux Sandbox, the Xguest user and the Secure Kiosk. Previously, Dan worked Netect/Bindview's HackerShield and BVControl for Unix, Vulnerability Assessment Products. Dan worked for Digital Equipment Corporation on the Athena Project along with designing and developing the AltaVista Firewall and AltaVista Tunnel (VPN) Products. Dan has a BA in Mathematics from the College of the Holy Cross and a MS in Computer Science from Worcester Polytechnic Institute.
Wednesday, March 14, 2012
Bacula: An Introduction to an Open Source backup system
K. M. Peterson
What should be a simple problem to solve - having a copy of important data in case of hardware error or human mistake - turns out to have complex (and expens ive) solutions. Bacula is an open-source application that runs in most popular environments, supports disk and tape-based backups, and utilizes a database for managing its catalog of file versions and backup media. This presentation will discuss Bacula's functionality, including its features and some implementation details, and provide a short example of a working configuration.
K. M. Peterson has worked as a manager, systems administrator, and consultant in academic, commercial, and non-profit environments. He's interested in topics in data management, networking, security and automation. Currently, he is seeking the next challenging role, and digging deeper into interesting technologies he has encountered over the past several years.
Wednesday, February 8, 2012
An informal overview of the oVirt project: status, goals and a brief demonstration.
Dave Allan
Dave Allan worn a number of hats: sysadmin, operations manager, field support staff, QA engineer, software developer. I'm currently a software development manager at Red Hat where I am the PHB of the libvirt team and interact freqently with the oVirt team.
Wednesday, January 11, 2012
No Meeting

Wednesday, December 14, 2011
LISA Recap
Adam Moskowitz
Wednesday, November 9, 2011
Tracking issues - experiences from the field
Christopher Allison, Tom Bechard, John Rouillard, Tony Rudie, Clarence Smith
Wednesday, October 12, 2011
Cloud Filesystem HekaFS
Jeff Darcy
Modern open-source distributed filesystems make it possible to provide file services at a scale and level of availability that's finally competitive with proprietary options. What they don't do - yet - is enable secure sharing of those resources between multiple user bases or organizations who pay for them. This talk will focus on how GlusterFS works to solve the first set of problems, and how HekaFS - which is based on GlusterFS - is solving the second. If you're tired of having to deal with umpteen departmental file servers, each configured differently, this approach might provide some relief.
Jeff Darcy has been working with network, cluster, and distributed filesystems for about twenty years - since DECnet was still relevant and NFSv2 was new. Since then he has gained scars from EMC's MPFS (for which he was one of the initial developers), Lustre, and GlusterFS. He is currently at Red Hat, where he's the project lead for HekaFS and all-around "cloud storage" expert.
slides (PDF, 97KB)
Wednesday, September 14, 2011
Converged Networks, Voice / Video / Storage / Data
Ryan Sutton
How do you keep everything running smoothly while giving technologies that require low or constant latency what they need and still be able to watch that dog skateboarding on YouTube. Will discuss how Quality of Service (QoS) works on both Layer 2 and Layer 3 devices and why you probably need both. How should traffic be tagged and retagged with policy decisions. If time permits we may even have some time to go into why net neutrality isn't as cut and dry as you may think. Technical details will be based off of Cisco devices but most concepts will translate to any modern equipment.
Our speaker, Ryan Sutton, is a Systems Engineer at a local Gold Certified Cisco Partner. Ryan specializes in large scale Routing and Switching, Voice, and Data Center designs and implementations. His solutions often include interworking between multiple networking vendors and technologies. Current projects include sub-second core convergence, and multi-city/multi-vendor VoIP installations.
Wed, July 13, 2011
Ruby: More Batteries, Fewer Brackets
Aaron D. Ball
Ruby may be most familiar as the language behind the Rails web framework, and Perl as the "Swiss Army chainsaw" that no sysadmin can live without, but they have a lot more in common than you might think. Ruby comes out of the box with a great set of sysadmin tools, from text processing to Unix system interfaces to TCP servers, and has a syntax about as terse as Perl but with object-oriented and functional-programming idioms that make your code easier to write and understand. Whether you're new to scripting or you've been typing line noise since 1987, this talk will show you another way.
Wed, June 8, 2011
Tuttle
Robert Thau
Robert Thau from Smartleaf will present their Tuttle system configuration tool.
Wed, May 11, 2011
Automating Inventory, Deployment and Configuration of Your Windows Infrastructure
Dan Stolts
Like most IT professionals, you are an administrator in a heterogeneous environment. You have a myriad of tools to inventory, deploy and configure your Unix/Linux machines but how do you do this for the rest of your machines? Come to this session to learn about the tools you must have in your toolbox to inventory, deploy, and remotely configure your windows desktops and servers. We will discuss the free tools as well as the top of the line fully automatable solutions available by Microsoft.
Dan Stolts is a technology specialist with more than 24 years in the industry. He is proficient with many Microsoft products especially those in the server area and holds many certifications including MCT, MCITP, MCSE, TS, etc. Dan is currently specializing in Systems Management and Security and is also very passionate about virtualization technologies. Dan is and has been a very active member of the community. He is the current president of Boston User Groups.
Wed, Apr 13, 2011
The Path to Senior Sysadmin
Adam Moskowitz
Being a senior system administrator is about more than knowing all the options to mount(8) or that modprobe is what's used to replace that buggy kernel module with the latest version. Rather, a good senior sysadmin will have a wide knowledge of relevant technical topics, in-depth knowledge of one or more technologies, good interpersonal skills, and the ability to manage "problem users" and will be comfortable making presentations to and negotiating with mid- and upper-level management. This talk will cover the skills a senior sysadmin needs and why they are necessary and will provide some suggestions for how to acquire these skills.

For nearly one-third of his sysadmin career, Adam Moskowitz held titles such as Senior System Administrator, System Architect, and IT Manager. Despite having returned to his roots as a programmer, Adam remains active in the sysadmin community, including running the LISA Advanced Topics Workshop and serving on the LOPSA Leadership Committee. He claims he does all of this only to support his hobby Advanced Topics Workshop and serving on the LOPSA Leadership Committee. He claims he does all of this only to support his hobby of judging barbecue contests and to keep food in his puppy's bowl.
slides (PowerPoint, 496KB)
speaking notes (powerPoint, 29MB)
Wed, March 9, 2011
How Splunk manages our Junk
Jim Donn and Tim Hartmann
As environments grow and systems become more complex, building and managing a usable centralized logging infrastructure can be a daunting task. In this talk, we will walk through our real-life experiences implementing Splunk as our centrali zed logging infrastructure for our Network, Systems, Security, and Application teams. Over the past three years, we have had to change our strategies and architecture to account for organic customer growth, changes in team requirements, and evolutions in technology.
Slides

Jim Donn, Harvard University Network Services Group (UNSG)
Senior Network Management Engineer
Tim Hartmann, Harvard University Network Services Group (UNSG)
Senior Systems Administrator
Wed, February 9, 2010
Project Caua: Private Sector, Environmentally Friendly Jobs with Free Software
Jon Hall
Project Caua is an Open project to create millions of private sector, environmentally friendly jobs utilizing FOSS in urban areas of Latin America, and millions more around the world. In addition, Project Caua will open an avenue for free (as in beer) wireless Internet to help defeat the digital divide, and to provide low-cost training to move people off unemployment and create taxpayers. The specifications for Project Caua can be found at http://www.projectcaua.org/. This talk will outline Project Caua and open the discussion for actual implementation details.

Jon Hall is the Executive Director of Linux International (www.li.org), an association of computer users who wish to support and promote the Linux Operating System. During his career in commercial computing w hich started in 1969, Jon has been a programmer, systems designer, systems administrator, product manager, technical marketing manager, author and educator. He currently works as an independent consultant, and is currently involved with bringing environmentally friendly computing to emerging marketplaces.
Wed, December 8, 2010
LISA Recap
Adam Moskowitz
Usenix LISA 2010 conference recap.
Wed, October 13, 2010
Using MySQLtuner 2.0 to monitor and improve mysql performance
Sheeri K. Cabral
With help from Major Hayden, mysqltuner's original author, Sheeri K. Cabral of the Pythian Group has modified mysqltuner to be more comprehensive, to output information and to have a "spreadsheet" mode where the results of mysqltuner are outputted as a single column, so that you can easily compare subsequent runs of the modified mysqltuner -- for example, running it monthly or weekly to see how performance is progressing (or degrading). There is also a truly offline mode that requires no database connectivity where files containing the output of SHOW GLOBAL STATUS and SHOW GLOBAL VARIABLES are used. This talk will go through how the the modified mysqltuner works including how easy it is to change what is checked and the thresholds, so that you can easily do one-off sanity checks as well as regular monitoring.

Sheeri K. Cabral (The Pythian Group) is a noted MySQL community activist who recently wrote The MySQL Administrator's Bible.
Keep up with her MySQL writings at http://www.pythian.com/news/author/sheeri/.

PDF Slides and Openoffice slides.
Wed, September 8, 2010
Ipswitch WhatsUpGold
Rich Makris
Built on a modular, yet integrated architecture, WhatsUp Gold is an affordable and easy-to-use solution that scales with the size and complexity of any physical or virtual IT infrastructure. From a single console, WhatsUp Gold supports standard IT management tasks including automated discovery, mapping, real-time monitoring, alerting, troubleshooting and reporting. Rich Makris will walk through the benefits of using WhatsUp Gold and how it can make your life easier. As a Sales Engineer for the Network Management division, Rich's focus is on helping customers solve their IT Management needs with WhatsUp Gold and Event Log Management products. He has held various systems and network positions for more than 15 years in government, manufacturing, financial services, and at service providers. Rich also holds certifications from Cisco, Microsoft, and Novell.
Wed, May 12, 2010
Using IPv6
Daniel Hagerty
Daniel has been using IPv6 for fun (and to get things done) for quite some time. Come learn what has and has not worked for him. Details of his personal dual-stack IPv6 setup will be presented as well as other experiences.
Wed, 14 Apr 2010
"Building 16 systems in 16 minutes with xCAT"
Ali Tayarani
Ali Tayarani will discuss how we use xCAT to manage several hundred hosts in our general-purpose LSF-based compute cluster. Slides
"Redefining Compute Nodes and Provisioning"
John Hanks
John Hanks will discuss his grand vision for the future (stateless compute nodes managed with Perceus), exemplified by our new genome-sequencing GridEngine cluster.

Wed, 10 Mar 2010
"How to Interview a System Administrator"
Adam Moskowitz
This will be a shortened version of Adam's LISA tutorial. The full description can be on the LISA 2007 web site.

Wed, 10 Feb 2010
(no meeting)

Wed, 13 Jan 2010
“I Got My Jet Pack and I'm Still Not Happy”
David Blank-Edelman
slides (PDF, 11.75MB)

Wed, 9 Dec 2009
Thirty Minute Tools
John Rouillard (and others)

Wed, 11 Nov 2009
LISA Recap
Adam Moskowitz

Wed, 14 Oct 2009
(was there a meeting?)

Wed, 09 Sep 2009
“Log Analysis with the Simple Event Correlator”
John P. Rouillard

Wed, 12 Aug 2009
Everything I Know About Sysadmin I Learned in the Back of an Ambulance
John P. Rouillard


webmaster at bblisa dot org