[BBLISA] accounting for I/O

[Rob Taylor]

Also, while I have never tried it, sysdig looks interesting, and might be able to do what you are looking for.

http://www.sysdig.org/

rgt

Whitehead Network/System Administrator

----- On Sep 1, 2016, at 4:54 PM, Rob Taylor(Whitehead) rgt at wi.mit.edu wrote:

> Have you tried iotop?
> It will tell you what processes are moving the most disk io at any given
> instant.
> Still might not get you what you want, but it might make it easier to narrow
> down.
> 
> rgt
> 
> Whitehead Network/System Administrator
> 
> ----- On Sep 1, 2016, at 3:05 PM, Daniel Feenberg feenberg at nber.org wrote:
> 
>> Apparently heavy random I/O overloaded our fileserver last week, and
>> response was very slow. We solved the problem with additional spindles,
>> but we are curious to know which process is doing the random I/O. Perhaps
>> we could approach that user with an offer to help improve their turnaround
>> time by changing the code. Our users are mostly inexperienced students so
>> the possibility of suboptimal code is certainly there. Most usage is
>> sequential access to very large files that does not load the fileserver
>> much at all so this has been a new experience for us.
>> 
>> We can easily track bytes/second but a process doing random I/O may use
>> very few bytes/second, but still occupy much of the fileservers capacity,
>> so it hasn't been fruitful to identify the processes doing the most reads
>> and writes. During the period of overload, few disks were showing more
>> than kilobytes/second of read or write, yet iostat revealed that several
>> disks were continuously at 100%.
>> 
>> A program such as iostat will tell us which physical disk is busy, lsof
>> will tell us which file is open by which process, netstat and nfstat will
>> give aggregate statistics over all processes, but I can't find a program
>> that will tell us which process is occupying the fileservers attention
>> with expensive requests.
>> 
>> We couldn't replace all the disks with SSD, but might be able to provide
>> SSD for some files, if we could identify the culprits.
>> 
>> Daniel Feenberg
>> 
>> _______________________________________________
>> bblisa mailing list
>> bblisa at bblisa.org
>> http://www.bblisa.org/mailman/listinfo/bblisa
> 
> _______________________________________________
> bblisa mailing list
> bblisa at bblisa.org
> http://www.bblisa.org/mailman/listinfo/bblisa