[BBLISA] ZFS Anyone?

Fri Apr 3 14:53:43 EDT 2009

Hi Dewey,

ZFS will go at spindle speed for sequential I/O. For random I/O it's best to
have flash memory for the ZIL (intent log). I assume you'd use NFS if you
went with an external device? If so that would be mostly random I/O and
you'd want the ZIL to be on flash. You could roll your own, say using the
Sun servers that support adding SSDs. But those SSDs are not as fast as the
ones that come in the Sun 7000. The Sun 7000 has write-optimized SSDs
(logzilla) for writes, which is essentially DRAM backed with flash and a
supercap to power the drain of the DRAM to the flash in case of power loss.

So performance-wise it would be difficult to roll your own equivalent.  Plus
in the S7000 you get rather astounding analytics functionality and easy
configuration, option of clustering, and so on with the S7000. My company
sells Sun (and Netapp, and 3Par) so I'm a little biased. Check out my blog
entryn on Analytics at
http://ctistrategy.com/2008/12/17/sun-storage-7000-analytics-overview/ and
my full column on it at
http://www.galvin.info/2009-02my-latest-column-at-login.

In terms of what performance you can expect, there is lots of good info on
S7000 performance now. Especially see Brendan Gregg's blog at
http://blogs.sun.com/brendan/, in particular the Jan 9 entry with a
performance results table (based on the 7410 model). Here are some of the
results:

> Workload Result 
> NFS streaming read from DRAM ~1.90 Gbytes/sec
> <http://blogs.sun.com/brendan/entry/up_to_2_gbytes_sec>
> NFS streaming read from disk 1.04 Gbytes/sec
> <http://blogs.sun.com/brendan/entry/1_gbyte_sec_nfs_streaming>
> NFS streaming write to disk 563 Mbytes/sec
> <http://blogs.sun.com/brendan/entry/1_gbyte_sec_nfs_streaming>
> NFS read IOPS from DRAM 281,000 IOPS
> <http://blogs.sun.com/brendan/entry/a_quarter_million_nfs_iops>
> CIFS streaming read from DRAM ~1.03 Gbytes/sec
> <http://blogs.sun.com/brendan/entry/cifs_at_1_gbyte_sec>
> CIFS streaming read from disk 849 Mbytes/sec
> <http://blogs.sun.com/brendan/entry/cifs_at_1_gbyte_sec>
> CIFS streaming write to disk 620 Mbytes/sec
> <http://blogs.sun.com/brendan/entry/cifs_at_1_gbyte_sec>
> CIFS read IOPS from DRAM 203,000 IOPS
> <http://blogs.sun.com/brendan/entry/cifs_at_1_gbyte_sec>

As to the general question of whether (I recommend) using ZFS, absolutely.
Except for a few corner cases (database clusters) it has the best
combination of price / performance / feature set going. I've been using it
since alpha / beta days and find it rock solid and feature rich.

Btw, Sun has a try-and-buy program which makes trying a 7000 (and many other
products) risk free. They pay to ship it to you and if you don't want it at
the end of 60 days of testing they will pay to ship it back.

Glad to help if you have any other questions (or need a Sun reseller).

--Peter

Peter Baer Galvin
CTO  IT Architecture
Corporate Technologies
Peter at cptech.com, peter at galvin.info
Corporate web site: www.cptech.com
Corporate blog: http://ctistrategy.com/
My website / blog: www.galvin.info
781-791-2112

On 4/3/09 1:46 PM, "Dewey Sasser" <dewey at sasser.com> wrote:

> Hello all,
> 
> I am in need of more, FAST storage.
> 
> 
> 
> The Brief Background:
> 
> I have 6 beefy VMWare ESX hosts (DL380, 2xQuad, 32GB), soon to expand to
> 9 (and all to go up to 48GB), on which I'm running VMWare ESX 3.5
> managed by LabManager (current capacity 180 virtual machines, expanding
> to around 400 with host and RAM expansion).  My storage so far has
> managed to usually keep up but occasionally it falls off a cliff.  I've
> 3 months of IOStat numbers but don't believe I have a very clear idea of
> where said cliff edge really is.
> 
> I currently have a 20 disk HP DL380 running RedHat EL 5.3 and have some
> stability problems which significantly interfere with performance tuning
> (right now my heavy load is on a single 6 disk RAID 10 and every time I
> try to resize the partition the volume hangs.)
> 
> I'm supporting development and test machines for a 40 person development
> group.  This system is *NOT* customer facing but is important for
> engineering productivity.
> 
> I've been talking to various VARs about NetApp, EquiLogic and Left Hand
> storage systems and the price/GB seems...high.  It's also clear that my
> existing system performance is pretty darn good for the base tech (990
> IOps average in 50% write load on 7.2k SAS drives).
> 
> We've recently started thinking about deploying a Sun ZFS based storage
> system, either on a "build it ourselves" or "buy a 7000 series storage
> box" basis.  I've read through Sun's "this is the best thing since the
> transistor" slide presentation and a fair amount of the admin manual,
> and it looks fairly cool and I can see some reasons it might be better
> than LVM2 on RAID.
> 
> 
> 
> The Questions:
> 
>     * Does anyone have experience with ZFS?  What has your experience been?
>     * Is it stable/reliable/etc.  Honestly, before this experience LVM2
>       has been rock solid for me across many deployments.
>     * What kind of performance should I expect?
>     * What are the worst/most annoying parts of dealing with ZFS?
>     * Why should I not use it?
>     * Why should I use it?
> 
> Thanks,
> 
> --
> Dewey
> 
> _______________________________________________
> bblisa mailing list
> bblisa at bblisa.org
> http://www.bblisa.org/mailman/listinfo/bblisa