<div>There are other costs/considerations to ESX nic bonding.</div>
<div>1) Traffic over the bonded links is not bi-directional until you do more switch and vkernel configuation.</div>
<div>2) You will have differe behavior to consider depending upon if your're using mac or src/destination IPs</div>
<div> </div>
<div>Yes, it stinks that the cost to boldly go beyond 10+ year old technology is so high but other hassels and complexities go away when you do not need to resort to convuluted configurations.</div>
<div>I think the perceived costs to upgrade are not being offset by the many other costs that disappear (such as the ESX details mentioned above and others).</div>
<div> </div>
<div>But beware, your bottle neck might move away from your 1gb ethernet to some place else, perhaps the spindles of your nifty ZFS storage system (a nice problem to have....)</div>
<div> </div>
<div>-w.b.</div>
<div> </div>
<div> <br><br></div>
<div class="gmail_quote">On Fri, Nov 12, 2010 at 12:00 PM, <span dir="ltr"><<a href="mailto:bblisa-request@bblisa.org">bblisa-request@bblisa.org</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">Send bblisa mailing list submissions to<br> <a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a><br>
<br>To subscribe or unsubscribe via the World Wide Web, visit<br> <a href="http://www.bblisa.org/mailman/listinfo/bblisa" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:bblisa-request@bblisa.org">bblisa-request@bblisa.org</a><br><br>You can reach the person managing the list at<br> <a href="mailto:bblisa-owner@bblisa.org">bblisa-owner@bblisa.org</a><br><br>When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of bblisa digest..."<br><br><br>Today's Topics:<br><br> 1. Faster than 1G Ether ... ESX to ZFS (Edward Ned Harvey)<br> 2. Re: Faster than 1G Ether ... ESX to ZFS (Tim Winter)<br> 3. Re: Faster than 1G Ether ... ESX to ZFS (John Orthoefer)<br>
<br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Fri, 12 Nov 2010 09:21:26 -0500<br>From: Edward Ned Harvey <<a href="mailto:bblisa3@nedharvey.com">bblisa3@nedharvey.com</a>><br>
Subject: [BBLISA] Faster than 1G Ether ... ESX to ZFS<br>To: <<a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a>><br>Message-ID: <000501cb8274$e75b7920$b6126b60$@<a href="http://nedharvey.com/" target="_blank">nedharvey.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br><br>Since combining ZFS storage backend, via nfs or iscsi, with ESXi heads, I'm<br>in love. But for one thing. The interconnect between the head & storage.<br>
<br><br><br>1G Ether is so cheap, but not as fast as desired. 10G ether is fast enough,<br>but it's overkill and why is it so bloody expensive? Why is there nothing<br>in between? Is there something in between? Is there a better option? I<br>
mean . sata is cheap, and it's 3g or 6g, but it's not suitable for this<br>purpose. But the point remains, there isn't a fundamental limitation that<br>*requires* 10G to be expensive, or *requires* a leap directly from 1G to<br>
10G. I would very much like to find a solution which is a good fit. to<br>attach ZFS storage to vmware.<br><br><br><br>Any suggestions?<br><br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>
URL: <a href="http://www.bblisa.org/pipermail/bblisa/attachments/20101112/54ebe2a9/attachment.html" target="_blank">http://www.bblisa.org/pipermail/bblisa/attachments/20101112/54ebe2a9/attachment.html</a><br><br>------------------------------<br>
<br>Message: 2<br>Date: Fri, 12 Nov 2010 10:30:21 -0500<br>From: Tim Winter <<a href="mailto:tim@gigageek.net">tim@gigageek.net</a>><br>Subject: Re: [BBLISA] Faster than 1G Ether ... ESX to ZFS<br>To: Edward Ned Harvey <<a href="mailto:bblisa3@nedharvey.com">bblisa3@nedharvey.com</a>><br>
Cc: <a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a><br>Message-ID:<br> <AANLkTikB+EB+4o-O3upZxOVygWMkEButuaiP=<a href="mailto:Qxnwvox@mail.gmail.com">Qxnwvox@mail.gmail.com</a>><br>Content-Type: text/plain; charset="utf-8"<br>
<br>What about bonding multiple 1G interfaces?<br><br><br>On Fri, Nov 12, 2010 at 9:21 AM, Edward Ned Harvey <<a href="mailto:bblisa3@nedharvey.com">bblisa3@nedharvey.com</a>>wrote:<br><br>> Since combining ZFS storage backend, via nfs or iscsi, with ESXi heads, I?m<br>
> in love. But for one thing. The interconnect between the head & storage.<br>><br>><br>><br>> 1G Ether is so cheap, but not as fast as desired. 10G ether is fast<br>> enough, but it?s overkill and why is it so bloody expensive? Why is there<br>
> nothing in between? Is there something in between? Is there a better<br>> option? I mean ? sata is cheap, and it?s 3g or 6g, but it?s not suitable<br>> for this purpose. But the point remains, there isn?t a fundamental<br>
> limitation that **requires** 10G to be expensive, or **requires** a leap<br>> directly from 1G to 10G. I would very much like to find a solution which is<br>> a good fit? to attach ZFS storage to vmware.<br>><br>
><br>><br>> Any suggestions?<br>><br>><br>><br>> _______________________________________________<br>> bblisa mailing list<br>> <a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a><br>> <a href="http://www.bblisa.org/mailman/listinfo/bblisa" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>
><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <a href="http://www.bblisa.org/pipermail/bblisa/attachments/20101112/3cfcd49e/attachment-0001.htm" target="_blank">http://www.bblisa.org/pipermail/bblisa/attachments/20101112/3cfcd49e/attachment-0001.htm</a><br>
<br>------------------------------<br><br>Message: 3<br>Date: Fri, 12 Nov 2010 10:39:39 -0500<br>From: John Orthoefer <<a href="mailto:jco@direwolf.com">jco@direwolf.com</a>><br>Subject: Re: [BBLISA] Faster than 1G Ether ... ESX to ZFS<br>
To: Back Bay LISA <<a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a>><br>Message-ID: <<a href="mailto:0F4F520B-B17A-42C6-A5D5-E669AE02C5F7@direwolf.com">0F4F520B-B17A-42C6-A5D5-E669AE02C5F7@direwolf.com</a>><br>
Content-Type: text/plain; charset=windows-1252<br><br>Between the two is link bonding/etherchannel. Which is gets expensive in it's self, since you are using multiple switch ports and card slots to increase your link speed. A quad GigE card is about $500 and uses a PCIe port, then 4 ports on your switch is going to be say 60$/port, so to get 4G out of a machine you are looking at $750. The card and a 10G port cost about $700 and then a switch with ports you are looking at about $1200/port for a switch to plug it into, so call it $2k/10G.<br>
<br>In general terms you 3xdollars buys you 10xspeed, without all the side arguments about rated speed and bus rates etc.<br><br>My basic though is that in general if 1G isn't enough goto 10G UNLESS you know you'll only need say 2-4G uplink it will save you from revisiting things in a year or two.<br>
<br>You can also look at what you might be able to do to squeeze a bit more performance out of 1G till the 10G comes down more, if the money really is a show stopper. Look at using Jumbo packets, and using a a P-t-P link (plug a straight though cable into each device without a switch, which BTW you can do with 10G too) for your traffic, make sure nothing but file access are going over your link etc...<br>
<br>johno<br><br>On Nov 12, 2010, at 9:21 AM, Edward Ned Harvey wrote:<br><br>> Since combining ZFS storage backend, via nfs or iscsi, with ESXi heads, I?m in love. But for one thing. The interconnect between the head & storage.<br>
><br>> 1G Ether is so cheap, but not as fast as desired. 10G ether is fast enough, but it?s overkill and why is it so bloody expensive? Why is there nothing in between? Is there something in between? Is there a better option? I mean ? sata is cheap, and it?s 3g or 6g, but it?s not suitable for this purpose. But the point remains, there isn?t a fundamental limitation that *requires* 10G to be expensive, or *requires* a leap directly from 1G to 10G. I would very much like to find a solution which is a good fit? to attach ZFS storage to vmware.<br>
><br>> Any suggestions?<br>><br>> _______________________________________________<br>> bblisa mailing list<br>> <a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a><br>> <a href="http://www.bblisa.org/mailman/listinfo/bblisa" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>
<br><br><br>------------------------------<br><br>_______________________________________________<br>bblisa mailing list<br><a href="mailto:bblisa@bblisa.org">bblisa@bblisa.org</a><br><a href="http://www.bblisa.org/mailman/listinfo/bblisa" target="_blank">http://www.bblisa.org/mailman/listinfo/bblisa</a><br>
<br>End of bblisa Digest, Vol 84, Issue 6<br>*************************************<br></blockquote></div><br>