[BBLISA] Large file transfer in 65ms latency

[Edward Ned Harvey]

> From: bblisa-bounces at bblisa.org [mailto:bblisa-bounces at bblisa.org] On
> Behalf Of Edward Ned Harvey
> 
> Also, you mentioned the data is oracle database.  This should be
irrelevant,
> as long as you stop the db server first.  Then it's just a simple file
> transfer.

If the source is a sparse file, or if you want the destination to be sparse,
tar can handle that (man tar and search for sparse).  It doesn't do a super
awesome job, but then, nothing above the filesystem layer does.  I guess it
does as well as anything, unless you want to use something like dump|restore
or "zfs send."  Which will intelligently preserve sparseness.

Holes in sparse files above the filesystem layer are seen as consecutive
0's, which is infinitely compressible, but still require the CPU to generate
all the 0's (not as fast as you would think).  The best thing you can do
with infinitely compressible consecutive zero's is to pipe it through the
super-fastest of super-fast compression algorithms.  The fastest I know is
lzop or pigz --fast.  (In some cases, threadzip, which is a python script I
wrote on google code, outperforms pigz, but if both the sending & receiving
systems have pigz and several cores available, I would say, just use pigz,
because with like 4 threads, it will saturate your 1Gbit link.  Threadzip is
pretty niche.)