[BBLISA] Long term data storage - archiving solutions - other ideas

[Robert Taylor]

Hi Stephen. We have ~450 tb of isilon, and we bought a solution from 
Fuji called dternity to help with archiving.
http://www.dternity.net/

It's basically a NAS with a disk buffer + tape drives + tape library 
that send the data to LTFS formatted tapes.

It has some terabytes of disk storage built in which it splits into read 
and write cache, (which is expandable via FibreChannel disk like a 
nexsan array).

The write cache eases ingest, so you are not writing directly to tape, 
you can nfs mount it and just copy or rsync the data on to it.

You can set policies on how long the data has to be in the cache before 
it writed it to tape, and how long to retain the disk copy before it is 
flushed(you can write to tape after 1hr in the cache, but you could keep 
it in the cache for a day if you wanted to). The metadata I believe is 
retained on disk(so ls's against it don't incur tape reads), and there 
is a setting for leaving a stub file of a preset size(globally, not per 
share), so if a client requests a file, while a tape is being loaded, 
the client can be fed some of the stub data from disk. I think their 
sweet spot in the market is large media files, like 4k video, but we 
have found it works ok for some of datasets that our users have 
generated and want to keep but don't need to be wasting the expensive 
space on the isilon to keep. We allow users to retrieve the files 
themselves(but we migrate the data into the archive for them), and let 
them know that if it's on the archive nfs mount that it will not have 
isilon level performance, and if they need that they need to copy it 
back onto the isilon.

If you're interested in hearing more about it, let me know, and I'll see 
if I can get you and one of our storage guys together to talk about it.


rgt



On 05/04/2015 12:18 PM, Stephen B Goldman wrote:
> Hello list members,
>
>                  We currently have Isilon NAS with 600 TB of data. We
> find we have 20 to 40 Tb of lab data we would like to offload spinning
> disk. We have TSM available from the campus IT group. They offer an
> archiving service.
>
>                  We would like to hear from others organizations that
> have the same need and listen to how they are handling this issue.
>
> Thanks in advance,
>
> Stephen
>
> Stephen Goldman
>
> Systems Administrator
>
> Massachusetts Institute of Technology
>
> Department of Biology
>
> 31 Ames Street 68-211
>
> Cambridge, Ma 02139
>
> sgoldman at mit.edu -617-452-2595
>
>
>
> _______________________________________________
> bblisa mailing list
> bblisa at bblisa.org
> http://www.bblisa.org/mailman/listinfo/bblisa
>