<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple style='word-wrap: break-word;
-webkit-nbsp-mode: space;-webkit-line-break: after-white-space'>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Actually, I have the same behavior – with files “vanished”
- But when I do my rsync, the source is a snapshot. So it’s
impossible for the file to have changed in between the initial scan and actual
attempted copy. So truly I have no explanation. And I am made
uneasy by this as well.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I agree with something someone else said though – Put the
command into a wrapper script to ensure you never have two concurrent rsync’s
doing the same thing.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>There is some real value also, in making sure you have snapshots
of the destination, or perhaps multiple destinations so you have some number of
days of history. It really stinks when somebody loses a file, and wants
to restore it, only to discover it was deleted from the backups too.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Since you mentioned dump & tar – Dump is written by
the people who write the filesystem (ufs, ext3, etc) so there’s no such
thing as dump for an NFS mount. You could only run dump on the actual
server that does the serving. When dump works right, it works well, but
it often doesn’t work right. Meaning – I’ve had serious
problems before, when the version of dump&restore which created&restored
the backup was different from the version of mkfs which formatted the filesystem,
which yielded corrupted restored files. No warnings, no errors … So
if you use dump, use religious obsession to ensure the same versions.
Also, you’re technically not allowed to dump a filesystem while it’s
mounted, but everybody does anyway. Personally, when I use dump, I always
prepare in advance by installing the OS twice. Once to be the real OS,
and one minimal version just for the purpose of doing dump on the original,
unmounted filesystem. By installing the OS twice, and using dump in the 2<sup>nd</sup>
to backup the 1<sup>st</sup>, I ensure the dump I use is the same as the
filesystem version, and I have a repeatable restore procedure if needed
sometime in the future. Tar doesn’t have the same problem (assuming
you stay with gnu tar all around) but I’ve seen some limitation on tar’s
ability to handle enormous backups.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>And lastly – If you’re using a mac to do the rsync,
I hope you remembered to format the filesystem case sensitive. By
default, OSX formats case insensitive. Yah, I got bit by that one too.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> bblisa-bounces@bblisa.org
[mailto:bblisa-bounces@bblisa.org] <b>On Behalf Of </b>Richard 'Doc' Kinne<br>
<b>Sent:</b> Friday, January 09, 2009 4:12 PM<br>
<b>To:</b> bblisa@bblisa.org<br>
<b>Subject:</b> [BBLISA] Odd Rsync Behavior...<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'>Folks:<o:p></o:p></p>
<div>
<p class=MsoNormal>Thanks much for all of the responses yesterday with regard
to my backup issue.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>One of the advantages I thought I had with regard to the
"cp" method was, if something turned up missing, I'd be able to
nearly "click-and-drag" the backup file back to the system thereby
restoring it. This was one of the reasons I'd not investigated "dump"
or "tar" more closely yet.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I decided to try "rsync" because not only did some
of you folks recommend and use it, but also I had a friend in MN who
recommended it.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>The partition I am trying to back up is coming from an NFS
mounted volume and going to an attached external Firewire drive. The command I
used in this test was:<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>/usr/bin/rsync -ral --progress --stats --times --perms
/Volumes/mira/scsi /Volumes/burly_1/Mira/ --delete<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Initially /Volumes/burly_1/Mira was empty.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>All went fine for a few hours. Then, seemingly toward the
end, with the "web" directory (probably the most important, of
course!) I started getting lines like:<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<div>
<p class=MsoNormal>file has vanished: "/Volumes/mira/scsi/web/vsx/submit/qry_getRemarks.php"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/vsx/submit/qry_getStar.php"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/vsx/submit/qry_getVarTypes.php"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/vsx/submit/qry_updateMethod.php"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>scsi/web/zapper/<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/zapper/.DS_Store"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/zapper/._index.shtml"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/zapper/._zapper.jnlp"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished: "/Volumes/mira/scsi/web/zapper/._zapperButton.jpg"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/zapper/._zapperdoc.shtml"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/zapper/._zapperhelp.shtml"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>file has vanished:
"/Volumes/mira/scsi/web/zapper/Zapper.jar"<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>This is a small sample toward the end. There were hundreds
of lines like this, almost as if rsync had just decided to give up.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I went to /Volumes/mira/scsi/web/* and I was able to
determine that these files were indeed there!<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Rsync ended with this data:<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<div>
<p class=MsoNormal>Number of files: 766415<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Number of files transferred: 746551<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Total file size: 66995757081 bytes<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Total transferred file size: 66995737633 bytes<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Literal data: 43101421090 bytes<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Matched data: 0 bytes<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>File list size: 17422528<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>File list generation time: 779.208 seconds<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>File list transfer time: 0.000 seconds<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Total bytes sent: 43129805144<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Total bytes received: 16543326<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>sent 43129805144 bytes received 16543326 bytes
6411523.66 bytes/sec<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>total size is 66995757081 speedup is 1.55<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>rsync warning: some files vanished before they could be
transferred (code 24) at /SourceCache/rsync/rsync-35.2/rsync/main.c(992)
[sender=2.6.9]<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>============<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I'm not sure where to go from here. The data says that
746551 files were transfered out of 766415, but I'm not even sure about that.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>The number of bytes in the "Total size" and in the
"Literal data" are confirmed by running "du" on those
areas, so those numbers are right.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I'm not sure what would stop the rsync, or what I may be
doing wrong. Any thoughts from those who've used this?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";
color:black'>---<o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";
color:black'>Richard 'Doc' Kinne, [KQR]<o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";
color:black'>American Association of Variable Star Observers<o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";
color:black'><rkinne @ aavso.org><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";
color:black'><o:p> </o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>