{"id":1889,"date":"2016-09-19T19:48:30","date_gmt":"2016-09-20T00:48:30","guid":{"rendered":"http:\/\/swildow.darktech.org\/wp\/?p=1889"},"modified":"2016-09-19T19:48:30","modified_gmt":"2016-09-20T00:48:30","slug":"the-non-beginners-guide-to-syncing-data-with-rsync","status":"publish","type":"post","link":"https:\/\/www.wildow.com\/blog\/?p=1889","title":{"rendered":"The Non-Beginner\u2019s Guide to Syncing Data with Rsync"},"content":{"rendered":"

The Non-Beginner\u2019s Guide to Syncing Data with Rsync<\/a><\/h2>\n
\n

\"\"<\/p>\n

The rsync protocol can be pretty simple to use for ordinary backup\/synchronization jobs, but some of its more advanced features may surprise you.\u00a0 In this article, we\u2019re going to show how even the biggest data hoarders and backup enthusiasts can wield rsync as a single solution for all of their data redundancy needs.<\/p>\n

Warning: Advanced Geeks Only<\/h3>\n

If you\u2019re sitting there thinking \u201cWhat the heck is rsync?\u201d or \u201cI only use rsync for really simple tasks,\u201d you may want to check out our previous article on how to use rsync to backup your data on Linux<\/a>, which gives an introduction to rsync, guides you through installation, and showcases its more basic functions.\u00a0 Once you have a firm grasp of how to use rsync (honestly, it isn\u2019t that complex) and are comfortable with a Linux terminal, you\u2019re ready to move on to this advanced guide.<\/p>\n

Running rsync on Windows<\/h3>\n

First, let\u2019s get our Windows readers on the same page as our Linux gurus.\u00a0 Although rsync is built to run on Unix-like systems, there\u2019s no reason that you shouldn\u2019t be able to use it just as easily on Windows.\u00a0 Cygwin<\/a> produces a wonderful Linux API that we can use to run rsync, so head over to their website and download the 32-bit<\/a> or 64-bit<\/a> version, depending on your computer.<\/p>\n

Installation is straightforward; you can keep all options at their default values until you get to the \u201cSelect Packages\u201d screen.<\/p>\n

\"\"<\/p>\n

Now you need to do the same steps for Vim and SSH, but the packages are going to look a bit different when you go to select them, so here are some screenshots:<\/p>\n

Installing Vim:<\/p>\n

\"\"<\/p>\n

Installing SSH:<\/p>\n

\"\"<\/p>\n

After you\u2019ve selected those three packages, keep clicking next until you finish the installation. Then you can open Cygwin by clicking on the icon that the installer placed on your desktop.<\/p>\n

rsync Commands: Simple to Advanced<\/h3>\n

Now that the Windows users are on the same page, let\u2019s take a look at a simple rsync command, and show how the use of some advanced switches can quickly make it complex.<\/p>\n

Let\u2019s say you have a bunch of files that need backed up \u2013 who doesn\u2019t these days?\u00a0 You plug in your portable hard drive so you can backup your computers files, and issue the following command:<\/p>\n

rsync -a \/home\/geek\/files\/ \/mnt\/usb\/files\/<\/code><\/p><\/blockquote>\n

Or, the way it would look on a Windows computer with Cygwin:<\/p>\n

rsync -a \/cygdrive\/c\/files\/ \/cygdrive\/e\/files\/<\/code><\/p><\/blockquote>\n

Pretty simple, and at that point there\u2019s really no need to use rsync, since you could just drag and drop the files.\u00a0 However, if your other hard drive already has some of the files and just needs the updated versions plus the files that have been created since the last sync, this command is handy because it only sends the new data over to the hard drive.\u00a0 With big files, and especially transferring files over the internet, that is a big deal.<\/p>\n

Backing up your files to an external hard drive and then keeping the hard drive in the same location as your computer is a very bad idea, so let\u2019s take a look at what it would require to start sending your files over the internet to another computer (one you\u2019ve rented, a family member\u2019s, etc).<\/p>\n

rsync -av --delete -e 'ssh -p 12345\u2019 \/home\/geek\/files\/ geek2@10.1.1.1:\/home\/geek2\/files\/<\/code><\/p><\/blockquote>\n

The above command would send your files to another computer with an IP address of 10.1.1.1.\u00a0 It would delete extraneous files from the destination that no longer exist in the source directory, output the filenames being transferred so you have an idea of what\u2019s going on, and tunnel rsync through SSH on port 12345.<\/p>\n

The -a -v -e --delete<\/code> switches are some of the most basic and commonly used; you should already know a good deal about them if you\u2019re reading this tutorial.\u00a0 Let\u2019s go over some other switches that are sometimes ignored but incredibly useful:<\/p>\n

--progress<\/code> \u2013 This switch allows us to see the transfer progress of each file.\u00a0 It\u2019s particularly useful when transferring large files over the internet, but can output a senseless amount of information when just transferring small files across a fast network.<\/p>\n

An rsync command with the --progress<\/code> switch as a backup is in progress:<\/p>\n

\"\"<\/p>\n

--partial<\/code> \u2013 This is another switch that is particularly useful when transferring large files over the internet.\u00a0 If rsync gets interrupted for any reason in the middle of a file transfer, the partially transferred file is kept in the destination directory and the transfer is resumed where it left off once the rsync command is executed again.\u00a0 When transferring large files over the internet (say, a couple of gigabytes), there\u2019s nothing worse than having a few second internet outage, blue screen, or human error trip up your file transfer and having to start all over again.<\/p>\n

-P<\/code> \u2013 this switch combines --progress<\/code> and --partial<\/code>, so use it instead and it will make your rsync command a little neater.<\/p>\n

-z<\/code> or --compress<\/code> \u2013 This switch will make rsync compress file data as it\u2019s being transferred, reducing the amount of data that has to be sent to the destination.\u00a0 It\u2019s actually a fairly common switch but is far from essential, only really benefiting you on transfers between slow connections, and it does nothing for the following types of files: 7z, avi, bz2, deb, g,z iso, jpeg, jpg, mov, mp3, mp4, ogg, rpm, tbz, tgz, z, zip.<\/p>\n

-h<\/code> or --human-readable<\/code> \u2013 If you\u2019re using the --progress<\/code> switch, you\u2019ll definitely want to use this one as well.\u00a0 That is, unless you like to convert bytes to megabytes on the fly.\u00a0 The -h<\/code> switch converts all outputted numbers to human-readable format, so you can actually make sense of the amount of data being transferred.<\/p>\n

-n<\/code> or --dry-run<\/code> \u2013 This switch is essential to know when you\u2019re first writing your rsync script and testing it out.\u00a0 It performs a trial run but doesn\u2019t actually make any changes \u2013 the would-be changes are still outputted as normal, so you can read over everything and make sure it looks okay before rolling your script into production.<\/p>\n

-R<\/code> or --relative<\/code> \u2013 This switch must be used if the destination directory doesn\u2019t already exist.\u00a0 We will use this option later in this guide so that we can make directories on the target machine with timestamps in the folder names.<\/p>\n

--exclude-from<\/code> \u2013 This switch is used to link to an exclude list that contains directory paths that you don\u2019t want backed up.\u00a0 It just needs a plain text file with a directory or file path on each line.<\/p>\n

--include-from<\/code> \u2013 Similar to --exclude-from<\/code>, but it links to a file that contains directories and file paths of data you want backed up.<\/p>\n

--stats<\/code> \u2013 Not really an important switch by any means, but if you are a sysadmin, it can be handy to know the detailed stats of each backup, just so you can monitor the amount of traffic being sent over your network and such.<\/p>\n

--log-file<\/code> \u2013 This lets you send the rsync output to a log file. We definitely recommend this for automated backups in which you aren\u2019t there to read through the output yourself.\u00a0 Always give log files a once over in your spare time to make sure everything is working properly.\u00a0 Also, it\u2019s a crucial switch for a sysadmin to use, so you\u2019re not left wondering how your backups failed while you left the intern in charge.<\/p>\n

Let\u2019s take a look at our rsync command now that we have a few more switches added:<\/p>\n

rsync -avzhP --delete --stats --log-file=\/home\/geek\/rsynclogs\/backup.log --exclude-from '\/home\/geek\/exclude.txt' -e 'ssh -p 12345' \/home\/geek\/files\/ geek2@10.1.1.1:\/home\/geek2\/files\/<\/code><\/p><\/blockquote>\n

The command is still pretty simple, but we still haven\u2019t created a decent backup solution.\u00a0 Even though our files are now in two different physical locations, this backup does nothing to protect us from one of the main causes of data loss: human error.<\/p>\n

Snapshot Backups<\/h3>\n

If you accidentally delete a file, a virus corrupts any of your files, or something else happens whereby your files are undesirably altered, and then you run your rsync backup script, your backed up data is overwritten with the undesirable changes.\u00a0 When such a thing occurs (not if, but when), your backup solution did nothing to protect you from your data loss.<\/p>\n

The creator of rsync realized this, and added the --backup<\/code> and --backup-dir<\/code> arguments so users could run differential backups.\u00a0 The very first example on rsync\u2019s website<\/a> shows a script where a full backup is run every seven days, and then the changes to those files are backed up in separate directories daily.\u00a0 The problem with this method is that to recover your files, you have to effectively recover them seven different times.\u00a0 Moreover, most geeks run their backups several times a day, so you could easily have 20+ different backup directories at any given time.\u00a0 Not only is recovering your files now a pain, but even just looking through your backed up data can be extremely time consuming \u2013 you\u2019d have to know the last time a file was changed in order to find its most recent backed up copy.\u00a0 On top of all that, it\u2019s inefficient to run only weekly (or even less often in some cases) incremental backups.<\/p>\n

Snapshot backups to the rescue!\u00a0 Snapshot backups are nothing more than incremental backups, but they utilize hardlinks to retain the file structure of the original source.\u00a0 That may be hard to wrap your head around at first, so let\u2019s take a look at an example.<\/p>\n

Pretend we have a backup script running that automatically backs up our data every two hours.\u00a0 Whenever rsync does this, it names each backup in the format of: Backup-month-day-year-time.<\/p>\n

So, at the end a typical day, we\u2019d have a list of folders in our destination directory like this:<\/p>\n

\"\"<\/p>\n

When traversing any of those directories, you\u2019d see every file from the source directory exactly as it was at that time.\u00a0 Yet, there would be no duplicates across any two directories.\u00a0 rsync accomplishes this with the use of hardlinking through the --link-dest=DIR<\/code> argument.<\/p>\n

Of course, in order to have these nicely- and neatly-dated directory names, we\u2019re going to have to beef up our rsync script a bit.\u00a0 Let\u2019s take a look at what it would take to accomplish a backup solution like this, and then we\u2019ll explain the script in greater detail:<\/p>\n

#!\/bin\/bash<\/code><\/p>\n

#copy old time.txt to time2.txt<\/code><\/p>\n

yes | cp ~\/backup\/time.txt ~\/backup\/time2.txt<\/code><\/p>\n

#overwrite old time.txt file with new time<\/code><\/p>\n

echo `date +\u201d%F-%I%p\u201d` > ~\/backup\/time.txt<\/code><\/p>\n

#make the log file<\/code><\/p>\n

echo \u201c\u201d > ~\/backup\/rsync-`date +\u201d%F-%I%p\u201d`.log<\/code><\/p>\n

#rsync command<\/code><\/p>\n

rsync -avzhPR --chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r --delete --stats --log-file=~\/backup\/rsync-`date +\u201d%F-%I%p\u201d`.log --exclude-from '~\/exclude.txt' --link-dest=\/home\/geek2\/files\/`cat ~\/backup\/time2.txt` -e 'ssh -p 12345' \/home\/geek\/files\/ geek2@10.1.1.1:\/home\/geek2\/files\/`date +\u201d%F-%I%p\u201d`\/<\/code><\/p>\n

#don\u2019t forget to scp the log file and put it with the backup<\/code><\/p>\n

scp -P 12345 ~\/backup\/rsync-`cat ~\/backup\/time.txt`.log geek2@10.1.1.1:\/home\/geek2\/files\/`cat ~\/backup\/time.txt`\/rsync-`cat ~\/backup\/time.txt`.log<\/code><\/p><\/blockquote>\n

That would be a typical snapshot rsync script.\u00a0 In case we lost you somewhere, let\u2019s dissect it piece by piece:<\/p>\n

The first line of our script copies the contents of time.txt to time2.txt.\u00a0 The yes pipe is to confirm that we want to overwrite the file.\u00a0 Next, we take the current time and put it into time.txt.\u00a0 These files will come in handy later.<\/p>\n

The next line makes the rsync log file, naming it rsync-date.log (where date is the actual date and time).<\/p>\n

Now, the complex rsync command that we\u2019ve been warning you about:<\/p>\n

-avzhPR, -e, --delete, --stats, --log-file, --exclude-from, --link-dest<\/code> \u2013 Just the switches we talked about earlier; scroll up if you need a refresher.<\/p>\n

--chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r<\/code> \u2013 These are the permissions for the destination directory.\u00a0 Since we are making this directory in the middle of our rsync script, we need to specify the permissions so that our user can write files to it.<\/p>\n

The use of date and cat commands<\/b><\/p>\n

We\u2019re going to go over each use of the date and cat commands inside the rsync command, in the order that they occur.\u00a0 Note: we\u2019re aware that there are other ways to accomplish this functionality, especially with the use of declaring variables, but for the purpose of this guide, we\u2019ve decided to use this method.<\/p>\n

The log file is specified as:<\/p>\n

~\/backup\/rsync-`date +\u201d%F-%I%p\u201d`.log<\/code><\/p><\/blockquote>\n

Alternatively, we could have specified it as:<\/p>\n

~\/backup\/rsync-`cat ~\/backup\/time.txt`.log<\/code><\/p><\/blockquote>\n

Either way, the --log-file<\/code> command should be able to find the previously created dated log file and write to it.<\/p>\n

The link destination file is specified as:<\/p>\n

--link-dest=\/home\/geek2\/files\/`cat ~\/backup\/time2.txt`<\/code><\/p><\/blockquote>\n

This means that the --link-dest<\/code> command is given the directory of the previous backup.\u00a0 If we are running backups every two hours, and it\u2019s 4:00PM at the time we ran this script, then the --link-dest<\/code> command looks for the directory created at 2:00PM and only transfers the data that has changed since then (if any).<\/p>\n

To reiterate, that is why time.txt is copied to time2.txt at the beginning of the script, so the --link-dest<\/code> command can reference that time later.<\/p>\n

The destination directory is specified as:<\/p>\n

geek2@10.1.1.1:\/home\/geek2\/files\/`date +\u201d%F-%I%p\u201d`<\/code><\/p><\/blockquote>\n

This command simply puts the source files into a directory that has a title of the current date and time.<\/p>\n

Finally, we make sure that a copy of the log file is placed inside the backup.<\/p>\n

scp -P 12345 ~\/backup\/rsync-`cat ~\/backup\/time.txt`.log geek2@10.1.1.1:\/home\/geek2\/files\/`cat ~\/backup\/time.txt`\/rsync-`cat ~\/backup\/time.txt`.log<\/code><\/p><\/blockquote>\n

We use secure copy on port 12345 to take the rsync log and place it in the proper directory.\u00a0 To select the correct log file and make sure it ends up in the right spot, the time.txt file must be referenced via the cat command.\u00a0 If you\u2019re wondering why we decided to cat time.txt instead of just using the date command, it\u2019s because a lot of time could have transpired while the rsync command was running, so to make sure we have the right time, we just cat the text document we created earlier.<\/p>\n

Automation<\/h3>\n

Use Cron on Linux<\/a> or Task Scheduler on Windows<\/a> to automate your rsync script.\u00a0 One thing you have to be careful of is making sure that you end any currently running rsync processes before continuing a new one.\u00a0 Task Scheduler seems to close any already running instances automatically, but for Linux you\u2019ll need to be a little more creative.<\/p>\n

Most Linux distributions can use the pkill command, so just be sure to add the following to the beginning of your rsync script:<\/p>\n

pkill -9 rsync<\/code><\/p><\/blockquote>\n

Encryption<\/h3>\n

Nope, we\u2019re not done yet.\u00a0 We finally have a fantastic (and free!) backup solution in place, but all of our files are still susceptible to theft.\u00a0 Hopefully, you\u2019re backing up your files to some place hundreds of miles away.\u00a0 No matter how secure that faraway place is, theft and hacking can always be problems.<\/p>\n

In our examples, we have tunneled all of our rsync traffic through SSH, so that means all of our files are encrypted while in transit to their destination.\u00a0 However, we need to make sure the destination is just as secure.\u00a0 Keep in mind that rsync only encrypts your data as it is being transferred, but the files are wide open once they reach their destination.<\/p>\n

One of rsync\u2019s best features is that it only transfers the changes in each file.\u00a0 If you have all of your files encrypted and make one minor change, the entire file will have to be retransmitted as a result of the encryption completely randomizing all of the data after any change.<\/p>\n

For this reason, it\u2019s best\/easiest to use some type of disk encryption, such as BitLocker<\/a> for Windows or dm-crypt<\/a> for Linux.\u00a0 That way, your data is protected in the event of theft, but files can be transferred with rsync and your encryption won\u2019t hinder its performance.\u00a0 There are other options available that work similarly to rsync or even implement some form of it, such as Duplicity, but they lack some of the features that rsync has to offer.<\/p>\n

After you\u2019ve setup your snapshot backups at an offsite location and encrypted your source and destination hard drives, give yourself a pat on the back for mastering rsync and implementing the most foolproof data backup solution possible.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

The Non-Beginner\u2019s Guide to Syncing Data with Rsync The rsync protocol can be pretty simple to use for ordinary backup\/synchronization jobs, but some of its more advanced features may surprise you.\u00a0 In this article, we\u2019re going to show how even … Read More »<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[],"class_list":["post-1889","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1889","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1889"}],"version-history":[{"count":1,"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1889\/revisions"}],"predecessor-version":[{"id":1890,"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1889\/revisions\/1890"}],"wp:attachment":[{"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1889"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1889"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1889"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}