Considering New Backup Options

[Katsmeow]

OK, this is a think going to be a "dumb" question. But, here it goes.

I actually have all kinds of cloud storage from Google Drive to Dropbox to OneDrive. I subscribe to Office 365 and they are just about to raise the OneDrive limit for each user from 20 GB to 1 TB making this suddenly very useful to me for backup purposes. So here is the question.

I set up One Drive as a folder and it wants me to put Documents, Pictures, etc. in there. Now, lets use Document as an example.

If put my Documents folder in the One Drive folder:

1. Are the Documents now going to automatically be synced to the One Drive folder on the cloud?

If so, if I delete a file on my notebook will that delete it on the cloud as well? If so, the what is on the cloud isn't really acting as a back up as it will get deleted there as well. What I want is to be able to have my documents on my notebook be on the OneDrive in the cloud storage, but if I delete something by mistake I want to be able to go get it from the cloud storage.

If the cloud storage is identical to what is in the Documents folder in the OneDrive folder, is the solution to the problem above to keep two copies of my documents. One in my "regular" Documents folder and then another in the One Drive documents folder and then periodically sync those two. That way if I delete something by mistake in my regular folder I can still get it from the Documents folder in the OneDrive folder?

2. Will the Documents in the OneDrive folder be only in the OneDrive folder on the cloud or will they be in both places? That is, locally on the hard drive and on the cloud?

3. Let's say I have another computer (which I do) and I put its Documents folder in the One Drive folder on that computer (I am the user on both computers). Will the documents in that Documents folder be synced on the cloud into a separate documents folder as the documents from my notebook or will all the documents be mixed together on the OneDrive in the cloud?

(I may have other questions, but they depend on the answers to these.)

[meierlde]

Note that you have to create a share point on the drive for file history to back up to the same drive, it is do able but takes more work it tries to avoid backing up to the same drive.
There is an MS tool called synctoy that will synchronize two file trees, but just keep the last copy. When I tried file history it kept backing up all files in the tree about every 3 days. While you can then clean up to all but the last copy, but then its just an automatic synctoy.

[imoldernu]

I don't do machine back ups... I have 8 working computers and a few tablets. Most of them have the same info, transferred at one time or another for some forgotten reason. All are old.
Last week after trying hard to reduce start up time on my vista machine, lost patience, shredded all and started from scratch. Out of curiosity, used search "Everything" for the stuff I'd like to keep... After stripping duplicate files with SlimCleaner and Glary, still have 42,000 jpg's, 7900 mp3's, and 8000 txt's.

Copies of all important documents on an 8G thumb drive and on Google drive.

Since I've never paid for software of any kind, anything I use I can download again for free.

I am reminded of something a w*rk enemy said to me as I was closing out the last of my company's franchise operations, and working myself out of a job.....

"See that pail of water? If I put my hand in it, and then take it out... that's how much you'll be missed."

It stuck with me, and now my philosophy of life has changed. What I had saved for my heirs and posterity... essays, writings, studies, arguments, photography, music, genealogies, and lifetime accomplishments, family history etc... all of this will go to the landfill. My kids have enough challenges in life, and I won't add to them by placing a guilt complex on their conscience.

[photoguy]

Quote:

Originally Posted by ERD50 A) First run my 'recently changed' script to capture all the files that were changed in the one day since I backed up to portable drives, and store to the cloud unzipped. B) Run my 'recently changed' script every day after that and capture files changed only within the last day (plus a few hours for overlap) - copy these temporarily to a directory on my hard drive. C) Now do an incremental update to the cloud from that temp directory. Delete the temp directory. Repeat from 'B'. That should build up all the changed files in one neat directory structure in the cloud. Baseline would be missing, but I have that on my backups. And I could sync from the cloud to my backups if my internal HD crashed, and have everything in one place. Assuming rsync works to the cloud, but I'm pretty sure it does. Since only the daily changes go to the cloud, this should be fast. Anyone see any problems? This should be easy to turn into a one or two button script. -ERD50

Do you have your own server to run as destination for rsync? I don't think the existing cloud backup services let you have as much control as rsync with the command line options.

You may not need to roll your own script. I think rsync has the ability to copy files by date and also put the incremental changes in a specific directory.

Rsync won't support file versioning so you lose that ability. I guess you could modify your script to put incremental backups in different directories by time.

Your script is based on date but I think there are some file operations that may not change the timestamp. E.g. if I copy/ftp/download a file from somewhere else (say another computer) will my copy have the original date and not the the date of the copy? When you download photos from your phone, will the file date be the date of the download? or the date you took the photo?

Overall I guess the main danger is that the process is more complex which means bugs/errors can come into play.

[TromboneAl]

Quote:

Originally Posted by Katsmeow OK, this is a think going to be a "dumb" question. But, here it goes. I actually have all kinds of cloud storage from Google Drive to Dropbox to OneDrive. I subscribe to Office 365 and they are just about to raise the OneDrive limit for each user from 20 GB to 1 TB making this suddenly very useful to me for backup purposes. So here is the question. I set up One Drive as a folder and it wants me to put Documents, Pictures, etc. in there. Now, lets use Document as an example. If put my Documents folder in the One Drive folder: 1. Are the Documents now going to automatically be synced to the One Drive folder on the cloud? If so, if I delete a file on my notebook will that delete it on the cloud as well? If so, the what is on the cloud isn't really acting as a back up as it will get deleted there as well. What I want is to be able to have my documents on my notebook be on the OneDrive in the cloud storage, but if I delete something by mistake I want to be able to go get it from the cloud storage. If the cloud storage is identical to what is in the Documents folder in the OneDrive folder, is the solution to the problem above to keep two copies of my documents. One in my "regular" Documents folder and then another in the One Drive documents folder and then periodically sync those two. That way if I delete something by mistake in my regular folder I can still get it from the Documents folder in the OneDrive folder? 2. Will the Documents in the OneDrive folder be only in the OneDrive folder on the cloud or will they be in both places? That is, locally on the hard drive and on the cloud? 3. Let's say I have another computer (which I do) and I put its Documents folder in the One Drive folder on that computer (I am the user on both computers). Will the documents in that Documents folder be synced on the cloud into a separate documents folder as the documents from my notebook or will all the documents be mixed together on the OneDrive in the cloud? (I may have other questions, but they depend on the answers to these.)

From what I've seen, OneDrive just acts as a another drive. There is no syncing. So "putting the documents folder on OneDrive" just means that you would decide that that is where you will keep all your documents. You could move what you have, and put new ones there in the future. If you access it with a different computer, it will just act as if you plugged that drive into the other computer.

That's how it works for me, though I keep my documents on my laptop's hard drive. I want to be able to access them in the internet is down.

[ERD50]

Quote:

Originally Posted by imoldernu I don't do machine back ups... I have 8 working computers and a few tablets. Most of them have the same info, transferred at one time or another for some forgotten reason. ... Copies of all important documents on an 8G thumb drive and on Google drive. .... What I had saved for my heirs and posterity... essays, writings, studies, arguments, photography, music, genealogies, and lifetime accomplishments, family history etc... all of this will go to the landfill. My kids have enough challenges in life, and I won't add to them by placing a guilt complex on their conscience.

If you have the same stuff on multiple machines, isn't that a 'machine backup'? And the 'Copies of all important documents on an 8G thumb drive and on Google drive', isn't that a 'machine backup'? I don't understand your distinction here.

I'm also not following you on the guilt trip for your kids? Who is going to send it to the landfill? How will they know if it should be kept or not?

I've procrastinated, but I should put the stuff that I think could be worth keeping on a drive, label it and let them know about it. If they want to trash it, that's fine, but at least they will know that's it. They don't need to worry about searching through all my other trivia, wondering if some important document is tucked away in there somewhere.

Quote:

Originally Posted by photoguy Do you have your own server to run as destination for rsync? I don't think the existing cloud backup services let you have as much control as rsync with the command line options. You may not need to roll your own script. I think rsync has the ability to copy files by date and also put the incremental changes in a specific directory. Rsync won't support file versioning so you lose that ability. I guess you could modify your script to put incremental backups in different directories by time. Your script is based on date but I think there are some file operations that may not change the timestamp. E.g. if I copy/ftp/download a file from somewhere else (say another computer) will my copy have the original date and not the the date of the copy? When you download photos from your phone, will the file date be the date of the download? or the date you took the photo? Overall I guess the main danger is that the process is more complex which means bugs/errors can come into play.

Good points, I'll check that out. This WIP plan for my 'cloud incrementals' is really just interim daily backups in between an (weekly?) incremental to my portable drive - that would pick up everything. So if I lose something in the cloud that was downloaded with an older date, it's probably not a big deal. But something to consider.

Writing the script was a learning experience for me, I had not done that in Linux, and wanted some experience. It's kind of fun to be able to customize something to do exactly what you want, and they seem to run so much faster than the equivalent GUI steps (in some cases).

But this has me thinking about another idea I had - let's see if I can explain this:

Could rsync be set up to create a baseline directory/catalog of all the files/folders, but without the actual files? The idea would be do a full backup to a portable drive, and tell rsync that today's catalog represents that baseline. The next day, point rsync to a new destination folder, and let rsync update that folder with the past day's changes, and update that baseline catalog to 'today'.

The idea there is, you could have say two full baseline backups on portable drives, but the 'changes only' incrementals would be small, and could be put on a USB stick or the cloud, as they would be small w/o the full baseline.

If that new destination was in the cloud, only the days changes would be uploaded, and you never had to upload the larger baseline to the cloud. But it also looks to be a little tricky to get rsync to work with google-drive in Linux, but there may be workarounds. Google drive isn't really as simple as a network drive, which I believe rsync handles fine.

-ERD50

[Katsmeow]

Quote:

Originally Posted by TromboneAl From what I've seen, OneDrive just acts as a another drive. There is no syncing. So "putting the documents folder on OneDrive" just means that you would decide that that is where you will keep all your documents. You could move what you have, and put new ones there in the future. If you access it with a different computer, it will just act as if you plugged that drive into the other computer. That's how it works for me, though I keep my documents on my laptop's hard drive. I want to be able to access them in the internet is down.

This is clear as mud to me.

So, the documents that are on OneDrive. Are they on my notebook's hard drive or are they only in the cloud?

If the internet goes down, does that mean I have no access to them while it is down?

If so (they are only on the cloud) then it would seem to me I only want to keep a copy of them on OneDrive).

[ERD50]

Quote:

Originally Posted by Katsmeow This is clear as mud to me. So, the documents that are on OneDrive. Are they on my notebook's hard drive or are they only in the cloud? If the internet goes down, does that mean I have no access to them while it is down? If so (they are only on the cloud) then it would seem to me I only want to keep a copy of them on OneDrive).

Then the best approach is to just try it and see what happens. Sometimes that adds clarity when word descriptions sound confusing.

But it sounds like you want to save what you work on to your local drive as always, then copy that to the cloud. Then you have a local copy for when you don't have internet access, and a cloud copy as back up.

Try it and see.

-ERD50

[rbmrtn]

Quote:

Originally Posted by Katsmeow This is clear as mud to me. So, the documents that are on OneDrive. Are they on my notebook's hard drive or are they only in the cloud?

If you COPY the file to onedrive it is available on both. If you MOVE the file to onedrive it is available on onedrive only. If you SAVE the file to onedrive, same thing. If you are syncing then changes in one location are copied to the other. GoogleDrive works about the same.

onedrive FAQ , OneDrive: FAQ - Windows Help

Why are some files in my OneDrive only available when I'm online, and how can I make them available offline?
The files you added to OneDrive from your PC or previously opened on your PC are automatically available offline on that PC. Any files you created on the OneDrive website or another computer are online-only to save space on your PC.

[rbmrtn]

Quote:

Originally Posted by ERD50 I Could rsync be set up to create a baseline directory/catalog of all the files/folders, but without the actual files? The idea would be do a full backup to a portable drive, and tell rsync that today's catalog represents that baseline. The next day, point rsync to a new destination folder, and let rsync update that folder with the past day's changes, and update that baseline catalog to 'today'.

The -dry-run option might be close to what you want. It runs rsync without actually touching the files, so can you see what rsync would have done. But rsync doesn't keep a catalog, it calculates what it needs to do each run.

You might have to make use of pipes and scripting. Maybe pipe the output of find to rsync. something like this bash - find and rysnc? - Unix & Linux Stack Exchange

[TromboneAl]

Quote:

Originally Posted by Katsmeow This is clear as mud to me. 1. So, the documents that are on OneDrive. Are they on my notebook's hard drive or are they only in the cloud? 2. If the internet goes down, does that mean I have no access to them while it is down? 3. If so (they are only on the cloud) then it would seem to me I only want to keep a copy of them on OneDrive).

Think of a little box taped to your computer. That's your computer's drive. Now think of another box on the shelf of you closet. That's the OneDrive.

You write on some paper and put it in the box on your computer. That's available to you even if you shut the door to the closet. You can take that paper out and put it in the box in the closet. If the door to the closet is closed, you can't get to it.

Note that you could also make a Xerox of the paper so that you have one in the computer box, and another in the closet box. But, if you change one, the other won't be changed.

And the answers to your questions are:

1.If you move the docs to onedrive, they are not on your computer, and they are only on onedrive.

2. Yes, if the internet is down, you have no access to the docs on onedrive (closet door is closed and locked).

3. Yes, if there's something you may need when the internet is down or unavailable, you'd only want copies on onedrive. That's why I use it for backup.

[ERD50]

Quote:

Originally Posted by TromboneAl The problem with the manual system is you are using the computer, so the machine will run slower. So then you think, "I'll finish what I'm doing, then do the backup," and it often doesn't happen. I used that system for years, and often the Friday backup didn't happen. ...

It always seemed lightening fast to me, and this thread has me looking back to check things out, so I ran the scripts and tried to time it, and I couldn't - it was done in a blink.

So I modified some more files, some buried deep in the directory, so I had a total of ~ 34MB of files modified in the last two days, and ran my '2 day' script. If you are just working on a few spreadsheets and word processing, you'd likely mod less than 34MB in a day, but of course that could be all over the map. And I limit this to my 'documents' and 'desktop' - I don't worry about daily backups for other stuff, that gets picked up less often.

The longest portion seems to be the zipping, and everything was completed in less than two seconds. That does not include copying it to the cloud or a USB stick, but that should also be pretty fast.

For ref, here is my script (I have 3x as many lines documenting what it does, but this is the code):

mkdir -p ~/000_Recent_File_Backup_*SAVE*
NOW=$(date +000-%Y%m%d--%d%b%Y-%T)
find ~/Documents/ -ctime -1 -exec cp --parents --preserve=timestamps {} ~/000_Recent_File_Backup_*SAVE*/ \;
find ~/Desktop/ -ctime -1 -exec cp --parents --preserve=timestamps {} ~/000_Recent_File_Backup_*SAVE*/ \;
cd ~/000_Recent_File_Backup_*SAVE*
zip -r ./$NOW home/

BTW, the 'ctime' command detects any change in the file attributes (copy, rename, move from one place to another), even if the "modified date" shows an older date.

I'm guessing that one reason this can run so fast is that I have an indexing program running (indexes the files and content, so you can do searches like the Spotlight feature on a Mac), and any file mod updates the index at the time you make that change. So I think it must access that index, so the changes are already recorded, it doesn't need to search the whole directory to 'find' what has changed.

-ERD50

[rbmrtn]

Quote:

Originally Posted by ERD50 I'm guessing that one reason this can run so fast is that I have an indexing program running (indexes the files and content, so you can do searches like the Spotlight feature on a Mac), and any file mod updates the index at the time you make that change. So I think it must access that index, so the changes are already recorded, it doesn't need to search the whole directory to 'find' what has changed. -ERD50

find doesn't use a index or database, but locate does. locate can be faster when searching more space. updatedb usually runs as a cron job to update locates database. locate can also be told which db to use.

[kcowan]

I have never backed up my computers. I do have multiple backups for any personal files, like spreadsheets, documents and pictures. I have 5 computers in two geographic locations. I use WD external drives (x2) to keep all the personal stuff synchronized.

Whenever I have had a problems with the system, I use a computer shop to get it right. This has only happened twice in 10 years so think that is pretty good return for not doing all those system backups.

I use Karen's replicator which does backup only changed files. After she died, I was concerned about support, but her app just keeps working, even for Win7.

For portable devices, we use iTunes and iCloud.

(I have been purging physical stuff extensively this year. Got rid of all those Zip Drives that I never had to use for backup. And my CDs and DVDs that I never used for backup. Sometimes I wonder about all the effort we expend, just in case.)

[ERD50]

Quote:

Originally Posted by rbmrtn find doesn't use a index or database, but locate does. locate can be faster when searching more space. updatedb usually runs as a cron job to update locates database. locate can also be told which db to use.

I was curious, so went back further to a 30 day incremental. This was still only ~ 55MB, but it was across 163 items. I added some timers at several points in the program, and created an empty dir with a label and that time appended.

It still took just 2 seconds for the find/copy, and 2 seconds to zip it. Fast enough, I won't worry about any optimization. It makes me wonder if all that catalog info is cached, that still seems very fast to me. No matter, it works great, I'm just curious.

Quote:

Originally Posted by kcowan I have never backed up my computers. I do have multiple backups for any personal files, like spreadsheets, documents and pictures. I have 5 computers in two geographic locations. I use WD external drives (x2) to keep all the personal stuff synchronized. ...

To my way of thinking, you are backing up your computers, but it sounds like you are saying you back up the data, not the system. Which is reasonable for some.

I prefer to have a total system back-up (clone) too. That was EZ on Macs with "SuperDuper!" The fantastic feature of Macs is you can clone to an external, and immediately re-boot right into that clone to verify it. I think in Windows, you need to re-install from the backup, which is a Catch-22 for testing your clone. And I have not found an EZ way to do that in Linux either - all the various things I've read about have limitations I don't like. But it really isn't that hard to re-install the system, and I recently made a video that captured most of my setup changes, so that procedure is documented, but I'd rather have a bootable clone. I do have some other Linux installs on portable drives that are configured pretty close to my daily system, so if I did crash, I could boot from those and be 99% of the way there anyhow.

-ERD50

[photoguy]

Making a bootable clone is a great idea and something I recommend for people on Macs (to do in addition to time-machine). With Superduper and CCC this is trivial.

On my old computer I had it setup (with CCC) to automatically mirror the boot drive to a different hard disk in my MP. So even if my main disk went down, I could be running again with a simple restart.

Is there a simple, free (and easy) tool to do this with windows 8.1 (asking for my father-in-law's computer)? or is ERD50 right that you need to reinstall from the backup?

[rbmrtn]

Maybe overlaping two concepts, cloning versus imaging. A clone is a duplicate of your existing HD onto another HD. You would swap out the HDs in the event of a failure. An image is a copy of the HD written to a file. In the event of a failure you would use software to write the image onto a new HD ( reinstalling )

Here's a link to some tools to look at, Five free and reliable cloning tools - TechRepublic, there are others like Acronis true image, easeus todo backup.

Howto article on cloning here, How to Clone a Hard Drive | PCMag.com

[ERD50]

Quote:

Originally Posted by rbmrtn Maybe overlaping two concepts, cloning versus imaging. A clone is a duplicate of your existing HD onto another HD. You would swap out the HDs in the event of a failure. An image is a copy of the HD written to a file. In the event of a failure you would use software to write the image onto a new HD ( reinstalling ) Here's a link to some tools to look at, Five free and reliable cloning tools - TechRepublic, there are others like Acronis true image, easeus todo backup. Howto article on cloning here, How to Clone a Hard Drive | PCMag.com

Yes, cloning is what I'm referring to on the Mac. And as I mentioned, the beauty is that you can clone to an external drive, then boot directly from that clone - just hold the "option" key at boot time, and all the boot-able systems will appear, and you choose what you want. Super EZ to test it out after you think you cloned it. And if you have not tested it, I do not consider it 'cloned'. And if the internal drive crashes, you can be up and running from that clone in minutes, with no fuss/muss. Saved my butt once way back on OS-8 ('Classic'), I think it was. I ran for a week or two, booting from my ZIP drive (remember those?), until I had time to re-install the crashed system on the internal HD.

AFAIK, it takes a 'pro' version in Windows to be able to boot from that clone as an external drive. And if you have to swap drives to test it, well that's a fair amount of trouble.

I know I can clone my Linux system, but AFAIK, it is not easy to boot from that clone w/o some playing around (and/or removing the internal drive). The external clone will have UUIDs that match the internal drive, and that will conflict. I think I once successfully managed to change the UUIDs on the clone, and get all the internal references updated to match, but it was a lot of work and confusing (for me). And things like Remastersys don't seem to handle a full system, with all the installs and config files (which I have on a separate /home partition). I could clone w/o my data, and merge that later, no sweat. But for me, the purpose of the clone is to have all my system changes and updates. Without that, a clean install is probably less trouble.

-ERD50

[Katsmeow]

Got it. I think what I am going to want to do is just make a copy of data and put it on One drive and then periodically update it.

[rbmrtn]

Quote:

Originally Posted by ERD50 AFAIK, it takes a 'pro' version in Windows to be able to boot from that clone as an external drive. And if you have to swap drives to test it, well that's a fair amount of trouble. -ERD50

The boot function is done by the BIOS and bootloader, so any version of windows can boot from a supported hardware device. On many PCs F12 will bring up the boot menu during BIOS load and it allows the boot device to be specified ( sounds similar to Mac "options" button) so that is done before windows starts.

I don't think superduper is doing anything with the boot portion. It's simply cloning the drive. It appears maybe apple boot loader is detecting any bootable device on the fly and allowing it the boot menu ? ( I don't have a mac to investigate ) maybe superduper is modifying the bootloader ?

It seems maybe an extra step in windows and linux is that you need to modify the bootloader configuration but it's same concept. I'm not familiar enough with Mac to know what it doing with the loader.

And yes I still have several zip drives, ditto parallel port tape drives, a DAT drive and closet full of other junk ( even a couple of XT motherboards )

And I think the UUID scheme is more confusing than the device id, there are places where it is useful and other where it is more trouble. Fix one thing... break something else.