Large Volume FOSS Backup Options

16 replies [Last post]
davidpgil
Offline
Joined: 08/26/2015

I've started to notice I will soon need to do some type of high volume backup in the bear future due ti the nature of wirking wuth video, audio and very large graphical source files. I need to be able to archive this work that could grow to be in the tens of terabytes large. What are my options? Tape backup? FOSS tape backup options? Any other suggestions?

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

An external hard drive you store away from the computer (so that they do not get robbed together, they do not burn in a fire in the same room, etc.) is probably the easiest option. Better (but slower): backup on a remote computer, through the network. You can use the "Backup" utility that is shipped by default with Trisquel.

davidpgil
Offline
Joined: 08/26/2015

Right now, I use BackInTime with an external RAID1. Its too little space.

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

Well, you can acquire an additional larger disk.

davidpgil
Offline
Joined: 08/26/2015

... thinking on this, Perhaps what I need is a backup solition that compresses the data as it backs up. Right now, it all uncompressed.

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

The Backup utility in Trisquel compresses, by default. But you are talking about video, audio and pictures: such files are typically already in compressed formats: there is basically no more space you can save.

davidpgil
Offline
Joined: 08/26/2015

Thats definitely not true. Especially for source files -- I know this from experience.

I will look into another program that compresses files as it backs them up. The default program in Trisquel is too simple for what I need.

Doh... I just remembered why I don't use compression. Its because the system I use, does a differential backup. I think that there is no way to compare files easily if they are compressed.

... Now I think I understand why tape drives are used. Lots of space on one medium for cheap. Those LTO drives are expensive...

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

We are not talking about the same thing. I thought you were talking about end-user multimedia files (jpeg, png, mp3, vorbis, mp4, webm, etc.): those are formats that are compressed with algorithms that take into account the specifics of the medium, thereby reaching better compression ratios that what generic algorithms (gz, bzip2, etc.) achieve.

Trisquel's default tool for backup, Déjà Dup, is incremental too. It is a graphical interface to 'duplicity'. Unless DéjàDup, 'duplicity' can be tuned and probably can do what you want.

davidpgil
Offline
Joined: 08/26/2015

To clarify, I create images, music, video, etc. So, naturally the output files I generate can be compressed or uncompressed. Sometimes when you make a final output file, you need to make a few uncompressed versions of the file before mastering. Regarding, BackInTime -- my current backup software, it does not compress the files do to a problem with hard-links when restoring the file.

I think the reason why I left Déjà Dup behind was that it does not "smart delete" backups when the drive is full. I used to use it but I stopped using it years ago in favor of BackInTime.

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

By default, Déjà Dup does "smart delete" if, by that, you mean removing the oldest backups if space is missing. At least, that is what is written in its "help" (I do not think I have ever reached the capacity of my backup disks).

davidpgil
Offline
Joined: 08/26/2015

I definitely have. Ok, ill take another look at Dejadup again to see if it can compress everything as youve stated. It could very well be that I switched to backintime for another reason in the past, but I can't recall. Thanks for the assistance.

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

You may have quit Déjà Dup because its automatic backup is buggy. When I plug the external hard drive, Déjà Dup almost always notifies of a problem. But, then, firing the backup from Déjà Dup's has always perfectly worked for me.

davidpgil
Offline
Joined: 08/26/2015

... Interesting. BackInTime's automatic backup is not buggy... Not sure I want to swap to that particular problem. I need hourly automatic backups, per day. Once the day passes, BackInTime determines if any of the backups were worth keeping ... Anyway, I did some research just now and it seems that 12 TB Hard Disks exist! Not cheap, but since I use my backups for professional purposes, I might need to invest in that.

aloniv

I am a translator!

Offline
Joined: 01/11/2011

16 TB hard dries have also been announced:

https://soylentnews.org/article.pl?sid=19/01/10/1722226&from=rss

davidpgil
Offline
Joined: 08/26/2015

Heh, wild.

davidpgil
Offline
Joined: 08/26/2015

By the way, I am now considering using a JBOD volume to save a lot of data for cheap. My thinking is I can afford to lose some data because I have multiple snapshot of a lot of data, so if one drive fails one day, I can just replace it with a new one and my backup program can just figure out the rest. I know I should probably do RAID 1 but, its kind of priceyor me... What do you think?

Magic Banana

I am a member!

Offline
Joined: 07/24/2010

RAID 1 looks nice indeed. I have never used RAID though. However, I have read software RAIS, as implemented in Linux, is simpler and more efficient than trying to use the embedded RAID controller that your motherboard may have. Then, there is pure hardware RAID but that is not the same price...