[mdlug] Compression and Archiving

[Ingles, Raymond]

> From: Drew

> I find myself with a lot of data to archive and back up - 120 GB of
it, with possibly more to
> come at a later date - and am considering DVDs and squashfs.

 Burned optical media has a less-than-stellar reputation for reliability
in my experience.
Hard drives are generally held to be considerably more reliable,
long-term. If you wish to
use DVD media, however, see here:

 http://news.slashdot.org/article.pl?sid=06/12/11/1714232

 This article covers various brands of optical media and their projected
reliability.

> Is there a way to specify better compression methods? Is there a
better filesystem to use?
> Or would I be better off using something like gzip or bz2? I'd like to
be able to mount
> these as loop devices.

 What compression works best depends critically on the type of data
being compressed. I
suggest trying some different schemes and seeing what works best.
However, I'm not sure that
you can count on being able to loopback-mount gzipped files, while
squashfs is designed for
that. And already-compressed files (and PDFs usually have some internal
compression) don't
compress all that well, though.

 As others have noted, though, compression impacts reliability. Small
errors can be
catastrophic, especially when all the files are globbed into one big
block - if you can't
recover the whole package, you potentially lose multiple files.

 One solution is to add some redundancy... but that means more data to
store. The 'par2'
utility allows you to add redundancy in a controlled manner. Given a
partially-readable
file and the par2 redundancy files, you should be able to recover the
entire file...
so long as the damage isn't too severe.

> Also, can an entire DVD be a squashfs filesystem? Or does it have to
> be wrapped in an iso?

Based on my understanding, to write onto optical media needs a
filesystem format specifically
designed for that. So, either iso9660 or UDF.

 Sincerely,

 Ray Ingles                                                  (313)
227-2317

 "A Georgia State University study shows that U.S. senators have an
uncanny
  knack for picking stocks that outpace the overall market. Professor
Alan
   Ziobrowski's analysis of senators' financial disclosure data found
that
   over a period of six years, the lawmakers outperformed the market by
12
      percent." - http://www.npr.org/features/feature.php?wfId=1751162

The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it.