The State of Backup and Cloning Tools under Mac OS X

March 5th, 2006

Introduction

Cloning sheep

Back in the days of OS 9, backing up files was fairly easy. One would just use the Finder to copy files and directories to another volume, and be done. The simplicity, unfortunately, is gone with OS X. Such a simplistic approach is no longer a guarantee to preserve all data faithfully (neither is it a simple or reliable approach for a regular backup procedure). The trouble on OS X is mostly related to metadata, i.e., data about files and directories (such as modification date, file creator/type, Unix permissions, etc.).

Another problem arises when a complete system partition shall be backed up and be bootable later on. Making a backup bootable is not trivial.

Superficially, one could nourish high expectations about the state of backup solutions on Mac OS X, because the underlying BSD Unix core has made all the mature backup and file copying tools that have been developed for Unix systems available on the platform. However, the fly in the ointment is that these tools are generally not aware of Mac OS X metadata and, hence, fail to produce a faithful backup.

This essay will first investigate means to copy files as completely and reliably as possible on Mac OS X, if possible with free and open-source tools. It will conclude with an (incomplete) survey of dedicated backup tools. The tools covered are not only relevant for backup purposes, but also for the case of migrating machines, when the content of one hard drive is to be cloned to another one.

I will not address common features of backup software such as scheduling, backup management, and incremental backups. This piece will be solely about the bare basics of copying files.

The analysis presented here assumes a recent install of OS X 10.4.5 (Tiger) with all updates. The state of backup and cloning tools has already been worse, so there is no need to shoot ourselves into the foot by using outdated tools.

Copying Files under Mac OS X

Paradoxically, copying a file and being sure that all information has been copied is not easy under Mac OS X. However, achieving this goal (generally termed cloning) is obviously paramount for backup purposes. There is one main culprit for all issues: metadata.

Types of Metadata under Mac OS X

A file does not only consist of the file data itself, but also of accompanying information, called metadata. Different operating systems have traditionally supported a wide range of metadata, with many headaches in cross-platform environments stemming from differences in metadata support. Mac OS (X) has traditionally supported rich metadata compared with other operating systems. Underlying support for this metadata is given by the HFS+ file system, which is the successor of the venerable HFS file system.

Classic Unix metadata — The classic Unix metadata includes the file name (HFS+ supports unicode strings with a maximum length of 255 characters), the file modification date, file owner/group, and the POSIX permissions and file flags (changeable via chmod and chflags, respectively). These types of metadata items are accessible to all usual unix command-line tools. However, some information such as POSIX permissions is not accessible to classic Carbon APIs.

Update 2006-04-23 Owner/group of symlinks — Symbolic links are somewhat special. Their permissions are irrelevant in UNIX systems, so they don’t need to be preserved on copy (nor can they). However, symlinks do have an owner and group that may be different from the file they’re pointing to. In most cases, this information is not too important, but (i) it tells who generated the symlink, and (ii) some software actually makes use of this information, e.g., Apache when the SymLinksIfOwnerMatch option is switched on. Until OS X 10.4 there was no possibility in Darwin to change the owner/group of a symlink, so it was impossible to preserve this information. However, fortunately Apple added a lchown(2) call in OS X 10.4.

Finder FlagsThese flags (and data fields) are a relic of OS 9, and are mostly used by the Finder. Apple is still actively using many of these features, although the technology is rather outdated. In these fields, there are a number of binary flags (file invisible, name locked, etc.). Also, the file creator and file type codes are part of the finder flags. These are each 32-bit constants that specify the creating program and file type. Under OS X, file type and creator are usually no longer used, but they are still honored. Finally, there is more, undocumented data used by the Finder for purposes such as file label and icon position. The Finder Flags are partially accessible by the GetFileInfo and SetFile command-line tools, which come with the Apple Developer Tools. Up to OS X 10.3 (Panther), there was no means to access the flags from regular BSD APIs.

Creation date — Unlike classical Unix file systems, files on HFS(+) volumes have a creation date. The creation date can be accessed via Carbon APIs. Finder displays the creation date in the “Get Info” window.

Finder Comments — Finder comments (nowadays called “Spotlight comments”) are arbitrary comments that can be affiliated with a file using the “Get Info” window of the Finder. However, these comments are not really stored together with the corresponding file.

Finder comments have, in fact, experienced somewhat of an odyssey throughout the history of Mac OS. These days, they are stored in an invisible file called .DS_Store in the file’s parent directory. Thus, it is crucial for the preservation of Finder comments to keep the .DS_Store files when performing a backup.

Resource Forks — OS 9 (and HFS) has always supported two forks of a file. Information could equally well be stored in the data fork and the resource fork. The normal content of a file, such as seen on Unix or Windows, is in the data fork. The resource fork was used by Apple for storing structured information in a proprietary database-like format. Although the use of resource forks is somewhat deprecated (cf. the infamous Technote #2034), Apple still uses resource forks, for example for storing custom icons and information about the application to be launched when a file is double-clicked. Thus, a proper backup needs to preserve resource forks, even if no classic OS 9 software is used any more. I haven’t tried, but a clone of an OS X system without resource forks probably doesn’t work any more. The problem for backup purposes is that, in general, no Unix tools are aware of resource forks. Apple has always made them semi-available for BSD APIs by the pseudopath /path/to/file/..namedfork/rsrc, and more recently by a hack to the xattr mechanism (see below).

In fact, with HFS+ came the possibility of storing an arbitrary number of forks for a file, not only two. As far as I know, this feature is not yet used.

HFS+ Extended Attributes — With OS X 10.4 (Tiger), Apple introduced even more metadata, called HFS+ Extended Attributes. These extended attributes are name:data pairs that can essentially carry arbitrary information. The attributes are accessible via BSD APIs, but not via higher-level Carbon, Cocoa, or Core Foundation interfaces. I am not sure to what extent the extended attributes are actually being used today (except for ACLs, see below). Does anyone have examples?

Access Control Lists (ACLs) — ACLs are a finer-granular way of setting file permissions. ACLs were introduced in OS X 10.4 (Tiger). Although ACLs are not in widespread use (yet), it is still desirable that a backup tool preserves these permissions. ACLs are stored in HFS+ Extended Attributes, but they are masked out for the xattr APIs (according to Ars Technica). ACLs must be enabled for a volume before they can be used.

Spotlight Metadata — For the purpose of searching, Spotlight under OS X 10.4 (Tiger) centrally stores key-value pairs of metadata about files. This data is accessible via the mdls command-line tool. Spotlight metadata is an orthogonal concept to HFS+ Extended Attributes. Spotlight metadata is extracted from the file contents and stored centrally. To quote Ars Technica:

Yes, Spotlight extracts, stores, and indexes information about file system objects. Yes, this information is properly called file metadata. But this information is extracted from the file contents and traditional file system metadata fields (file name, dates, size, etc.) and is stored in external plain-file indexes.

The only way actual, arbitrarily extensible file system metadata is involved at all is if an application chooses to write extended attributes when it saves a file, and then a Spotlight metadata importer plug-in reads these extended attributes and passes their values off to Spotlight for storage in its index files. At the time of Tiger’s launch, no existing applications or metadata importer plug-ins do this.

Spotlight merely extracts, stores, and indexes file metadata. It does not and cannot be used to add arbitrary metadata to files. It can read a file and add metadata to the Spotlight index on behalf of that file, but the metadata is not “physically” attached to the file itself.

Hence, there is no need (neither is there a way) to backup Spotlight metadata belonging to a specific file.

Update 2006-04-04 As barefootguru rightly points out, Spotlight metadata is used by Apple to store data that is not stored in the corresponding file itself. In particular, Safari saves the download location of downloaded files in the kMDItemWhereFroms property. One could argue that this is a design flaw, since the information is lost upon copying the file or moving to another partition. Ideally, all Spotlight metadata for a file should be recoverable from the contents of the file. A more appropriate location of permanent metadata would be suitable HFS+ Extended Attributes. I could imagine, however, that this design decision was due to a trade-off between functionality and security/privacy; perhaps Apple didn’t want to create a metadata security nightmare such as it exists for Word documents. The lack of high-level APIs for HFS+ Extended Attributes might also have contributed to the “improper” implementation of this feature.

inode number (a.k.a. file ID) — Each file and directory on a file system is identified by a unique number called inode in the file system catalog. Strictly speaking, this number is not really metadata, and ideally it should be irrelevant for backup purposes. However, Apple, in the OS 7 days, invented an ingenious concept called Alias, which is somewhat of a smarter Unix symlink. An alias not only stores the path to the file or directory it points to, but also the inode number. Thus, if the target is moved in the file system (a situation where every Unix symlink chokes fatally), the target can still be identified uniquely. Ideally, the alias record is updated with new path information once the target is moved, so that a complete set of redundant information is available again. However, there is no automatic mechanism for such an update. Finder sometimes performs an update if an alias is followed explicitly.

The problem in the context of backups now is that once a backup is performed and restored to another volume, the original inode numbers have become obsolete. Thus, every alias has effectively been degraded to a symlink. Once the target file is moved, the alias is worthless. I’ve also seen some cases where aliases would randomly associate with different files. An ideal OS X backup would, therefore, preserve inode numbers; however, this is not possible short of a device-level clone of an entire volume. In many cases, one will have to put up with aliases breaking slowly after a backup restore.

Analysis of Low-Level Copying Tools

Now, on to an analysis of the available tools for copying files on OS X. Ideally, we would wish to have a universal file copying (or even better, synching) tool that preserves directory structure (taken for granted), file contents (also taken for granted), and all categories of metadata described above (very hard to achieve).

In the following table, for commonly used tools, I have depicted what categories of metadata the corresponding tools preserve.

ownSOpermBFFFlckMDCDFC [a]RFEAACLind
cp -RpApple (10.4.6)x- [e]xxxxx-(x)xxx-
CpMac -r -pApplex-x-xxxx(x)x---
FinderApple (10.4.6)--x-xxxxxxxx-
dittoApple (10.4.6)x- [f]x-x-x-(x)x---
rsync -aEApple (10.4.6)xxx-x-- [b]-(x)xx--
rsync_hfs –eahfs -ax-x(x) [d]x- [c]xx(x)x---
Superduper enginexxxxxxxxxxxx-
psync/MacOSX::Filex-x-xxxx(x)x---
ASR (dev mode)Apple (10.4.5)xxxxxxxxxxxxx
ASR (file mode)Apple (10.4.5)x- [g]x?xxx-xxxx-

Column headers:

Update 2006-04-23 added BSD flags, symlink owner

own — owner information for regular files and directories.<br/> SO — symlink owner information.<br/> perm — POSIX permissions.<br/> BF — BSD Flags, which can be set via chflags (see man page).<br/> FF — Finder Flags.<br/> lck — Locked flag (this is part of Finder Flags).<br/> MD — Modification date.<br/> CD — Creation date.<br/> FC — Finder comments.<br/> RF — Resource fork.<br/> EA — HFS+ extended attributes.<br/> ACL — ACLs.<br/> ind — inode.

Tools notes:

  1. cp -p is the cp command provided by Apple in Tiger.
  2. CpMac is contained in the Developer Tools. It was known to be even buggier, but some bugs appear to have been fixed recently.
  3. Finder stands for a drag-and-drop copy operation in Finder.
  4. rsync -aE is the rsync command provided by Apple in Tiger. Apple’s version of rsync has a history of being badly buggy. I wouldn’t trust it for serious backup purposes. Update 2006-04-23 In OS X 10.4.6 and with ACLs enabled, Apple’s rsync fails pretty much completely on me, almost no metadata is preserved; weird error messages such as “file has vanished” appear.
  5. rsync_hfs --eahfs -a is part of the RSyncX package available at http://archive.macosxlabs.org/rsyncx/rsyncx.html.
  6. SuperDuper is a commercial backup solution available at http://www.shirt-pocket.com/SuperDuper/SuperDuperDescription.html.
  7. psync is available at http://www.dan.co.jp/cases/macosx/psync.html. Update 2006-04-23 I’ve now tested psync.
  8. ASR is Apple Software Restore, available as the asr command-line utility and the Disk Utility as graphical frontend. [ Update 2006-04-26: asr's behavior is fundamentally degraded in OS X 10.4.6, use with care. ]

Footnotes:

[a] Finder comments are usually preserved provided that the .DS_Store files are copied. Only Finder copies individual comments.

[b] Apple’s rsync has a bug with respect to preservation of modification dates. For files with resource forks, the modification date is clobbered. This has already been noted here and here.

[c] rsync_hfs refuses to copy locked files at all. Update 2006-04-23 I couldn’t observe this problem any more under OS X 10.4.6; but I still wouldn’t trust the tool. rsync_hfs behaves seriously buggy when the uappnd flag is set on directories.

[d] rsync_hfs doesn’t copy the opaque flag.

[e] filed as # 4523881.

[f] filed as # 4523882.

[g] filed as # 4523924.

Preservation of Ownership

To be able to unconditionally preserve file ownership, a copying engine must be run as root. All command-line tools and utilities that have explicit authorization facilities support such a mode. The only exception is the Finder, which is usually run under some nonroot user account. Hence, one cannot expect the Finder to preserve file ownership.

Preservation of Creation Date

[clarifying paragraph added 2008-02-07] When the creation time is not explicitly set on a file, it defaults to the modification date. Hence, the nonpreservation of creation dates only manifests if creation date and modification date of the original file were different to begin with. Should you want to reproduce my tests, keep this in mind.

Tiger and Apple’s BSD Tools

With the advent of Tiger, Apple touted that they had enhanced all BSD file manipulation utilities to be metadata-aware; i.e., all utilities such as cp, mv, tar, etc. should handle metadata transparently. Basically, Apple patched the BSD utilities to fall back to facilities provided in copyfile.c/copyfile.h, which is provided in Darwin’s libc, for copying metadata. copyfile in turn uses the APIs exposed by the xattr facilities, ergo, the HFS+ extended attributes. Apple also extended the kernel-level file system code to support “pseudoattributes” com.apple.FinderInfo for extended Finder flags and com.apple.ResourceFork for the resource fork. That is, though neither Finder flags nor resource forks are HFS+ extended attributes, the kernel exposes this data via the extended attributes (xattr) API. This is the only way for BSD tools to directly access this data. However, Apple chose only to expose the 32 bytes of FileInfo/ExtendedFileInfo or FolderInfo/ExtendedFolderInfo structures. The result is that finally finally all metadata is accessible on a BSD API level except for the creation date. Hence, whenever one uses Tiger’s new shiny tools that are based on copyfile, the creation date gets clobbered. There is no way for a BSD-level tool to preserve the creation date, unless Apple fixes the Darwin kernel. Update 2006-04-23: The bug is filed as # 4506951.

Update 2006-06-27:After considering comment #30 below, it seems that my diagnosis actually comes to the wrong conclusion. With the getattrlist(2) and setattrlist(2) calls, there actually are BSD-level APIs for accessing the creation date.

Stress Testing

It should be obvious from the above that file copying on Mac OS X under the most simple circumstances is already a fairly complex task. In real-world situations, many more issues may appear. For example, Apple’s rsync is known to have issues with Spotlight, and weird things may happen when copying files that are being written. I haven’t spent any effort in stress-testing the tools; so, there may be more bugs lurking that prevent some of the tools from actually functioning in practice. It is advisable to use only tools that have a good track record of being used for backup purposes and that are actively supported.

The Good, The Bad, and the Ugly

Unfortunately, the state of low-level file copying tools on the Mac is sad. There is no all-round solution. Every tool has its drawbacks.

The Good

I’d make the following recommendations:

  1. Whenever a full device-level backup can be afforded, ASR is the tool of choice, since it guarantees to preserve absolutely all metadata including inode numbers.
  2. Apple’s new cp tool would be my second tool of choice. The only drawback, as with all copyfile-based tools, is that the creation date is not preserved. One could see as another downside that cp has not traditionally been used for large-scale backups on Mac OS X, so there is relatively little experience with the reliability of this solution.
  3. If a commercial solution is acceptable and no command-line tool is needed, I would recommend the SuperDuper engine. The engine seems to be rather bug-free and preserves all metadata.
The Bad and the Ugly
  1. Having a reliable rsync would be highly desirable. However, Apple’s rsync tool has severely fallen short of expectations, with numerous bugs still present, even in the sixth subrelease of Tiger. I wouldn’t trust Apple’s rsync for backup purposes. There are numerous efforts to patch various sources of rsync for Mac OS X compatibility, but I have lost track of the current status.
  2. ditto used to be the workhorse of many backup solutions, among them the venerable CarbonCopyCloner. ditto does not preserve the creation date, just as Apple’s cp. But it also doesn’t copy the locked flag (which is a minor issue). The show stopper, though, is that it does not copy HFS+ extended attributes. Thus, ditto will increasingly become insufficient for backup purposes as the extended attributes are being put to use.
  3. CpMac and psync have severe issues that preclude their use for backup purposes.
  4. Although the Finder copying engine has become good with respect to metadata preservation, it is still not a useful backup tool. The only reason is that file ownership is generally not preserved, unless all files belong to the user running the Finder.

Bootability

Once a full system backup is restored, it needs to be made bootable. The basic steps are described on Mike Bombich’s site. Most current backup/cloning tools are capable of restoring a bootable backup/clone.

Overview of Dedicated Backup/Cloning Tools

After I have reviewed the available low-level file copying engines above, I will now give a short overview of available backup/cloning solutions. Many of their properties are a direct function of the underlying engine. This is not intended to be a complete review of backup software functionality. I merely discuss the applications’ basic capabilities of faithfully cloning Macintosh files.

Apple Disk Utility

Disk utility comes with OS X.

Engine: asr<br/> Pros: Free, easy to use<br/> Cons: only allows full volume backups

Recommendation: Disk utility is easy to use for a quick volume backup. Since it uses the asr engine, its metadata preservation properties are generally good. In device-level copy mode, it can’t get better. [ Update 2006-04-23: this observation, if correct, would invalidate my recommendation for asr. ]

SuperDuper

Engine: SuperDuper proprietary<br/> Pros: Probably best cloning engine around, slick user interface, relatively bug-free, has scheduling support, incremental update functionality, great support, active development<br/> Cons: Commercial software; it’s cumbersome to selectively clone only parts of a volume; engine not usable as command-line tool.

Recommendation: The best cloning/backup solution around, but not free.

Carbon Copy Cloner

The veteran among free OS X cloning tools.

Engine: ditto<br/> Pros: not commercial (donation-ware), easy to deselect subdirectories of volume to be cloned<br/> Cons: all downsides of the ditto engine; written in AppleScript studio (i.e., doesn’t feel too polished); no active development; closed source

Recommendation: Still acceptable to use, but one should be aware of the limitations of the ditto engine.

RSyncX

Engine: rsync_hfs<br/> Pros: free, easy to deselect subdirectories of volume to be cloned<br/> Cons: all downsides of the rsync_hfs engine; no active development;

Recommendation: Don’t use.

Apple Backup

I have no experience with Apple Backup. Feedback would be appreciated.

Engine: ?<br/> Pros: ?<br/> Cons: requires .Mac account

Recommendation: ?

Finder

Engine: Finder<br/> Pros: quick and easy<br/> Cons: generally does not preserve file ownership

Recommendation: Use the Finder only to backup small amounts of user data. The Finder is not suitable for backing up an entire volume.

Conclusion and Outlook

In my above list, many backup tools are still missing, such as Unison and Retrospect. I have no further information about these tools at this time. What is also missing is a discussion of archive formats such as tar, cpio, and disk images. Archives can be potentially useful for encapsulating backups and for storing backups on foreign file systems that do not support OS X metadata.

In my eyes, there is no silver bullet for backup and cloning under Mac OS X. If you have any better solutions than the ones presented above, please let me know in the comments.

Update 2006-04-23 I have now posted an extensive analysis of free and commercial GUI-based tools.

Categories: hacking, macosx, unix

Tags: , , , , , , , , ,

118 Comments Add your own

  • 1. rick  |  March 28th, 2006 at 00:52

    check this one. Tri-Backup http://www.tri-edre.com/english/tribackup.html

    especially the regroup feature.

  • 2. barefootguru  |  April 3rd, 2006 at 08:56

    Good article thanks. I use Retrospect, but it still makes interesting reading.

    Couple of corrections:

    1. The Finder does copy invisible files if they’re inside one of the folders you’re copying.

    2. When you download a file, Safari saves the source of the download in the kMDItemWhereFroms attribute — e.g. http://www.apple.com/macosx/

  • 3. maurits  |  April 4th, 2006 at 01:07

    barefootguru – thanks for your comments. You’re of course right about Finder and invisible files; I’ve removed the mistakable phrase, which was not really important anyway. Also thanks for the pointer to the kMDItemWhereFroms attribute. I’ve added a discussion in the article.

  • 4. Justin Blanton  |  April 24th, 2006 at 17:49

    Nice roundup Maurits.

  • 5. Justin Blanton | The stat&hellip  |  April 24th, 2006 at 17:53

    [...] The state of backup and cloning tools under Mac OS X. [...]

  • 6. soeren says » Blog A&hellip  |  April 24th, 2006 at 18:48

    [...] Via Shirt Pocket Watch (the blog of the makers of SuperDuper!): two posts about backups on Mac OS X; the first primarily a description of the problems encountered (mainly preserving metadata) and the second a practical comparison of how actual tools performed — with devastating results. The surprising conclusion is that almost all Macintosh backup or cloning programs do not fulfill their primary purpose, i.e., they are not able to restore files with all associated metadata. This is despite the fact that many of the tools are advertised as “safe”, “accurate”, “bug-free”, etc. The tools that fail are harmful because they generate a false sense of security. Even more exasperating is that many of these tools cost (significant amounts of) money. The only laudable exception is the great SuperDuper application, which performs flawlessly. [...]

  • 7. Mac Backup » The We&hellip  |  April 24th, 2006 at 21:46

    [...] You back up your Mac datafiles regularly and you think your data is safe, right? Wrong. Maurits writes about the (abysmal) state of Mac backups here and here. Hint: If you’re using Retrospect, you should be worried. [...]

  • 8. Sean  |  April 24th, 2006 at 22:05

    Do you have any advice for command line tools that are as complete as Super Duper?

  • 9. Peter of the Norse  |  April 24th, 2006 at 22:08

    I’m surprised that my favorite Unix tool for backups, “dd”, wasn’t listed. It works with everything and is perfect for tape backups.

  • 10. maurits  |  April 24th, 2006 at 22:10

    Sean: No, I wish myself an engine as strong as SD existed. In theory, it’s possible to call the SD binary from the command-line, they do have an interface. But Dave Nanian told me that this interface is neither documented nor stable nor supposed to be used in the first place.

    For the lack of better alternatives, my current recommendation is to use cp -Rp.

  • 11. maurits  |  April 24th, 2006 at 22:15

    Peter of the Norse: good point. dd would be comparable to ASR in device mode, so it’s trivial to assume that it preserves all metadata. However, whenever you want to perform a backup short of backing up your entire disk, dd is useless. The focus here has been on file-by-file copying engines, which are of course much harder to build.

  • 12. Matt Johnston  |  April 25th, 2006 at 11:35

    Where do Finder background colours/icon sizes fit – I’m guessing Finder Flags?

    I’ve been testing hfstar, which seems to preserve Unix things well (as gnu tar should) except symlink owners, and doesn’t seem to handle Finder Flags (GetFileInfo stuff). It’s saving Finder Comments though (with a .DS_Store), but doesn’t seem to handle extended attributes either.

  • 13. Matt Johnston  |  April 25th, 2006 at 11:52

    hfstar -p actually does do symlink owners, forgot that I needed to sudo when extracting, duh.

  • 14. maurits  |  April 25th, 2006 at 11:52

    Matt: I’d assume that background colors/icon sizes for folders are stored in .DS_Store files, but I’m not entirely sure.

  • 15. blurbomac » Blog Ar&hellip  |  April 25th, 2006 at 20:44

    [...] This kind of recap scares me to death. Especially with valuable data at stake. Mac admins everywhere need to read this and evaluate if their processes are as bulletproof as they might think. [...]

  • 16. Arek Dreyer  |  April 26th, 2006 at 05:29

    12 & #14: It looks like .DSStore does not contain the Color labels. I created a test directory with a folder and a text file with Color labels, Spotlight Comments, and custom Finder icons. I closed the test folder in Finder, removed .DSStore in Terminal, and Forced Quit Finder. When I opened the test folder again, only the Spotlight Comments were gone – the Color labels and custom Finder icons were still there.

  • 17. maurits  |  April 26th, 2006 at 05:35

    Arek: yeah, that’s to be expected. Color labels are stored in the Finder flags, custom icons in the resource fork. What Matt eluded to was the window background color and the icon size setting for the folder. As I said, I’m not entirely sure about those. My bet would still be .DS_Store.

  • 18. Arek Dreyer  |  April 26th, 2006 at 05:37

    With my above scenario, when I used ditto to duplicate a file with a Color label and a custom Finder icon, both the Color label and the custom Finder icon were preserved. cp -Rp would preserve the custom Finder icon, but not the Color label.

  • 19. Arek Dreyer  |  April 26th, 2006 at 05:39

    Maurits, (#17) how much do you want to bet ;) Set a background color and change the icon size, close the folder, remove .DS_Store, and open the folder. The background color and icon sizes are preserved.

  • 20. maurits  |  April 26th, 2006 at 05:41

    Arek: your ditto results are in line with mine. cp -Rp not copying the label sounds weird to me – works here. I guess it might be a weird Finder caching problem.

  • 21. Arek Dreyer  |  April 26th, 2006 at 05:45

    Maurits: You’re right about the Finder caching with ditto. Gotta keep remembering to Relaunch the Finder to get rid of that. Sorry!

  • 22. maurits  |  April 26th, 2006 at 05:45

    Arek (#19): hm, definitely not a month’s salary… your test doesn’t say anything: Finder caches like crazy. But I’d still be happy to be convinced that my bet was wrong…

  • 23. Arek Dreyer  |  April 26th, 2006 at 06:04

    I ran some tests of changing icon size and folder background color while watching fslogger (from http://www.kernelthread.com), and noticed that the custom icon size and folder background color are being kept in the .DS_Store of the enclosing folder.

  • 24. Arek Dreyer  |  April 26th, 2006 at 06:05

    18 Funny how the Finder caches some information but not others. haha.

  • 25. maurits  |  April 26th, 2006 at 06:11

    The Finder is weird.

    Did you know the fs_usage tool, built-in?

  • 26. Arek Dreyer  |  April 26th, 2006 at 06:28

    fsusage is useful to see what files are being read, accessed, and written to, but I like to use http://www.kernelthread.com/software/fslogger when I’m really only interested in what is being _written. Please let me know if you have figured out the flags to make fs_usage a little less, er, verbose.

  • 27. Matt Johnston  |  April 26th, 2006 at 12:25

    Yeah, I had a look at the contents of the .DS_Store file, if you set a custom background image then the path gets stored in there. Seems that it’s the Finder caching (both read and write) that confuses things – shall have to investigate how SuperDuper makes things work right :) It could be something special WRT it writing to disk images rather than just tar files that I’ve been testing with.

  • 28. Arek Dreyer  |  April 26th, 2006 at 16:56

    What do you use to look inside the .DS_Store file?

  • 29. maurits  |  April 26th, 2006 at 17:10

    Arek: well, nothing. The format of the .DS_Store file is, as far as I know, not documented and hasn’t even been reverse-engineered.

  • 30. Chris  |  April 28th, 2006 at 10:38

    You certainly can get at the creation date using a BSD-level call – call getattrlist() and request the ATTR_CMN_CRTIME attribute. The API’s a bit hairy, but it works :-)

    This call has a man page as of Tiger, though the call existed back in 10.0.

    Man page

    [edited to stop overeager Markdown, sorry -- maurits]

  • 31. Aaron  |  April 28th, 2006 at 18:55

    It’s extremely difficult to perform tests on .DSStore functionality, as most (all?) of their data is cached by the Finder. The only fool-proof way I’ve found to reset DSStore info is to log out, ssh in from another box (or log in as “>console”), and delete the file in question. (There may even be some merit to clearing various Cache directories as well; it’s been quite a while since I’ve done this.) On logging back in, you should have a clean folder.

  • 32. maurits  |  May 2nd, 2006 at 00:21

    Chris: wooot, I didn’t know that getattrlist() call. Thanks for proving me wrong! I had studied the kernel source (finding nothing for the creation date), implicitly assuming that if they’d implemented it, it would have been through the xattr API. But that was of course stupid.

    This makes it all the less understandable that Apple’s copyfile call doesn’t preserve the creation date. Interestingly, Apple has never pointed out this call in reponses to my bug reports to them.

  • 33. Jon W.  |  May 4th, 2006 at 19:19

    Two pieces of Unix-level metadata I don’t see mentioned are the access time (ls -lu) and the change time (ls -lc). A file-level backup utility will necessarily modify the access time, unless it specifically resets it (e.g., tar –atime-preserve), in which case it will modify the change time.

    Also, I wouldn’t let backup utilities off the hook when it comes to aliases with dangling inodes. Why can’t the utility update the alias?

  • 34. Buggy-Net.de » Back&hellip  |  May 7th, 2006 at 10:11

    [...] Sehr gut: Vergleich der Backup-Strategien und Möglichkeiten unter OSX Diesen Artikel drucken [...]

  • 35. Chris  |  May 13th, 2006 at 16:37

    I can’t remember where I first found the getattrlist()… The “Advanced Mac OS X Programming” (Dalrymple/Hillegass) book has a section on it though.

    You’d think Apple would have known about it…

    It is actually a good call to make to return metadata in bulk, or if you want to be very selective about what you want. Or if you want the creation date!

  • 36. Muskblog » Blog Arc&hellip  |  May 19th, 2006 at 21:26

    [...] Also recently I read a blog posting about Macintosh backup software. It was an exhaustive comparison, but the conclusion was that only SuperDuper! backed up everything correctly. As someone who has paid for Retrospect and Carbon Copy Cloner this was a bit disappointing. Both Retrospect and Carbon Copy Cloner have worked OK for me. Retrospect is far from perfect but you can create a lot of custom scripts. My complaint with Carbon Copy Cloner is the automation. It just doesn’t seem to work (at all?) like I would expect it to. I will keep using Retrospect for regular user directory backups and maybe switch to SuperDuper! when I clone my entire hard drive to an external firewire volume. [...]

  • 37. It’ll Never Fly &ra&hellip  |  May 25th, 2006 at 13:19

    [...] plasticsfuture » The State of Backup and Cloning Tools under Mac OS X A really nice round-up of well-known or popular backing up software for the mac, with some arguably surprising recommendations and warnings. (tags: apple osx backup comparison filesystem metadata software mac review unix tiger tool) [...]

  • 38. Nathaniel Gray  |  May 27th, 2006 at 21:37

    I think it would be really, really useful if somebody would create a set of unit tests for backup tools. (Maybe you already have?) Something like this: Create a directory with a bunch of files (or programs that create files), each of which tests one aspect of file copying. Also write a program (perhaps one for each file) that can analyze the results of a copy, either by testing directly for the attribute in the copy or by comparing before vs. after. Make it modular so that new tests can be added as new failure modes are detected. This would give developers (including Apple!) a way of validating their tools and would, I believe, lead to much more reliable backup software on OS X.

  • 39. Nathaniel Gray  |  May 27th, 2006 at 21:40

    Oh, and you have my deepest gratitude for this article! I’ve spent far too many hours trying to find out just this information, shocked and dismayed at the fragmented, pathetic state of free/open-source backup software for MacOS.

  • 40. Robert J. Berger  |  May 28th, 2006 at 01:51

    Has anyone analyzed ChronoSync? It sounds good but don’t know how it meets some of the criteria discussed in the article.

    http://www.econtechnologies.com/site/Pages/ChronoSync/chrono_overview.html

  • 41. Bill  |  May 29th, 2006 at 23:52

    Has anyone tried Qdea’s Synchronize Pro in a mixed Windows and Mac server environment?

  • 42. Matthew, SF, CA  |  June 4th, 2006 at 05:52

    It’ll be interesting to see the results if you test BRUclone ( http://www.google.com/search?q=mac+bruclone&hl=en&lr=&c2coff=1&safe=off&filter=0 or click my name) works well. BRU made bold claims about it… But it will be a commercial item (or at least it requires one).

  • 43. tmk  |  June 20th, 2006 at 03:37

    Has anyone testing scp on 10.4.x?

    According to the man pages the -E option “Preserves extended attributes, resource forks, and ACLs. Requires both ends to be running Mac OS X 10.4 or later.”.

    Would be interesting to know how it performs.

    = tmk =

  • 44. Galfromdownunder  |  June 21st, 2006 at 15:02

    from a complete non-geek that just needs to maintain a synchronized clone each day, thank you. The Readers Digest Condensed Version therefore, for non-geeks, would be…

    • Super Duper is so far the best of the rest but you need to pay for it.
    • CCC was great but until v.3 comes out and hopefully does not use ditto, it is flawed.

    That about right?

  • 45. jack  |  June 26th, 2006 at 19:45

    You do not mention hdiutil anywhere. I was under the impression that Disk Utility used hdituil to create images from devices or folders. I have been using this as a backup method for sometime and have successfully restored bootable disks from these images using Disk Utiltity, which does use asr for this purpose.

  • 46. maurits  |  June 26th, 2006 at 20:15

    jack: What do you mean by “successful”? If you mean the data fork is intact, then all the tools tested here would be “successful”. However, in general you point out into an interesting direction. I wouldn’t be surprised if hdiutil create -srcfolder used the same engine as asr, and thus suffered from the same shortcomings. I’d have to check, though.

  • 47. Chris  |  June 27th, 2006 at 22:01

    What about SilverKeeper? It’s free too! http://www.silverkeeper.com/

  • 48. Jaharmi  |  June 28th, 2006 at 12:51

    I’d be interested in throwing rdiff-backup into the mix. I’ve seen information on the ‘net that it has been patched to support Mac OS X extended attributes (even on non-Mac OS X file systems, and file systems that have their own EAs), but not necessarily ACLs. Validation would be useful.

    Given that ACLs are explicitly not enabled on Tiger clients, perhaps we’re jumping the gun just a little bit there. EAs are solidly here, so they are a different situation. But I understand and agree with your point that we’re only going to have to deal with this more in the future, not less. In this respect, my view that Tiger is a transitional OS is reinforced — a lot of technology has been thrown into it, and not all of it has been realized yet.

  • 49. Thomas von Eyben  |  June 29th, 2006 at 21:53

    Chronosync 3.3.1 is out – based upon this article I am in the middle of testing it and sofar it looks very promising! Meta data is not that big an issue for me (yet – if at all…) as the capability to make incremental backups are. But based on the post from CS’ author that all the metadata “bugs” allready should have been fixed I think that’s a non-issue (have not tested it though).

    PS.: Have anyone done testing with bacula? It’s looks very promising for a X-platform networkbased backup system! URL: http://www.bacula.org/

  • 50. girtby.net » Blog A&hellip  |  July 22nd, 2006 at 14:15

    [...] File-system metadata-aware tar, cp, rsync, and other tools. Manipulating the bewildering array of complex filesystem metadata is a constant chore for MacOS users, and these tools are a step in the right direction, if nothing else. [...]

  • 51. Blogaholic: Getting thing&hellip  |  July 31st, 2006 at 19:11

    [...] http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-%20cloning-tools-under-mac-os-x/ [...]

  • 52. Den eviga frågan, backup&hellip  |  August 8th, 2006 at 17:34

    [...] Den eviga frågan, backup på bästa sätt Har precis plockat upp min nya fina LaCie-disk ur kartongen och kopplat in den för att börja göra något åt det man borde gjort från början; backuper. Som vanligt när man vill veta något började jag söka i forumen för att hitta ett vettigt backupprogram. Det alternativ som lät bäst var att använda OS X egna terminalkommando asr som beskrivs i tråden Backa upp ett helt system till extern firewiredisk. Efter att ha gjort en sökning på google angående asr så hittade jag dessa tre artiklar: The State of Backup and Cloning Tools under Mac OS X Apple’s asr Badly Broken in Mac OS X 10.4.6? Mac Backup Software Harmful För er som inte orkar läsa så kan det kort sammanfattas i att i princip alla backupmjukvaror döms ut, även Apples egna asr. Så hjälp mig, någon OS X guru lär det ju finnas här på forumet som kan förklara; är det så illa ställt som han påstår? Tillexempel så får Carbon Copy Cloner betyget Not recommended och Retrospect det än värre Avoid at all cost. Apples asr får samma betyg förutsatt att man kör 10.4.6 eller senare. Tre program som väldigt många verkar använda om man kollar i gamla trådar. Det jag vill ha är helt enkelt en mjukvara som kan klona disken för att skapa en bootbar kopia. LaCie disken som jag kommer använda vill jag partitionera upp i två delar, en på 80 GB för min klon (PowerBook) och en 160 GB partition för lite annat. Kan det bli problem att starta från en partitionerad FW-disk? [...]

  • 53. Ralph  |  August 11th, 2006 at 08:27

    What would be really great was, if you could explain where all those nifty metadata are actually stored in the filesystem. You tell it for some of them but for example not for the “Finder Flags” (they are stored in the HFS+ directory structure).

    When you are at it you could also elaborate where some structures had been stored in OS 9. ;o)

    All this would help people to fit together low level implementation with higher level abstraction.

  • 54. Agent  |  August 29th, 2006 at 12:29

    Also if you could review this version of rsync that would be nice: http://www.onthenet.com.au/~q/rsync/

  • 55. Niall O Broin  |  September 3rd, 2006 at 00:08

    I found this page when investigaing the following problem iwth rysnc on OS-X – file ownerships are not preserved when writing to a HFS+ volume in a disk image file.

    This does not seem to be exclusively an rsync problem – chown doesn’t work either. Basically, ALL files on the HFS+ filesystem in the disk aimage have ownership of unknown/unknown.

    This is not very useful :-(

  • 56. maurits  |  September 3rd, 2006 at 01:01

    Niall,

    that’s a pretty trivial issue and well known. It falls into the category “it’s a feature, not a bug”: disable the “ignore ownership” flag in the Get Info… window of the volume in the Finder, and all will be good.

  • 57. PeterW  |  September 8th, 2006 at 11:28

    As an tech.support end-user, this has been a very useful learning curve for me – one question: when I use Carbon Copy Cloner/ASR to create a bootable disk image, am I using what you list as “dev mode” or “file mode”?

  • 58. maurits  |  September 8th, 2006 at 11:43

    CCC always does “file mode” copies, since it uses ditto.

    From man asr:

                --source       can be a disk image, /dev entry, or volume
                               mountpoint. In the latter two cases, the volume
                               must be unmountable in order for a erase block-
                               copy to occur.
    
                --target       can be a /dev entry, or volume mountpoint. Must
                               be unmountable in order for a erase blockcopy
                               to occur.
    
    
                --erase        erase erases target and is required if a fast
                                   block-copy restore is desired.
    

    In this case, “erase blockcopy” is synonymous with my term “dev mode”. Says it all.

  • 59. PeterW  |  September 8th, 2006 at 14:34

    I think, having read more of your recent blogs, I can answer my own question. I thank you for what has been quite an eye-opener.

  • 60. FredrikW  |  September 14th, 2006 at 19:50

    I just found that with the developer tools is included a small program called pbxcp, /System/Library/PrivateFrameworks/DevToolsCore.framework/Resources/pbxcp From the help you get when you run it without parameters, it seems like it can preserve a lot of this… What do you think of that?

  • 61. Dead Reckoning » Ar&hellip  |  September 18th, 2006 at 17:42

    [...] For the next two days Dave Nanian of Shirt Pocket Software is offering a 10% discount on his award-winning SuperDuper! backup and system recovery software for Mac OS X because ex-Apple superstar Guy Kawasaki recently suffered an expensive hard drive failure on his MacBook with no backup system in place. (The coupon code “getsmart” brings the price down to US$25.) SuperDuper! is an unregrettable purchase. Elena and I use it to back up our PowerBooks and lucky thing too: just 18 months after replacing the original hard drive on Yiddel — which died of catastrophic MFA thesis overload back in March last year — he is now again in the shop to replace the replacement. Thankfully this time we have a full disk image from the day before meltdown sitting on a firewire drive awaiting Yiddel’s return. When your hard drive dies (as everything must) you will attain enlightenment: but it’s easier if you just take it from the people who’ve been there and back up your computer. SuperDuper! is the best way to do that on a Mac. [...]

  • 62. Matt & Jen Simerson &&hellip  |  September 19th, 2006 at 20:43

    [...] I owned Synchronize! Pro years ago but gave up on it during the switch to OS X. CCC was the perfect (and only) tool for duplicating OS X drives for quite a while. I used and recommended it for a few years. It’s so good and cheap that I paid the suggested donation for it several times. It “just works.” However, CCC has grown rather long in the tooth. As new versions of OS X arrived, it has been slow to get updated and even the latest version today does not support all of OS X’s file metadata features. Say hello to SuperDuper. [...]

  • 63. Peter N Lewis  |  October 6th, 2006 at 10:18

    You asked if anyone is using the xattr’s. Since we added support for transparent file converters in Interarchy 8.2 and include a Backup converter that preserves most/all the meta data, including ACLs and HFS+ Extended Attributes, I figured we should use the attributes as well, so we added support for using an extended attribute com.stairways.interarchy.ignored as another option to avoid having a file mirrored.

    I also agree with Jon W in comment 33 – a backup program can update the alias when it restores it. This is not without precident, we did this in Assimilator about a decade ago when we copied aliases off a network volume!

  • 64. JimJ  |  October 6th, 2006 at 18:25

    maurits:

    Thanks for the article!

    I have been using psync for several years now with no problems, but I have never tested a full restore. I have only needed to recover an accidentally deleted file now & again. I have tested that the backup is bootable & that seems OK.

    I like psync because it is a command line tool I can script in a usual unix way and make daily backups part of the periodic daily process, but your comments about psync have me worried.

    You said “CpMac and psync have severe issues that preclude their use for backup purposes.” I suppose the connection between the two is that the psync perl script uses the CpMac command for copying. Correct?

    However, you didn’t say much else about psync and you didn’t mention it in your final overview section. What are the “severe issues” you are worried about?

    Would the issues be fixed by rewriting psync (or something like it) to use cp instead of CpMac?

  • 65. marc  |  October 10th, 2006 at 19:43

    very interesting article… i use Retrospect at work and Synk at home. I would highly appreciate if the author would add the following tools to his comparison:

    Synk (www.decimus.net) Retrospect (www.dantz.com) iMsafe (http://homepage.mac.com/sweetcocoa/imsafe/)

    Thanks

  • 66. marc  |  October 10th, 2006 at 19:44

    very very interesting article… i use Retrospect at work and Synk at home. I would highly appreciate if the author would add the following tools to his comparison:

    Synk (www.decimus.net) Retrospect (www.dantz.com) iMsafe (http://homepage.mac.com/sweetcocoa/imsafe/)

    Thanks

  • 67. Marc  |  October 11th, 2006 at 20:38

    1) Has anyone tested under MOSX 10.4.8?

    2) Is the test methodology available anywhere? Perhaps something as simple as mtree?

  • 68. maurits  |  October 11th, 2006 at 20:51

    The answer to both questions is no at this time. I’m just swamped with other stuff, so no ETA is available either.

  • 69. Matthew  |  December 11th, 2006 at 18:10

    We have syncsort backup express 2.35, and we have tested with MAC. and that tested okay.

    http://www.syncsort.com

  • 70. Dave Nelson  |  December 12th, 2006 at 23:26

    Hello.

    To Comment #69. I took a look at Syncsort BE and their website says it will only backup a Mac Mac OS X v10.3.3+ client – won’t perform on a master server or device server under Mac OS X v10.3.3+. Have you actually tested metadata from backups to determine it faithfully retains the original metadata as described in the above table?

    Also, would Maurits be willing to test an out-of-the box Retrospect v5.1 Workgroup if I provided a copy? You had a question about whether it would perform differently if you had used full version.

    Dave

  • 71. Grant Jacobs  |  December 18th, 2006 at 02:10

    A few thoughts:

    1. It’d be nice to see how you check the metadata, so people can do their own tests. Or better still…

    2. Release your file and a checking script as a simple minded test suite.

    3. I’d love to see BRU LE tested.

    4. If a test quite were available, I’d add file/symlink/alias/directory tests: in my experience these are (very) poorly handled too.

    Thanks for the article.

  • 72. Ira  |  December 23rd, 2006 at 23:38

    Has anyone tried an app called NTI Shadow, by a company called New Tech Infosystems? It’s very fast and seems to do a good job of basic copying. I don’t know anything about its metadata capabilities, though.

  • 73. Daniel Axelrod  |  December 27th, 2006 at 20:55

    Thank you so much for compiling and testing all this.

    I understand that you’re busy, but is there any way you could just make the disk image you used for testing available? I understand documenting your methodology is more difficult but it might give some of us interested in our own testing a head start.

  • 74. Ben Smith  |  January 15th, 2007 at 01:28

    Brilliant run-down of a few of the options! Thanks for all the time this would have taken.

    I use SuperDuper! (and love it) but as there are some restrictions that it places on how one does things I’ve got a few scripts running after each scheduled job it does to tidy up loose ends. Reading this article and finding out which of the command line options was the least evil (cp -Rp for what I wanted) was a great help.

  • 75. Ross Brown  |  January 25th, 2007 at 21:02

    Wow. Thanks for your thorough investigation of backup tools. I’ve been looking for something like this for a while. There are an overwhelming number of options out there for OS X and, it seems, many pitfalls.

    If I had one wish for Leopard, it would be a version of rsync that gracefully handled all OS X file metadata. I find rsync tremendously useful in so many situations, but one must wade one’s way through a swamp of third-party tools to find something than can reliably preserve all of the fancy metadata OS X stores outside of your data files. It’s hard to feel comfortable using these metadata features if the operating system isn’t serious about preserving it.

    It’s not clear what Leopard’s bringing in terms of new filesystem and backup features (ZFS? Time Machine?). But it seems the feature plateau might again be moving out of rsync’s reach?

  • 76. cornelius  |  January 31st, 2007 at 03:41

    Thanks for the very thorough review. Some of it is a bit beyond me, but I appreciate not being talked down to, and like the openness of your approach.

    I tried Retrospect for a while but could never get the thing to do what I wanted it to do. That’s when I turned to CCC.

    I used CCC for a long time, but found it a bit cumbersome, and after a while the synchronization feature became an issue. I switched to SuperDuper, registered it and now backing up is a breeze. I do wish, though, that it afforded greater flexibility of selection beyond entire disk or Users Folder, like CCC does. Yet, I am happy with it, and glad to know I am not missing anything.

    It’s nice to have one’s prejudices confirmed! Thanks.

  • 77. Jimmy  |  March 25th, 2007 at 19:01

    Very good and interesting site!!!

  • 78. Diigo daily 04/09/2007 &l&hellip  |  April 9th, 2007 at 20:00

    [...] plasticsfuture » The State of Backup and Cloning Tools under Mac OS X [...]

  • 79. Stephan  |  April 14th, 2007 at 11:55

    well done overview of important topic for a newbie does asr allow incremental backups – or do you have to back up the whole hard drive every time? my powerbook should have 74GB hard drive space – but I can use only about 55GB – is that a sign of disk failure and how can I check?

  • 80. gopher  |  April 21st, 2007 at 19:27

    I hear that Carbon Copy Cloner has improved its metadata support, and it would be interesting to hear if think the latest version of Carbon Copy Cloner has as good support for meta data as Superduper? Some people are going to balk at even the $28 of Superduper and would rather pay the $5 on CCC.

  • 81. Karen  |  April 25th, 2007 at 00:24

    Thank you for the outstanding articles. If you get the time, I’d love to learn how this patched version of rsync stacks up against your testcases:

    http://www.lartmaker.nl/rsync/

  • 82. n8gray.org: Introducing B&hellip  |  April 27th, 2007 at 22:50

    [...] But then someone (or something?) named maurits actually tested these tools (and quite a few others), and the results weren’t pretty. This was a valuable wake-up call for many in the Mac community, but maurits never got around to releasing his test suite so we could do our own testing. Backup-Bouncer is here to remedy this omission. [...]

  • 83. plasticsfuture » Ba&hellip  |  April 28th, 2007 at 00:37

    [...] A year has passed since I published my articles on the state of metadata conservation in Mac backup/file copying software (here, here, here, and here). [...]

  • 84. isaac  |  May 1st, 2007 at 15:35

    Has anyone reviewed BakBone’s NetVault for its ability to get all the perms & flags right?

  • 85. mickintosh  |  May 28th, 2007 at 14:14

    Regarding BruClone desktop version (http://www.tolisgroup.com/products/macosx/bruclone/) worked flawlessly, even with the incremental cloning

    While other apps like Intego Personnal Backup, fails to deliver even on a simple cloning.

  • 86. MacUser  |  June 22nd, 2007 at 15:42

    Latest version of Personal Backup X4 worked flawlessly for me, preserving all metadata (except inodes number)

  • 87. Peter da Silva  |  July 28th, 2007 at 00:12

    OK, here’s the deal. You never did explain what the basis of your comparisons are. You “partially recommend” a program that I know preserves less of what I would have thought was genuinely critical metadata than one you “don’t recommend”, but you do make one point… you really recommend SuperDuper. Since it is free for basic use, it can’t hurt to try it.

    Of course… it’s not really a backup program, it’s a mirroring program, but given the price of backup media these days that seems to be what you’re stuck with.

    So I go to make a backup using SuperDuper.

    You know what?

    It’s missing what I would have thought was totally basic functionality for a mirroring program. You can’t select the folder the mirror is going to be rooted in. You either have to blow away the whole destination disk, or create a disk image (which uses more disk space and memory). You can’t select the folder you’re going to back up, it’s the whole disk or nothing.

    Is this what they consider advanced functionality? It doesn’t seem like it.

    PLEASE, could you go into more detail why you recommend some programs over others? What metadata are you considering essential, and what metadata are you letting slide, and why? And what basic functionality are you assuming any program should have… because SuperDuper is about the most basic non-functional backup program I’ve ever fired up.

  • 88. mark_usha  |  July 30th, 2007 at 00:56

    @ Peter da Silva Some of SuperDuper’s features you’re missing are available after registering. It’s has some shortcomings, agreed, but it copies all metadata.

    I think maurits recommendations are based on “how good are they at metadata preservation”. If someone can do without BSD-Flags, Extended Attributes or ACL (ACL-support in TIger is normally turned off) he’ll be happy with old RsyncX’ rsync (which preserves the creation date!!) and find workarounds for known bugs (e.g. avoiding uappnd-flags on directories).

    Someone here asked for a report on rdiff-backup: the development version of rdiff-backup (1.1.12) loses the Z-Finderflag (busy-flag), the “locked” or BSD-ish “uchg”-flag, all BSD-flags, extended attributes on symlinks (attributes elsewhere are preserved, if py-xattr 0.4 is installed!!) and ACLs (1). rdiff-backup cannot copy device files and fifos.

    This is promising for a development version, and rdiff-backup offers some “time machine” functions (restore to any given date of hour in the past)!

    (1) rdiff-backup actually does support ACLs on other platforms, but not (yet??) on MacOS X (pylibacl refuses to install on a Mac).

  • 89. Keith Dancey  |  August 2nd, 2007 at 23:01

    I am very grateful for your efforts. I have just tried, for the first time, to creat a bootable copy of my internal disc on an external drive, using Disk Utility.

    It failed because my copy of Disk Utility (Mac OS X version 10.4.10) cannot drag-and-drop the source and destination devices!

    A known but unfixed bug, apparently:-(

    I started searching for alternatives and uncovered your can of worms! I am rather surprised that such a basic requirement as backing up bootable versions of the filestore is in such a mess:-(

  • 90. dp  |  November 6th, 2007 at 16:39

    any thoughts on synk pro?

  • 91. Articles  |  November 11th, 2007 at 16:53

    I am very grateful for your efforts. I have just tried, for the first time, to creat a bootable copy of my internal disc on an external drive, using Disk Utility.

    It failed because my copy of Disk Utility (Mac OS X version 10.4.10) cannot drag-and-drop the source and destination devices!

    A known but unfixed bug, apparently:-(

    I started searching for alternatives and uncovered your can of worms! I am rather surprised that such a basic requirement as backing up bootable versions of the filestore is in such a mess:

  • 92. Matt & Jen Simerson &&hellip  |  November 12th, 2007 at 17:14

    [...] I owned Synchronize! Pro years ago but gave up on it during the switch to OS X. CCC was the perfect (and only) tool for duplicating OS X drives for quite a while. I used and recommended it for a few years. It’s so good and cheap that I paid the suggested donation for it several times. It “just works.” However, CCC has grown rather long in the tooth. As new versions of OS X arrived, it has been slow to get updated and even the latest version today does not support all of OS X’s file metadata features. [...]

  • 93. Anders  |  November 16th, 2007 at 17:01

    You should look at “Unison” http://www.cis.upenn.edu/~bcpierce/unison/ I use it to keep my laptop files synced. I used macports to get and install. http://www.macports.org/ enjoy!

  • 94. Charles Young  |  December 14th, 2007 at 00:54

    I’ve had a lot of success with Bacula. [www.bacula.org].

    It’s great if you’ve got a few machines to backup and can spare a few hours to configure it.

    There is a good resource for bacula for OSX at http://www.haught.org/howto/detail/tag/tech/item/bacula_osx/

    and a prebuilt binary client at http://www.pixelchaos.net/

    As for most backup applications (as opposed to DR), it is predicated on the idea of saving your data rather than your machine, but there is some flexibility to recover from bare metal if you want to go down that path. Personally I wouldn’t generally bother.

  • 95. Blog 2&hellip  |  December 19th, 2007 at 16:36

    [...] The State of Backup and Cloning Tools under Mac OS X [...]

  • 96. New Home Server | Robert &hellip  |  January 2nd, 2008 at 04:30

    [...] Next I wanted to replicate data across the drives on a cron. Initially I was thinking rsync, since as of 10.4, it’s resource-fork aware. It turns out that’s not really true. I ended up going back to SuperDuper to copy between the drives. It only copies changed files, and once a week will delete removed files (so if you accidentally delete something, there’s still a chance to recover, unless you do it at the wrong time). Not a bad solution IMHO. Still would prefer rsync more. Initial backup took less than 1/2 hour. Just a few minutes should be enough to keep the disks in sync. I briefly considered setting up RAID, but decided against it since RAID is not backup. It doesn’t protect against things like corruption. [...]

  • 97. chipwreck | blog » &hellip  |  January 26th, 2008 at 19:55

    [...] More details can be found here: blog.plasticsfuture.org [...]

  • 98. Rob  |  February 6th, 2008 at 20:31

    I can confirm that asr in its file-copy mode is still badly broken in Tiger 10.4.11. (The asr version is 72).

    I created two disk images and put some files in the first. Then I used asr to copy the files from the first disk image to the second.

    1. No extended attributes were copied.
    2. No BSD flags were copied.
    3. No locked flags were copied (which are just a type of BSD flag).
    4. Symlink ownership was not copied.

    And it really messed up the Symlink ownership. One of symlinks had the owner of rob:dummy. When the symlink was copied, it became root:rob. How it did this is beyond my comprehension.

    Come on Apple. Why don’t you fix these long-standing bugs!

  • 99. Rob  |  February 6th, 2008 at 20:34

    I don’t know why the author is tracking the “locked Flag”. Mac OS X uses the BSD flag “uchg” to lock or unlock a file.

    All you need to do is check to see whether the BSD flags are being copied. If they are, then the “locked” status of the file will also be copied.

  • 100. Rob  |  February 7th, 2008 at 00:49

    P.S. Carbon Copy Cloner 3.0.1 is now based on asr. But even though asr does not always preserve metadata, Carbon Copy Cloner does copy metadata including BSD Flags, Locked, Extended Attributes, Symlink Ownership, etc.

    But there may be a bug in 3.0.1. If the file is locked, CCC may not always copy extended attributes. I hope that gets fixed soon.

  • 101. PG  |  March 24th, 2008 at 20:00

    Carbon Copy Cloner 3.1 has now been released: http://www.bombich.com/software/ccc.html

    According to an independent backup-integrity testing tool (Backup-Bouncer: http://www.n8gray.org/blog/2007/04/27/introducing-backup-bouncer/ ), it successfully preserves ALL metadata.

    maurits – I saw your posts on the Backup-Bouncer site: I’m aware of your time constraints, so I don’t expect a full update of the test results. But, I think it would be of great benefit if you would add a brief “update” note, simply saying that (a) some of the tools (e.g. CCC) have been updated, with much better results, and (b) a link to Backup-Bouncer.

    Otherwise, people coming across this page are going to be misinformed about the current state of affairs, and might spend a fair amount of time tracking current information down.

    Thanks for this great work – you really started something, and a lot of good results have come from it!

    PG

  • 102. Rsyncx to backup Mac | To&hellip  |  April 28th, 2008 at 06:39

    [...] Plasticfutures.org has a good deeper analysis of the various tools. [...]

  • 103. n8gray.org: Backup Bounce&hellip  |  June 2nd, 2008 at 21:06

    [...] The FIFO/device file tests are not included in either the “critical” or “important” set of tests. So I’m saying (and have always said) that most users, even most power users, won’t care about them. So why test for them at all? Consider the situation before Maurits’ blogged on backups and BB existed. Most people, myself included, didn’t even know what the full set of OS X filesystem object types and metadata was. Maurits filled us in on what existed, but knowing how to test for preservation was still a black art. BB was meant to democratize that testing, but also to act as an exhaustive test set. (I don’t claim to have achieved even close to 100% coverage, but that’s the goal.) So it’s important to me that BB include tests for any metadatum or filesystem object that can conceivably have a reason to be backed up. [...]

  • 104. lucidsystems  |  September 17th, 2008 at 11:45

    LBackup uses rsync to copy data. The latest stable version of rsync (version 3.0.4) is capable of preserving a great deal of the Mac OS X related metadata.

  • 105. rptb1  |  September 29th, 2008 at 19:49

    See http://www.bombich.com/mactips/rsync.html for updated information on rsync, which now checks all the boxes (except inode) in the table in the article.

  • 106. psychotron.net » Bl&hellip  |  October 3rd, 2008 at 00:08

    [...] I tre capitoli sul Filesystem sono poi molto interessanti, perche’ oltre a ripassare le tipiche funzioni per la lettura e scrittura dei files, si parla di tutti gli attributi e metadati che puo’ avere un file. Con Mac OS X siamo ai massimi storici come quantita’ di metadati perche’ oltre ad avere i classici permessi POSIX e proprietari, abbiamo da tener conto di attributi del Finder, Resource Forks, Extended Attributes, ACL e altro. Tutto questo poi purtroppo non puo’ essere gestito da una sola API, ma da un miscuglio di BSD,Carbon e Cocoa. Se si e’ pignoli, fare un backup corretto non e’ proprio un gioco da ragazzi. [...]

  • 107. Discussions on Mac Backup&hellip  |  November 4th, 2008 at 17:15

    [...] The CCC homepage pointed to this article, which is a fantastic resource on how the file metadata system introduced with the Mac OS X operating system introduces a ton of complications for those who think all that is needed to back up a Mac’s file system is to do a drag files and directories to another volume. [...]

  • 108. Super Flexible File Synch&hellip  |  November 22nd, 2008 at 14:08

    [...] I have been a very happy user of SuperDuper for many years – and have recommended it widely but I have run into a limitation that has made me look elsewhere. To be specific what I was looking for was a detailed way to run backups not just on a schedule (which SD can do if registered) but if a scheduled backup is missed then at the next earliest opportunity. Furthermore, I wanted to be emailed with a summary of the backup success or failure – and emailed if the backup had not run at all. All of these items and many more I have found in the app Super Flexible File Synchronizer. However, and this is the big but – has anyone had any long term experience with this app and what are your thoughts of using it following a restore. How does it play with large package format files on a Mac such as Aperture & iPhoto Libraries? Also, does it copy file attributes with anything like the success of SuperDuper – check out this article for an in-depth examination of what I am really trying to get at: plasticsfuture Blog All thoughts gratefully appreciated… [...]

  • 109. Disk Utility - First Aid &hellip  |  January 11th, 2009 at 21:14

    [...] Slowly going insane while waiting for a fix As for the $28, most people know that SD still does full clones for free — and it always has. The $28 buys incremental (or "differential" if you prefer that terminology) backup features. What else did SD do differently? Well, according to these articles (from 2006), SD was the top cat in the "preserving the most metadata" department: The State of Backup and Cloning Tools under Mac OS X [...]

  • 110. naechtliches backup auf k&hellip  |  January 20th, 2009 at 16:26

    [...] naechtliches backup auf kleines externes Medium Hallo, was waere euer Vorgehen, um z.B. naechtlich mittels cron-job auf einem USB-Stick ein backup laufen zu lassen? Anwendung ist z.B. ein laptop oder Mac Mini mit z.B. 40 GB Platte und ein externer USB-Stick mit z.B. 8 GB. Recht einfach ist die Kombination mit einem cron-job und tar, um alle Dateinen der letzten 24 Stunden komprimiert zu archivieren. Interessanter waere vielleicht rsync u.a. Nach der Lektuere von http://blog.plasticsfuture.org/2006/…nder-mac-os-x/ stelle ich aber fest: MacOSX und HFS+ scheinen doch noch erhebliche ‘Tricks’ auf Lager zu haben, wo resource forks und HFS+ Extended Attributes eher mehr Probleme schaffen, als ein backup zu vereinfachen. Wichtigere Randbedingung: das backup soll komprimiert werden. Ich vermute, gerade daran scheitern die sonst besten Ansaetze? ditto und damit CarbonCopyCloner kommen in dem genannten blog recht schlecht weg. SuperDuper war eher auf der Empfehlungsseite. Ueberhaupt scheint der Finder vielen Programmen gerne ein Bein zu stellen, wenn Daten noch nicht im .DS_Store stehen, sondern noch immer nur im cache vorliegen? Schoenen Gruss Martin [...]

  • 111. Xag  |  August 24th, 2009 at 00:30

    This can be of interest: automatic back up on a USB key without a click. It’s a do-it-yourself. It uses rsyncx, no other third party app. http://mogrifiers.blogspot.com/2009/08/quick-usb-backup-from-mac.html

  • 112. RealTime - Questions: "Cl&hellip  |  December 1st, 2010 at 21:08

    [...] – Rapidshare Download How to use SysPrep to Generate Unique SID before Cloning | Raymond.CC Blog plasticsfuture » The State of Backup and Cloning Tools under Mac OS X Intel Data Migration Software “Security Accounts Manager initialization failed …” [...]

  • 113. OS X: Incomplete and part&hellip  |  February 24th, 2011 at 04:53

    [...] blog posts, The State of Backup and Cloning Tools under Mac OS X and Extended Attributes led me to playing with the xattr and mdls [...]

  • 114. Good Apps for a Utility D&hellip  |  February 27th, 2012 at 14:37

    [...] [...]

  • 115. maximo playstation 2 walk&hellip  |  November 22nd, 2012 at 05:25

    maximo playstation 2 walkthrough cheats codes…

    plasticsfuture » The State of Backup and Cloning Tools under Mac OS X…

  • 116. Mac OS X : Why Does Chown&hellip  |  April 21st, 2013 at 11:56

    [...] flags.Some of the file attributes/metadata and how they are handled by different copy tools are here.After a lot of struggling, here’s what I had to do to get the problem fixed:Moved the file to [...]

  • 117. ATP 022: Full Brichter | &hellip  |  December 16th, 2013 at 17:31

    [...] Backing up Mac filesystem metadata, Backup Bouncer, and current scores of online backup apps. [...]

  • 118. All the Show Notes! <&&hellip  |  December 17th, 2013 at 02:40

    [...] Backing up Mac filesystem metadata, Backup Bouncer, and current scores of online backup apps. [...]

Leave a Comment

hidden

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackback this post  |  Subscribe to the comments via RSS Feed


Feeds

Categories

Archives

Most Recent Posts