A Disaster Recovery Nightmare from ‘Toy Story 2’

“I can’t look. Could somebody please cover my eyes?” says Rex, the toy dinosaur in Pixar’s “Toy Story 2.”

Oren Jacob, who was then an associate technical director for “Toy Story 2,” was probably thinking the same thing as Rex when he watched a moviemaker’s worst nightmare unfold before his eyes.

After two months and hundreds of hours of work, someone executed an incorrect command on the servers where all the movie’s files were stored. Jacob and his colleagues were forced to watch most of the movie vanish before their eyes as the files were deleted.

The team turned to Pixar’s backup tapes to restore the files. All seemed well until the filmmakers realized a week later that there was a problem: They weren’t working on the most recent version of the movie. At the time, the backups were not continuously tested, and the team realized too late that the tape backups had been failing for the past month.

But by a stroke of luck, Supervising Technical Director Galyn Susman, who had been working from home, had a copy of the entire movie stored on a computer at her house. If it hadn’t been for her, the company wouldn’t have had a backup at all.

After this disaster recovery debacle, the systems administrators re-evaluated their data backup strategy.

Pixar learned some important lessons the hard way:

  • Test your backups.
  • Store copies of your files in multiple locations.
  • Test your backups again.

Click here for the full story.