[VIDEO] Real Stories about Real Declarations — Disaster Recovery Sessions Part 2

In Part 1 of our "Disaster Recovery Sessions" series, we invited our disaster recovery solution engineers to reveal why testing is so important. Referencing their vast experience over the past two decades, they gave us four good reasons why clients should test.

Later on in the interview, we asked these longtime disaster recovery coordinators to divulge the most memorable disaster declarations they have ever been a part of.

Here are their real stories about real declarations.

Chris Day — Hurricane Through Houston

Chris Day discusses his most memorable declaration experience from 2012. He relates how Houston was suffering from the aftermath of a major hurricane.

The storm had blown out the high-rise windows in one of our client’s buildings, causing rain to soak several of the offices, rendering them uninhabitable.

The business’s employees relocated into our mobile trailers, equipped with communications technology and onboard generators, and returned to business as usual.

Dan Seyer — Hurricane Ike

Hurricane Ike entered Galveston, TX on September 13, 2008, bringing 110-mile-per-hour winds, mowing down 50-foot trees and flooding homes and businesses along the coast.

Dan Seyer remembers “being one of the first companies down into Galveston after the hurricane came through.”

We were able to deliver a banking client's precontracted solutions, helping it become one of the first functioning branches in Galveston after the storm.

Daniel Pritchard — Ruptured Pipeline

Daniel Pritchard describes a declaration for an oil company that required a rural recovery site. The company’s pipeline had ruptured, and its employees needed to remain in the area until they located the source of the leak. We were able to deploy two Mobile Recovery Centers and establish communications in the middle of a nearby cornfield.

Glen Boote — Hurricane Katrina

After Hurricane Katrina, communications across  Louisiana were shut down. The telephone company’s central offices, which multiple refineries along the Gulf Coast depend on, were underwater.

Oil production for the region could not continue until the refineries’ communications were re-established. One of Rentsys Recovery’s larger clients, who owned refineries along the coast, declared a disaster, which Glen Boote had the privilege to be a part of.

He and his team were able to get the refineries’ communications back online using satellite equipment. The day after communications were established, crude oil and gas prices dropped as the media announced the refineries were back online producing gasoline. “It was… one time that we really felt that we really made a difference,” he said.

What to Do When... The Lights Go Out

As a business owner or disaster recovery planner, you probably have provisions in your disaster recovery plan for natural disasters or the loss of your data infrastructure because you expect those types of interruptions to cost you a lot of money. But what about when the power goes out?

According to the U.S. Department of Energy, power outages cost businesses $100 billion a year.

Here are some tips on what to do before, during and after a power outage to prepare your employees and reduce downtime.

Before the Lights Go Out

  • Know the location of the main breaker switch.
  • Provide your employees with emergency flashlights.
  • Determine which equipment will be affected.
  • Plug computers, phones and security systems into surge protectors.
  • Prepare a list of equipment that will need to be reset when the power is restored.
  • Back up your data off-site and test your backup systems regularly.
  • Keep a generator and fuel on hand.
  • Plan to reroute business calls to employee cellphones.
  • Contact your disaster recovery vendor about a mobile solution for long-term outages.

During a Power Outage

  • Use your backup power source for emergency power.
  • Place generators in well-ventilated areas, as they can be a major cause of carbon monoxide poisoning.
  • Call your disaster recovery vendor to declare a disaster.
  • Relocate your employees to mobile recovery trailers.
  • Stay updated on how long the outage will last and estimate your recovery needs.

After a Power Outage

  • Refer to your previously prepared checklist and reset equipment.
  • Wait 10 to 15 minutes before reconnecting other equipment to avoid overloading the electrical system.
  • Check your facility of make sure everyone is accounted for and equipment is operating correctly.
  • Review your preparedness plan to determine what worked and what areas needed improvement.

The key to recovering from a power outage is knowing what to do when the lights go out. For more tips, check out our hurricane preparedness checklist.

[VIDEO] The Importance of Testing — Disaster Recovery Sessions Part 1

At Rentsys Recovery, we always encourage our clients to test, because testing is the key to making the most of our disaster recovery solutions. We recently sat down with our disaster recovery solution engineers, who have been a part of countless tests during the past couple of decades, to pick their brains about why testing is so important.

Here’s what they had to say.

It Reveals Problems with Your DR Plan

The purpose of testing is to identify any problems within your DR plan that need to be ironed out — before you find yourself in a high-pressure disaster situation in which you can’t afford for your plan to go wrong. Identifying solutions to these problems early on will, of course, benefit your business in the event of a disaster, but it’s also useful for improving your day-to-day operations.

It Allows for Proactive Troubleshooting

You never know when your business will be impacted by a hurricane, earthquake or massive power outage, so when you contract disaster recovery solutions, it’s important to schedule a test as soon as possible and test your solutions under conditions that are as close to real-world conditions as possible (in other words, test everything).

During this process, you might discover that certain applications don’t work on a specific hardware device the way you thought they would, that you need to involve certain people who weren’t initially included in the disaster recovery plan or that a certain recovery location doesn’t work as well as an alternate site.

When you identify these issues ahead of time, you have the advantage of being able to prepare for as seamless a recovery as possible.

It Helps Meet Both Your and Our Expectations

Our goal is, of course, to help our clients get back up and running as soon as possible following a disaster declaration. But sometimes we and our clients will have different expectations about the declaration process in terms of setup time, functionality, etc. And because every client’s situation and required resources are different, testing is the most reliable way to set realistic objectives for a declaration.

By taking the solutions through a trial run, you have the opportunity to experience a declaration for yourself, giving you a concrete experience upon which to base your expectations, and we’re able to work with you to make sure we meet those expectations.

It Gives Management the Confidence That You Can Recover in an Emergency

One of the most common challenges of engaging a disaster recovery vendor is getting management on board. (For more on how to address this issue, download our list of common misconceptions about disaster recovery here.) Testing your disaster recovery solutions allows you to optimize your RTOs and even your day-to-day operations (see above), which reiterates to management the value of having a disaster recovery vendor on standby.

Stay tuned for part 2 of the series to hear our disaster recovery solution engineers share some of their most memorable disaster declarations.

[INFOGRAPHIC] Workplace Fire Safety

Workplace Fire Safety InfographicDid you know that 6,000 business caught fire in the U.S. in 2011, causing over $17 billion in damage?

Piper Fire Protection of Florida released this infographic covering the most common causes last year, but the advice on how to mitigate these fire dangers is still sound.

Remember, an ounce of prevention is better than a pound of cure, and just because you have a great recovery plan that covers all the bases doesn't mean you want your business to suffer a disaster just so you can use it.

Review Your Company’s Disaster Recovery Plan

Regularly reviewing your company's disaster recovery plan is vital to staying up to date with your company’s current needs and keeping the plan fresh in your mind in case you need to implement it. But actually reviewing it in depth can be a tiresome and thankless task.

To help you get through the next review with your sanity intact, let’s compare your plan to Jurassic Park's.

Plan for More Than Just Natural Disasters

T-Rex in the hotel
Image courtesy of Warner Bros.
Park founder Jon Hammond’s management plan rests on Dennis Nedry, the programmer in charge of everything from the phones to electric fences. When Nedry decides that Hammond isn't paying him enough and locks everyone out of the system, he illustrates just how much the park relied on him.

At this point, the park's recovery plan should have kicked in, but there wasn't one.

This serves as a reminder that natural disasters aren't the only ones that can negatively impact your business. After you plan for the high-profile natural disasters, make sure to plan for the everyday, human-caused disasters. Does more than one person know where the natural gas line shutoff valves are? In the event of an earthquake, a ruptured gas line can cause a devastating fire. If a disgruntled employee shuts down the fence keeping the T-Rex out of your hotel, does anyone else know how to turn it back on?

Be Aware of Your Company’s Vital Needs

It doesn't start off well for Jurassic Park. Besides the guy getting eaten in the first five minutes, Samuel L. Jackson says the park “has all the problems of a major theme park and a major zoo, and we’re not even open yet.” Being aware of those problems should have led the park to develop a flexible disaster recovery plan.

Most companies don’t have the needs of either a theme park or a zoo, but each company has operations that must function properly, whether it’s phone systems, computer workstations or equipment for a paleoveterinarian. Make sure your plan includes the equipment and resources necessary to keep your business running.

Acknowledge What Can Befall Your Company

Image courtesy of Warner Bros.

Remember, the definition of a disaster is up to you. Your definition of a disaster can be a long-term power outage, the complete loss of your facility or a velociraptor infestation.

To make sure that everyone agrees on what disturbances pose a threat to your business, create a list of natural and everyday disasters that could affect your company.

Then classify each event from high to low likelihood depending on your area. For example, if you were a certain dinosaur-themed park off the coast of Costa Rica, you would rank tropical storms and hurricanes as highly likely, triceratops attacks as somewhat likely and blizzards as not likely.

For more tips on creating and maintaining a definitive disaster recovery plan, download our Business Continuity Plan Checklist.

Popular Posts