BC/DR Plan Testing: Two Companies Doing It Right

Here at Rentsys, we stress the importance of testing your BC/DR plan, because without running through a simulated disaster scenario, you can't know if your plan works properly. Testing gives you the opportunity to work out any kinks and evaluate the efficiency of your plan — not to mention it helps you better prepare for disasters.

Men setting up computers in mobile workspaceA little more than 30 percent of companies test their DR plans once a year, which is the minimum we suggest. Unfortunately, 81 percent of these companies discover issues, and 11 percent report major problems or complete failure. Around 17 percent never test their plans at all, so if they encounter problems during a full-blown recovery, they might not be able to resolve the issues properly, if at all.

To demonstrate the benefits of planning ahead and testing your plan, we're highlighting two companies that do it right.

Wal-Mart


"We have such a big footprint that we are going to deal with something every day." — Jason Jackson, Director of Emergency Management for Wal-Mart

Wal-Mart has a team of nearly 40 people dedicated to emergency recovery, business continuity and preparedness. These people are separated into four teams, each dedicated to a different facet of planning and recovery.

Alarm Operations: Handles day-to-day business interruptions such as fire alarms, burglaries and the company emergency hotline. 
Response and Recovery: Manages the Emergency Operations Center.
Business Continuity: Covers planning, disaster recovery, business impact analyses and more.
Preparedness: Organizes implementation, validation and testing of the BC/DR plans.

By dedicating teams to different stages in the process, Wal-Mart is able to streamline its emergency preparedness and recovery, enabling the company to successfully prepare for and recover from countless disasters while also helping the communities where disasters strike.

AT&T


"Lots of people do paper exercises, but we roll out the assets and do the actual exercises so our people know exactly what to do." — Mark Francis, VP of Global Network Operations Planning and Support for AT&T

AT&T has been a leader in disaster preparedness for over 20 years, and completed its 70th full field technology recovery exercise in September 2013. The AT&T Network Disastery Recovery (NDR) team believes they need to test their plans before disasters occur to ensure that employees know what to do in the event of a disaster and that management has the right teams selected for each job.

AT&T regularly runs full-scale test scenarios because different events could cause different snags and call for special recovery equipment. Testing this often allows AT&T to respond quickly during real disasters. The necessary teams immediately gather and review the severity of the disaster at hand. AT&T also has an emergency hotline for employees to call in the event of a disaster to let management know where they are and that they're safe. As a result of its hard work, AT&T become the first private-sector organization to receive the PS-PrepTM certification.

By following the examples of Wal-Mart and AT&T, your company can fully prepare for different disasters by running through scenarios regularly. For more information about how to improve your DR preparedness, check out our post "How to Get a Passing Grade on DR Preparedness."

Best Practices for Implementing Cloud Recovery

digitized cloud with technology inside
By Eric Thompson, solutions architect for Rentsys Recovery Services, Inc. 

Today, almost every newspaper or tech magazine you pick up is either singing the praises of the cloud or pointing out its shortcomings. The challenge is transitioning from talking about cloud to actually implementing a cloud-based solution so you can judge its usefulness for yourself.

If you're ready to take the cloud plunge, follow these three steps to be best prepared.

Step 1: Complete a Business Impact Analysis


In a business impact analysis (BIA), you identify your most critical business functions, map out the applications that support each function and then designate maximum allowable downtimes for each function. To classify downtime, we typically use the periods defined in the FFIEC IT Examination Handbook:

  • Nonessential — 30 days
  • Normal — seven days
  • Important — 72 hours
  • Urgent — 24 hours
  • Critical — less than 24 hours

As you assess the maximum allowable downtime for each function, consider designating separate recovery time objectives (RTOs) for each function depending on if you experience a regional disaster (e.g., hurricane, flood) or an operational interruption (e.g., crashed server, power outage).

By separating the functions into these two categories, you can significantly reduce the cost of recovery. The reason is that if a regional disaster like a hurricane hits your organization, people are more likely to be empathetic to your situation and understand that it may take you a couple days to be up and running again. Instead of allocating resources to maintain a short RTO, you can designate a smaller set of resources for these scenarios.

On the other hand, customers tend to be less forgiving when a server crashes and they don't have access to their accounts. Fortunately there are many recovery solutions you can use to restore applications for a short period that don't require the expensive resources needed for a full-blown disaster.


Step 2: Categorize Data and Data Size


Once you've completed a BIA, the next critical step is determining if there are any legal or regulatory obligations dictating how data must be handled. If your critical business data is defined as sensitive, your cloud vendor must prove that it can back up and restore your information within the laws and regulations governing your organization. Here are some basic questions to ask when evaluating a cloud recovery provider's ability to safeguard your customers' information:

  • Is the cloud service provider familiar with your industry's legal and regulatory requirements for safeguarding customer information and other sensitive data?
  • Has an auditor evaluated the vendor's internal controls to determine if those controls are functioning appropriately?
  • Does the provider appropriately encrypt or otherwise protect nonpublic personal information (NPPI) and other data that could harm your business or customers if disclosed?
  • What controls does the vendor have to ensure the integrity and confidentiality of your institution's data?
  • Is customer data stored or processed overseas?

After determining that your cloud provider can securely back up and restore you and your customers' information, evaluate the amount of data that you'll need to recover after an interruption. The following charts provide a guideline of how long it takes to move different amounts of data across a variety of common connection types (note that these figures don't factor in latency or regional problems affecting bandwidth speed).

Estimated Data Transfer Speeds


Data transfer speeds for 5, 10 and 100 GBs of data


Step 3: Align Cloud Recovery Solutions With Business Functions


After completing a BIA and categorizing your data, you'll better understand costs as they relate to recovery time, enabling you to make informed decisions about the solutions that are right for your business.

If you're concerned about equipment failure and need quick recovery for a single server, an ideal solution is to back data up to an appliance hosted at your primary site in addition to vaulting data in the cloud. If a server crashes or data is accidentally deleted, the data can be immediately pulled from the on-site appliance across a local area network instead of over the wide area network, which significantly increases recovery speed. This solution allows you to handle less complicated recoveries without declaring a disaster and taking on unnecessary fees.

For high-priority applications with a recovery window of 24 hours, a traditional cloud recovery model in which backup data is vaulted directly to the cloud may be sufficient. At time of event the data is recovered to virtual machines within the same cloud network, significantly improving the recovery time. The data moves at local area network speeds and you don't have to acquire physical hardware, deliver a tape or transfer people to an alternate location to start the restore. However, if the system is critical during a major disaster and the recovery time remains less than four hours, you should consider a solution using replication with standby virtual resources.

For a more in-depth analysis of  how cloud services can help you meet your business's specific recovery times, work with your cloud vendor's solutions architect to identify a solution to best fit your recovery needs.

Implementing cloud solutions doesn't have to be daunting. Follow these three steps, and soon you'll be singing the praises of the cloud.

How to Get a Passing Grade on DR Preparedness

Earlier this year, the Disaster Recovery Preparedness (DRP) Council released the results of an annual benchmark survey that graded businesses worldwide on their state of DR preparedness using a scale of A (best) to F (worst). The report revealed some disturbing news: 3 in 4 companies are at risk due to incomplete or nonexistent disaster recovery plans. Fortunately, the DRP Council offered this nugget of encouragement: We're starting to identify DR best practices. Specifically, the survey results showed that businesses that scored an A or B had three things in common:

  • They built detailed DR plans.
  • They defined specific DR metrics for RTOs and RPOs.
  • They tested DR plans more frequently.

The report is very clear that these goals are key to being a good student of DR preparedness. Now let's take a look at what solutions you can use to get a passing grade on your business's DR plan.

Goal: Build a Plan 


Signpost with signs reading "crisis," "recovery," "growth""Build a DR plan for everything you need to recover, including applications, networks and document repositories, business services such as the entire order processing system, or even your entire site in the event of an outage or disaster."

More than 60 percent of participants don't have a fully documented DR plan, and 40 percent said the DR plan they did have couldn't weather the business's worst outage. These stats are troubling, considering 65 percent of companies are required to produce DR reports for compliance purposes. The problem is that 43 percent find reporting "overly difficult, manual and expensive."

Solution: Business Continuity Software
Using continuity planning software can simplify the planning process by providing a step-by-step road map of the planning process to ensure you don't overlook key elements of the plan, such as important applications for each department, employee and vendor contact information and IT/business relations. Software that allows you to upload data from your production databases reduces the amount of data entry required as well.


Goal: Define Metrics


Ticking clock on top of calendar"Define Recovery Time Objectives (RTO) & Recovery Point Objectives (RPO) for critical applications. Without these important metrics, you cannot set proper expectations and assumptions from management, employees, and customers about your DR capabilities and how to improve them."


Defining metrics for every critical business service is a crucial part of the planning process. The DRP Council found that the highest scoring businesses had established RTOs and RPOs for functions such as customers orders, finance and email. Determining these metrics can get sticky, however, especially when departments don't agree on business priorities and when regulatory requirements enter the picture.

Solution: Professional Planning Services
Using professional planning services can help you manage the planning process with as few headaches as possible. Business continuity (BC) professionals can serve as arbiters between departments, offering objective recommendations for priorities. If compliance is an issue, they can point out any specific areas that are required by law to have specific RTOs or RPOs.

Goal: Test the Plan Frequently

Businessman touching blue cloud with lock in it
"Test critical applications frequently to validate they will recover within RTOs/RPOs. For DR preparedness to improve, companies must begin to automate these processes to overcome the high cost in time and money of verifying and testing their DR plans." 


Test more and test faster is the principle DR superstars live by. Historically, testing has been a cumbersome and expensive process, which explains why 23 percent of those surveyed don't test their plans. To improve efficiency of verifying and testing DR plans, businesses are automating processes.

Solution: Cloud Vaulting and Testing Solutions
Vaulting data in the cloud allows you to quickly and securely back up large amounts of data, including transaction records, images, videos, logs and more. Some businesses, especially in the healthcare and financial industries, have expressed concern over storing data in the cloud due to compliance restrictions. However, trained compliance experts can help you create a strategy for testing IT systems and applications to stay compliant.

Overall, the key to getting a passing grade is similar to what it takes to pass your college exams: Take notes, set goals, plan ahead and use failures as learning opportunities.


Bankers As Buyers: 2014 Tech Trends for Disaster Recovery

Businessmen shaking handsEvery year the William Mills Agency releases a Bankers As Buyers report containing essential information and statistics about the technology trends that are popular in the U.S. financial services industry.

In this year's report, we found three key takeaways that your firm should keep in mind as you update your 2014 disaster recovery (DR) plan.

Outsourcing Is Gaining Momentum


"...institutions are doing everything possible to utilize system functions to make their employees more efficient."

As margins become slimmer due to factors such as declining fee income and larger expenses, institutions have to think leaner and find ways to run more efficiently. One strategy is outsourcing technology management and system maintenance to a trusted service provider.

According to Jerry Silva from IDC Financial Insights, spending on third-party providers has increased by 17 percent in the last 10 years. Using an outside provider for IT hardware and services, including DR solutions, is becoming more popular due to a new mindset among financial firms: "You run my technology, I'll run my institution."

This year, forward-thinking financial firms are looking for sophisticated, high-availability technology, particularly cloud solutions. They are working with an experienced technology and cloud services provider in areas like data vaulting and recovery to eliminate the cost associated with maintaining secondary or tertiary data centers.

Compliance Strategies Are Evolving


"With an additional 4,000 pages of new regulations that went into effect on Jan. 10, banks are planning how to keep up with the changes."

Compliance and regulation is getting more costly for financial institutions with no signs of relief. The CEB TowerGroup mentioned how the Consumer Financial Protection Bureau has only put into effect 42 percent of the new rules defined by the Dodd-Frank Act as of December 2, 2013.

The compounding layers of regulations are forcing banks to find alternate solutions for managing the expense of maintaining compliance. Technology partners and co-sourcing compliance functions are helping institutions drive down costs and improve efficiency. It's important to look for vendors who understand how compliance relates to their role within an institution and add value by minimizing the compliance burden.

Vendor Consolidation Is Increasing


"...banking technology has become more complex, uptime more critical and integration essential to success."

Historically, banks and credit unions purchased technology solutions as needed from multiple vendors (e.g., a VAR for hardware, a cloud services provider for data vaulting and recovery, etc.). When it comes to disaster recovery, meeting recovery objectives for a fast-paced, complex infrastructure paired with compliance and security concerns is driving the need to consolidate vendors.

Aligning with a technology partner that understands your industry and has the capability to optimize integration can improve performance and drive down cost. Using a disaster recovery provider that offers off-site data backup, compliance monitoring and more improves data security and reduces complications and vendor conflict at time of event.

What DR tech trends are you getting on board with this year? 

Counting the Cost: Are You Prepared for a Business Interruption?

The Business Continuity Institute has announced that this year's Business Continuity Awareness Week will be March 17-21, and the theme is "Counting the Cost."

We've touched on a few examples of how much disasters can cost businesses (remember Pixar's nightmare and South Park's first missed deadline?), but this month we'll go more in-depth about how to prevent those expenses and prepare for workplace disasters.

We'll be posting a series of blogs that demonstrate the costs of not having a BC/DR plan and cover topics such as DR technology trends, investing in compliance and cloud-based solutions.

Check back in with us throughout the next few weeks to count the cost of being ill-prepared. In the meantime, we're going to kick off the month with this infographic that shows which natural disasters are the most economically destructive.

The Cost of Catastrophe infographic



Popular Posts