Q&A: Black Knight Financial Services Talks BC/DR Testing on the Go

Four men sitting around table with TV screens in background
Black Knight Financial Services' BC/DR Test 
When most people attend the Disaster Recovery Journal (DRJ) Spring World conference, they plan to attend sessions to enhance their knowledge of business continuity and disaster recovery (BC/DR) best practices or browse through the exhibit hall to check out technological advances in the industry. The team at Jacksonville, FL-based Black Knight Financial Services had a more ambitious schedule — they decided to perform a BC/DR test between show activities. 

Black Knight, which is a customer of ours, had heard that we’d be deploying one of our Mobile Recovery Centers (MRCs) to DRJ Spring World to showcase our new Crisis Command Center configuration. With some test deadlines looming, Black Knight approached us about scheduling a test at the show. When we say we have a flexible testing schedule, we mean it, so we made it happen.

After the test, we had a chat with William Russ, Business Continuity Analyst for Black Knight, to talk about Black Knight’s experience with testing on the go. Here’s what he had to say.

Q: What was the objective of the test?
A: Our objective was to simulate a disaster in our primary facility requiring recovery of the enterprise business continuity office at a remote facility to direct crisis management operations and any critical business continuity support functions.

Q: Who participated?
A: Five business continuity specialists and one call center support manager participated in the exercise.

Q: What functions did you test?
A: The tested functions included:
  • VPN connectivity into our backup data center network
  • Network speed test — both Wi-Fi and cabled Ethernet
  • Emergency notification system activation
  • Five-way live video conferencing between Little Rock, AR; Jacksonville, FL; and Orlando, FL MRC locations
  • VoIP softphone capability
  • Logging in to five critical systems to verify data entry and reporting capability

Q: What did you learn from the test?
A: This was the first time most of the team had ever utilized an MRC and we were quite
pleased with the facility, its capabilities and the Rentsys support team. 

Q: What was the most surprising thing the test revealed?
A: The most surprising thing about our exercise is that everything went off without even one hitch!  Also, we were impressed by the network speed back to our company network and the helpfulness of the Rentsys team.  

Quote from William Russ, Business Continuity Analyst, Black Knight Financial ServicesQ: What will you do differently next time?
A: While management was invited to participate in this exercise, a last-minute scheduling conflict required changing some of the participants. We will invite more management to participate next time for higher corporate visibility.

Have you had a unique BC/DR testing experience? We want to hear about it! Let us know in the comments. 

Do You Revoke Access Privileges After an Employee Leaves?

Application password
There were no auto dealership sales reps milling around when a man returned the red muscle car he'd been driving to the dealership's lot. Nobody was there to ask him what he thought of his test drive or to discuss the price. That's because it was 5 a.m. on a Sunday, and the dealership was closed. The man, a former employee of the dealership, never should have had access to the car in the first place.

Lingering access privileges for former employees is a growing problem across all industries. But not all privilege abuses are detected as easily as the dealership ex-employee's joyride — especially when digital assets are involved.

According to a recent study by Osterman Research [PDF], almost 90 percent of former employees retained login credentials for at least one business application, such as PayPal, WordPress or Facebook, after they left the company. Almost half still had access to confidential business data. Forgetting to reset passwords, disable accounts and revoke network access puts your business at serious risk of data and cybersecurity breaches.

An FBI warning to businesses issued in 2014 revealed that costs incurred due to data breaches involving disgruntled or former employees ranged from $5,000 to $3 million. No matter the size of your business, can you afford to risk that much by allowing former employees to retain data access after they leave?

Here are three organizations that had to deal with data breaches at the hands of disgruntled, retiring or former employees and tips for what you should do to avoid a similar breach.


What Happened: In 2010, an employee of fashion brand Gucci created a fake VPN token in the name of a nonexistent employee and later tricked Gucci's IT staff into activating the token after he was fired. He used the access to do about $200,000 worth of damage to the Gucci network, deleting data and shutting down servers.

What You Should Do: Perform regular reviews of employee access privileges. If something seems fishy — such as an account for a fake employee — or if a real employee has access to something that isn't needed for their job duties, terminate the account or the access. You should also terminate all accounts associated with a former employee or contractor and change passwords to group accounts immediately after their departure.

Office of the Comptroller of Currency

What Happened: The U.S. Office of the Comptroller of Currency (OCC), which supervises all national banks, was sent scrambling in 2016 when it discovered that a former employee had downloaded a large number of files onto two removable memory devices prior to retiring from the bureau the year before.

Though the data was encrypted and was not believed to have been misused, the OCC still considered it a major incident. The former employee had misplaced the memory devices, meaning the unrecovered files could still fall into the wrong hands.

What You Should Do: The OCC didn't discover the incident when it happened because it didn't have a policy concerning the use of external media devices. Even when employees feel like they're downloading harmless data such as personal photos, they can still represent a risk. Consider implementing a policy that prevents the download of information to a removable device without supervisor approval. Regularly reviewing what data is being downloaded can also help you react quickly to potential breaches.

Houston Astros

What Happened: A former St. Louis Cardinals employee was recently sentenced to 46 months in prison for his part in hacking into the Houston Astros' player information database. The employee had left to be the Astros general manager and used a similar password between the two teams, giving the hacker an open door to the Astros' confidential research.

What You Should Do: When hiring new employees, be sure you educate them on password security and encourage them to not reuse a password they've used for any other employer or personal application. Implement a policy that requires unique passwords that are frequently changed to combat the possibility of a password falling into the wrong hands.

Unrestricted network access and poor password security aren't the only things that can cause security breaches. The use of personal devices such as smartphones and tablets for business purposes can represent another major security risk for businesses. Read our post "Is Your BYOD Policy Prepared for Pokémon GO?" to find out the importance of a bring your own device (BYOD) policy.

[Webinar Recap] The Cure for Your HIPAA Headache

Culture of Compliance screenshot
As of February 2017, there are more than 1,800 healthcare providers listed on the breach portal — known as the “wall of shame” in the healthcare industry — maintained by the U.S. Department of Health & Human Services Office for Civil Rights (OCR). In 2016, cyber attacks against healthcare organizations increased by 63 percent. These numbers are symptomatic of a growing problem in the healthcare industry: ever-evolving cyber risks and a struggle to adhere to Health Insurance Portability and Accountability Act (HIPAA) security requirements.

In a recent webinar with the Disaster Recovery Journal, Robert Felps, CEO/CISO for compliance and risk management firm Third Rock, and Brandon Tanner, senior manager for Rentsys Recovery Services, discussed what healthcare providers can do about this “HIPAA headache.”

To discover the cure, check out the recording of the webinar here.

Business Continuity 2016: A Year in Review

There was no shortage of challenges for business continuity professionals in 2016. As we move into a new year, we wanted to highlight some of the themes from last year, as we fully expect to see more of the same this year. Here are our top six observations.

Ransomware Was the Most Talked-About Cyber Threat

It’s responsible for shutting down transit systems. It’s cut off communications among hospital staff. It’s extorted millions of dollars from banks. “It” is ransomware — a type of malware that hackers deploy to encrypt data. The hacker then demands a ransom in exchange for a decryption key.

Back in March, the Los Angeles Times declared that 2016 was “shaping up as the year of ransomware.”  The prediction rang true — Kaspersky Lab confirmed that ransomware attacks against businesses increased threefold in 2016. Healthcare is by far the most targeted industry, with telecom and transportation trailing behind.

Ransoms can reach into the range of thousands of dollars. Hollywood Presbyterian Medical Center, for example, paid $17,000 worth of bitcoin to quickly regain access to its data. The FBI, however, has recommended against this strategy, saying that giving in to criminals’ demands only encourages further criminal activity and there’s no guarantee businesses will receive the decryption key after paying the ransom.

The best defense is the one the U.S. Department of Health and Human Services recommends: regularly back up data (so you can restore it in case primary copies are encrypted by ransomware), use security software and educate employees on cybersecurity best practices.

Data Breaches Continued Unabated

Data breaches have spent plenty of time in the spotlight during the past few years. Whether they involved a hacker exploiting a vulnerability while a client moves from one online services vendor to another; a healthcare vendor losing hard drives containing patient data; an employee falling for a phishing attempt and exposing employee W-2s; or a hospital employee accessing files without authorization over a period of several years, data breaches put countless Americans’ data at risk this year.

The exact cost is debatable, but the risks are clear: Businesses risk not only data loss but also intellectual property theft, exposure of company secrets, source code sabotage, investigations by regulatory authorities, reputation damage and costly litigation. The list of consequences goes on and on.

The top three sectors targeted in 2016 were government, healthcare and business. Businesses in the healthcare industry are a prime target, because unlike credit card numbers, personal data like Social Security numbers and medical records can’t be easily changed, so they fetch a premium on the black market.

Some businesses take out data breach insurance policies as protection, but this strategy is no substitute for business continuity planning.

Businesses Lack the Resources for a Well-Rounded Business Continuity Program

Companies aren’t always well equipped to deal with the business continuity threats they’re facing, especially when it comes to disaster recovery (DR) and cybersecurity.

Cybersecurity technology might be top of mind for the global C-suite, but finding the security talent to accompany it is another story. A combination of rapidly evolving cyber threats and inadequate education programs has led to a shortage in security talent. The security professionals that do exist command such high salaries that they’re inaccessible to SMBs and industries that don’t pay as much for cybersecurity talent as others (the financial services industry pays more than healthcare, for example).

Disaster recovery, on the other hand, simply isn’t a top priority for half of C-level execs. Perhaps this is because 65 percent of execs are already confident in their organizations’ DR plans. Problematically, only 31 percent of IT managers agree with this assessment.

Considering these shortcomings, it’s not surprising that only 51 percent of businesses report having a comprehensive business continuity plan.

Vendor Due Diligence Became a Larger Part of Compliance

As always, compliance plays a prominent role in business continuity. In 2016, businesses that are subject to guidelines set by the Federal Financial Institutions Examination Council (FFIEC) and the Health Insurance Portability and Accountability Act (HIPAA) saw their regulatory burden increase. Now, these businesses are being held responsible for performing due diligence on any outsourced service providers that provide essential technology services and/or handle protected health information (PHI) or personally identifiable information (PII). What’s more, they must also perform due diligence on any third parties and their subcontractors used to provide the contracted services.

HIPAA is especially strict — it holds business associates (i.e., subcontractors handling PHI and PII) directly responsible for observing HIPAA requirements. In fall 2016, the Office of Civil Rights (OCR) launched phase 2 of a round of audits, which resulted in the first penalty against a business associate.

The emphasis on vendor management is largely driven in part by the increasing number of cyber threats. The OCR issued an alert on the importance of cyber awareness, and the FFIEC issued a warning about cyber attacks.

The DRaaS Market Continued to Expand

The cloud market — particularly the disaster recovery as a service (DRaaS) market has continued to expand. In 2016, the DRaaS market was worth $1.68 billion and is expected to be worth $11.1 billion by 2021, growing at an estimated CAGR of 45.9 percent.

Data breaches are hastening the move to the cloud, and MSPs are realizing the vast revenue potential of offering DRaaS to their customers. It’s especially appealing to SMBs that lack the resources and expertise to manage a cloud solution, large enterprises that want their dedicated IT staff to spend more time on revenue-generating projects, and organizations that want to leverage multiple clouds (e.g., private and public).

New Weather Challenges Emerged

Data security garnered plenty of attention in the media during 2016, but unique weather threats and natural disasters put business continuity plans to the test as well.

For instance, this year’s hurricane season was a significant one on many levels. It boasted the most hurricane activity since 2012 (there were 15 named storms) and had the most major hurricanes (three) since 2011. Additionally, the Atlantic saw its first Category 5 hurricane in nine years. While hurricane season officially begins in June and ends in November, this season was extra long. Hurricane Alex made an early appearance in the Atlantic in mid-January, and Hurricane Otto showed up in the Caribbean on Thanksgiving. We can expect more of the same in years to come, as some say hurricane season could be extended by as much as a day every year.

NASA reports that fire seasons are getting longer and more frequent as well, with dry landscapes and hotter temps creating prime conditions for fires. To make matters worse, the Forest Service is underfunded and struggling to accommodate fire suppression efforts. The 2016 season included deadly fires such as the Clayton fire in California, which consumed 300,000 acres and destroyed 175 structures. In August, the National Interagency Fire Center reported that California ranked highest for the number, size and severity of wildfires in the West.

The Southeast also experienced significant wildfire activity, which is uncharacteristic of the region. As of November 20, forest fires had burned 119,000 acres across eight states. These numbers don’t include the deadly fires in Gatlinburg, TN — the worst the state has experienced in 100 years. Those blazes alone destroyed more than 2,400 structures and scorched 20,000 acres, killing 14 and injuring 175.

Outlook for 2017

Considering the threats we faced in 2016, our advice for 2017 is to be vigilant, as threats exist on all fronts, from natural disasters to cyber breaches. When planning for cyber threats, be sure you don’t neglect your physical infrastructure. With severe weather threats and natural disasters always on the horizon, you need to consider the impact of not having access to your primary facility. Train your employees well and invest in third-party help if your internal resources aren’t adequate for ensuring you’re protected.

[Webinar Recap] Lessons Learned: Call Center Recovery Testing

The Need for Call Center Continuity slide
Gone are the days of the call center being treated as a cost center. Both customer demands and compliance obligations are bringing the call center to the forefront in business continuity plans for businesses in many industries.

In a recent webinar with the Association for Continuity Professionals (ACP), Brandon Tanner, senior manager for Rentsys, discussed some industry trends that show the role call centers play in addressing customers’ expectations for on-demand service and in meeting compliance requirements for availability.

Brandon was joined by Rentsys customer Steve Hamilton, who’s the business continuity manager for Fiserv, a provider of technology solutions to the financial world. Steve explained the lessons his organization learned during a recent call center recovery test. These takeaways included the importance of manager participation in tests and making adjustments to daily operations when working in an alternate environment.

If you missed the live webinar, you can watch the recording here. Be sure to stick around for the Q&A session at the end. Attendees had plenty of questions about testing logistics, whether work-from-home strategies work for call centers and more.

Cybersecurity: Spend Big Bucks, Outsource or Be Hacked

When it comes to cybersecurity, businesses now have three choices:

    Blue cybersecurity concept
  • Pay a premium for full-time security talent
  • Outsource
  • Be hacked

These choices may sound extreme, but they’re the logical responses to a perfect storm of rapidly evolving cyber threats and inadequate education programs. This combination of factors has resulted in a shortage of skilled security talent for nearly 80 percent of organizations.

A recent article by NewsFactor painted this picture of the cybersecurity landscape, citing research by Intel Security with the Center for Strategic and International Studies (CSIS).

While several top universities offer cybersecurity programs, the curriculum is unable to keep pace with the evolution of security threats. When students leave these programs and enter cybersecurity roles, they’re unprepared to deal with current cyber threats, according to the vast majority (76 percent) of lT professionals.

It’s not surprising, then, that knowledgeable cybersecurity professionals are in high demand and that these positions pay an average of $6,500 more than other IT professions.

If you can’t afford in-house resources, outsourcing can give you access to the cybersecurity skills you require for functions such as ongoing risk assessment and mitigation, network monitoring and access management, and repair of compromised systems. You’ll be in good company — nearly 60 percent of organizations say they’ve outsourced cybersecurity work.

So what will it be for your organization: spend the money for full-time security talent, outsource or be hacked?

Don't Wait. Communicate.

Family looking at digital tabletSeptember is National Preparedness Month, and the timing couldn’t be better. The Predictive Services National Interagency Fire Center predicted a fire season of above-normal risk [PDF], and Hurricane Hermine made landfall on the Southeast coast on Friday.

When disasters like these strike, every second counts. That’s why the Red Cross has chosen “Don’t Wait. Communicate.” as the theme of this year’s National Preparedness Month. After all, you might have a perfectly plotted preparedness plan for responding to disasters, but that plan is useless if the right people don’t know the details of those plans. It’s also important for people to be able to communicate with each other in the midst of a disaster.

For tips on creating a family game plan, visit redcross.org, and then check out this blog post for tips on how to include communications in your business continuity and disaster recovery plan.