Prepare to fail… elegantly

Guardians and Warehouses?

A Data Guardian, in the sense that I use the term – is a Business Intelligence governance role which is responsible for the quality & integrity of corporate data. The final collection point of this data is usually an enterprise Data Warehouse, and what the data warehouse stores is often referred as 'the truth'.

In any single day, while working with corporate data, there are a thousand things that can affect the integrity of our precious facts as they progresses through business intelligence workflows that make up our BI structures.

End users can (and often do!) input erroneous data into core systems, infamous network issues are never far from reality, power outages, hardware failures and more are just a few of the day to day challenges one will have to cater for when attempting to protect the veracity of a factual payload.

think of enterprise data as a living entity..., breathing, undulating, never static, always changing, always challenging and as cunning as a fox.

All of these potential errors are cause for concern and can affect any link in our information supply chain, whether in the ETL (the data extraction and integration process), the data warehouse or the presentation layer.

It's Alive!!

The best BI Architects and BI systems utilize sophisticated data cleansing and error handling routines which prevent unwanted artifacts from contaminating the data warehouse.

But... from experience, I also know that no matter how perfectly architected and developed a BI solution is, corporate data has an uncanny ability to find a way past the extremes of any error trapping, ingeniously bypassing them, to find an unwelcomed home in our disinfected pool of wondrous data. To me, I think of enterprise data as a living entity..., breathing, undulating, never static, always changing, always challenging and as cunning as a fox.

We take great care massaging and cleansing data from our source systems to integrate into our data warehouse, we build fancy pants error handling and workflow, and yet no matter how much precaution we take, we also need to prepare for inevitable failure.

It’s important to remember that there are always potential conditions beyond our control. We have to set our professional egos aside and prepare for the worst. We can’t always predict hardware failure, or how every one of your users will react, or every overload situation on the data warehouse, or every and all security scenarios or whether or not your network decides to hold up, or even mother nature throwing a tantrum. Potentially any of these could cause your systems or processes to fail.

Speaking exclusively from an data integrations viewpoint, it is imperative that these processes are able to restart and restore elegantly... The significance of which is they don’t then propagate the BI system with corrupt data.

It’s important to remember that there are always potential conditions beyond our control. We have to set our professional egos aside and prepare for the worst.

So how do we get around this?

Prepare for failure.

Architect your processes to restart elegantly. Always consider what will happen to your information payload in case of sudden and unexpected outage.

Use Stage-gates.

Another good way to mitigate risk is to use a “stage-gate” approach and to heavily audit your data before it reaches it's destination.

These approaches take extra time and can cost, and too many times I’ve seen clients looking to cut corners to save time and costs, and when they do, they end up with corrupt data and a lot of issues.

Spend effort on prevention.

A useful guide to live by is what's known as the 1-10-100 rule. The rule estimates that every dollar spent on prevention, can save 10x that in repair costs, and in turn save 100x the preparation cost of a catastrophic failure.

Think about the recent United Airlines debacle. Going forward they’re going to be offering $10,000 to re-board and re-schedule an overbooked passenger. Which makes me wonder why they overbook the flights for the sake of an extra $1,000 in the first place, as the failure of which might end up costing $100,000’s from a PR perspective (or in United’s case, millions).

Data Guardians and BI professionals may be a cautious and somewhat serious bunch, but we need to always be thinking about what could go wrong as well as well as how it should go right.

In summary, as Benjamin Franklin once said: “If you fail to plan, you are planning to fail!”.

…or in our case “Plan to fail... elegantly.”

Prevention is the key.

Dave Anderson is a senior BI architect who excels at sculpting raw data into interesting & useful information. He helps forge that critical bond between business and technology. His career has included endeavors for blue-chip Australian banks, international retailers and professional services firms. Where other consultants might step back, Dave steps up to crunch the numbers and solve the knottiest and gnarliest of challenges.