Top 5 Reasons for Data Warehouse Modernization
Many paths lead to the improvements users need for analytics, big data, real time speed, productivity, and costs.
- By Philip Russom, Ph.D.
- May 20, 2014
In recent surveys by TDWI Research, roughly half of respondents report that they will replace their primary data warehouse (DW) platform and/or analytic tools within three years. Ripping out and replacing a DW or analytics platform is expensive for IT budgets and intrusive for business users. This raises the question: What circumstances would lead so many people down such a dramatic path?
It's because many organizations need a more modern DW platform to address a number of new and future business and technology requirements. In a nutshell, organizations that seek to modernize their data warehouse environment do so to improve advanced analytics, scale, speed, productivity, or economics. Each of the five reasons listed here has multiple meanings, they are all interrelated, and users sort the five into varying priority orders, based on their needs.
Even so, in general, the list constitutes the top five reasons for data warehouse modernization, and they can provide some guidance for users facing modernization.
Advanced analytics. Many organizations have invested heavily in reporting and OLAP, but now they need to invest in advanced forms of analytics to leverage big data, find new customer segments, and stay competitive.
Speed. Organizations likewise need the data warehouse and related systems to operate faster because speed contributes to scale, supports agile development and discovery analytics, and brings analytics closer to real-time business operations.
Scale. This continues to be an issue with big data and other burgeoning enterprise datasets as well as with growing numbers of concurrent users, reports, analyses, and data structures.
Productivity. Traditional requirements gathering, prototyping, and development takes months, which is too long for a modern business. That's why agile development methods are now the norm in DW/BI and analytics. Likewise, users are adopting agile tool types, including those for data exploration and discovery, data profiling, and data visualization.
Costs. The good news that modernization is not only a chance to increase speeds and feeds in your data warehouse environment, but it is also a golden opportunity to rethink DW overall costs, as users seek to save money in some areas (storage, CPUs, upgrades, admin) so they can invest in others (new data platforms, analytic tools, and developing new solutions).
Achieving the top five goals of DW modernization demands the acquisition of new data platforms and tool types, usually columnar databases, DW appliances, NoSQL databases, Hadoop, and so on. For example, according to a recent TDWI survey about Hadoop, only 10 percent of respondents report having the Hadoop Distributed File System (HDFS) in production today, while a whopping 63 percent expect to deploy HDFS within three years.
Hence, the result of some DW modernizations is a multi-platform DW environment. The benefit is that users can choose the best platform for a given data workload or analytic goal, plus offload certain workloads from the data warehouse. The challenge is to establish and maintain a broad data warehouse architecture that unifies data and its processing, despite being strewn across multiple platforms.
For more information about trends in data warehouse modernization, attend the upcoming TDWI Webinar: The Data Warehouse Modernization Tipping Point, to be broadcast May 28, 2014. Register online at http://tdwi.org/webcasts/2014/05/the-data-warehouse-modernization-tipping-point.aspx. You can also read the 2014 TDWI Best Practices Report: Evolving Data Warehouse Architectures in the Age of Big Data, available for download at tdwi.org/bpreports.
Other articles of interest include the 2013 TDWI Checklist Report: The Modern Data Warehouse, available for download here
and the 2013 TDWI Checklist Report: Hadoop Best Practices for Data Warehousing, Data Integration, and Analytics, available here.