The two impediments to real-time data warehousing

The two impediments to real-time data warehousing

In order to achieve real-time data warehousing, defined loosely as no more than five minutes from an event's occurrence to its appearance in the data warehouse, there are two main issues that shops must have a plan for dealing with. The first is the cooperation and lack of intrusion into the applications being sourced. Many legacy applications are fragile, expensive and unruly to update. Many are non-relational and ill prepared to support extracts of any kind except during nightly batch windows. These limitations are driving many "operational" functions into the warehouse environment. I know several shops that list the inability to source in real-time for the data warehouse as being a key driver for their proposed replacement.

Of course, once source systems are upgraded, numerous reporting and real-time access capabilities usually accompany such upgrade diminishing the need for real-time data warehousing.

The second impediment is the impact on usage of the data warehouse itself. It stands to reason that if real-time sourcing and loading is a requirement that the data access would be rather active as well. User usage of the warehouse should not (can not) be disrupted by loading activity. If you cannot achieve real-time data warehousing without taking the warehouse down or severely restricting usage of the warehouse throughout the day, keep trying. Usually partitioning strategies, whether manual or automatic/DBMS-driven, are sufficient for achieving

    Requires Free Membership to View

    Login

    When you register, you'll begin receiving targeted emails from my team of award-winning editorial writers on the latest customer relationship management (CRM)and call center technology issues today. Our goal is to keep you informed on the hottest issues facing this fast-changing industry.

    Hannah Smalltree, Editorial Director

    By submitting your registration information to SearchCRM.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchCRM.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

the right lock level for the load operation which will result in minimal impact on warehouse usage for the minimalist "load" that occurs frequently with a real-time warehouse.

Overcoming these two impediments, whether with a more exercised and frequently scheduled "batch" ETL or with the inclusion of EAI in the architecture provides a rich reward to the successful program -- a real-time data warehouse.

For more information, check out SearchCRM.com's Best Web Links on Business Intelligence.


This was first published in October 2002

Join the conversationComment

Share
Comments

    Results

    Contribute to the conversation

    All fields are required. Comments will appear at the bottom of the article.

    Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.

    Back to top