What to use for ETL: Tool or spool

What to use for ETL: Tool or spool

For ETL jobs, two methods are commonly adopted -- the use of an ETL tool or the process of spooling data out to a flat file, transforming data using scripts or a programming language (most common is C) and the use of a native utility like SQL Load or BCP to put the data into its respective tables.

A shop serious about data warehousing and in for the long haul would be wise to use an ETL tool but there is always controversy as to which to use -- tool or spool. There are times, however, when it will almost be mandatory to use to an ETL tool:

  1. Cleansing of operational data is required.
  2. Frequent data massage during transformation is required.
  3. Duplication and/or migration are necessary.
  4. Tables at target should be populated/updated taking data from different tables from different databases.
  5. Target database should contain the ETL procedures/packages for system integrity.
  6. Data repository containing metadata and Data mining is important for OLAP or analytical cubes.

There are also times when, unfortunately, it's possible to delay the decision to purchase, for a brief time:

  1. Huge amount of mathematical and statistical calculations required during the process of transformation.
  2. Frequent and periodic update at the target when data is coming from a production database and performance is really an issue.
  3. Transformation rules vary time to

    Requires Free Membership to View

    Login

    When you register, you'll begin receiving targeted emails from my team of award-winning editorial writers on the latest customer relationship management (CRM)and call center technology issues today. Our goal is to keep you informed on the hottest issues facing this fast-changing industry.

    Hannah Smalltree, Editorial Director

    By submitting your registration information to SearchCRM.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchCRM.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

  1. time or depends on the nature of the coming source data.
  2. Data cleansing is necessary that involves intelligent behavior of the transformation process to avoid duplication at the target for the same data entered differently at different sources.

For more information, check out SearchCRM's Best Web Links on Business Intelligence and Data Analysis.


This was first published in March 2002

Join the conversationComment

Share
Comments

    Results

    Contribute to the conversation

    All fields are required. Comments will appear at the bottom of the article.

    Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.

    Back to top