Data load for the snowflake data warehouse

Here are a few points to remember for the initial data load for the snowflake data warehouse:

  1. After development of the source-target map for the initial load, start from the top-level dimensions and then come to the fact through the intermediate level dimensions for loading.
  2. Have a staging area where source data lands before going into production. The staging area may contain the operational data model and some other supplementary tables to perform joins to implement the connection of surrogate keys and also to acquire the metadata from the ETL jobs.
  3. The metadata for the initial load may include:
    • Job Name
    • Date
    • Package / Procedure Name
    • Source database and Table
    • Destination Database and Table
    • No of rows loaded
    • No of rows rejected
    • Time consumed
  4. For ETL jobs, E-mail service or proper message escalation keeps track of the process and alerts users and other concerned parties on any type of mishaps. These messages may contain the metadata collected from the ETL job.
  5. If any tool is being used for the ETL, it is always advisable to use SQL more than any other external language. This will be good for performance, reliability and further modifications.
  6. Data cleansing, transformation and implementation of data quality should be implemented in the staging area and not on the production server

    Requires Free Membership to View

    Login

  1. of operational systems.
  2. The ETL processes should be flexible enough to cope with ongoing updates regularly. Using global variables and parameters, rather than hard coded codes, enhances the process.
  3. Well document everything for future use and maintenance.

For more information, check out SearchCRM's Best Web Links on Data Warehousing.


This was first published in April 2002

Join the conversationComment

Share
Comments

    Results

    Contribute to the conversation

    All fields are required. Comments will appear at the bottom of the article.

    Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.

    Back to top