Skip to main content

Enterprise Data Management (Data Warehouse/ ETL)

 A data warehouse, also known as an enterprise data warehouse (EDW), is a system that stores and analyzes data from multiple sources.

Data warehouses are used for reporting and data analysis, and are considered a core component of business intelligence.

Case study: 

Trends, an affordable fashion & lifestyle retailer which operates more than 2,300 stores in over 1,100 towns and cities across India.

They might have goods stored in different warehouse, They might have enterprise application to track the goods in warehouse and analyze them for improving their business.


A data warehouse has four main components: refer above diagram
  1. Central database:
  2. ETL (extract, transform, load) tools
  3. Metadata (Data sources)
  4. Access tools 
Some examples of data warehouses include:
  1. Snowflake
  2. Google BigQuery
  3. Amazon Redshift
  4. Azure Synapse Analytics
  5. IBM Db2 Warehouse
  6. Firebolt 

Benefits of Data Warehouse:
  1. Understand business trends and make better forecasting decisions.
  2. Data Warehouses are designed to perform well enormous amounts of data.
  3. The structure of data warehouses is more accessible for end-users to navigate, understand, and query.
  4. Queries that would be complex in many normalized databases could be easier to build and maintain in data warehouses.
  5. Data warehousing is an efficient method to manage demand for lots of information from lots of users.
  6. Data warehousing provide the capabilities to analyze a large amount of historical data.

Enterprise Data Management ( EDM ) governance and control:

Enterprise data management (EDM) is the process of defining, integrating, and retrieving data for internal applications and external communication. 

EDM also involves managing the people who access the data, ensuring that they have the right information and follow the organization's standards for storing data. 

EDM focuses on creating accurate, consistent, and transparent content. 


EDM Architecture:

Data Management Maturity Model:


Extract, transform, and load (ETL) is the process of combining data from multiple sources into a large, central repository called a data warehouse. ETL uses a set of business rules to clean and organize raw data and prepare it for storage, data analytics, and machine learning (ML).
Here are some advantages of ETL:
  1. Data governance: ETL fosters data democracy, which broadens data accessibility and makes it available to all the stakeholders to analyze it and use it for business
  2. Reduce delivery time: ETL can reduce delivery time
  3. Reduce unnecessary expenses: ETL can reduce unnecessary expenses
  4. Automate complex processes: ETL can automate complex processes
  5. Validate data before migration: ETL can validate data before migration 
Amazon Redshift:





Comments

Post a Comment

Popular posts from this blog

Certified Enterprise Architect Professional (CEAP) - Module 5 - Architecture Frameworks

Architecture Frameworks: An Architecture Framework is a theoretical structure that has the purpose of developing, executing, and maintaining an Enterprise Architecture. Advantages of EA framework: Simplify Breaks down areas of the business process Organise business components and create and identify relationships between business Determine the scope Customization in the existing framework Disadvantages of EA framework: Need to follow process Provides only direction and not information It's based on goal and objective Need creativity and proactive thinking Zachman Framework: The Zachman Framework is a widely used model in Enterprise Architecture (EA) that provides a structured way to classify and organize an organization's information infrastructure by defining different perspectives from various stakeholders, allowing for a holistic view of the enterprise and facilitating alignment between business needs and technology solutions; essentially acting as a template to organize arc...
Post a Comment
Read more

Daily Agile Scrum stand-up meeting guidelines

Followers of the Scrum method of project management will typically start their day with a " stand-up meeting ". In short, this is a quick daily meeting (30 minutes or less) where the participants share the answers to the three questions with each other: • What did I accomplish yesterday?  • What will I do today?  • What obstacles are impeding my progress?  Some people are talkative and tend to wander off into Story Telling .  Some people want to engage in Problem Solving immediately after hearing a problem. Meetings that take too long tend to have low energy and participants not directly related to a long discussion will tend to be distracted. These are the minimum number of questions that satisfy the goals of daily stand-ups. Other topics of discussion (e.g., design discussions, gossip, etc.) should be deferred until after the meeting.  Here are few tips for running a smooth daily meeting:  • Everyone should literally stand-up and no one should sit down ...
Post a Comment
Read more

Empiricism (Scrum)

Empiricism asserts that knowledge comes from experience and making decisions based on what is observed. Pillars of  Empiricism . Various practices exist to forecast progress, like burn-downs, burn-ups, or cumulative flows. While proven useful, these do not replace the importance of empiricism . In complex environments, what will happen is unknown. Only what has already happened may be used for forward-looking decision making. Each artifact contains a commitment to ensure it provides information that enhances transparency and focus against which progress can be measured: ● For the Product Backlog it is the Product Goal. ● For the Sprint Backlog it is the Sprint Goal. ● For the Increment it is the Definition of Done. These commitments exist to reinforce empiricism . The sum of the Increments is presented at the Sprint Review thus supporting empiricism .
Post a Comment
Read more