📘 Free DP-750 Sample Questions
This is a case study. Case studies are not timed separately from other exam sections. You can use as much exam time as you would like to complete each case study. However, there might be additional case studies or other exam sections. Manage your time to ensure that you can complete all the exam sections in the time provided. Pay attention to the Exam Progress at the top of the screen so you have sufficient time to complete any exam sections that follow this case study.
To answer the case study questions, you will need to reference information that is provided in the case. Case studies and associated questions might contain exhibits or other resources that provide more information about the scenario described in the case. Information provided in an individual question does not apply to the other questions in the case study.
A Review Screen will appear at the end of this case study. From the Review Screen, you can review and change your answers before you move to the next exam section. After you leave this case study, you will NOT be able to return to it.
To start the case study -
To display the first question in this case study, select the “Next” button. To the left of the question, a menu provides links to information such as business requirements, the existing environment, and problem statements. Please read through all this information before answering any questions. When you are ready to answer a question, select the “Question” button to return to the question.
Overview -
Company Information -
Contoso, Inc. is a renewable energy provider that operates solar and wind farms across North America. Existing Environment -
Azure Environment -
Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.
Workspace1 contains all-purpose clusters for both development and production workloads. The company's Azure environment contains:
In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub
A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data
An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data Data Environment -
Contoso ingests the following operational and business data:
Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.
Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.
ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information.
Problem Statements -
The company’s existing analytics environment has several issues:
Ingestion -
Telemetry pipelines fall behind during peak loads. Telemetry ingestion fails when schema drift occurs. Streaming pipelines reprocess events after a pipeline restarts. Compute -
Production and development workloads run on the same all-purpose clusters.
Production and development workloads do NOT support autoscaling or workload isolation. Governance -
The ERP data is duplicated across systems and development teams.
Naming conventions are inconsistent across development teams, regions, and products.
Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership. Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.
Pipeline operations -
Pipelines lack resiliency, alerting, and centralized scheduling. Requirements -
Planned Changes -
Contoso plans to implement the following changes: Implement scalable data pipeline orchestration.
Create a managed analytics catalog in Unity Catalog. Implement a consistent approach to creating curated datasets.
Establish a centralized governance model across ingestion, cleansed, and curated layers. Grant data engineers access to the ERP tables by using minimal development effort. Adopt a compute strategy that isolates production workloads and supports autoscaling.
Adopt a slowly changing dimension (SCD) approach to address current data modeling issues. Technical Requirements -
Contoso identifies the following environment and compute requirements:
Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes.
Provide fast and consistent performance for business intelligence (BI) workloads. Prevent development activity from affecting production pipelines.
Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.
Contoso identifies the following data ingestion and processing requirements:
Auto-scale ingestion pipelines to handle bursty workloads. Handle schema drift for the maintenance and telemetry data. Ingest file-based telemetry data by using minimal operational effort.
Store all the ingested data in a format that supports incremental processing.
Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics. Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server. Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog.
Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data.
Contoso identifies the following data modeling and optimization requirements:
Build curated tables that standardize business logic.
Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.
Contoso identifies the following pipeline deployment and operation requirements: Orchestrate multi-step ingestion and transformation workflows.
Define a clear execution order and dependencies. Automatically retry failed steps and notify operators.
Schedule ingestion and transformation workloads consistently. Governance Requirements -
Contoso identifies the following governance requirements:
Centralize the metadata catalog.
Provide isolated development areas that follow standard naming conventions. Establish a consistent structure for organizing raw, cleansed, and curated data. Provide a read-only mechanism to reference the ERP data through a foreign catalog. Business Requirements -
Contoso identifies the following business requirements:
Improve ingestion reliability and reduce operational effort. Standardize data definitions across development teams.
You need to configure compute for the ingestion of telemetry data. The solution must meet the data ingestion and processing requirements.
What should you do?
A
Move the ingestion pipelines to shared compute.
B
Enable Photon acceleration for a job compute cluster.
C
Increase an all-purpose cluster to a larger fixed node type.
D
Disable autoscaling for a job compute cluster.
Correct Answer:
B. Enable Photon acceleration for a job compute cluster.
Explanation:
Why B is the best choice:
Scalability and Autoscaling: The requirement specifies that production ingestion workloads must run on compute clusters that can scale automatically during telemetry spikes. Job compute clusters in Azure Databricks are designed for this purpose, and enabling Photon acceleration further enhances the performance of these workloads by providing a native, in-memory query processing engine that accelerates Spark workloads without requiring any code changes.
Workload Isolation and Autoscaling Support: Job clusters are ideal for scheduled, non-interactive pipelines (as required), isolating production workloads from development activities. Photon acceleration on a job cluster ensures fast and consistent performance, crucial for BI workloads and meeting the exactly-once semantics for telemetry data ingestion.
Meeting Specific Requirements:
Auto-scale ingestion pipelines: Supported by job clusters.
Handle schema drift: While not directly related to compute, the overall architecture (e.g., using Delta tables with merge capabilities) can handle this, and Photon doesn’t hinder this capability.
Supports incremental processing and exactly-once semantics: Photon-accelerated job clusters can efficiently handle these requirements.
Why Other Options are Less Suitable:
A. Move the ingestion pipelines to shared compute:
Contradicts Requirements: Shared compute would not isolate production workloads from development, potentially affecting pipeline performance and reliability.
Lacks Autoscaling for Production Spikes: Shared clusters might not scale as effectively for sudden production workload increases.
C. Increase an all-purpose cluster to a larger fixed node type:
Inefficient Scaling: Fixed node sizes do not offer the dynamic scaling needed for handling bursty, spike-based workloads efficiently.
No Workload Isolation: All-purpose clusters mix development and production workloads, contrary to the requirements.
D. Disable autoscaling for a job compute cluster:
Directly Opposes Requirements: Disabling autoscaling would prevent the cluster from scaling automatically during telemetry spikes, contradicting a primary technical requirement.
Inefficient Resource Utilization: Would lead to under or over provisioning, affecting cost and performance.
s
Azure Databricks: Job Compute Clusters Azure Databricks: Photon Acceleration
You have an Azure Databricks workspace that is attached to a Unity Catalog metastore named metastore1, metastore1 contains a catalog named catalog1.
You need to create a new schema named schema2 that meets the following requirements: Is contained in catalog1 -
Uses abfss://[email protected]/data as the managed location Which SQL statement should you execute?
A
CREATE SCHEMA catalog1.schema2 - LOCATION ‘abfss://[email protected]/data’;
B
CREATE SCHEMA catalog1.schema2 -
MANAGED LOCATION ‘abfss://[email protected]/data’;
C
CREATE CATALOG schema2 -
MANAGED LOCATION ‘abfss://[email protected]/data’;
D
CREATE SCHEMA catalog1.schema2 -
WITH DBPROPERTIES (LOCATION-’abfss://[email protected]/data’);
Correct Answer:
B. CREATE SCHEMA catalog1.schema2 -
MANAGED LOCATION ‘abfss://[email protected]/data’;
Explanation:
Technical Justification for Correct Answer: B
To create a new schema named schema2 within catalog1 on Azure Databricks with Unity Catalog, meeting the specified requirements, the correct SQL statement is B. CREATE SCHEMA catalog1.schema2 -MANAGED LOCATION 'abfss://@[storage_account_name].blob.core.windows.net/data';. Here’s why:
You have an Azure Databricks workspace named Workspace1.
You create a compute cluster named Cluster1 that will be used to ingest data.
You need to install the required libraries on Cluster1. The solution must use Unity Catalog for access control. What should you do?
A
Install the libraries by using pip3.
B
Create a custom dependency management script and run the script from a Databricks notebook.
C
Upload the libraries to Workspace1 and install the libraries on Cluster1.
D
Install the libraries on Cluster1 and manually restart the cluster.
Correct Answer:
C. Upload the libraries to Workspace1 and install the libraries on Cluster1.
Explanation:
Technical Justification for Correct Answer (C)
Why C is the best option:Installing libraries by uploading them to the Workspace and then installing on the Cluster (Option C) is the most suitable approach for several reasons, especially when Unity Catalog is involved for access control:
Centralized Management: Uploading libraries to the Workspace facilitates centralized management, making it easier to track and update libraries across multiple clusters.
Security and Access Control via Unity Catalog: Since the solution must use Unity Catalog for access control, uploading libraries to the Workspace aligns with managing resources in a unified manner. This approach ensures that access to libraries can be controlled and audited similarly to other resources managed by Unity
Catalog.
Cluster Management Efficiency: Installing libraries from a centralized workspace location reduces the overhead of managing different library versions across clusters and ensures consistency.
Automation and Scalability: This method is more amenable to automation, particularly when integrating with CI/CD pipelines for library updates, enhancing the overall scalability of the data engineering workflow.
Why Other Options are Less Suitable:
A. Install the libraries by using pip3:
Lack of Central Control: Installing directly via pip3 on the cluster doesn’t leverage Workspace-level management, making version control and access management more challenging.
Security Concerns: Direct internet access for pip3 might not be desirable in all environments due to security
policies.
B. Create a custom dependency management script and run from a Databricks notebook:
Complexity: Introduces unnecessary complexity for library installation, which can be managed more straightforwardly through Workspace uploads.
Dependency on Notebook Execution: Ties library installation to notebook runtime, which might not be ideal for cluster provisioning workflows.
D. Install the libraries on Cluster1 and manually restart the cluster:
Inefficient: Manual intervention for installation and restart is inefficient and prone to human error, especially in automated deployment scenarios.
Lacks Integration with Unity Catalog’s Access Control: Does not align with the requirement for unified access control through Unity Catalog.
s
Azure Databricks: Managing Libraries Azure Databricks: Unity Catalog
You have an Azure Databricks workspace that contains an all-purpose cluster named Cluster1. You need to configure Cluster1 to meet the following requirements:
Scale up automatically when workloads increase Scale down automatically when workloads decrease Minimize costs -
Which two actions should you perform? Each correct answer presents part of the solution. NOTE: Each correct selection is worth one point.
A
Disable Photon acceleration.
B
Enable autoscaling for Cluster1.
C
Apply a compute policy that enables users to manage the cluster settings.
D
Specify a fixed number of workers.
E
Configure Cluster1 to terminate after 30 minutes of inactivity.
Correct Answer:
B. Enable autoscaling for Cluster1.
Explanation:
Technical Justification for Correct Answer: BE
To meet the requirements of scaling up/down automatically based on workload and minimizing costs for Cluster1 in an Azure Databricks workspace, the following actions are justified:
B. Enable Autoscaling for Cluster1
Why It's Correct: Enabling autoscaling allows Cluster1 to dynamically adjust the number of workers in response to changes in workload. When workloads increase, the cluster scales up to handle the demand, and when workloads decrease, it scales down. This directly addresses the first two requirements.
Technical Basis: Autoscaling in Azure Databricks is based on the current workload, ensuring resources are allocated efficiently. It leverages Databricks' internal metrics to decide when to add or remove nodes.
Cost Minimization: By only provisioning additional resources when necessary and reducing them during lulls, autoscaling plays a crucial role in minimizing costs.
E. Configure Cluster1 to Terminate After 30 Minutes of Inactivity
Why It's Correct: Configuring the cluster to terminate after a period of inactivity ensures that no resources are consumed (and thus, no costs incurred) when the cluster is not in use. This significantly contributes to minimizing costs.
Technical Basis: Azure Databricks clusters, especially those not configured with manual termination settings, can incur unnecessary costs if left idle. Setting an inactivity timeout aligns with a cost-optimized strategy. Alignment with Requirements: While not directly scaling, this action ensures that when there's no workload (the ultimate decrease), costs are fully eliminated for that cluster.
Why Other Options Are Less Suitable
A. Disable Photon Acceleration:
Irrelevant to Scaling and Cost: Photon acceleration is related to query performance optimization, not scaling or cost management. Disabling it would not impact the ability to scale or reduce costs.
C. Apply a Compute Policy to Enable Users to Manage Cluster Settings:
Indirect and Potentially Counterproductive: While useful for governance, this doesn't directly achieve the scaling or cost objectives. It might even lead to inefficient scaling if users aren't optimized for cost.
D. Specify a Fixed Number of Workers:
sAzure Databricks Autoscaling Documentation:
s
Azure Databricks Autoscaling Documentation:
Azure Databricks Cluster Lifecycle Policies:
You have an Azure Databricks workspace that is enabled for Unity Catalog.
You have an Apache Spark Structured Streaming job that writes data to a Delta table. After the cluster restarts, the streaming job reprocesses previously ingested data.
You need to prevent the streaming job from reprocessing the data after the cluster restarts. What should you do?
A
Configure a checkpoint location for the streaming query.
B
Increase the trigger interval of the streaming query.
C
Enable change data feed (CDF) for the target table.
D
Configure a watermark for the streaming query.
Correct Answer:
A. Configure a checkpoint location for the streaming query.
Explanation:
Technical Justification for Correct Answer: A
Why A is the Best Option:Configuring a checkpoint location for the streaming query (Option A) is the most suitable solution to prevent reprocessing of previously ingested data after a cluster restart. Here's why:
Checkpointing in Apache Spark Structured Streaming allows the job to save its progress (e.g., the last offset processed) to a designated location. Upon restart, the job can resume from the last checkpoint, avoiding redundant processing of already ingested data.
This approach directly addresses the issue by maintaining the stream's processing state, ensuring continuity post-restart without altering the job's logical behavior or dependencies.
Why Other Options are Less Suitable:
B. Increase the trigger interval of the streaming query:
Increasing the trigger interval might reduce the frequency of processing but does not prevent reprocessing of previously ingested data upon restart. It's more about throughput control than state persistence.
C. Enable change data feed (CDF) for the target table:
Change Data Feed (CDF) on Delta tables is useful for capturing changes (inserts, updates, deletes) made to the table. However, it doesn't directly solve the problem of a streaming job restarting and reprocessing data from its source; CDF is more relevant for tracking changes within the Delta table itself.
D. Configure a watermark for the streaming query:
Watermarks are used to handle late-arriving data in event-time processing, ensuring that the query can distinguish between late arrivals and the end of the stream. While crucial for certain streaming scenarios,
watermarks do not prevent the reprocessing of already consumed data upon a cluster restart.References:
watermarks do not prevent the reprocessing of already consumed data upon a cluster restart.
s:
HOTSPOT -
You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a managed Delta table named Table1.
Table1 is written by batch jobs every hour and is queried frequently by filtering two columns named CustomerId and EventDate.
You expect Table1 to grow significantly over time.
The rows in Table1 are frequently updated and deleted to support compliance requests.
You need to keep query performance consistent as Table1 grows. The solution must minimize update and deletion effort.
What should you include in the solution? To answer, select the appropriate options in the answer area. NOTE: Each correct selection is worth one point.
A
Correct Answer:
A.
Explanation:
Liquid clustering: This is the modern, recommended data layout technique in Delta Lake and Azure Databricks. Unlike legacy partitioning or static Z-ordering, Liquid clustering dynamically groups data based on high-cardinality or frequently queried columns without creating fragmented small files. It ensures consistent query performance automatically as the table size grows over time without requiring repetitive manual tuning or optimization re-runs.
Enable deletion vectors: Deletion vectors drastically minimize update and deletion effort by utilizing a "merge-on-read" mechanism. When a row is modified or deleted, Databricks writes a tiny, highly compressed bitmap file indicating which rows were removed, rather than executing a heavy "copy-on-write" operation that forces the entire data file to be rewritten. This reduces write amplification and accelerates transaction execution.
Incorrect answer:
Partitioning: This legacy directory-based strategy frequently results in the "small-file problem" or uneven data distribution when applied to high-cardinality fields, degrading query performance as tables expand.
Z-ordering: While excellent for multi-dimensional data skipping, Z-ordering is a static operation. It requires running periodic, high-overhead OPTIMIZE commands to maintain its performance benefits as new data is continually appended.
Disable deletion vectors: Disabling this feature forces Databricks to fall back to traditional copy-on-write execution, significantly increasing latency, infrastructure compute usage, and storage costs for frequent updates or deletes.
Run the VACUUM command after each batch load: The VACUUM command physically purges stale data files that are no longer referenced by the current state of the transaction log. Running it continuously after every batch load adds severe, unnecessary storage engine overhead and limits your ability to use "Time Travel" queries.
You have an Azure Databricks workspace that contains a cluster named Cluster1.
Performance monitoring shows that Cluster1 is consistently overprovisioned for its batch workload: CPU utilization remains below 20 percent, including peak processing periods.
The workload is highly predictable and does not spike.
The current node type already meets the workload requirements. You need to reduce compute costs without increasing job duration. What should you do?
A
Enable Photon acceleration.
B
Configure Cluster1 to use a larger node type.
C
Decrease the autotermination timeout of Cluster1.
D
Disable autoscaling and reduce the number of worker nodes.
Correct Answer:
D. Disable autoscaling and reduce the number of worker nodes.
Explanation:
Technical Justification for Correct Answer: D
Why D is the Best Option:Reducing compute costs without increasing job duration for an overprovisioned Azure Databricks cluster (Cluster1) with a predictable, non-spiking batch workload can be most effectively achieved by D. Disable autoscaling and reduce the number of worker nodes.
Cost Reduction: Directly decreasing the number of worker nodes reduces the overall compute resources utilized, leading to lower costs.
No Increase in Job Duration: Since the workload is highly predictable and the current node type meets requirements, reducing the number of nodes (not changing the node type) won’t impact job processing time, assuming the reduced node count still sufficiently handles the workload’s peak (which, at <20% CPU utilization, is likely).
Autoscaling Disablement: Given the predictability of the workload, autoscaling (which is designed for variable or unpredictable workloads) is unnecessary and disabling it ensures the cluster doesn’t inadvertently scale up, incurring additional costs.
Why Other Options are Less Suitable:
Enable Photon acceleration:
Purpose: Photon is designed to accelerate query performance by improving processing speeds, not directly reducing costs.
Impact on Cost: Enabling Photon might reduce job durations, potentially leading to indirect cost savings if nodes can be reduced afterward. However, it does not directly address the overprovisioning issue described. Conclusion: Not the primary solution for reducing compute costs in this scenario.
Configure Cluster1 to use a larger node type:
Impact: Upgrading node types would increase costs due to the larger resource footprint, contrary to the objective.
Conclusion: Directly opposes the goal of reducing compute costs.
Decrease the autotermination timeout of Cluster1:
Purpose: Reduces the time a cluster stays idle before shutting down, which can save costs if the cluster is frequently idle.
Limitation: The question implies the cluster is consistently overprovisioned during peak processing periods, not just idle times. Thus, the cost savings would be minimal compared to reducing node count.
Conclusion: Less effective for the described scenario focusing on overprovisioning during active processing.
s
[Azure Databricks: Cluster Configuration Best Practices](
You have an Azure Databricks solution that was developed by multiple engineers and deployed to development, test, and production environments.
You need to implement an application lifecycle management (ALM) process that meets the following requirements:
Deploys and provides source control for Databricks notebooks, workflows, and jobs in a consistent and repeatable way
Parameterizes the target deployment environment What should you include in the solution?
A
the export and import of Databricks workspaces
B
Databricks resources managed by using the Databricks Terraform provider
C
Git folders and manual promotion
D
Declarative Automation Bundles in Git
Correct Answer:
D. Declarative Automation Bundles in Git
Explanation:
Technical Justification for Correct Option (D) and Analysis of Other Options Correct Option: D. Declarative Automation Bundles in Git
Why D is the best choice:
Declarative Automation Bundles in Git directly address the requirements by providing a consistent and repeatable way to deploy Databricks notebooks, workflows, and jobs.
Source Control Integration: Storing these bundles in Git (e.g., Azure DevOps, GitHub) inherently provides source control for all components, ensuring versioning and collaboration among multiple engineers.
Environment Parameterization: Declarative Automation Bundles can be easily parameterized for target environments (development, test, production) using variables in the Git repository, which can be switched or overridden based on the deployment target.
Automation and Consistency: Automation Bundles automate the deployment process, reducing manual errors and ensuring consistency across environments.
Why Other Options are Less Suitable:
Export and Import of Databricks Workspaces
Limitation: This approach is more manual and not scalable for managing individual notebooks, workflows, and jobs across multiple environments efficiently.
Lack of Native Parameterization: Doesn’t inherently support environment parameterization without additional, potentially complex scripting.
Source Control: While workspaces can be exported/imported, this method doesn’t natively integrate with source control for the deployment process itself.
Databricks Resources Managed by using the Databricks Terraform Provider
Partial Solution: Excellent for provisioning Databricks infrastructure but less focused on the deployment of notebooks, workflows, and jobs themselves.
Additional Complexity for Deployment Logic: Would require integrating Terraform with another deployment tool/script for the actual deployment of Databricks artifacts, adding complexity.
Parameterization and Source Control: While Terraform configs can be parameterized and stored in Git, this option doesn’t directly address the deployment of Databricks content in a simple, repeatable manner.
C. Git Folders and Manual PromotionHighly Manual: Involves too much manual intervention for deployment, increasing the risk of human error.Lack of Automation: Does not provide a automated, repeatable process for deployments.Parameterization Challenges: Environment parameterization would require additional, potentially error-prone steps.ReferencesAzure Databricks Documentation: Declarative Automation BundlesDatabricks Blog: Implementing CI/CD for Databricks with Declarative Automation Bundles
C. Git Folders and Manual Promotion
Highly Manual: Involves too much manual intervention for deployment, increasing the risk of human error.
Lack of Automation: Does not provide a automated, repeatable process for deployments.
Parameterization Challenges: Environment parameterization would require additional, potentially error-prone steps.
s
Azure Databricks Documentation: Declarative Automation Bundles
Databricks Blog: Implementing CI/CD for Databricks with Declarative Automation Bundles
You have an Azure Databricks workspace that contains the objects shown in the following table.
Users often use the following words to refer to a sale: transaction, event, order, and invoice.
You need to create a knowledge store. The solution must ensure that when the users use any of the words in Space1, Genie queries the Sales1 table. Any other Genie spaces must remain unaffected.
To which object should you add the instructions?
A
Sales1
B
Schema1
C
Space1
D
Catalog1
E
Notebook1
Correct Answer:
C. Space1
Explanation:
Space1.
In Databricks AI/BI Genie, a knowledge store is used to add custom business terms, synonyms, and specific text instructions to guide the AI assistant in querying data correctly.
Space-level Scoping: All knowledge store configurations and text instructions are strictly scoped to a specific Genie Space (in this case, Space1).
Isolation: Modifying Space1 directly ensures that the specific synonyms (transaction, event, order, invoice) map
to the Sales1 table only within that workspace, leaving all other Genie spaces in the organization completely unaffected. Adding metadata or definitions at the Catalog, Schema, or Table level would apply globally across multiple spaces.
This is a case study. Case studies are not timed separately from other exam sections. You can use as much exam time as you would like to complete each case study. However, there might be additional case studies or other exam sections. Manage your time to ensure that you can complete all the exam sections in the time provided. Pay attention to the Exam Progress at the top of the screen so you have sufficient time to complete any exam sections that follow this case study.
To answer the case study questions, you will need to reference information that is provided in the case. Case studies and associated questions might contain exhibits or other resources that provide more information about the scenario described in the case. Information provided in an individual question does not apply to the other questions in the case study.
A Review Screen will appear at the end of this case study. From the Review Screen, you can review and change your answers before you move to the next exam section. After you leave this case study, you will NOT be able to return to it.
To start the case study -
To display the first question in this case study, select the “Next” button. To the left of the question, a menu provides links to information such as business requirements, the existing environment, and problem statements. Please read through all this information before answering any questions. When you are ready to answer a question, select the “Question” button to return to the question.
Overview -
Company Information -
Contoso, Inc. is a renewable energy provider that operates solar and wind farms across North America. Existing Environment -
Azure Environment -
Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.
Workspace1 contains all-purpose clusters for both development and production workloads. The company's Azure environment contains:
In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub
A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data
An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data Data Environment -
Contoso ingests the following operational and business data:
Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.
Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.
ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information.
Problem Statements -
The company’s existing analytics environment has several issues: Ingestion -
Telemetry pipelines fall behind during peak loads. Telemetry ingestion fails when schema drift occurs. Streaming pipelines reprocess events after a pipeline restarts. Compute -
Production and development workloads run on the same all-purpose clusters.
Production and development workloads do NOT support autoscaling or workload isolation. Governance -
The ERP data is duplicated across systems and development teams.
Naming conventions are inconsistent across development teams, regions, and products.
Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership. Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.
Pipeline operations -
Pipelines lack resiliency, alerting, and centralized scheduling. Requirements -
Planned Changes -
Contoso plans to implement the following changes:
Implement scalable data pipeline orchestration. Create a managed analytics catalog in Unity Catalog.
Implement a consistent approach to creating curated datasets.
Establish a centralized governance model across ingestion, cleansed, and curated layers. Grant data engineers access to the ERP tables by using minimal development effort. Adopt a compute strategy that isolates production workloads and supports autoscaling.
Adopt a slowly changing dimension (SCD) approach to address current data modeling issues. Technical Requirements -
Contoso identifies the following environment and compute requirements:
Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes.
Provide fast and consistent performance for business intelligence (BI) workloads. Prevent development activity from affecting production pipelines.
Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.
Contoso identifies the following data ingestion and processing requirements:
Auto-scale ingestion pipelines to handle bursty workloads. Handle schema drift for the maintenance and telemetry data. Ingest file-based telemetry data by using minimal operational effort.
Store all the ingested data in a format that supports incremental processing.
Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics. Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server. Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog.
Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data.
Contoso identifies the following data modeling and optimization requirements:
Build curated tables that standardize business logic.
Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.
Contoso identifies the following pipeline deployment and operation requirements: Orchestrate multi-step ingestion and transformation workflows.
Define a clear execution order and dependencies. Automatically retry failed steps and notify operators.
Schedule ingestion and transformation workloads consistently. Governance Requirements -
Contoso identifies the following governance requirements:
Centralize the metadata catalog.
Provide isolated development areas that follow standard naming conventions. Establish a consistent structure for organizing raw, cleansed, and curated data. Provide a read-only mechanism to reference the ERP data through a foreign catalog. Business Requirements -
Contoso identifies the following business requirements:
Improve ingestion reliability and reduce operational effort. Standardize data definitions across development teams.
You need to organize Unity Catalog. The solution must meet the governance requirements. What should you do?
A
Use a single shared schema for all the development teams and rely on table-level permissions for isolation.
B
Enable the development teams to create objects directly in the default catalog and schema.
C
Create a separate catalog for each development team and enable each team to choose its own schema names.
D
Create a shared development catalog, enforce a standardized naming convention, and assign each development team its own schema.
Correct Answer:
D. Create a shared development catalog, enforce a standardized naming convention, and assign each development team its own schema.
Explanation:
Technical Justification for Correct Answer D Why D is the Best Option:
Governance Requirements Alignment: Option D directly addresses Contoso's governance requirements by providing a "consistent structure for organizing raw, cleansed, and curated data" through a standardized naming convention.
Isolation with Efficiency: By assigning each development team its own schema within a shared development catalog, D balances isolation (preventing naming conflicts and ensuring team ownership) with efficiency (avoiding catalog proliferation).
Centralized Metadata Catalog: Although D suggests a "shared development catalog," the correct interpretation in the context of Unity Catalog's capabilities and the question's intent is that this shared catalog exists under the unified management of Unity Catalog, thereby centralizing metadata management as required.
Scalability and Management: This approach is scalable for managing multiple teams and easier to administer than managing multiple catalogs or lacking any structure.
Why Other Options are Less Suitable:
Single Shared Schema with Table-Level Permissions:
Inadequate Isolation: Relying solely on table-level permissions for isolation can become cumbersome to manage across large teams and does not enforce naming conventions.
Naming Conflicts: A single schema increases the risk of naming conflicts among development teams.
Development Teams Creating Objects in the Default Catalog and Schema:
Lack of Governance: This approach fails to establish any form of isolation or naming convention, leading to potential chaos in the metadata catalog.
Security and Organization Concerns: Insecure and disorganized, contradicting governance requirements.
Separate Catalog for Each Development Team:
Over-Complication: Managing multiple catalogs for each team would overly complicate the metadata landscape, contrary to the goal of centralizing governance.
Scalability Issues: Less scalable for large numbers of teams and does not inherently enforce a consistent naming structure across teams.
Correct Action for Organizing Unity Catalog (Answer D):Create a shared development catalog, enforce a standardized naming convention, and assign each development team its own schema.
s(Two valid and working documentation links for further reading on Unity Catalog and Azure Databricks governance)
Unity Catalog Documentation:
Azure Databricks Governance Best Practices:
Questions: 1-10 out of 162
Continue Full Practice..
GET ALL 162 QUESTIONS