AWS Certified Data Engineer - Associate (DEA-C01) Dumps July 2026
PDF + Test Engine
$139 $97
Test Engine
$119 $83
PDF
$99 $69
Here are Amazon Data-Engineer-Associate PDF available features:
302 questions with answers
Updation Date : 16 Jul, 2026
1 day study required to pass exam
100% Passing Assurance
100% Money Back Guarantee
Free 3 Months Updates
Amazon Data-Engineer-Associate Sample Questions
Question # 41
A data engineer must orchestrate a data pipeline that consists of one AWS Lambda function and one AWS Glue job. The solution must integrate with AWS services. Which solution will meet these requirements with the LEAST management overhead?
A. Use an AWS Step Functions workflow that includes a state machine. Configure the statemachine to run the Lambda function and then the AWS Glue job. B. Use an Apache Airflow workflow that is deployed on an Amazon EC2 instance. Define adirected acyclic graph (DAG) in which the first task is to call the Lambda function and thesecond task is to call the AWS Glue job. C. Use an AWS Glue workflow to run the Lambda function and then the AWS Glue job. D. Use an Apache Airflow workflow that is deployed on Amazon Elastic Kubernetes Service(Amazon EKS). Define a directed acyclic graph (DAG) in which the first task is to call theLambda function and the second task is to call the AWS Glue job.
Answer: A Explanation: AWS Step Functions is a service that allows you to coordinate multiple AWSservices into serverless workflows. You can use Step Functions to create state machinesthat define the sequence and logic of the tasks in your workflow. Step Functions supportsvarious types of tasks, such as Lambda functions, AWS Glue jobs, Amazon EMR clusters,Amazon ECS tasks, etc. You can use Step Functions to monitor and troubleshoot yourworkflows, as well as to handle errors and retries.Using an AWS Step Functions workflow that includes a state machine to run the Lambdafunction and then the AWS Glue job will meet the requirements with the least managementoverhead, as it leverages the serverless and managed capabilities of Step Functions. Youdo not need to write any code to orchestrate the tasks in your workflow, as you can use theStep Functions console or the AWS Serverless Application Model (AWS SAM) to define and deploy your state machine. You also do not need to provision or manage any serversor clusters, as Step Functions scales automatically based on the demand.The other options are not as efficient as using an AWS Step Functions workflow. Using anApache Airflow workflow that is deployed on an Amazon EC2 instance or on AmazonElastic Kubernetes Service (Amazon EKS) will require more management overhead, asyou will need to provision, configure, and maintain the EC2 instance or the EKS cluster, aswell as the Airflow components. You will also need to write and maintain the Airflow DAGsto orchestrate the tasks in your workflow. Using an AWS Glue workflow to run the Lambdafunction and then the AWS Glue job will not work, as AWS Glue workflows only supportAWS Glue jobs and crawlers as tasks, not Lambda functions. References:AWS Step FunctionsAWS GlueAWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide,Chapter 6: Data Integration and Transformation, Section 6.3: AWS Step Functions
Question # 42
A security company stores IoT data that is in JSON format in an Amazon S3 bucket. The data structure can change when the company upgrades the IoT devices. The company wants to create a data catalog that includes the IoT data. The company's analytics department will use the data catalog to index the data. Which solution will meet these requirements MOST cost-effectively?
A. Create an AWS Glue Data Catalog. Configure an AWS Glue Schema Registry. Create anew AWS Glue workload to orchestrate the ingestion of the data that the analyticsdepartment will use into Amazon Redshift Serverless. B. Create an Amazon Redshift provisioned cluster. Create an Amazon Redshift Spectrumdatabase for the analytics department to explore the data that is in Amazon S3. CreateRedshift stored procedures to load the data into Amazon Redshift. C. Create an Amazon Athena workgroup. Explore the data that is in Amazon S3 by usingApache Spark through Athena. Provide the Athena workgroup schema and tables to theanalytics department. D. Create an AWS Glue Data Catalog. Configure an AWS Glue Schema Registry. CreateAWS Lambda user defined functions (UDFs) by using the Amazon Redshift Data API.Create an AWS Step Functions job to orchestrate the ingestion of the data that theanalytics department will use into Amazon Redshift Serverless.
Answer: C Explanation: The best solution to meet the requirements of creating a data catalog that includes the IoT data, and allowing the analytics department to index the data, most costeffectively,is to create an Amazon Athena workgroup, explore the data that is in AmazonS3 by using Apache Spark through Athena, and provide the Athena workgroup schemaand tables to the analytics department.Amazon Athena is a serverless, interactive query service that makes it easy to analyzedata directly in Amazon S3 using standard SQL or Python1. Amazon Athena also supportsApache Spark, an open-source distributed processing framework that can run large-scaledata analytics applications across clusters of servers2. You can use Athena to run Sparkcode on data in Amazon S3 without having to set up, manage, or scale anyinfrastructure. You can also use Athena to create and manage external tables that point toyour data in Amazon S3, and store them in an external data catalog, such as AWS GlueData Catalog, Amazon Athena Data Catalog, or your own Apache Hive metastore3. Youcan create Athena workgroups to separate query execution and resource allocation basedon different criteria, such as users, teams, or applications4. You can share the schemasand tables in your Athena workgroup with other users or applications, such as AmazonQuickSight, for data visualization and analysis5. Using Athena and Spark to create a data catalog and explore the IoT data in Amazon S3 isthe most cost-effective solution, as you pay only for the queries you run or the compute youuse, and you pay nothing when the service is idle1. You also save on the operationaloverhead and complexity of managing data warehouse infrastructure, as Athena and Sparkare serverless and scalable. You can also benefit from the flexibility and performance ofAthena and Spark, as they support various data formats, including JSON, and can handleschema changes and complex queries efficiently.Option A is not the best solution, as creating an AWS Glue Data Catalog, configuring anAWS Glue Schema Registry, creating a new AWS Glue workload to orchestratetheingestion of the data that the analytics department will use into Amazon RedshiftServerless, would incur more costs and complexity than using Athena and Spark. AWSGlue Data Catalog is a persistent metadata store that contains table definitions, jobdefinitions, and other control information to help you manage your AWS Gluecomponents6. AWS Glue Schema Registry is a service that allows you to centrally storeand manage the schemas of your streaming data in AWS Glue Data Catalog7. AWS Glueis a serverless data integration service that makes it easy to prepare, clean, enrich, andmove data between data stores8. Amazon Redshift Serverless is a feature of AmazonRedshift, a fully managed data warehouse service, that allows you to run and scaleanalytics without having to manage data warehouse infrastructure9. While these servicesare powerful and useful for many data engineering scenarios, they are not necessary orcost-effective for creating a data catalog and indexing the IoT data in Amazon S3. AWSGlue Data Catalog and Schema Registry charge you based on the number of objectsstored and the number of requests made67. AWS Glue charges you based on the computetime and the data processed by your ETL jobs8. Amazon Redshift Serverless charges youbased on the amount of data scanned by your queries and the compute time used by yourworkloads9. These costs can add up quickly, especially if you have large volumes of IoTdata and frequent schema changes. Moreover, using AWS Glue and Amazon RedshiftServerless would introduce additional latency and complexity, as you would have to ingestthe data from Amazon S3 to Amazon Redshift Serverless, and then query it from there,instead of querying it directly from Amazon S3 using Athena and Spark.Option B is not the best solution, as creating an Amazon Redshift provisioned cluster,creating an Amazon Redshift Spectrum database for the analytics department to explorethe data that is in Amazon S3, and creating Redshift stored procedures to load the datainto Amazon Redshift, would incur more costs and complexity than using Athena andSpark. Amazon Redshift provisioned clusters are clusters that you create and manage byspecifying the number and type of nodes, and the amount of storage and computecapacity10. Amazon Redshift Spectrum is a feature of Amazon Redshift that allows you toquery and join data across your data warehouse and your data lake using standardSQL11. Redshift stored procedures are SQL statements that you can define and store inAmazon Redshift, and then call them by using the CALL command12. While these features are powerful and useful for many data warehousing scenarios, they are not necessary orcost-effective for creating a data catalog and indexing the IoT data in Amazon S3. AmazonRedshift provisioned clusters charge you based on the node type, the number of nodes,and the duration of the cluster10. Amazon Redshift Spectrum charges you based on theamount of data scanned by your queries11. These costs can add up quickly, especially ifyou have large volumes of IoT data and frequent schema changes. Moreover, usingAmazon Redshift provisioned clusters and Spectrum would introduce additional latencyand complexity, as you would have to provision andmanage the cluster, create an externalschema and database for the data in Amazon S3, and load the data into the cluster usingstored procedures, instead of querying it directly from Amazon S3 using Athena and Spark.Option D is not the best solution, as creating an AWS Glue Data Catalog, configuring anAWS Glue Schema Registry, creating AWS Lambda user defined functions (UDFs) byusing the Amazon Redshift Data API, and creating an AWS Step Functions job toorchestrate the ingestion of the data that the analytics department will use into AmazonRedshift Serverless, would incur more costs and complexity than using Athena andSpark. AWS Lambda is a serverless compute service that lets you run code withoutprovisioning or managing servers13. AWS Lambda UDFs are Lambda functions that youcan invoke from within an Amazon Redshift query. Amazon Redshift Data API is a servicethat allows you to run SQL statements on Amazon Redshift clusters using HTTP requests,without needing a persistent connection. AWS Step Functions is a service that lets youcoordinate multiple AWS services into serverless workflows. While these services arepowerful and useful for many data engineering scenarios, they are not necessary or costeffectivefor creating a data catalog and indexing the IoT data in Amazon S3. AWS GlueData Catalog and Schema Registry charge you based on the number of objects stored andthe number of requests made67. AWS Lambda charges you based on the number ofrequests and the duration of your functions13. Amazon Redshift Serverless charges youbased on the amount of data scanned by your queries and the compute time used by yourworkloads9. AWS Step Functions charges you based on the number of state transitions inyour workflows. These costs can add up quickly, especially if you have large volumes ofIoT data and frequent schema changes. Moreover, using AWS Glue, AWS Lambda,Amazon Redshift Data API, and AWS Step Functions would introduce additional latencyand complexity, as you would have to create and invoke Lambda functions to ingest thedata from Amazon S3 to Amazon Redshift Serverless using the Data API, and coordinatethe ingestion process using Step Functions, instead of querying it directly from Amazon S3using Athena and Spark. References:What is Amazon Athena?Apache Spark on Amazon AthenaCreating tables, updating the schema, and adding new partitions in the DataCatalog from AWS Glue ETL jobsManaging Athena workgroupsUsing Amazon QuickSight to visualize data in Amazon AthenaAWS Glue Data Catalog AWS Glue Schema RegistryWhat is AWS Glue?Amazon Redshift ServerlessAmazon Redshift provisioned clustersQuerying external data using Amazon Redshift SpectrumUsing stored procedures in Amazon RedshiftWhat is AWS Lambda?[Creating and using AWS Lambda UDFs][Using the Amazon Redshift Data API][What is AWS Step Functions?]AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide
Question # 43
A company uses Amazon Athena to run SQL queries for extract, transform, and load (ETL) tasks by using Create Table As Select (CTAS). The company must use Apache Spark instead of SQL to generate analytics. Which solution will give the company the ability to use Spark to access Athena?
A. Athena query settings B. Athena workgroup C. Athena data source D. Athena query editor
Answer: C Explanation: Athena data source is a solution that allows you to use Spark to accessAthena by using the Athena JDBC driver and the Spark SQL interface. You can use theAthena data source to create Spark DataFrames from Athena tables, run SQL queries onthe DataFrames, and write the results back to Athena. The Athena data source supportsvarious data formats, such as CSV, JSON, ORC, and Parquet, and also supportspartitioned and bucketed tables. The Athena data source is a cost-effective and scalableway to use Spark to access Athena, as it does not require any additional infrastructure orservices, and you only pay for the data scanned by Athena.The other options are not solutions that give the company the ability to use Spark to accessAthena. Option A, Athena query settings, is a feature that allows you to configure variousparameters for your Athena queries, such as the output location, the encryption settings,the query timeout, and the workgroup. Option B, Athena workgroup, is a feature that allowsyou to isolate and manage your Athena queries and resources, such as the query history,the query notifications, the query concurrency, and the query cost. Option D, Athena queryeditor, is a feature that allows you to write and run SQL queries on Athena using the webconsole or the API. None of these options enable you to use Spark instead of SQL togenerate analytics on Athena. References:Using Apache Spark in Amazon AthenaAthena JDBC DriverSpark SQLAthena query settings[Athena workgroups][Athena query editor]
Question # 44
A company needs to set up a data catalog and metadata management for data sources that run in the AWS Cloud. The company will use the data catalog to maintain the metadata of all the objects that are in a set of data stores. The data stores include structured sources such as Amazon RDS and Amazon Redshift. The data stores also include semistructured sources such as JSON files and .xml files that are stored in Amazon S3. The company needs a solution that will update the data catalog on a regular basis. The solution also must detect changes to the source metadata. Which solution will meet these requirements with the LEAST operational overhead?
A. Use Amazon Aurora as the data catalog. Create AWS Lambda functions that willconnect to the data catalog. Configure the Lambda functions to gather the metadatainformation from multiple sources and to update the Aurora data catalog. Schedule theLambda functions to run periodically. B. Use the AWS Glue Data Catalog as the central metadata repository. Use AWS Gluecrawlers to connect to multiple data stores and to update the Data Catalog with metadatachanges. Schedule the crawlers to run periodically to update the metadata catalog. C. Use Amazon DynamoDB as the data catalog. Create AWS Lambda functions that willconnect to the data catalog. Configure the Lambda functions to gather the metadatainformation from multiple sources and to update the DynamoDB data catalog. Schedule theLambda functions to run periodically. D. Use the AWS Glue Data Catalog as the central metadata repository. Extract the schemafor Amazon RDS and Amazon Redshift sources, and build the Data Catalog. Use AWSGlue crawlers for data that is in Amazon S3 to infer the schema and to automaticallyupdate the Data Catalog.
Answer: B Explanation: This solution will meet the requirements with the least operational overheadbecause it uses the AWS Glue Data Catalog as the central metadata repository for datasources that run in the AWS Cloud. The AWS Glue Data Catalog is a fully managedservice that provides a unified view of your data assets across AWS and on-premises data sources. It stores the metadata of your data in tables, partitions, and columns, and enablesyou to access and query your data using various AWS services, such as Amazon Athena,Amazon EMR, and Amazon Redshift Spectrum. You can use AWS Glue crawlers toconnect to multiple data stores, such as Amazon RDS, Amazon Redshift, and Amazon S3,and to update the Data Catalog with metadata changes. AWS Glue crawlers canautomatically discover the schema and partition structure of your data, and create orupdate the corresponding tables in the Data Catalog. You can schedule the crawlers to runperiodically to update the metadata catalog, and configure them to detect changes to thesource metadata, such as new columns, tables, or partitions12.The other options are not optimal for the following reasons:A. Use Amazon Aurora as the data catalog. Create AWS Lambda functions thatwill connect to the data catalog. Configure the Lambda functions to gather themetadata information from multiple sources and to update the Aurora data catalog.Schedule the Lambda functions to run periodically. This option is notrecommended, as it would require more operational overhead to create andmanage an Amazon Aurora database as the data catalog, and to write andmaintain AWS Lambda functions to gather and update the metadata informationfrom multiple sources. Moreover, this option would not leverage the benefits of theAWS Glue Data Catalog, such as data cataloging, data transformation, and datagovernance.C. Use Amazon DynamoDB as the data catalog. Create AWS Lambda functionsthat will connect to the data catalog. Configure the Lambda functions to gather themetadata information from multiple sources and to update the DynamoDB datacatalog. Schedule the Lambda functions to run periodically. This option is also notrecommended, as it would require more operational overhead to create andmanage an Amazon DynamoDB table as the data catalog, and to write andmaintain AWS Lambda functions to gather and update the metadata informationfrom multiple sources. Moreover, this option would not leverage the benefits of theAWS Glue Data Catalog, such as data cataloging, data transformation, and datagovernance.D. Use the AWS Glue Data Catalog as the central metadata repository. Extract theschema for Amazon RDS and Amazon Redshift sources, and build the DataCatalog. Use AWS Glue crawlers for data that is in Amazon S3 to infer the schemaand to automatically update the Data Catalog. This option is not optimal, as itwould require more manual effort to extract the schema for Amazon RDS andAmazon Redshift sources, and to build the Data Catalog. This option would nottake advantage of the AWS Glue crawlers’ ability to automatically discover theschema and partition structure of your data from various data sources, and tocreate or update the corresponding tables in the Data Catalog.References:1: AWS Glue Data Catalog2: AWS Glue Crawlers: Amazon Aurora: AWS Lambda: Amazon DynamoDB
Question # 45
A manufacturing company wants to collect data from sensors. A data engineer needs to implement a solution that ingests sensor data in near real time. The solution must store the data to a persistent data store. The solution must store the data in nested JSON format. The company must have the ability to query from the data store with a latency of less than 10 milliseconds. Which solution will meet these requirements with the LEAST operational overhead?
A. Use a self-hosted Apache Kafka cluster to capture the sensor data. Store the data inAmazon S3 for querying. B. Use AWS Lambda to process the sensor data. Store the data in Amazon S3 forquerying. C. Use Amazon Kinesis Data Streams to capture the sensor data. Store the data inAmazon DynamoDB for querying. D. Use Amazon Simple Queue Service(Amazon SQS) to buffer incomingsensor data. UseAWS Glue to store thedata in Amazon RDS for querying.
Answer: C Explanation: Amazon Kinesis Data Streams is a service that enables you to collect, process, and analyze streaming data in real time. You can use Kinesis Data Streams tocapture sensor data from various sources, such as IoT devices, web applications, or mobileapps. You can create data streams that can scale up to handle any amount of data fromthousands of producers. You can also use the Kinesis Client Library (KCL) or the KinesisData Streams API to write applications that process and analyze the data in the streams1.Amazon DynamoDB is a fully managed NoSQL database service that provides fast andpredictable performance with seamless scalability. You can use DynamoDB to store thesensor data in nested JSON format, as DynamoDB supports document data types, such aslists and maps. You can also use DynamoDB to query the data with a latency of less than10 milliseconds, as DynamoDB offers single-digit millisecond performance for any scale ofdata. You can use the DynamoDB API or the AWS SDKs to perform queries on the data,such as using key-value lookups, scans, or queries2.The solution that meets the requirements with the least operational overhead is to useAmazon Kinesis Data Streams to capture the sensor data and store the data in AmazonDynamoDB for querying. This solution has the following advantages:It does not require you to provision, manage, or scale any servers, clusters, orqueues, as Kinesis Data Streams and DynamoDB are fully managed services thathandle all the infrastructure for you. This reduces the operational complexity andcost of running your solution.It allows you to ingest sensor data in near real time, as Kinesis Data Streams cancapture data records as they are produced and deliver them to your applicationswithin seconds. You can also use Kinesis Data Firehose to load the data from thestreams to DynamoDB automatically and continuously3.It allows you to store the data in nested JSON format, as DynamoDB supportsdocument data types, such as lists and maps. You can also use DynamoDB Streams to capturechanges in the data and trigger actions, such as sendingnotifications or updating other databases.It allows you to query the data with a latency of less than 10 milliseconds, asDynamoDB offers single-digit millisecond performance for any scale of data. Youcan also use DynamoDB Accelerator (DAX) to improve the read performance bycaching frequently accessed data.Option A is incorrect because it suggests using a self-hosted Apache Kafka cluster tocapture the sensor data and store the data in Amazon S3 for querying. This solution hasthe following disadvantages:It requires you to provision, manage, and scale your own Kafka cluster, either onEC2 instances or on-premises servers. This increases the operational complexityand cost of running your solution.It does not allow you to query the data with a latency of less than 10 milliseconds,as Amazon S3 is an object storage service that is not optimized for low-latencyqueries. You need to use another service, such as Amazon Athena or AmazonRedshift Spectrum, to query the data in S3, which may incur additional costs andlatency.Option B is incorrect because it suggests using AWS Lambda to process the sensor dataand store the data in Amazon S3 for querying. This solution has the followingdisadvantages:It does not allow you to ingest sensor data in near real time, as Lambda is aserverless compute service that runs code in response to events. You need to useanother service, such as API Gateway or Kinesis Data Streams, to trigger Lambdafunctions with sensor data, which may add extra latency and complexity to yoursolution.It does not allow you to query the data with a latency of less than 10 milliseconds,as Amazon S3 is an object storage service that is not optimized for low-latencyqueries. You need to use another service, such as Amazon Athena or AmazonRedshift Spectrum, to query the data in S3, which may incur additional costs andlatency.Option D is incorrect because it suggests using Amazon Simple Queue Service (AmazonSQS) to buffer incoming sensor data and use AWS Glue to store the data in Amazon RDSfor querying. This solution has the following disadvantages:It does not allow you to ingest sensor data in near real time, as Amazon SQS is amessage queue service that delivers messages in a best-effort manner. You needto use another service, such as Lambda or EC2, to poll the messages from thequeue and process them, which may add extra latency and complexity to yoursolution.It does not allow you to store the data in nested JSON format, as Amazon RDS isa relational database service that supports structured data types, such as tablesand columns. You need to use another service, such as AWS Glue, to transformthe data from JSON to relational format, which may add extra cost and overheadto your solution.References:1: Amazon Kinesis Data Streams - Features2: Amazon DynamoDB - Features3: Loading Streaming Data into Amazon DynamoDB - Amazon Kinesis DataFirehose[4]: Capturing Table Activity with DynamoDB Streams - Amazon DynamoDB [5]: Amazon DynamoDB Accelerator (DAX) - Features[6]: Amazon S3 - Features[7]: AWS Lambda - Features[8]: Amazon Simple Queue Service - Features[9]: Amazon Relational Database Service - Features[10]: Working with JSON in Amazon RDS - Amazon Relational Database Service[11]: AWS Glue - Features
Question # 46
A company receives .csv files that contain physical address data. The data is in columns that have the following names: Door_No, Street_Name, City, and Zip_Code. The company wants to create a single column to store these values in the following format:
Which solution will meet this requirement with the LEAST coding effort?
A. Use AWS Glue DataBrew to read the files. Use the NEST TO ARRAY transformation tocreate the new column. B. Use AWS Glue DataBrew to read the files. Use the NEST TO MAP transformation tocreate the new column. C. Use AWS Glue DataBrew to read the files. Use the PIVOT transformation to create thenew column. D. Write a Lambda function in Python to read the files. Use the Python data dictionary typeto create the new column.
Answer: B Explanation: The NEST TO MAP transformation allows you to combine multiple columns into a single column that contains a JSON object with key-value pairs. This is the easiestway to achieve the desired format for the physical address data, as you can simply selectthe columns to nest and specify the keys for each column. The NEST TO ARRAYtransformation creates a single column that contains an array of values, which is notthesame as the JSON object format. The PIVOT transformation reshapes the data bycreating new columns from unique values in a selected column, which is not applicable forthis use case. Writing a Lambda function in Python requires more coding effort than usingAWS Glue DataBrew, which provides a visual and interactive interface for datatransformations. References:7 most common data preparation transformations in AWS GlueDataBrew (Section: Nesting and unnesting columns)NEST TO MAP - AWS Glue DataBrew (Section: Syntax)
Question # 47
A company is planning to upgrade its Amazon Elastic Block Store (Amazon EBS) General Purpose SSD storage from gp2 to gp3. The company wants to prevent any interruptions in its Amazon EC2 instances that will cause data loss during the migration to the upgraded storage. Which solution will meet these requirements with the LEAST operational overhead?
A. Create snapshots of the gp2 volumes. Create new gp3 volumes from the snapshots.Attach the new gp3 volumes to the EC2 instances. B. Create new gp3 volumes. Gradually transfer the data to the new gp3 volumes. When thetransfer is complete, mount the new gp3 volumes to the EC2 instances to replace the gp2volumes. C. Change the volume type of the existing gp2 volumes to gp3. Enter new values forvolume size, IOPS, and throughput. D. Use AWS DataSync to create new gp3 volumes. Transfer the data from the original gp2volumes to the new gp3 volumes.
Answer: C Explanation: Changing the volume type of the existing gp2 volumes to gp3 is the easiestand fastest way to migrate to the new storage type without any downtime or data loss. Youcan use the AWS Management Console, the AWS CLI, or the Amazon EC2 API to modifythe volume type, size, IOPS, and throughput of your gp2 volumes. The modification takeseffect immediately, and you can monitor the progress of the modification usingCloudWatch. The other options are either more complex or require additional steps, suchas creating snapshots, transferring data, or attaching new volumes, which can increase theoperational overhead and the risk of errors. References:Migrating Amazon EBS volumes from gp2 to gp3 and save up to 20% oncosts (Section: How to migrate from gp2 to gp3)Switching from gp2 Volumes to gp3 Volumes to Lower AWS EBS Costs (Section:How to Switch from GP2 Volumes to GP3 Volumes)Modifying the volume type, IOPS, or size of an EBS volume - Amazon ElasticCompute Cloud (Section: Modifying the volume type)
Question # 48
A company created an extract, transform, and load (ETL) data pipeline in AWS Glue. A data engineer must crawl a table that is in Microsoft SQL Server. The data engineer needs to extract, transform, and load the output of the crawl to an Amazon S3 bucket. The data engineer also must orchestrate the data pipeline. Which AWS service or feature will meet these requirements MOST cost-effectively?
A. AWS Step Functions B. AWS Glue workflows C. AWS Glue Studio D. Amazon Managed Workflows for Apache Airflow (Amazon MWAA)
Answer: B Explanation: AWS Glue workflows are a cost-effective way to orchestrate complex ETL jobs that involve multiple crawlers, jobs, and triggers. AWS Glue workflows allow you tovisually monitor the progress and dependencies of your ETL tasks, and automaticallyhandle errors and retries. AWS Glue workflows also integrate with other AWS services,such as Amazon S3, Amazon Redshift, and AWS Lambda, among others, enabling you toleverage these services for your data processing workflows. AWS Glue workflows areserverless, meaning you only pay for the resources you use, and you don’t have to manageany infrastructure.AWS Step Functions, AWS Glue Studio, and Amazon MWAA are also possible options fororchestrating ETL pipelines, but they have some drawbacks compared to AWS Glueworkflows. AWS Step Functions is a serverless function orchestrator that can handledifferent types of data processing, such as real-time, batch, and stream processing.However, AWS Step Functions requires you to write code to define your state machines,which can be complex and error-prone. AWS Step Functions also charges you for everystate transition, which can add up quickly for large-scale ETL pipelines.AWS Glue Studio is a graphical interface that allows you to create and run AWS Glue ETLjobs without writing code. AWS Glue Studio simplifies the process of building, debugging,and monitoring your ETL jobs, and provides a range of pre-built transformations andconnectors. However, AWS Glue Studio does not support workflows, meaning you cannotorchestrate multiple ETL jobs or crawlers with dependencies and triggers. AWS GlueStudio also does not support streaming data sources or targets, which limits its use casesfor real-time data processing.Amazon MWAA is a fully managed service that makes it easy to run open-source versionsof Apache Airflow on AWS and build workflows to run your ETL jobs and data pipelines.Amazon MWAA provides a familiar and flexible environment for data engineers who arefamiliar with Apache Airflow, and integrates with a range of AWS services such as AmazonEMR, AWS Glue, and AWS Step Functions. However, Amazon MWAA is not serverless,meaning you have to provision and pay for the resources you need, regardless of yourusage. Amazon MWAA also requires you to write code to define your DAGs, which can bechallenging and time-consuming for complex ETL pipelines. References:AWS Glue WorkflowsAWS Step FunctionsAWS Glue Studio Amazon MWAAAWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide
Question # 49
A company uses Amazon S3 to store semi-structured data in a transactional data lake. Some of the data files are small, but other data files are tens of terabytes. A data engineer must perform a change data capture (CDC) operation to identify changed data from the data source. The data source sends a full snapshot as a JSON file every day and ingests the changed data into the data lake. Which solution will capture the changed data MOST cost-effectively?
A. Create an AWS Lambda function to identify the changes between the previous data andthe current data. Configure the Lambda function to ingest the changes into the data lake. B. Ingest the data into Amazon RDS for MySQL. Use AWS Database Migration Service (AWS DMS) to write the changed data to the data lake. C. Use an open source data lake format to merge the data source with the S3 data lake toinsert the new data and update the existing data. D. Ingest the data into an Amazon Aurora MySQL DB instance that runs Aurora Serverless.Use AWS Database Migration Service (AWS DMS) to write the changed data to the datalake.
Answer: C Explanation: An open source data lake format, such as Apache Parquet, Apache ORC, or Delta Lake, isa cost-effective way to perform a change data capture (CDC) operation on semi-structureddata stored in Amazon S3. An open source data lake format allows you to query datadirectly from S3 using standard SQL, without the need to move or copy data to anotherservice. An open source data lake format also supports schema evolution, meaning it canhandle changes in the data structure over time. An open source data lake format alsosupports upserts, meaning it can insert new data and update existing data in the sameoperation, using a merge command. This way, you can efficiently capture the changes fromthe data source and apply them to the S3 data lake, without duplicating or losing any data.The other options are not as cost-effective as using an open source data lake format, asthey involve additional steps or costs. Option A requires you to create and maintain anAWS Lambda function, which can be complex and error-prone. AWS Lambda also hassome limits on the execution time, memory, and concurrency, which can affect theperformance and reliability of the CDC operation. Option B and D require you to ingest thedata into a relational database service, such as Amazon RDS or Amazon Aurora, whichcan be expensive and unnecessary for semi-structured data. AWS Database MigrationService (AWS DMS) can write the changed data to the data lake, but it alsocharges you forthe data replication and transfer. Additionally, AWS DMS does not support JSON as asource data type, so you would need to convert the data to a supported format before usingAWS DMS. References:What is a data lake?Choosing a data format for your data lakeUsing the MERGE INTO command in Delta Lake[AWS Lambda quotas][AWS Database Migration Service quotas]
Question # 50
A data engineer must manage the ingestion of real-time streaming data into AWS. The data engineer wants to perform real-time analytics on the incoming streaming data by using time-based aggregations over a window of up to 30 minutes. The data engineer needs a solution that is highly fault tolerant. Which solution will meet these requirements with the LEAST operational overhead?
A. Use an AWS Lambda function that includes both the business and the analytics logic toperform time-based aggregations over a window of up to 30 minutes for the data in Amazon Kinesis Data Streams. B. Use Amazon Managed Service for Apache Flink (previously known as Amazon KinesisData Analytics) to analyze the data that might occasionally contain duplicates by usingmultiple types of aggregations. C. Use an AWS Lambda function that includes both the business and the analytics logic toperform aggregations for a tumbling window of up to 30 minutes, based on the eventtimestamp. D. Use Amazon Managed Service for Apache Flink (previously known as Amazon KinesisData Analytics) to analyze the data by using multiple types of aggregations to perform timebasedanalytics over a window of up to 30 minutes.
Answer: A Explanation: This solution meets the requirements of managing the ingestion of real-time streaming data into AWS and performing real-time analytics on the incoming streamingdata with the least operational overhead. Amazon Managed Service for Apache Flink is afully managed service that allows you to run Apache Flink applications without having tomanage any infrastructure or clusters. Apache Flink is a framework for stateful streamprocessing that supports various types of aggregations, such as tumbling, sliding, andsession windows, over streaming data. By using Amazon Managed Service for ApacheFlink, you can easily connect to Amazon Kinesis Data Streams as the source and sink ofyour streaming data, and perform time-based analytics over a window of up to 30 minutes.This solution is also highly fault tolerant, as Amazon Managed Service for Apache Flinkautomatically scales, monitors, and restarts your Flink applications in case of failures.References:Amazon Managed Service for Apache FlinkApache FlinkWindow Aggregations in Flink
