Category : Microsoft Azure Data Engineering | Sub Category : Practice Assessment for Exam DP-203 - Data Engineering on Microsoft Azure | By Prasad Bonam Last updated: 2023-09-10 03:09:57 Viewed : 296
You create an Azure Stream Analytics job. You run the job for five hours.
You review the logs and notice multiple instances of the following message.
{"message Time":"2019-02-04 17:11:52Z","error":null, "message":"First Occurred: 02/04/2019 17:11:48 | Resource Name: ASAjob | Message: Source ASAjob had 24 data errors of kind LateInputEvent between processing times 2019-02-04T17:10:49.7250696Z and 2019-02-04T17:11:48.7563961Z. Input event with application timestamp `2019-02-04T17:05:51.6050000` and arrival time `2019-02-04T17:10:44.3090000` was sent later than configured tolerance.","type":"DiagnosticMessage","correlation ID":"49efa148-4asd-4fe0-869d-a40ba4d7ef3b"}
You need to ensure that these events are not dropped.
What should you do?
Increasing the tolerance for late arrivals ensures that late arrivals are not dropped.
The error is about late arrivals, not out-of-order events.
Increasing the number of SUs to 12 will not change how late arrivals are handled.
Decreasing the number of SUs to 3 will not change how late arrivals are handled.
Configuring event ordering policies for Azure Stream Analytics | Microsoft Learn
Get started with Azure Stream Analytics - Training | Microsoft Learn
The error message you are seeing in the logs indicates that you have late arriving events in your Azure Stream Analytics (ASA) job, and these events are being marked with the "LateInputEvent" error. The error message also mentions that the input event with application timestamp is later than the configured tolerance.
To ensure that these late arriving events are not dropped and are processed correctly, you should adjust the late arrival tolerance settings in your ASA job. The late arrival tolerance defines how long ASA should wait for late events before considering them as out-of-order or dropping them.
Here is what you should do:
Open your Azure Stream Analytics job in the Azure portal.
In the job settings, go to the "Inputs" section, which represents the input source you are reading data from (e.g., Event Hub, IoT Hub, etc.).
In the input configuration, find the "Event ordering" or "Late arrival" settings. The exact name and location of these settings may vary depending on your input source.
Adjust the "Late arrival tolerance" setting to a value that accommodates the expected lateness of your events. You should set this value to a duration that allows your late arriving events to be included in the processing window. For example, if you consistently expect events to arrive up to 5 minutes late, set the tolerance accordingly.
Save the configuration changes and restart your ASA job to apply the new settings.
By increasing the late arrival tolerance, you instruct ASA to wait longer for late arriving events before considering them as errors. This should ensure that events arriving within the specified tolerance period are not dropped and are included in the processing. Be cautious not to set the tolerance too high, as it can affect the overall processing latency of your job. Find a balance that suits your specific use case and expected lateness of events.