ePrivacy and GPDR Cookie Consent by Cookie Consent Skip to main content

Processing Diagnostics

The Diagnostics page provides an overview of how customer data is currently being processed. It allows administrators to monitor the health of the data processing pipeline, understand whether data is being processed normally, and identify potential processing issues or errors.

Every event that enters the system, such as a website visit, purchase, or form submission, goes through several processing stages before it becomes available for segmentation, analytics, and activations. The Diagnostics page visualizes these stages and shows how many items are waiting to be processed, how quickly they are being processed, and whether the system is operating normally.

You can open the Diagnostics page in Administration → Diagnostics.


Data Processing Pipeline

image.png

At the top of the page, you will see the Data Processing Pipeline section. This section displays the three main stages that your customer data passes through as it is processed by the system.

These stages are displayed as cards arranged from left to right, showing the sequential flow of data through the platform. Data first passes through Identity Stitching, then Profile Recalculation, and finally Profile Synchronization.

Each card contains several metrics that help you understand the current state of that processing stage.

The values shown in these cards are updated regularly to reflect the current state of the system.


Identity Stitching

image.png

Identity stitching is the first stage of the data processing pipeline. In this step, incoming events are matched with existing customer profiles. The system uses identifiers such as email address, cookie ID, or phone number to determine which customer profile an event belongs to.

For example, if a user visits your website and later makes a purchase using the same email address, the system will connect those events to the same customer profile.

The Identity Stitching card displays two main metrics: the number of events waiting to be processed and the current processing speed.

  1. The Events in queue value shows how many incoming events are currently waiting to be stitched to customer profiles. This number increases when new events arrive faster than they can be processed. Short spikes are normal, especially during periods of high activity.
  2. The Processing speed value shows how many events are being processed per minute. This number indicates the current processing throughput of the identity stitching engine.

If the queue grows significantly while the processing speed remains low, this may indicate that the system is temporarily processing a backlog of events.


Profile Recalculation

image.png

After an event has been successfully linked to a customer profile, the system may need to update that profile’s attributes. This happens in the Profile Recalculation stage.

Attributes are calculated values stored on the customer profile. Examples include the total number of purchases, the date of the last activity, or the number of times a user visited your website.

Whenever an event changes data that affects these attributes, the system schedules a recalculation for the affected profile.

  1. The Profiles in queue value shows how many customer profiles are currently waiting for attribute recalculation. If many profiles are affected by incoming events, this number may temporarily increase.
  2. The Processing speed value shows how many profiles are recalculated per minute.

If this queue grows continuously while the processing speed remains low, it may indicate that attribute recalculations are taking longer than expected.


Profile Synchronization

image.png

The final stage of the pipeline is Profile Synchronization. Once profile attributes have been recalculated, the updated profiles must be synchronized with the search index used by the platform.

This synchronization ensures that the most recent profile data is available for segmentation, targeting, and activation workflows.

  1. The Profiles in queue value shows how many profiles are waiting to be synchronized.
  2. The Processing speed value shows how many profiles are being synchronized per minute.

When synchronization is complete, updated profile data becomes available for segmentation and campaign execution.


Status Indicators

image.png

Each pipeline card displays a status badge that helps you quickly understand whether processing in that stage is running normally or whether a backlog is starting to build up.

The status is based on the current queue size, which represents how many events or profiles are waiting to be processed in that stage. In general, smaller queues indicate that the system is keeping up with incoming data, while larger queues may indicate that processing is delayed.

The following statuses can appear on the Diagnostics page:

Status Queue size What it means
Processing is running well Up to 25,000 The system is keeping up with incoming data, and the queue is within a normal range. No action is usually needed.
Profiles are piling up 25,001–50,000 The queue is growing, and processing may be starting to slow down. This can happen temporarily during periods of high activity, large imports, or increased traffic. If the queue remains high for a longer period, data updates may be delayed.
System is overloaded More than 50,000 The queue is significantly backed up, and the system is no longer keeping up with incoming data. In this situation, customer data may take longer to become available in the platform. If this status continues, further investigation may be needed.

These status indicators are intended to provide a quick visual summary of system health. A temporary warning status does not always mean there is a problem, especially during short periods of heavy processing. However, if a queue remains large for a long time or continues to grow, it is recommended to review the Diagnostics page in more detail and check the Error Log for related issues.


Queue Update Information

Above the pipeline, you will see a message indicating when the queue size values will be updated.

For example, the system may display a message such as “Next queue size update in 4 minutes.” Queue size values are refreshed every 10 minutes, and this message shows how much time remains until the next update starts.
image.png

To correctly interpret the data on this page, it is important to understand that not all metrics are updated at the same time:

Metric How often it updates What it shows
Queue size Every 10 minutes The number of events or profiles currently waiting to be processed in each stage. This value remains the same until the next scheduled update.
Processing speed Continuously (approximately every 30 seconds) How fast the system is processing data over time. This value is calculated as an average over a longer period, which makes it stable and less affected by short spikes.

Because these two metrics are updated differently, their values may temporarily appear inconsistent. For example, the queue size may stay the same while the processing speed continues to change.

During the queue update process, the message may temporarily change to “Queue size update is running.”
image.png

This behavior is expected and does not indicate an issue. It simply reflects how and when each metric is refreshed.


Error Log

image.png

The lower section of the Diagnostics page contains the Error Log. This section lists processing errors that occurred while the system was handling events or recalculating profile attributes.

Errors can occur for various reasons, such as incorrect attribute definitions, unexpected data formats, or conflicts in identity resolution.

Each error entry contains a short description of the issue, the type of processing step where the error occurred, and the time when the error was recorded.


Filtering Errors

image.png

You can use the filtering controls above the error list to locate specific errors.

The search field allows you to search for errors by keyword or message content. To perform a search, you must enter at least three characters.

You can also filter errors by type, which allows you to view only errors related to identity stitching or attribute recalculation.

The time range filter allows you to limit results to specific time periods, such as the last hour, last 24 hours, or the last 7 days.

image.png


Viewing Error Details

image.png

Each error row can be expanded to view additional information about the issue.

When expanded, the system displays the detailed context of the error, including the related customer entity ID, attribute information when applicable, and the technical data used by the system during processing.

This information helps administrators understand the root cause of the issue and identify potential configuration problems.


When to Use the Diagnostics Page

The Diagnostics page is particularly useful when investigating delays in data updates or troubleshooting processing issues.

If segments are not updating as expected, campaign targeting appears outdated, or profile attributes seem incorrect, this page can help you determine whether the system is processing data normally or experiencing errors.

By monitoring queue sizes, processing speeds, and error logs, administrators can quickly identify potential issues and ensure that customer data is processed reliably.


Frequently Asked Questions

Question Answer
Why is the queue not zero? A non-zero queue is normal. Customer data is constantly entering the system, and processing happens continuously in the background. A small queue simply means items are waiting to be processed.
The queue size has not changed in a while. Is something wrong? Not necessarily. Queue size values are refreshed every 10 minutes. If the same number appears across multiple updates and processing speed is also very low or zero, it may be worth investigating further.
Why does the processing speed change so often? Processing speed is updated more frequently and represents an average over time. It may fluctuate depending on incoming data volume, system load, and the complexity of calculations. Small changes are expected.
Why is processing speed changing while the queue size stays the same? This is expected. Queue size updates every 10 minutes, while processing speed updates more frequently. Because of this, the two values may temporarily appear out of sync.
A card shows “System is overloaded”. What does that mean? This means the queue has grown beyond 50,000 items. The system is no longer keeping up with incoming data, and processing delays are likely.
What should I do if a card shows “System is overloaded”? This can happen during large imports or high traffic. If the status does not return to normal after some time, check the Error Log and contact your technical team or Meiro support.
Does a high queue always mean there is a problem? No. Temporary increases are normal, especially during busy periods. It becomes a concern if the queue continues to grow or stays high for a long time.
What does “Profiles are piling up” mean? This indicates that the queue is growing beyond the normal range, but processing is still ongoing. It is a warning that delays may occur if the trend continues.
Why can processing speed be zero even when there are items in the queue? Because processing speed is calculated as an average, it may temporarily show zero even when data is being processed. However, if it remains zero for a longer time while the queue is high, it may indicate an issue.
Why are my latest data changes not visible yet? Your data may still be in one of the processing stages. Updates become available only after passing through all stages of the pipeline.
When should I check the Error Log? Check the Error Log when queues remain high, processing seems delayed, or data is not updating as expected.
What kind of errors can appear in the Error Log? Errors may be related to identity stitching, attribute recalculation, or other processing steps. They help identify issues affecting data processing.