Data Quality - The Foundation of Effective AI

Why Data Quality Is Critical for AI Success

Even the best artificial intelligence algorithm, designed by the most brilliant experts, won't help if you feed it garbage data. The "garbage in, garbage out" (GIGO) principle is particularly true in industry, where data often comes from many different systems, has different formats, units, time zones, and contains various types of errors and irregularities.

According to Gartner research, poor data quality costs companies an average of 12-15% of revenue. For AI projects, these costs can be even higher - flawed models lead to flawed decisions that can cost hundreds of thousands or even millions. Imagine a situation where an AI system misinterprets data and generates false alarms, leading to unnecessary downtime, or worse, fails to detect real problems, resulting in costly failures.

Data quality is not just a technical issue - it's the foundation of the entire AI project. Without good data, even the most advanced algorithms won't be able to learn correct patterns and generate reliable predictions.

Common Problems with Industrial Data

1. Missing Values

Sensors stop working, connections break, systems restart. Gaps in data are an everyday reality:

• Random gaps - individual measurements lost sporadically
• Continuous gaps - entire periods without data (e.g., sensor failure)
• Structural gaps - data was never collected

2. Outliers

Abnormal values that distort analysis:

• Sensor errors - spikes, zeros, saturation values
• Transmission errors - bit corruption
• Real anomalies - genuine events that may be valuable

3. Format Inconsistencies

Data from different sources has different formats:

• Units - temperatures in °C or °F, pressures in bar, psi, kPa
• Time zones - UTC vs local time, daylight saving time
• Separators - period vs comma for decimal numbers
• Encoding - UTF-8, ISO-8859-2, Windows-1250

4. Duplicates and Conflicts

The same measurement recorded multiple times or different values for the same timestamp.

5. Lack of Context - Data Without Meaning

Numeric values alone without metadata are of little use to AI algorithms. If you see a value of 75, you don't know whether it's temperature in Celsius, pressure in bars, or speed in RPM. Similarly, if you see an increase in value, you don't know whether it's a problem signal or a normal change related to the production cycle.

Key questions you need to answer include: What type of sensor is it? Where is it mounted? What's its measurement range? What's its accuracy? When was it last calibrated? What do error codes mean? Without answers to these questions, AI algorithms won't be able to properly interpret the data and generate reliable predictions.

Data Quality Assurance Process - A Systematic Approach

Ensuring data quality is not a one-time activity but a systematic process that should be integrated into daily operations. Below we present a comprehensive process, refined through numerous GC AIT implementations, that enables systematic management of data quality from the moment of collection through to use in AI models.

Step 1: Data Profiling - Know Your Data

Before you start cleaning data, you first need to understand what you actually have. Data profiling is the process of analyzing dataset characteristics that allows you to identify problems and understand data structure.

Key metrics you should analyze include data volume - how many records you have, what's the daily growth rate, how quickly datasets are growing. This allows you to plan infrastructure and understand the scale of the challenge.

Data completeness - what percentage of values are missing, where gaps occur, whether there are patterns in missing data. This allows you to identify problems with specific sensors or systems.

Uniqueness - what percentage of data are duplicates, whether there are conflicts between different data sources. This allows you to identify data integration problems.

Consistency - whether values fall within sensible ranges, whether there are physically impossible values, whether there are anomalies in value distribution. This allows you to identify measurement quality issues.

Timeliness - how old the data is, whether there are delays in data collection, whether all systems are time-synchronized. This allows you to identify infrastructure problems.

For data profiling, you can use various tools - Python with libraries like pandas-profiling or Great Expectations, SQL for analysis directly in databases, or dedicated data quality (DQ) tools that offer advanced automatic problem detection features.

Step 2: Defining Quality Rules - What Does "Good Quality" Mean?

Determining what "good quality" means for your data is crucial for effective quality management. Quality rules should be defined based on domain knowledge - understanding the physical limitations of machines, normal parameter ranges, and typical behavioral patterns.

Quality rules can be simple - for example, temperature should be in the range of -40°C to +150°C for most industrial applications, pressure cannot be negative, timestamp cannot be from the future. But they can also be more complex - for example, rotational speed cannot exceed the maximum value for a given machine, vibrations above a certain threshold should be flagged as alerts.

It's worth remembering that quality rules should be flexible and tailored to the specifics of your machines and processes. What's normal for one machine may be an anomaly for another. That's why it's important for rules to be defined in collaboration with domain experts - maintenance technicians who know the specifics of your machines.

# Temperature should be in range -40°C to +150°C
temperature >= -40 AND temperature <= 150

# Pressure cannot be negative
pressure >= 0

# Timestamp cannot be from the future
timestamp <= NOW()

# Rotational speed cannot exceed max for machine
rotational_speed <= machine.max_rpm

# Vibrations above 10 mm/s are an alert
IF vibration > 10 THEN flag_as_anomaly()
                

Step 3: Data Cleaning - Transforming Raw Data into Usable Data

Data cleaning is the process of transforming raw, often incomplete and incorrect data into high-quality data that can be used by AI algorithms. This is one of the most time-consuming but also most important stages of data preparation.

Filling Missing Values - Different Strategies for Different Situations

Missing values are inevitable in industrial data, but they can be filled in various ways depending on the nature of the data and the cause of the gaps.

Interpolation is ideal for continuous time series where values change smoothly. If you have a missing value between two known values, you can estimate it based on the trend between those values. This works well for parameters like temperature or pressure that change gradually.

Forward fill (repeating the last known value) or backward fill (using the next known value) are simple but can introduce errors, especially if values change rapidly. However, they're useful for short gaps in data.

Mean or median of surrounding values can be used for short gaps, especially when values are relatively stable. Median is more resistant to outliers than mean.

Deleting data may be necessary when the gap is too large or critical. If several hours of data from a key period are missing, it's better to delete that data than try to fill it in, as incorrect filling can lead to incorrect conclusions.

Handling Outliers - Separating Errors from Real Anomalies

Outliers can be measurement errors that should be removed, or real anomalies that are very valuable for analysis. The key is properly distinguishing between these cases.

Z-score (standard deviation) allows identifying values that are significantly distant from the mean - typically values greater than 3 standard deviations are considered outliers. This works well for normally distributed data.

IQR (Interquartile Range) uses quartiles to identify outliers - values outside 1.5× the interquartile range are considered outliers. This is more resistant to outliers than z-score and works well for data with various distributions.

Domain rules use knowledge of physical machine limitations - for example, if temperature physically cannot exceed 200°C, any value above this threshold is an error, regardless of statistics.

Outlier isolation - instead of removing outliers, you can keep them for separate analysis. Sometimes outliers are exactly what we're looking for - real signals of problems that AI algorithms should detect.

Standardization - Unifying Formats and Units

Standardization is the process of unifying data formats so that all data is in the same format, units, and encoding. This is absolutely necessary if data comes from many different sources.

Converting units to standards - all temperatures should be in Celsius, all pressures in the same units, all speeds in the same units. This allows for proper comparison and analysis of data.

Normalizing times to UTC - all timestamps should be in UTC, which allows for proper synchronization of data from different sources and avoiding problems with daylight saving time.

Unifying numeric formats - all numbers should use the same decimal separator (period or comma), the same thousands separator, the same date format.

Mapping codes to readable names - instead of numeric codes or abbreviations, use readable names that are understandable to AI algorithms and humans. For example, instead of code "ERR_001", use "Sensor_Failure".

Step 4: Data Enrichment - Adding Context

Clean data is just the beginning. To increase the analytical value of data, it's worth adding context that allows AI algorithms to better understand the situation and generate more precise predictions.

Machine metadata - machine type, age, service history, recent repairs, modifications. This allows AI algorithms to account for the specifics of a particular machine in analysis. For example, older machines may have different wear patterns than new ones, which should be considered in models.

Operational context - shift (first, second, third), operator, product recipe, process parameters. This helps understand whether changes in machine parameters are related to a specific product or operator, which can be important for problem diagnosis.

External conditions - ambient temperature, humidity, season. This allows accounting for the impact of environmental conditions on machine operation. For example, machines may operate differently in summer than in winter, which should be considered in analysis.

Events - downtime, changeovers, failures, maintenance. This allows AI algorithms to recognize that changes in parameters may be related to specific events rather than machine degradation. For example, increased vibration during changeover is normal and should not be treated as a problem signal.

Step 5: Quality Monitoring - A Continuous Process

Data quality is a continuous process, not a one-time activity. Data changes every day, new problems arise, old problems may return. That's why continuous monitoring of data quality and quick response to problems is necessary.

Automatic validations should be performed with each data load into the system. This allows detecting problems immediately before they affect analysis quality. Validations should check all defined quality rules and automatically flag data that doesn't meet requirements.

Quality dashboards should show key data quality indicators - completeness, consistency, percentage of outliers, percentage of data meeting quality rules. This allows quick identification of problems and tracking data quality trends over time.

Alerts should be generated automatically when data quality falls below a certain threshold. For example, if data completeness falls below 95%, an alert should be generated informing the team about the problem and allowing quick response.

Regular audits - quarterly data quality reviews allow identification of long-term trends and problems that may not be visible in daily monitoring. It's also an opportunity to review and update quality rules to reflect changing conditions and requirements.

Data Architecture for AI - Designing the Storage System

Proper data storage architecture is crucial for effective use of AI in industry. Different types of data require different approaches to storage and processing, and choosing the right architecture can significantly impact system performance and costs.

Data Lake vs Data Warehouse - A Hybrid Approach

For industrial AI, the GCAIT team recommends a hybrid approach that combines the advantages of Data Lake and Data Warehouse, enabling efficient storage and processing of different types of data.

Data Lake is a place for storing raw sensor data - huge volumes of data in their original form, without preprocessing. The flexible schema allows storing data in various formats, which is particularly important in an industrial environment where data may come from many different sources. Data Lake is ideal for high-frequency data such as vibration measurements or video streams.

Data Warehouse is a place for storing processed data - aggregates, reports, data prepared for business analysis. Data here is organized in a schema structure, which facilitates quick queries and analysis. Data Warehouse is ideal for data that's already processed and prepared for use by reporting systems or dashboards.

Feature Store is a special repository of ready-made features prepared for use in machine learning models. Features are calculated once and stored, allowing quick use by different models without the need to reprocess data. This significantly speeds up the development and deployment of new AI models.

Processing Layers - Medallion Architecture

Medallion architecture is a popular approach to organizing data in Data Lake, dividing data into three processing layers, each with a specific role.

The Bronze layer is raw data, without any modifications - exactly as received from sensors or systems. This is the source of truth that you can always return to if something goes wrong during processing. Data in the Bronze layer is immutable - it's never modified, only new data is appended.

The Silver layer is cleaned, validated, and standardized data. This is where all data cleaning operations are performed - removing duplicates, filling missing values, handling outliers, standardizing formats. Data in the Silver layer is ready for use in analysis but may still be too detailed for some applications.

The Gold layer is aggregated data, enriched with context, and ready for analysis. This is where data is aggregated to appropriate levels of detail, enriched with metadata and operational context, and prepared for use by AI models or reporting systems. Data in the Gold layer is optimized for quick access and analysis.

Data Quality Management Tools - Choosing the Right Solution

Many data quality management tools are available on the market, from simple programming libraries to comprehensive enterprise platforms. Choosing the right tool depends on project scale, budget, and functionality requirements.

Open Source Solutions - Flexibility and Control

Great Expectations is a popular data validation framework in Python that allows defining and executing data quality tests. It offers a wide range of features, from simple value range tests to advanced statistical and relational tests. It's particularly popular in the data science environment where Python is the standard tool.

Apache Griffin is a data quality platform designed to work at scale, especially in Big Data environments. It offers advanced real-time data quality monitoring features and integration with the Apache ecosystem (Spark, Hadoop). It's ideal for projects requiring processing of huge data volumes.

dbt (data build tool) is a tool for data transformation and testing in SQL that allows defining data transformations as code and automatically executing quality tests. It's particularly popular in the data engineering environment where SQL is the standard data processing language.

Commercial Solutions - Comprehensive Enterprise Platforms

Informatica Data Quality is an enterprise-grade solution offering comprehensive data quality management features, from profiling through cleaning to monitoring. It's particularly popular in large organizations that require advanced features and vendor support.

Talend Data Quality offers integration with ETL (Extract, Transform, Load) processes, allowing automatic data quality checking during processing. This makes it particularly useful in projects requiring continuous data processing.

Ataccama ONE uses artificial intelligence to automatically detect data quality problems and suggest solutions. This modern approach can significantly speed up the data quality assurance process, especially in projects with many different data sources.

GC AIT Summary - Data Quality as the Foundation of Success

Data quality is the foundation of every AI project. Without solid, high-quality data, even the best algorithms will produce incorrect results that can lead to costly decision errors. Investing in data quality at the beginning of a project pays off many times over in the form of more accurate predictive models that better detect problems and generate fewer false alarms.

Moreover, good quality data builds user trust in the system. If the AI system generates many false alarms due to poor data quality, users will stop trusting the system and start ignoring alerts, even real ones. On the other hand, a system based on high-quality data will generate reliable predictions, which builds trust and encourages system use.

Finally, good quality data enables faster deployment of subsequent use cases. If data is already prepared, cleaned, and validated, you can quickly use it in new AI models without having to go through the entire data preparation process again.

If you're planning an AI implementation and don't know how to assess the quality of your data or how to prepare it for analysis - contact us. GC AIT experts will conduct a comprehensive data quality audit and help prepare data for use in AI systems. This is the first and most important step to AI project success.