Highlights
- Feature-Rich AI Platform for Defect Detection: The system performs multimedia analysis and provides inspectors with tools for defects recognition, digital twins, actionable insights on maintenance, support chatbot, and reporting.
- Leveraged Multiple ML Algorithms: The system benefits from NLP, object recognition, anomaly detection, video and image analysis, sensor data analysis, and other ML-driven capabilities.
Client
Our client is a US-based company that provides inspection services for equipment used in different industries, including oil and gas, power transmission, construction, and manufacturing. The company focuses on offering AI-driven solutions for prescriptive maintenance, visual inspection, and actionable recommendations. The client’s approach to machinery inspection brings more efficiency and safety as well as cost reduction for factories, critical infrastructure companies, production facilities, and other industries where heavy industrial equipment can be used.
Country 
Industry 
Team Size: Product
The product is an AI-driven platform capable of predicting, detecting, and managing equipment/asset defects. Some of the examples of the machinery that can benefit from this inspection include pipelines, assembly lines, and solar panels. The platform works by collecting data from the equipment (text, visual, or sensor information). Then the platform processes this data and highlights defects that need to be addressed or provides predictions for potential malfunctions.
Goals and objectives
The final goal of creating this AI-driven inspection platform was to automate and, hence, accelerate the work of asset inspectors and reduce human error. This was expected to be translated into drastic cost savings since inspectors could more optimally use their labor time.
To achieve such an outcome, we needed to equip the system with the following functionality:
- Predictive Maintenance through IoT Data Analysis: we needed to make the platform gather and analyze metrics from various sensors attached to equipment. Some of those metrics included vibration, temperature, pressure, and power consumption. By analyzing trends and patterns in this data, the platform was set up to anticipate potential machinery failures.
- Anomaly/Defect Detection using Multimedia Data: our team were required to develop and fine-tune anomaly detection algorithms that can analyze images, videos, or text documents to identify different defects. Those defects include, but are not limited to, cracks, corrosion, or misalignment.
- User Guidance and Insights: another task was to make the platform generate actionable insights to guide users through the inspection process. Based on the equipment defect analysis, the platform needed to produce hints and recommendations about maintenance and repairs.
Project challenge
- Data Integration and Correlation: The intelligent agent needed to seamlessly integrate data from diverse sources: sensor readings, images and videos, historical maintenance records. Correlating this data to identify anomalies and pinpoint root causes required robust data management and analysis techniques.
- Model Interpretability and Accuracy: Striking a balance between object and defect detection models complexity (leading to high accuracy) and interpretability (allowing inspectors to understand how the model arrives at its conclusions) were crucial.
- Digital Twin Accuracy and Maintenance Actionability: The digital twin’s value hinged on its accuracy in reflecting the asset’s real-world condition. Additionally, the system needed to translate detected defects into actionable insights for maintenance staff, prioritizing repairs and optimizing maintenance schedules.
Solution
Data Collection and Analysis
To develop an AI-driven solution, we needed to collect, process, and analyze vast amounts of data. This data needed to be collected from different sources, including sensors, images, videos, and maintenance logs. To achieve that, we developed a modular data collection system that is compatible with several sensors (vibration, temperature, pressure, etc.) and can capture images, including RGB and thermal, and videos.
Another source of data was an Android application that provided inspectors with the ability to capture images and videos and send them to the system. The app also contained notes from maintenance reports that were also sent to the system.
Once the platform received the data, natural processing algorithms helped us to identify textual data and extract it for further analysis. YOLO models and convolutional neural networks allowed us to set up the visual data processing for identifying defects. Plus, our team uses various supervised or unsupervised machine learning algorithms utilized for data recognition, when dealing with complex sensor data.
Defect Detection
Once we had our data collected, we moved on to the defect detection task. At first, we leveraged anomaly detection models to capture normal behavior of the equipment based on historical data. Any deviation from the predicted behavior was a signal of potential anomalies. The platform was designed in a way that workers on sites could receive a notification about possible equipment failures.
On top of that, we utilized deep learning models specifically trained on a versatile dataset of industrial equipment images and videos containing a number of defects (cracks, corrosion, misalignment, etc.). So, in case a system spotted some kinds of these defects, it would notify the workers about required maintenance. The platform was also equipped with an AI-powered asset health score to each inspected machine. This health score was needed for the client to ensure a big picture of the equipment’s condition.
Digital Twins
Another crucial element of the platform, that made it possible to inspect equipment, was digital twins. We leveraged a 3D visualization engine to create a digital twin of each inspected asset. This digital twin feeded on sensor data, defects spotted during image or video analysis, and records from maintenance logs. As a result, we created an interactive platform where inspectors could visualize defects in their 3D context on the equipment virtual model.
Staff Support AI Chatbot
While the system was designed as an advanced AI tool that could transform the work with heavy machinery or any other equipment, the staff that worked with this system still needed help with its adoption. Therefore, we developed a chatbot, using NLP and RAG (Retrieval-Augmented Generation) approach. NLP helped to recognize contextual information, while RAG was utilized for generating concise answers in natural language.
Automated Reporting
The platform was configured to automatically generate detailed inspection reports. These reports could include captured images/videos, identified defects with severity levels, predicted maintenance needs based on the asset health score, and historical maintenance data. Additionally, inspectors received the possibility to customize reports to include specific details relevant to the inspection and share them electronically with stakeholders for informed decision-making.
Tech Stack
-
Amazon AWS -
Java -
Spring -
Python -
PyTorch -
PostgreSQL -
ONNX -
OpenDroneMap
Our results
We successfully delivered the platform, where multiple capabilities, including defect detection and recognition, digital twins, staff recommendations on maintenance, reporting, and support, were executed thanks to AI and ML algorithms.
This system contributes to enhanced asset inspection of heavy machinery and other equipment. Its additional benefits include:
- Improved Safety: With predictive maintenance, inspectors can spot potential asset failures and malfunctions early to prevent them.
- Data-Driven Decision Making: Based on data analysis from images, videos, sensors, and textual information, the platform can craft actionable recommendations for maintenance schedules or workforce allocation.
- Increased Efficiency: Automated tasks and real-time insights allow inspectors to optimize their work and complete inspections faster.
- Reduced Costs: Early detection of problems prevents costly breakdowns and unplanned maintenance activities.
Highlights Client Our client, the Israeli telecom technology vendor BD Innovations, was acquired by the UK-based innovative connectivity provider...
Explore Case