Creating a Computer Vision-Powered API for a Petcare Mobile App

# API development # Data Analytics # Mobile Application
Discover more

Highlights

  • Two Significant Challenges in Data Structure: successfully addressing noisy labeled data and its high imbalance between classes, ensuring data consistency and reliability for subsequent neural network development.
  • Two Advanced Neural Network Models: created an instance segmentation and deep imbalance regression models that laid the foundation for the computer vision-enabled API.
  • Lightning-Fast Image Analysis: enabled the company to significantly reduce operational costs and deliver effective user experience with almost-instant in-app image processing.

Client

Established in 2015 as a startup, our client has rapidly evolved to become a prominent leader in the Israeli pet tech industry. The company designed a technology platform to actively engage dog owners in monitoring and improving their pets’ well-being. Their solution goes beyond conventional tracking, fostering engagement with gamification. The users can earn coins for being proactively involved in their dogs’ lives, and then, exchange them for dog food, treats, toys or veterinary services within the partners’ network.

Country:
Industry:
Team Size:

Product

The company offers mobile applications for Android and iOS. Dog’s fitness tracker is the main feature of the app; it allows the owners to track walk duration and dog activity, competing with other members within their neighborhood. Active users can earn coins that can be redeemed for discounts for pet care products, and for every mile walked with the dog, the company donates a meal to a shelter.

In 2022, the client decided to bring artificial intelligence capabilities to the app, enchanting the pet care service being provided and solidifying the company’s position as a leader in the Israeli pet tech industry. SPD Technology became the mobile development services provider of choice.

Goals and objectives

  • Introduce a Remote Veterinary Service

To make a client’s app more innovative and effective for users, we were tasked to automate remote veterinary service delivery by bringing artificial intelligence capabilities.

  • Build an Image Analysis Neural Network

As the main goal, we had to develop a computer vision-embedded feature for canine excrement analysis and further medical suggestions that should be deployed as an API.

  • Drive User Engagement

Since before the development started, the application had a dog fitness tracker feature only, driving user engagement by expanding service offering became one of the primary goals as well. 

Challenges

  1. Noisy Labeled Data: the presence of incorrect or mislabeled instances in a dataset can have detrimental effects on the model’s performance and reliability. To address the issue, we had to use a combination of careful data curation, employing robust learning algorithms, and implementing strategies to identify and mitigate the impact of mislabeled instances.
  2. High Imbalance Between Data Classes: since non-uniform distribution of labels across different classes can lead to biases towards the majority classes, we had to choose the right tactic to address the issue.

Solution

The solution we proposed to the customer involved unfolding the project in the next distinct stages.

  1. Data Collection: at the first stage, we gathered a dataset of images containing dog excrement samples with a variety of textures, colors, and conditions representing different health states. We also ensured that the images within the dataset are of high quality, being captured from different angles and light conditions.
  2. Data Exploration and Analysis:  our initial focus was on understanding the intricacies of the dataset. Through thorough data exploration and analysis, we identified instances of noise in the labels and recognized the significant imbalance between different data classes. This critical step laid the foundation for devising effective strategies to handle the challenges.
  3. Instance Segmentation Model Trainingat the third stage, we embarked on training an instance segmentation model. This model excelled at identifying and segmenting individual instances within images, offering a more granular understanding of the data. In such a way, this algorithm identifies and segments the fecal sample in the image, distinguishing it from other objects and ensuring further analysis correctness. At this stage, we also annotated the images to mark the areas of interest (the poop) for training the instance segmentation model and to label the conditions or health states for regression model training. The outcomes of this stage included improved object localization accuracy, laying the groundwork for subsequent phases.
  4. Deep Imbalanced Regression Model Training: the next move was to train a deep imbalanced regression model. We assigned a condition score to each pool instance, and then proceeded with training the model to assess the health of the dog based on its excrement appearance or other factors. A regression model is particularly suitable for this task because it can predict continuous values, such as the degree of healthiness or abnormalities in the poop. The result was a robust model capable of making accurate predictions across diverse data classes.
  5. QA and Data Labeling Review: after the models were trained, we conducted thorough testing and validation of the feature before deployment. We checked it for false positives and false negatives to refine the model’s accuracy and ensured the model’s predictions align with expert veterinary opinions. In this critical quality assurance step, our team also conducted a meticulous review of the data labeling. This involved identifying and rectifying instances of noisy labels that could potentially impact the model’s performance. 
  6. Integration and Testing: at the last stage, we integrated the trained models into the mobile application, ensuring they can process images and provide results in real-time. For this goal, we tested the feature with a separate validation dataset and made sure that it provides accurate results for a variety of poop samples. As a result, we delivered a computer vision-powered API that incorporated the trained instance segmentation and imbalanced regression models.
  7. Feedback Loop Implementation: to ensure continuous improvement of the model’s performance, we implemented a feedback loop so that the users can share their opinion on the accuracy of the analysis.

Tech Stack

  • Python Python
  • PyTorch  PyTorch

Our results

Our collaboration with the customer goes on. Our ML developer is proactively engaged in image analysis model support and its further training. For now, SPD Tech delivered the next results: 

  1. Shortened Image Analysis Time to Several Seconds:
    Our team has successfully coped with data inconsistency challenges and built a robust neural network, capable of an almost-instant image analysis.
  2. Rid Company’s Employees From Manual Work:
    The customer shared that after releasing an image recognition model, the time employees spent on manual image processing was reduced significantly. 
  3. Created a Revenue Stream for the Customer:
    An AI-supported remote veterinary service was introduced as a paid feature, thereby creating an additional source of income for our client. 
Next Project
# AWS infrastructure, # Data migration, # Fintech, # Legacy Application Improvement
How We Helped BlackHawk Network with Distributed Systems Unification and Integration 

Highlights Client BlackHawk Network Inc. (BHN) is a U.S.-headquartered, globally leading provider of branded payment technology solutions, whose history...

Explore Case