Highlights
- Automated Real Estate Advertisement Distribution: successfully implemented a fully automated system for the distribution of real estate advertisements across diverse platforms such as Craigslist, Facebook Marketplace, and more.
- End-to-End Development Process Automation: accomplished complete automation of all development processes, streamlining workflows for increased efficiency and productivity.
- Self-Updating and Self-Healing Desktop Application: introduced a cutting-edge desktop application with self-updating and self-healing capabilities, ensuring a seamless and resilient user experience.
Client
A MarTech company dedicated to assisting apartment owners and operators in streamlining and enhancing their lease conversion through automation and optimization.
Country: 
Industry: 
Team Size: Product
The product revolutionizes ad distribution, specifically designed to amplify lead generation. This service seamlessly extracts data from diverse rental platforms and autonomously disseminates it across various advertising platforms, such as Craigslist and Facebook Marketing, including social media channels. Through this process, it references rental ads from existing posts on specific platforms. Additionally, this system incorporates basic chatbot functionalities to engage and communicate with potential tenants effectively.
This solution tackles common challenges faced in rental advertising: lackluster prospect responses, inaccuracies, outdated listings, and the absence of robust ad analysis and strategic planning.
Goals and objectives
When we initially received the project, many of the development processes, including code deployment, were largely manual. A few dedicated developers managed the lead generation process from their local machines, leading to significant time consumption and an inability to meet future scaling demands.
In response, the SPD Technology team was tasked with automating these processes and transitioning all workload to the AWS cloud. The new solution aimed to offer enhanced visibility into the ad posting process while being resilient against and capable of bypassing website blocking mechanisms. Deployment was streamlined to a single-click process for ease.
As part of the solution, one of the distribution applications was installed on clients’ local servers in their offices, ensuring automatic updates.
Project challenge
- Overcoming Website-Blocking Mechanisms: the website-blocking mechanisms of robots are hard to bypass, it is a complicated process, requiring a nuanced and sophisticated approach.
- Server-Based Application Installation and Updates: an application that distributes ads to one of the advertisement platforms must be installed on the client’s servers in their offices and be automatically updated.
Solution
Regarding the ease of use in CI/CD tools, Github Actions stands out prominently. It offers a rich array of built-in actions catering to diverse deployments and testing scenarios, delivering an exceptional user experience, especially for projects managed within GitHub. We seamlessly incorporated pipelines for tests and code deployment into AWS.
For Python application deployment, we utilized Fabric as our tool of choice. Upon a successful release, a new release tag became visible in GitHub, signifying the completion and success of the deployment process.
The original website navigation, executed using Selenium for Python, encountered visibility issues upon deploying functionality to an AWS server, hindering ongoing scraping processes. To resolve this, we integrated Selenoid, a trusted tool extensively used in end-to-end UI testing, ensuring seamless monitoring and visibility.
To augment our solution, we developed Terraform scripts adopting an Infrastructure as Code approach. These scripts facilitated the automatic creation of new servers pre-equipped with the Selenoid service. This allowed developers to monitor all clicks, navigations, and server actions within a new browser tab. This automation streamlined the website scraping process while providing essential visibility into the underlying operations. Selenoid, being inherently scalable, efficiently manages a high volume of concurrent processes, enabling the execution of thousands of tasks effortlessly.
The most challenging part of the project’s implementation was the need to deal with multiple issues associated with automated browsers getting blocked by target websites. The automated scripts could be blocked for many reasons: blacklisted IPs, requests rate-limit checks, suspicious user-agent header, and many more. Websites could display a Captcha or block the user / IP address altogether. Notably, Craigslist has invested extensively in its protection mechanisms, resulting in the deletion of a majority of automatically generated ads. While certain issues only surfaced when the automated process was scaled up, preemptive measures were still feasible to mitigate potential risks.
An integral part of the success of automated scripts is a proper use of IP addresses. There are a lot of services that provide IP proxy functionality and one of the most popular among them is Bright Data. It offers a wide range of proxies, including Residential Proxies, Datacenter Proxies, and more. Our team has integrated the Residential proxy into automated browsers, which has significantly improved the success rate of the jobs. The solution had specific pauses to simulate human activities in between the steps and used browser-like user agents.
Implementing the auto-updating Python desktop application presented a thrilling challenge.
The client distributed an installer for the leads generator app, ensuring users could run it on their laptops. Considering that most users operated Windows or MacOS, we opted to provide packages for both platforms. Beyond ensuring cross-platform compatibility, our goal was to make the app self-updating, self-healing, and perpetually operational even after an OS reboot. It had to seamlessly run in the background and update itself upon a new version release.
After exploring various options, we selected the PyUpdater library. Its integration with AWS S3 facilitated secure app distribution to clients while enabling seamless self-updating. Adapting to the specific requirements of each operating system was essential. For background operation in MacOS, we utilized the launchctl instrument, and for Windows, Docker Desktop was our solution. Both methods automatically initiated the service upon laptop restart, ensuring complete autonomy. We crafted user-friendly instructions for app installation, ensuring accessibility even for non-technical clients.
Our results
- Fully automated development process with one-click CI/CD, code and infrastructure management as a code.
- Infrastructure has been successfully migrated from the local environment to AWS.
- Development of a cross-platform desktop application installed on client servers, ensuring seamless, automatic updates.
- Managed to overcome most of the websites guard systems, enabling automated ad posting and driving towards the accomplishment of company business objectives.
Highlights Client The client is BlackHawk Network Inc. (BHN), a U.S.-headquartered, globally leading provider of branded payment technology solutions,...
Explore Case