SquareTrade: Automated real-time product tracking to optimise warranties across vendor platforms
Project Overview
SquareTrade, a subsidiary of Allstate Protection Plans, partnered on the development of a custom Scraper Portal—an automated system designed to collect and update product data in real time from major retailers such as Costco, Walmart, and Amazon. This data powers accurate warranty pricing and product listings for TVs, furniture, and other electronics sold across various retail channels. Built with scalability, flexibility, and speed in mind, the platform automates a key operational function that previously required significant manual effort.
Problem Statement
SquareTrade required a robust, scalable system to automatically gather and update product data from various e-commerce vendors. This information needed to be posted in real-time to the client portal, ensuring accurate warranty plans and up-to-date pricing. The solution needed to be flexible, allowing quick integration of new vendors without disrupting operations.
Key Findings
- Frequent Product Updates: E-commerce platforms often update product information, making it essential for SquareTrade to have real-time data to offer accurate warranty plans and competitive pricing.
- Scalable Data Scraping: A scalable solution was necessary to handle the constant influx of product data from multiple vendors, ensuring efficient scraping and importing without manual intervention.
- Task Management Efficiency: Efficient scheduling and task management were critical for handling large volumes of data and ensuring that no product details were overlooked or delayed.
Implemented Solution
The portal was designed with the following key modules to address the problem:
-
Microservices Architecture:
Built the portal on a microservices foundation, enabling each component to operate independently. This architecture supported easy scalability and vendor-specific customisation without disrupting the overall system.
-
Discovery & Crawler Modules:
The Discovery Module (built with Scrapy) continuously scans vendor websites to identify new product URLs, ensuring complete and up-to-date coverage. The Crawler Module performs targeted data scraping—capturing details like price, warranty, SKU, and descriptions—using Scrapy scripts triggered by a scheduler for efficient and timely extraction.
-
AWS Lambda Scheduling:
Deployed AWS Lambda to schedule scraping tasks efficiently. The serverless model offered cost-effective, lightweight scheduling while maintaining high uptime and responsiveness.
-
Importer & Data Cleaning:
The Importer Module processed and cleaned incoming data, performing validations before submitting to the client portal via API, ensuring accuracy and uniformity across datasets.
-
CI/CD with Jenkins:
Integrated a CI/CD pipeline using Jenkins, AWS EC2, and ECR, automating the deployment process and supporting continuous updates to microservices.
Results
The Scraper Portal delivered transformative outcomes for SquareTrade. It enabled real-time updates of product details, ensuring warranty plans remained accurate and competitive. The flexible architecture allowed new vendor sources to be added quickly, significantly reducing onboarding time. Built using Scrapy, AWS Lambda, and a CI/CD pipeline, the platform automated the entire workflow—minimising manual intervention, improving development velocity, and maintaining high system availability. The use of serverless infrastructure also optimised costs while supporting rapid scale, making the Scraper Portal a reliable, future-proof solution for continuous product data integration.