All work
2024Data
Product & Review Crawlers
One walks product pages and extracts the structured fields — title, price, image links, highlights. The other goes after the review stream attached to a product.
Data
Approach
Scrapy handles scheduling, concurrency, retries, and the item pipeline, so the work reduces to selectors that survive small markup changes and shaping the output into a consistent item.
Products and reviews are separate spiders because the traversals genuinely differ: one is page-per-item, the other is pagination over a stream hanging off a single item.
What it taught me
- The fragile part of a crawler is never the HTTP — it is the selectors, and semantic attributes outlive layout classes.