Shopify Store Scraping: Track Any Competitor's Catalog, Prices & New Launches
Shopify store scraping is one of the cleanest ways to watch a direct-to-consumer rival. Most Shopify storefronts publish their catalog in a structured form, so you can see what a brand sells, how it prices it, which items are marked down and when new products appear, without guessing from screenshots.
To make this concrete, we collected a free sample of 100 products from the public Allbirds catalog. You can download it from the Allbirds Shopify catalog dataset page. Every number in this article is computed from that file, so you can check it yourself.
Why Shopify catalogs are worth tracking
DTC brands compete on assortment and price, and both change quietly. A new colorway appears, a bestseller sells out, a line gets cut by 40%. None of this arrives as a press release. If you run a rival brand, a marketplace or a retail buying team, those moves matter.
A catalog feed gives you the same facts in a repeatable format. Compared with reading pages by hand, structured data lets you compare week against week and see differences instead of impressions. It also pairs well with the broader approach we describe in competitor analysis and market research.
What our Allbirds sample shows
We took the first 100 products returned by the store's catalog feed and computed the following. All figures describe this sample only, not the store's full catalog.
- Product type mix: 92 products are tagged "Shoes", 1 is "Apparel", and 7 have an empty product type. That blank group is a data-quality finding in itself: the Cruiser Coastal styles, for example, carry no type, so a type-based filter would miss them.
- Variant depth: the average product has 10.9 variants (sizes), the median is 13 and the maximum is 14. 61 of 100 products have 13 or more variants, and none has a single variant.
- Price ladder: the lowest variant price is $49 and the highest is $165. The median of the minimum price is $105 and the average is $102.54.
- Markdown share: 15 of 100 products (15%) carry a compare-at price higher than the selling price. On those items the average discount is 46.4% off the compare-at price.
- Availability: 92 products have at least one variant in stock, so 8 are fully sold out at the time of the scrape.
- Launch rhythm: 57 of 100 products were created in 2025 and 8 in 2026. 30 products share an August 2025 creation month, which points to a large single drop. The oldest product in the sample was created in November 2018 and the newest in May 2026.
- Audience split from tags: 63 products are tagged womens, 28 mens, 1 toddler, and 8 have no merchandising tags.
Here is a five-row excerpt:
| Product | Variants | Min price (USD) | In stock | Created (month) |
|---|---|---|---|---|
| Women's Allbirds Flip Flop - Dusty Pink | 7 | 50 | true | 2026-03 |
| Men's Canvas Runner NZ - Deep Navy Stripes | 13 | 100 | true | 2025-12 |
| Women's Canvas Runner NZ - Deep Navy Stripes | 13 | 100 | true | 2025-12 |
| Men's Cruiser Terry - Warm White | 13 | 115 | true | 2025-10 |
| Men's Cruiser Slip On - Natural Black | 13 | 100 | true | 2025-08 |
One caution on the sample: it is the first 100 rows of a 250-product feed page, so it is a snapshot, not a statistically drawn slice of the brand.
How scrape Shopify products at scale
If you want to scrape Shopify products yourself, the useful fact is that many stores expose a products feed at a predictable path. It returns product titles, handles, tags, vendor, product type, creation date and a list of variants with price, compare-at price and availability.
That is far more stable than parsing page layouts. But a few steps are still needed to make it useful:
- Collapse variants to the product level. The feed lists one entry per size. For comparison you usually want the number of variants, the minimum and maximum price, and whether any variant is in stock.
- Normalize prices. Prices arrive as strings. Convert them to numbers and keep the currency in a separate field.
- Detect markdowns. A product is discounted when the compare-at price is higher than the selling price. A null compare-at price means no markdown.
- Clean the tags. Tags mix merchandising labels (gender, material, silhouette) with internal operations labels. Keep the former and drop the latter.
- Store a timestamp. Without
scraped_at_utc, you cannot tell today's snapshot from last month's.
Which fields matter and why
- product_id: the stable key. Titles change; IDs do not. Use it to match rows between snapshots.
- product_type and tags: your category structure. Check for blanks, as we found 7 in this sample.
- variant_count: a proxy for size depth and inventory commitment. A brand that cuts sizes is often winding a line down.
- min_price and max_price: the price ladder. A gap between them means variants are priced differently.
- compare_at_price: the markdown signal. Track the share of discounted products and the depth of the discount.
- available: stock status. A product that flips to unavailable and stays there is usually sold out or retired.
- created_at: the launch clock. Counting products per month shows when a brand drops new ranges.
Use cases for competitor catalog monitoring
Assortment gap analysis. Compare product types, materials and price tiers against your own range to see where a rival is thick and where it is absent.
Markdown tracking. The 15% discount share in our sample is a single snapshot. Repeat the scrape weekly and you can see when sales start, which lines are discounted first and how deep the cuts go. For a wider view of this practice, see our e-commerce price monitoring playbook.
New launch alerts. Compare today's product IDs with last week's. New IDs are launches; missing IDs are retirements. The August 2025 cluster in our sample shows how a drop appears in created_at.
Marketplace and retail buying. Buyers can check whether a brand's wholesale line matches what it sells direct, and how many sizes it really supports.
Investor and analyst research. Product count, launch pace and markdown share are practical indicators of how a DTC brand is managing its range. They are not a substitute for financial data, but they are public and timely.
Common challenges
Anti-bot measures. Not every request is answered cleanly. In our own run, the first plain request returned a browser verification page rather than data, and a retry through residential routing returned the full feed. Build retries and block-page detection into any recurring job.
Pagination. Feeds are capped per page. Larger catalogs need paged requests, and a stable order helps avoid duplicates. Deduplicate by product ID.
Store customization. Some stores disable the public feed or change its fields. A layout or settings change can break a pipeline overnight, so validate the schema on every run.
Data quality. Blank product types, inconsistent tags and variant-level quirks (for example, a compare-at price on some sizes only) are normal. Decide your rules once and apply them every time.
Legal and ethical limits. Stick to public product data and avoid anything behind a login or tied to individuals. Our guide to small-business web scraping covers practical ground rules.
How often to refresh
- Weekly is enough for assortment and launch tracking.
- Daily suits markdown tracking during sale seasons.
- Several times a day is only worth it for fast-moving drops or stock-sensitive categories.
Whatever the cadence, keep every snapshot. The value comes from comparing dates, and you cannot rebuild yesterday's prices afterwards.
FAQ
Can I scrape any Shopify store? Many stores expose a public products feed, but not all do, and some restrict access. Always limit yourself to public data and respect the site's terms.
What does a compare-at price mean? It is the original price a seller shows before a markdown. When it is higher than the current price, the product is on sale. In our sample, 15 of 100 products had one.
Is dtc brand data reliable for market analysis? It is accurate for what a store lists: products, prices and stock status. It does not show sales volume, so combine it with other sources before drawing conclusions about demand.
How many products do I need for a useful sample? A hundred rows is enough to test your pipeline and see the shape of a catalog. Full monitoring needs the complete product list.
Get your own catalog data
If you want a competitor's full catalog, or several DTC brands tracked on a schedule, request a free sample of your own target. Our team will show you the fields before you commit. Datasets start from $49; see pricing for details. You can also start with the Allbirds sample to see the format.
Author: DataHarbor Team