How do you normalize product data from different retailer websites into one schema?
Define one target schema with three levels (product, variant, and each store's offer), extract every store from its most structured source first (JSON-LD, then platform data such as Shopify's product JSON, then HTML), and map each field with explicit rules for prices, availability, sizes, colors and categories. Then match the same item across stores by GTIN, falling back to brand plus part number, and re-extract on a schedule, because stores change their pages and a normalized record goes stale.
Last updated · Vendor facts checked
Step 1
Start from the schema, not the stores
Pick the target before writing any extractor. Model three levels: the product (what the item is), the variant (the color and size combination someone actually buys), and the offer (one store's price and stock for that variant). Most normalization bugs come from flattening these: a price with no variant, or a "product" that is really one color of one.
{
"product": {
"title": "Men's Tree Runner",
"brand": "Allbirds",
"category_path": ["Apparel & Accessories", "Shoes", "Sneakers"],
"attributes": { "material": ["eucalyptus fiber"] },
"identifiers": { "gtin": null, "mpn": null, "group_id": "mens-tree-runners" }
},
"variants": [
{
"sku": "33179624669264",
"options": { "color": "Jet Black (White Sole)", "size": "8", "size_system": "US" },
"color_family": "black",
"gtin": null,
"offer": {
"store": "allbirds.com",
"url": "https://www.allbirds.com/products/mens-tree-runners?variant=33179624669264",
"price": { "amount": "100.00", "currency": "USD" },
"original_price": null,
"availability": "in_stock",
"retrieved_at": "2026-11-06T14:02:11Z",
"source": "json_ld"
}
}
]
}- Prices are a decimal amount plus an ISO 4217 currency code, never a display string. Keep the current and the original price.
- Availability is an enum per variant:
in_stock,out_of_stock,preorder, orunknown. Unknown is not the same as out of stock. - Provenance on every offer: the URL, when it was retrieved, and which source each field came from.
- Raw beside normalized: keep the store's own label ("Oatmeal Heather") next to the value you mapped it to (
beige), so you can re-map later without re-crawling.
You don't have to invent field names. schema.org's Product, ProductGroup and Offer, and Google's Merchant Center product data specification (gtin, mpn, item_group_id, size_system), are good starting vocabularies, and stores already publish data in their shape.
Step 2
Extract from the most structured source on each page
A retailer's product page usually carries the same product several times over. Read the sources in order of reliability, and fall back only when a field is missing:
- JSON-LD (
<script type="application/ld+json">with schema.orgProduct,ProductGrouporOffer). The most common source, but it often describes only the default variant, or carries a price that lags the page. - Platform data. Shopify stores serve
/products/<handle>.jswith every variant, its price in cents and its availability. Other platforms expose their own JSON endpoints or embed state in the page, such as__NEXT_DATA__. - The page's own API calls for price and stock, which are often the only place per-size stock appears.
- Rendered HTML last. Selectors are the first thing to break when a store redesigns.
Record which source each field came from. When sources disagree (JSON-LD says $120, the page shows $96 on sale), prefer the one closest to what a shopper sees and log the disagreement. Disagreement rates per store are the earliest signal that an extractor has drifted.
Step 3
Map every field with explicit rules
| Field | What varies across stores | Normalize to |
|---|---|---|
| Price | $1,299.00, 1299, 129900 (cents), "from $49", VAT-inclusive EU prices | Decimal amount + ISO 4217 currency; "from" prices as ranges; note whether tax is included |
| Availability | InStock, in_stock, "Only 2 left", per-size flags, nothing at all | Enum per variant; missing becomes unknown |
| Color | "Oatmeal Heather", OAT, a swatch image and no text | The store's label, plus a color family from a fixed list |
| Size | M, "Medium", 38, "US 8 / EU 39", 32x30 | The value plus its size system (US, EU, UK, alpha, waist × inseam) |
| Category | Store breadcrumbs such as "Women > Sale > Tops" | One taxonomy (for example Google's product taxonomy); drop merchandising nodes like Sale and New |
| Brand | NIKE, "Nike Inc.", "Nike Sportswear", a retailer's own label | A canonical brand ID with an alias table |
| Units | "16 oz", "1 lb", "453 g" | One unit system, with the original kept |
| Images | Relative URLs, CDN resize parameters, per-variant swatches | Absolute URLs, attached to the variant when the store does that |
Use deterministic parsers for prices, currencies and units. An LLM earns its place on the long tail: mapping store categories into your taxonomy and pulling attributes such as material or fit out of descriptions. Validate its output against your schema's enums, and keep the input so you can re-run it when the prompt or model changes.
Step 4
Match the same product across stores
Normalized records only become comparable once you know which ones are the same item. Use a ladder, strongest key first, and stop at the first rung that answers:
- GTIN (UPC, EAN, ISBN). Left-pad every GTIN to 14 digits and validate the check digit before trusting it. Many stores publish none, and some publish one variant's GTIN on the whole product.
- Brand + MPN (manufacturer part number), after normalizing the brand and stripping punctuation and spaces from the MPN.
- Brand + model + key attributes (model tokens from the title, capacity, color, size), scored for similarity, with a threshold you tune on a few hundred labeled pairs.
- Image similarity as a tiebreaker only: retailers reuse manufacturer images across colors and sizes.
Match variants, not pages. "128 GB" and "256 GB" often share one product page, and a page-level match would compare their prices as if they were the same thing.
Step 5
Model variants explicitly
The same jacket is one page with a color picker on one store, five pages (one per color) on another, and one listing per color and size on a marketplace. Normalize all three into one product group whose variants are keyed by option values, each with its own SKU, GTIN, price, stock and image. Group with the store's own group ID when there is one (schema.org productGroupID, the Shopify product ID, Google's item_group_id); otherwise group by brand, a normalized title stem and shared images.
Stock belongs to the variant. A product that reads "in stock" because one size is left is the most common wrong answer a shopping assistant gives.
Step 6
Keep it fresh, and validate every run
- Refresh on a schedule that matches how fast things change: daily for most catalogs, more often for price-sensitive categories and during sales, and just in time before you show a price to someone.
- Detect changes instead of re-crawling everything: sitemap
lastmod,ETagandLast-Modifiedheaders, and product feeds where a store offers one. - Validate each run against the schema and alert on drift. A sudden share of products with no price, no variants or a price of zero usually means a store redesigned its pages.
- Keep `retrieved_at` on every offer and show it. A price without a timestamp can't be trusted.
Options
Ways to get normalized product data
You can build every step above yourself, use an extraction API for steps 1 and 2 and keep matching and freshness in house, or use a product data service that does all of it for the stores it covers.
| Option | What it is | Good at | Trade-offs | Pricing |
|---|---|---|---|---|
| Build it yourself | Your own crawlers and per-site parsers, or a headless browser plus an LLM that fills your schema. | Full control over fields, sources and refresh schedule. | Every store is a parser to maintain. Anti-bot, rendering, matching and freshness are all yours. | Engineering time, proxies, browsers and LLM tokens |
| Firecrawl | Scrapes any URL into Markdown, JSON shaped by a schema you pass, or a product object. Also searches the web. | Any page on the open web, including stores no product index covers. The product format returns variants with price and availability without an LLM. | One page per request. You supply the URLs, match the same product across stores, and schedule refreshes. Scrapes are cached for up to 2 days by default. | 1 credit per page; JSON mode adds 4 |
| Zyte API | Web data API: send a URL, get an AI-extracted product record, with ban handling and browser rendering built in. | Dependable extraction at scale from sites you choose, including hard anti-bot targets. | No cross-store product search. You bring the URLs, schedule refreshes, and match products yourself. | Per successful response, priced by site difficulty |
| Apify | Marketplace of scraping Actors, including a general E-commerce Scraping Tool and many store-specific Actors. | Configurable bulk collection, keyword search across marketplaces, and exports to JSON, CSV or Excel. | You run and schedule Actors and store the datasets. Field names differ between Actors, so mapping them to one schema is yours. | Pay per result, from $1 / 1k product details |
| Datafiniti | Product database that merges listings sharing an identifier (UPC, EAN, ASIN) into one record per product. | Identifier-keyed records with price observations across merchants, bulk downloads for analytics, and an MCP server. | One merged record per product rather than each store's current variants and stock. Queries use field filters, not plain language. | Per record, from $99 / month for 5,000 |
| Channel3 | Product search and URL lookup API for AI shopping, with an MCP server. Reports about 100M products. | Agent product search with merchant offers and commission-earning purchase links out of the box. | Answers from its own catalog, so check that it covers the retailers you need. Affiliate offers are part of the model. | 1,000 free credits / month, then about $0.007 per call |
| StoreRouter | Index of covered stores' catalogs behind search and parse, in one enriched schema. REST, SDKs and MCP. | Cross-store search and normalized product detail (variants, per-size stock, GTIN where the store publishes it) without running crawlers. | Answers best from covered stores. Other URLs can fall back to a shallower live parse, and onboarding a new store takes hours to days. | $2 / 1k parse · $14 / 1k search; $5 free credit / month |
How to choose
- A few stores and full control over the schema: build it. The six steps above are the whole job.
- Long-tail stores or non-commerce pages, in your own schema: an extraction API such as Firecrawl or Zyte, with your own matching and refresh jobs.
- Configurable bulk collection from specific marketplaces: Apify Actors, with your own field mapping.
- Identifier-keyed datasets with price history for analytics: Datafiniti.
- An app or agent that needs current products from many stores in one schema now: a product data API such as StoreRouter or Channel3. Check each one's coverage of the stores you need first.
Prices and capabilities are from each vendor's public docs and pricing pages as of November 6, 2026. Side-by-side detail: StoreRouter comparisons.
StoreRouter
Where StoreRouter fits
StoreRouter runs steps 2 to 6 for the stores it covers. It crawls them, extracts from structured sources first, maps every product into one schema (variants with per-size stock, current and original price, GTIN and group ID where the store publishes them, a normalized category path and enriched attributes such as material and style), and re-crawls a store when it changes. Call parse with any product URL and the same shape comes back, whichever store it came from.
A practical way to adopt it: use it as the normalized source for covered stores, keep your own pipeline for the rest, and compare both outputs on a sample of your URLs first.
curl -sS https://api.storerouter.ai/v1/products/parse \ -H "Authorization: Bearer $STOREROUTER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://www.allbirds.com/products/mens-tree-runners"}'// npm install @storerouter/sdk
import { OctogenClient } from "@storerouter/sdk";
const client = new OctogenClient({
apiKey: process.env.STOREROUTER_API_KEY,
baseUrl: "https://api.storerouter.ai/v1",
});
// Product pages from different stores...
const urls = [
"https://www.allbirds.com/products/mens-tree-runners",
// add product URLs from other stores here
];
for (const url of urls) {
// ...come back in one schema (POST /v1/products/lookup, the same as /parse).
const { catalogDisplayName, product } = await client.lookupProduct(url);
console.log(catalogDisplayName, product.title, product.brand?.name);
console.log(product.currentPrice, product.currency, product.identifiers?.gtin);
for (const variant of product.variants ?? []) {
console.log(" ", variant.color?.label, variant.size, variant.inStock);
}
}# Claude Code; any MCP client works with the same URL (OAuth, no key to paste)claude mcp add --transport http storerouter https://mcp.dev.storerouter.ai/mcp# Then ask your agent:# Parse these product URLs and give me title, brand, price, currency and in-stock sizes for each as one table.StoreRouter normalizes into its own schema, not yours, so you still map its fields to your model, once rather than once per store. It does not cover every store: check catalog coverage for the domains you need. A URL outside coverage can still resolve through a shallower live parse of the page's structured data.
FAQ
Frequently asked questions
- What is product data normalization?
Turning product records from different sources into one consistent shape: the same field names, the same units and currencies, the same category taxonomy, and the same product → variant → offer structure, so products from different stores can be searched, filtered and compared together.
- Which schema should I normalize product data into?
Start from schema.org Product, ProductGroup and Offer, and borrow field definitions from Google's Merchant Center product data specification (gtin, mpn, item_group_id, size_system). Stores already publish data in those shapes, which keeps mapping simple. Add your own fields only for what your app needs, such as a color family or a normalized category path.
- How do I match the same product across different stores?
Use GTIN first, after padding it to 14 digits and validating its check digit. Fall back to brand plus manufacturer part number, then to brand, model and key attributes with a tuned similarity threshold. Match at the variant level, and send low-confidence matches to review rather than merging them on title similarity alone.
- Can an LLM normalize product data for me?
It helps with the long tail: mapping store categories into your taxonomy and extracting attributes like material or fit from free text. Prices, currencies, units and identifiers are better parsed deterministically. Validate LLM output against your schema's enums and keep the raw input, so you can re-run it when the prompt or model changes.
- How often should normalized product data be refreshed?
It depends on the field. Titles, descriptions and attributes change rarely, while prices and stock change daily or faster. Refresh offers daily as a baseline, more often for price-sensitive categories, and check just in time before showing a price to a user.
- How is this different from web scraping?
Scraping gets you each store's data; normalization makes it comparable. A scraper or extraction API covers fetching and field extraction (steps 1 and 2 above). Mapping rules, product matching, variant modeling and freshness are separate work you own unless you use a service that does them.