2026
a multi-marketplace e-commerce dashboard was reporting 100% margin on two of three sales channels. that's not a healthy business — that's a broken parser.
the original dashboard was a single python script: read excel files, calculate numbers, spit out HTML. when the marketplace export format changed (or when certain files didn't have the right columns), the parser didn't crash. it just output zeros. silently.
two channels showed zero cost of goods sold. zero COGS means 100% margin. the dashboard looked great. the numbers were wrong.
the core problem: missing SKU cross-referencing.
the monolith had no validation layer. no intermediate state to inspect. the raw file went in, HTML came out, and nobody checked the middle.
instead of patching the monolith, i rebuilt the entire pipeline as a 6-agent system with inspectable intermediate JSON at every stage.
[1] Collect → download files from cloud storage [2] Analyze → LLM auto-detects file schema [3] Clean → raw files → normalized JSON [4] Margin → calculate profit per channel [5] Generate → JSON → static HTML [6] QC → validate numbers + LLM anomaly check
the key insight: agent 2 (LLM) auto-detects file schema. when marketplace changes their export format, the LLM reads the first 20 rows and maps columns automatically. no hardcoded column names.
but the real fix was in agent 3 (Clean): cross-referencing the Completed Order CSV (which HAS Seller SKU) against the COGS lookup table, then filtering to only orders that appeared in the income file.
the function that caught the discrepancy:
def build_completed_order_sku_map(csv_path):
"""
Read Completed Order CSV
→ {month: {channel: {order_id: [(sku, qty)]}}}
Used to match TikTok/Tokopedia orders
to COGS via Seller SKU.
"""
df = pd.read_csv(csv_path)
mapping = {}
for _, row in df.iterrows():
order_id = row['Order ID']
sku = row['Seller SKU']
qty = int(row['Quantity'])
channel = row['Purchase Channel'].lower()
month = parse_date(row['Created Time']).month
mapping[month][channel][order_id] \
.append((sku, qty))
return mapping
this function bridges the gap: income files give us which orders were settled, completed order CSVs give us which SKUs were in each order, and the COGS lookup gives us cost per SKU.
three data sources, one join. the discrepancy became visible.
after the fix:
the pipeline runs on GitHub Actions. operator uploads files to Google Drive, triggers workflow, deploys to Cloudflare Pages. total cost per run: under $0.03 (LLM calls for schema detection + QC anomaly detection).
the real value isn't just the fix — it's the inspectable intermediate state:
data/raw/ — exact files downloaded from Google Drivedata/schema/ — how the LLM interpreted each file (cached, re-used)data/clean/ — normalized JSON per channel per monthdata/margin/ — final P&L with per-channel breakdownwhen something looks wrong, you don't debug a 800-line monolith. you inspect the specific JSON layer where the problem lives.
data pipelines need validation at every layer, not just at the end.
the v1 monolith was a black box: excel in, html out. when the numbers were wrong, there was nothing to inspect. no intermediate state. no audit trail.
the v2 pipeline generates inspectable JSON at every stage. the QC agent uses both rule-based checks (total not zero, COGS within range) and LLM anomaly detection ("does this margin look weird for this business?").
the lesson: silent failures are worse than crashes. a parser that outputs zeros instead of raising an error will let bad data deploy to production.