Indexing on-chain data without losing your mind

I have spent a good chunk of the last few years turning raw blockchain data into something a product can actually use. Indexers for a token swap and a bridge while I was at Shiba Inu. A subgraph and on-chain indexing for ElasticSwap, a DeFi exchange. The work sounds dry until the data is wrong at 11pm on a Friday and real users are staring at the wrong balance.

If you are choosing an indexing stack right now, here is what I wish someone had handed me before I started.

Reading the chain is the easy part

The naive version of indexing is a loop that pulls blocks and writes rows. That version works in a demo and falls over in production. The hard parts show up later.

• Reorgs. The block you indexed an hour ago can get orphaned, so your writes have to be reversible.
• Ordering. Logs inside a transaction are not always in the order you assume, and assuming wrong corrupts state quietly.
• Backfill. Indexing from genesis on a busy contract can take days, and you only find out after you have committed to an approach.
• RPC limits. Your provider will rate limit you right when you need throughput the most.

Every tool you pick is really a set of answers to these four problems. Judge them on that, not on the landing page.

When The Graph is the right call

I reach for The Graph when the data is public, mostly read-heavy, and other people will want to query it too. Subgraphs are a known quantity, the schema-to-query path is short, and you inherit an ecosystem that already understands your data shape. For a standard DEX or an NFT collection where the queries are predictable, it gets you to useful fast.

The cost is control. You live inside the mapping model, and anything that wants heavy custom processing or off-chain joins starts to feel like a fight.

When Subsquid earns its keep

I lean on Subsquid when throughput matters, when I need my own database, or when the processing is more than a thin map from event to row. Backfills are fast because it batches aggressively, and once the data lands in my own Postgres I can join, aggregate, and reshape it however the product needs.

At Shiba Inu the indexers were feeding real product surfaces, not a public explorer, so owning the pipeline end to end was worth the extra setup. That is the trade. You get control and speed, and you carry more of the operational weight.

The traps that cost me weekends

• Treating reorgs as an edge case. Handle them on day one or you will rewrite half your pipeline later.
• Underestimating backfill time. Measure it on a real contract before you promise a launch date.
• Trusting a single RPC endpoint. Have a fallback and a backoff, or your indexer dies under load.
• Schema migrations on live data. Plan how you reprocess history before you change a column, not after.

How I choose now

If the data is public and the queries are standard, I start with The Graph and ship. If I need throughput, custom processing, or my own database, I go with Subsquid and accept the operational cost. The wrong move is picking based on what is trendy that month, then discovering the mismatch three weeks in when switching is expensive.

Most indexing pain is not about the tool. It is about treating a distributed, reorg-prone data source like a normal database. Respect that early and the rest gets a lot calmer.