January 12, 2026 at 2:39 PM
← back to writings

The N+1 Query Problem: A Painful Lesson from Building FinBuddy

When I was building FinBuddy, my stock analysis platform that crunches data for hundreds of stocks across the NSE universe, I ran headfirst into one of the most insidious performance problems in database-driven applications. What started as a simple feature to display stock fundamentals turned into a masterclass in database optimization.

Let me tell you about the N+1 query problem and how I solved it.

What Exactly Is This Problem?

Picture this: you're building a feature that needs to fetch a list of items from your database, and each item has some related data. Seems straightforward, right? Here's where things go sideways.

The N+1 query problem happens when your application makes one initial query to grab your main data (let's say, a list of stocks), and then makes N additional queries to fetch related information for each of those items. So if you're fetching 100 stocks and want to show their sector information, you end up making 101 database queries instead of just one or two.

Think of it like going to the grocery store. The efficient approach is to grab everything on your list in one trip. The N+1 approach is like driving back to the store separately for every single item on your list. Exhausting and wildly inefficient.

How I Discovered This Performance Killer

During the early days of FinBuddy, I was working on a dashboard that would display fundamental metrics for stocks in a user's watchlist. The feature seemed simple enough:

  1. Fetch all stocks in the watchlist
  2. For each stock, display its latest financial ratios
  3. Show sector-specific benchmarks

In my local development environment with maybe 10-15 stocks, everything felt snappy. The page loaded instantly. I was happy with the implementation and pushed it to our staging environment.

Then reality hit.

When I tested with a realistic dataset analyzing 500 stocks from the NSE universe, the dashboard took over 8 seconds to load. Eight seconds! For a fintech application where users expect instant insights, this was unacceptable.

I dove into the Postgres logs and saw the smoking gun: my application was making 501 database queries for a single page load. One query to fetch the watchlist stocks, and then 500 individual queries to grab financial ratios for each stock.

Why This Problem Matters (And Why It's So Common)

The N+1 query problem is particularly sneaky because:

It hides during development. With small test datasets, those extra queries happen so fast you don't notice. It's only when you scale to production volumes that the problem explodes.

ORMs make it easy to fall into this trap. Modern frameworks like Django, Rails, or SQLAlchemy use lazy loading by default. This means they only fetch related data when you actually access it, which sounds efficient but often leads to the N+1 pattern without you realizing it.

The performance impact is exponential. One extra query might add 5ms of latency. But when you're dealing with hundreds or thousands of records, those milliseconds compound into seconds of delay.

In my case, each individual query to Postgres was taking about 15ms on average. That doesn't sound like much, but 500 queries × 15ms = 7.5 seconds just in database time, not counting network overhead and application processing.

This kind of performance degradation leads to:

  • Frustrated users who abandon your application
  • Excessive load on your database servers
  • Higher infrastructure costs as you try to scale your way out of the problem
  • Angry phone calls from your operations team when the database starts falling over during peak hours

The Solution: Eager Loading to the Rescue

The fix for the N+1 problem is conceptually simple but requires you to think differently about how you fetch data. Instead of letting your ORM lazily load related data, you explicitly tell it to fetch everything you need upfront.

This is called eager loading, and it's a game-changer.

How I Fixed It in FinBuddy

In my Postgres-backed application, I was using SQLAlchemy as the ORM. Here's what my original, problematic code looked like:

# The N+1 approach (inefficient)
watchlist_stocks = session.query(Stock).filter_by(watchlist_id=user_watchlist_id).all()

for stock in watchlist_stocks:
    # This line triggers a NEW query for each stock!
    print(f"{stock.symbol}: P/E = {stock.financial_ratios.pe_ratio}")

Every time I accessed stock.financial_ratios, SQLAlchemy fired off another query to fetch that data. With 500 stocks, that's 500 additional queries.

The solution was to use SQLAlchemy's joinedload to fetch everything in one go:

# The optimized approach (efficient)
watchlist_stocks = (session.query(Stock)
    .options(joinedload(Stock.financial_ratios))
    .filter_by(watchlist_id=user_watchlist_id)
    .all())

for stock in watchlist_stocks:
    # No additional query! Data is already loaded
    print(f"{stock.symbol}: P/E = {stock.financial_ratios.pe_ratio}")

This single change reduced my query count from 501 to just 1. The page load time dropped from 8 seconds to under 300ms. That's a 25x performance improvement with just one line of code change.

Other Approaches That Work

Eager loading isn't the only tool in your arsenal. Depending on your situation, you might also consider:

Batching queries: If you can't use eager loading for some reason, you can at least batch your queries. Instead of querying for each stock individually, fetch data for all stocks in one query using WHERE stock_id IN (1, 2, 3, ...).

Caching: For data that doesn't change frequently, caching can help. I implemented Redis caching for sector benchmark data in FinBuddy since that only updates daily. But remember, caching is a band-aid, not a cure. Fix the underlying query pattern first.

Database views or materialized views: For complex data aggregations that you access frequently, consider pre-computing results in a database view. I used this approach for some of FinBuddy's more complex screening queries.

Denormalization: Sometimes, storing redundant data is the right call. If you're constantly joining the same tables, consider denormalizing your schema to avoid those joins altogether.

How to Spot N+1 Queries Before They Bite You

After that painful experience, I made N+1 detection part of my development workflow. Here's what I do:

Enable query logging in development. I configure my local environment to log every SQL query. If I see a pattern of similar queries in a loop, I know I've got a problem.

Use database profiling tools. Tools like Django Debug Toolbar or Rails' Bullet gem automatically detect N+1 patterns and warn you during development.

Write performance tests. I added tests that specifically check query counts for critical endpoints. If a test expects 5 queries but suddenly sees 500, it fails loudly.

Code review with a critical eye. Whenever I see a loop that accesses related model data, I immediately ask: "Is this going to trigger multiple queries?" More often than not, the answer is yes.

Lessons Learned

Building FinBuddy taught me that database performance isn't something you bolt on at the end. It needs to be baked into how you think about data access from day one.

The N+1 query problem is just one of many performance pitfalls, but it's one of the most common and most damaging. The good news is that once you know what to look for, it's straightforward to fix.

These days, every time I write code that fetches related data, I pause and think about the query pattern. Is this going to scale? Am I being smart about how I'm accessing the database? That habit has saved me countless hours of performance debugging.

If you're building any kind of data-intensive application, make eager loading your default approach. Your database will thank you, your users will thank you, and your future self will definitely thank you when you're not scrambling to fix performance issues at 2 AM.

Trust me on this one. I learned it the hard way so you don't have to.