My weekend hobby project to scrape Reddit for business pain points

I Built a Weekend Project to Find Real Business Pain Points that People were Talking About. It Found 80 in the First Run.

Or: How I Accidentally Created a Goldmine While Procrastinating on Actual Work

You know that feeling when you’re supposed to be doing something productive, but instead you get obsessed with solving a completely different problem? Yeah, that was my weekend.

I was staring at a blank document, trying to write one more proposal or guide. Writer’s block hit hard. So naturally, I did what any reasonable person would do: I spent the entire weekend building an app to find business pain points.

Brilliant? Maybe. Necessary? Probably not. Fun? Absolutely.

(Complete transparency… since most of my work revolves around strategy, operations, and administration, I like to stay in touch with technology by building weekend hobby projects. This is a story about one of those projects…)

The Problem (That Led to Building a Problem-Finder)

Here’s the thing about AI strategy and consultation: you need to solve real problems people actually have. Not problems you think they have. Not problems that sound important in a LinkedIn post. Real, frustrating, “I’m complaining about this on Reddit at 2am” problems.

And where do people complain about their real problems? Reddit.

So I thought: could I scrape Reddit for business pain points, use AI to analyze them, and automatically surface the best opportunities for content?

Narrator: He could.

What I Built (aka My Weekend Rabbit Hole)

The app I built does two main things:

Part 1: Find the Right Subreddits

It searches Reddit for business-related communities and uses AI to assess whether they’re likely to contain actual pain points or just memes about hating Mondays. (Both are valuable, but only one works for this project. Maybe next weekend I’ll tackle the memes.)

The AI looks at each subreddit and answers:

  • Is this relevant for finding business problems?
  • Do people actually complain here, or just post inspirational quotes?
  • What kind of audience hangs out here?

Then I manually review the suggestions and add the good ones to my “monitored” list.

Part 2: Hunt for Pain Points

Once I have my curated list of subreddits, the app:

  1. Searches for posts with keywords like “frustrated with,” “manual process,” “waste time on,” etc.
  2. Uses AI to extract the actual business problem from each post
  3. Generates embeddings to detect duplicate complaints (so I don’t count “invoice follow-ups are terrible” twelve different times)
  4. Scores each pain point on urgency and automation potential

Then I review the results and decide which ones are actually worth considering further.

The First Run: Holy Crap, It Works

I loaded up about 20 business-related subreddits: r/smallbusiness, r/entrepreneur, r/marketing, r/sales, that sort of thing. Set it to search posts from the last week. (Yes, ONLY the last week.) Hit “Go.” And waited.

Five minutes later (thanks, AI APIs), it came back with results.

80 distinct business pain points to consider…

Not 80 posts. 80 unique, actual problems that real people were complaining about, asking for help with, or desperately seeking solutions to.

Some highlights from the first batch:

  • Small business owners spending 10+ hours per week manually following up on unpaid invoices
  • Marketing teams copying data from five different platforms into spreadsheets every week for reports
  • Real estate agents retyping MLS listing data into their CRM because the systems don’t talk to each other
  • Restaurant owners manually calculating food costs and margins in Excel
  • Freelancers losing track of project communications across email, Slack, text messages, and carrier pigeons (okay, maybe not that last one)

Each one scored on how urgent it seemed (based on upvotes, comments, and the language used) and how automatable it was.

The Accidental Goldmine

Here’s what I didn’t expect: the quality.

These weren’t vague complaints like “my business is hard.” They were specific, actionable problems with clear contexts:

  • What they’re doing now (usually manual, time-consuming processes)
  • Why it’s frustrating (losing time, making errors, missing opportunities)
  • What they’ve already tried (often nothing, or basic tools that don’t quite fit)

In other words, perfect information to begin building AI and Automation proposals.

(To make this project even more helpful, I built in a way to create outlines of different types of documents based on the information in the posts and comments (Strategy, Implementation, Executive Summary, etc), with customizable instructions (tones, audience, etc).)

What Makes This Actually Useful

1. It’s Real Market Research

These aren’t problems I think businesses have. These are problems people are actively complaining about, right now, with multiple people agreeing and discussing solutions in the comments.

It’s like having a focus group that runs 24/7 and volunteers their pain points for free.

2. It Finds Patterns

The AI embeddings thing? Game-changer. Instead of reading through hundreds of posts that basically say “invoicing sucks” in different words, it clusters similar problems together and tells me: “Hey, this pain point has shown up in 7 different posts across 3 subreddits. People really care about this.”

That frequency signal is gold for prioritizing content.

3. It Surfaces Non-Obvious Opportunities

Some of the pain points I found were things I never would have thought to write about. Like, there’s apparently a whole community of people struggling with organizing their digital recipe collections for meal planning businesses. Who knew? (Reddit knew.)

The Tech Stack (For the Nerds)

Since this started as a weekend project, I kept it simple:

  • React + TypeScript frontend (because I like seeing pretty things)
  • FastAPI backend (Python is great for this kind of data wrangling)
  • PostgreSQL with pgvector (for that sweet, sweet vector similarity search)
  • Celery + Redis (because scraping takes time and I didn’t want to sit there watching a loading spinner)
  • Amazon Bedrock (Nova Lite for analysis, Titan for embeddings)
  • PRAW (Python Reddit API Wrapper for the heavy lifting)

All running in Docker locally. No cloud deployment, no user auth, no production readiness. Just a scrappy tool that does one thing well.

The vector similarity search is what makes the duplicate detection work. Each pain point gets turned into a 1024-dimension embedding, and then I can find similar ones using cosine similarity. It’s like having a really good intern who reads everything and says “hey, you already captured this complaint.”

What I Learned

1. Weekend Projects Don’t Have to Be Perfect

I didn’t optimize for scale. I didn’t build a fancy UI (although it is quite colorful!). I didn’t even add authentication (it’s local-only, sue me). But it works, and it’s useful, and that’s what matters.

2. AI is Weirdly Good at This

I was skeptical about using AI to assess “relevance” and “urgency,” but it’s actually pretty accurate. Not perfect (I still manually review everything) but good enough to save me hours of reading posts.

3. Reddit is an Untapped Goldmine

Everyone’s on Twitter/X and LinkedIn looking for trending topics. But Reddit? Reddit is where people go to actually solve problems and complain about their day. It’s more honest, more specific, and way more useful for finding real pain points.

4. Scope Creep is Real (and Fun)

This was supposed to be a simple scraper. It turned into a full-stack app with AI integration, vector search, background jobs, and a curation workflow. And OAuth for Reddit, of course. I regret nothing. (Nerd!)

What’s Next?

Honestly? I’m going to use this thing. I’ve already started… and I’ll do more after I work through these 80 business pain points and potential AI and automation use cases.

I might expand it to monitor more subreddits, maybe add sentiment analysis, or build a proposal creator. Or I might just keep it as-is because it already does what I need.

The beautiful thing about weekend projects is they don’t have to become startups. Sometimes they can just be useful tools that make your life easier.

Want to Build Your Own?

The full architecture is pretty straightforward if you want to replicate this:

  1. Find your data source (Reddit, forums, support tickets, customer feedback, etc.)
  2. Use AI to extract insights (what’s the actual problem being described?)
  3. Use vector embeddings to detect duplicates (pgvector makes this stupidly easy)
  4. Score and prioritize (urgency, frequency, automation potential)
  5. Manual curation layer (AI is good but not perfect; you still need human judgment)

The whole thing cost me almost nothing in API fees for the first run. Amazon Bedrock pricing is wild: Titan embeddings are like $0.0001 per 1,000 tokens, and Nova Lite is similarly cheap.

The Real Lesson

The best projects solve problems you actually have.

I didn’t build this because I thought it would be a cool demo or a portfolio piece. I built it because I wanted to keep my development skills sharp.

And now I have 80 business pain points.

And now have a fun story about how I spent my weekend building a robot to find business problems.

Worth it.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *