How I stopped worrying about evals: a practical guide to testing AI features for indie hackers

· 6 min read

Most indie hackers test AI features the same way: tweak the system prompt, hit "Generate" three or four times in the sandbox, see a decent result, and push to production.

This works for pet projects. But when your product is a SaaS like VYUD Publisher, where a cron scheduler sends generated posts straight to real Telegram channels, VK groups, or LinkedIn profiles, the approach becomes a ticking time bomb. Sooner or later the LLM will spit out broken markdown to a network that doesn't support it, invent a fact about the client's business, or write 4000 characters instead of a concise post for X.

The enterprise approach, with heavy frameworks and hundreds of benchmarks, doesn't suit a solo developer: it eats weeks of development and thousands of dollars in API costs. Below is a system of lightweight evals (quality checks) you can deploy in an evening, and then sleep soundly while background workers publish content.

1. The deterministic layer: free checks in 5 ms

The most common mistake when adding AI is trying to validate everything with yet another LLM. It's expensive, slow, and non-deterministic. Most critical production failures get caught by plain code before the post ever reaches the database with a scheduled status.

Publisher uses a combination of Pydantic schemas and platform linters:

  • Syntax blacklists by platform. Telegram and the blog handle clean Markdown fine. But feed asterisks and header hashes to VK, Instagram, X, or Odnoklassniki and the output is visual mush. The rule is simple: isolate formatting pipelines and check against a NO_MARKDOWN_PLATFORMS dictionary. Any Markdown headed for those channels is either stripped with something like markdown_to_plain_text or rejected at the generation stage.
  • Social media length limits. If the prompt asked for a 280-character post for X and the model got carried away and produced 350, the post will fail with an API error at publish time. The validator checks hard limits character by character right after generation.
  • Forbidden stop-patterns. Plain regexes for filler phrases that give away a bot: "Sure! Here is a post for your blog:", "Hope this text was helpful", or unfilled placeholders like [Insert link].

If the result fails the deterministic linter, the backend doesn't even save a draft. It retries immediately with a stricter system message.

2. Content plans: strict schemas against structural hallucinations

Generating a content plan a week or a month ahead is a hard task for a model. Models love to lose days of the week, duplicate topics, or suddenly switch from JSON to list markup.

The quick-launch solution is Structured Outputs with a strict schema contract:

from pydantic import BaseModel, Field
from typing import List
 
class ContentPlanItem(BaseModel):
    day_offset: int = Field(ge=1, le=30)
    topic: str = Field(min_length=10, max_length=120)
    hook: str = Field(description="First line of the post, the hook")
    call_to_action: str
    target_platform: str
 
class ContentPlan(BaseModel):
    strategy_summary: str
    schedule: List[ContentPlanItem] = Field(min_items=7, max_items=30)

The eval for these features is a synthetic run across 10 different niches, from a local coffee shop to a B2B SaaS. The metric is binary: Schema Pass Rate, the share of responses that parsed into a valid object on the first try. If it drops below 98%, the prompt goes back for revision: lower the temperature, remove ambiguous instructions.

3. LLM-as-a-Judge: cheap quality assessment without the academia

Once syntax and schema are valid, the question of meaning arises: is the text alive, does it follow the user's editorial policy, is it free of blatant nonsense?

Instead of subjective "rate quality from 1 to 10" scales, which confuse the models themselves, a quick binary checklist run by a cheap judge model (a Haiku- or mini-class model) works better.

The indie hacker's golden dataset. You don't need thousands of examples. A JSON file with 20-30 real cases where generation previously stumbled is enough: tricky niches, a specific tone of voice, long context.

The judge gets the original context, the target audience, the generated post, and a list of binary questions (True/False):

  1. Relevance: does the post answer the stated topic without drifting into abstract philosophy?
  2. Hallucination Check: does the text contain specific facts or figures that were not in the provided context?
  3. Tone Match: does the style match the requested tone (expert, provocative, friendly)?

Running these 30 tests before deploy takes under a minute and costs a few cents. If the share of "True" drops by even 5% after a prompt change, the change is rolled back.

4. Image generation: control at the entrance and the exit

Evaluating generated images programmatically is the hardest part. A user won't forgive an abstract mess with six fingers on a hand instead of an illustration for an IT article.

Instead of building complex CV analysis for finished images, I focus on two stages:

  1. Prompt sanitization (pre-eval). The model that writes prompts for diffusion networks goes through a filter for negative styles, artifacts, and stop words. If a user writes "make a stylish banner about a product launch", the intermediate LLM should expand it into a detailed technical description for the image generator, with the aspect ratio (16:9 for social, 1:1 for the feed) and the color palette fixed.
  2. Post-generation check. Basic metadata validation: successful upload to Supabase Storage (post-media), valid response headers, non-zero file size, correct mime-type. If the generator returns a broken URL or times out, the scheduler moves the status to retrying with exponential backoff, without breaking the publishing chain.

The architecture of safe production

The last line of defense is built into the lifecycle of entities in the database. In VYUD Publisher a post goes through a strict chain of states:

[Prompt + Context]
       │
       ▼
[Generation] ──(linters + Pydantic)──► [Draft / Scheduled]
                                              │
                                     (CAS-claim by scheduler)
                                              │
                                              ▼
                                        [Publishing]
                                              │
                     ┌────────────────────────┴────────────────┐
                     ▼                                         ▼
          [Success on social media]                [Partial / Full Error]
                     │                                         │
                     ▼                                         ▼
               [Published]                  [Retrying (up to 3 times)] ──► [Failed]
  1. Draft and Scheduled are separate. If the model's confidence or the validation result lands in a gray area, the post is not scheduled for automatic publication. It stays a draft and waits for one click of confirmation from the user.
  2. Atomic CAS-claim. Before sending, the scheduler runs an atomic UPDATE ... WHERE status = 'scheduled' RETURNING *. Even with parallel workers, no post goes to a channel twice.
  3. Graceful degradation. If an external provider fails on an attachment or the text, the error is logged in last_error with a clear, human-readable description instead of a raw stack trace.

Testing an AI product solo doesn't require academic metrics like BLEU or ROUGE. Three things are enough: strict linters on input, Pydantic schemas in the middle, and 30 synthetic tests through Claude or GPT on output. With them I can sleep soundly while LLMs run clients' social media on autopilot.

Tags:EvalsAI

← All posts