Guide to Building a World-Class AI Strategy for Your Custom Software

AI is not a feature. It is an engineering decision with architectural implications, ongoing costs, and quality maintenance requirements. Treat it like infrastructure, not decoration. For more on this, see what's included in development costs.

Editorial note:  Founder quotes throughout this article are composites drawn from multiple EB Pearls engagements. The numbers and decisions are real. Identifying details have been changed.

Why We Wrote This

The AI conversation has shifted from "should I use AI?" to "how do I use AI effectively?" This guide cuts through the hype to answer the practical questions: where does AI add value, which approach is right for your product, how much does it cost, and how do you maintain quality over time.

Introduction: The AI Decision Framework

Not every product needs AI. Not every task in an AI product needs AI. The most expensive AI mistake is using machine learning for a problem that traditional code solves better. The second most expensive mistake is avoiding AI for a problem where it creates a decisive competitive advantage. For more detail, see AI flows: Vibe Code vs. Agents vs. Agentic Workflows.

McKinsey's 2023 State of AI report found that organisations that achieved significant value from AI had one thing in common: they started with a specific business problem, not with the technology. The companies that started with "we need an AI strategy" underperformed those that started with "we need to solve this specific problem, and AI might be the best approach."

When AI Adds Value (And When It Does Not)

The Three AI Approaches

Approach 1: Pre-Built API Integration

Use: GPT-4, Claude, Gemini, or specialised APIs (vision, speech, translation). You send data, the API returns a result. No model training required.

Best for: Products where AI is a feature, not the core product. Content generation, summarisation, classification, chatbot functionality, document extraction.
Cost: AU$10K to $40K integration cost. AU$100 to $5,000+/month API fees (usage-based). Timeline: 2 to 6 weeks.

Advantage: Fastest to market. Lowest upfront cost. Models improve without your effort (the API provider updates them).
Risk: Vendor dependency. Privacy concerns (data sent to external servers). Cost unpredictability at scale.

Approach 2: Fine-Tuned Model

Use: Start with a pre-built model and fine-tune it on your domain data. The model learns your specific use case while retaining general capabilities.

Best for: Products where the generic API output is not domain-specific enough. Medical terminology, legal language, industry jargon, company-specific patterns.

Cost: AU$20K to $80K. Requires a labelled dataset (100+ examples minimum, 1,000+ for best results). Timeline: 4 to 10 weeks.

Approach 3: Custom Model

Use: Train a model from scratch (or from a base model) on your proprietary data. Full control over architecture, data, and deployment.

Best for: Products where AI is the core competitive advantage and you have proprietary data that creates a moat. Recommendation engines at scale, domain-specific prediction models, specialised computer vision.

Cost: AU$80K to $200K+. Requires significant data (thousands to millions of examples). Timeline: 8 to 16+ weeks.

Designing the Human-AI Interaction

The most common AI UX mistake: presenting AI output as final. In v1 of any AI feature, design for human-in-the-loop. The user should be able to see the AI output, understand how it was generated (at a high level), verify it, correct it, and provide feedback that improves future output.

The AI Quality Pipeline

1. Define Evaluation Criteria Before Building
   What does 'good' output look like? For text: accuracy, relevance, tone, length. For classification: precision and recall. For predictions: accuracy and confidence calibration.
   
   

2.  Create a Test Dataset
   50 to 100 representative examples with known correct outputs. This dataset is used to evaluate every model change, prompt update, or API version migration.

3. Automate Evaluation
   Run the test dataset through the AI pipeline on every change. Compare output against expected results. Flag regressions before they reach users.
   
   

4.  Monitor in Production
   Track quality metrics on real user data. AI models degrade over time as the world changes (model drift). Production monitoring catches drift before users notice. For more detail, see from prototype to product.

5. Iterate on Prompts and Data
   Prompt engineering is not a one-time activity. It is an ongoing optimisation process. Collect user corrections, add them to the test dataset, and refine prompts quarterly.

•  AI-appropriate tasks identified (unstructured data, generation, prediction)

•  Approach selected: pre-built API, fine-tuned, or custom model

•  Human-in-the-loop UX designed for v1

•  Evaluation criteria defined before development

•  Test dataset created (50-100 examples with expected outputs)

•  Automated evaluation pipeline built

•  Production quality monitoring in place

•  Ongoing AI maintenance budgeted (15-25% of build cost annually)

Frequently Asked Questions

Free Founder Resources

1.   AI Decision Framework (PDF)
   A one-page decision tree for determining whether AI is right for each task in your product, and which approach to use.

2.  AI Cost Calculator (Google Sheets)
   Estimate integration cost, API fees, and ongoing maintenance for each AI approach. Includes cost scaling projections by user volume.

3. AI Evaluation Template (Notion)
   Pre-built template for defining evaluation criteria, creating test datasets, and tracking quality metrics over time.

Final Thought

AI strategy is not about using AI everywhere. It is about using AI where it creates disproportionate value, choosing the right approach for your stage and data, and building the evaluation infrastructure that ensures quality over time.

The best AI features are invisible. Users do not marvel at the machine learning. They marvel at how the product seems to understand them.