Free lesson · AI Engineering

When to Use AI (and When Not To)

Medium14 min readaillmrag

When to Use AI (and When Not To)

The Feature We Killed After Six Months of Work

In 2023, my team at Meta built an AI-powered code comment generator. The idea was elegant: a developer writes a function, clicks a button, and an LLM generates a docstring explaining what the function does. We integrated it with the internal IDE. We fine-tuned the model on Meta's codebase. We ran evals that showed 91% of generated comments were technically accurate. We shipped it to 2,000 engineers in a controlled rollout.

Six months later, we killed it. Not because it did not work - it worked exactly as designed. We killed it because the problem it solved was not worth solving with AI. Here is what we discovered.

Engineers who wrote clean code with descriptive function names did not need generated comments - the code was already self-documenting. Engineers who wrote unclear code needed to refactor, not to paper over the confusion with an AI-generated explanation. The 9% of comments that were inaccurate created a new problem: engineers started trusting generated comments without reading the code, which led to three production incidents where incorrect comments misled debugging efforts. And the feature cost $47,000 per month in inference - for a tool that, when we measured carefully, saved each developer approximately 90 seconds per day.

That is the AI tax in action. Every AI feature carries costs that deterministic software does not: inference costs, quality monitoring, hallucination risk, the cognitive overhead of probabilistic outputs, and the organisational cost of maintaining a system that is right most of the time but wrong in unpredictable ways. The question is never "can AI do this?" The question is "does AI do this well enough, cheaply enough, and reliably enough to justify the tax?"

This topic gives you the decision framework for answering that question before you build.

The AI Tax: What Every AI Feature Actually Costs

Before evaluating whether a problem is a good fit for AI, you need to understand the full cost of adding AI to a product. Most teams dramatically underestimate these costs because they measure only inference spend and ignore the operational overhead.

The seven components of the AI tax:

1. Inference cost. The direct cost of model calls. This is the cost most teams plan for. At production scale, it ranges from negligible (a few cents per day for simple classification) to substantial ($50,000-$500,000 per month for high-volume generation). The key variable is cost per call multiplied by volume multiplied by average token count.

2. Quality monitoring cost. AI outputs require ongoing quality evaluation that deterministic features do not. Someone - a team, a system, or both - must continuously assess whether the AI's outputs remain acceptable. This includes building evaluation pipelines, maintaining golden test sets, reviewing flagged outputs, and investigating quality regressions. At Meta, our AI quality monitoring for a single feature consumed roughly 0.5 FTE of engineering time per quarter.

3. Hallucination liability. When a deterministic feature has a bug, it produces the same wrong output every time and the bug is reproducible. When an AI feature hallucinates, it produces different wrong outputs for different inputs, the errors are not reproducible, and each one may affect a different user. The support burden, the legal exposure (for regulated industries), and the trust damage are qualitatively different from traditional software bugs.

4. Latency overhead. LLM calls add 500ms to 30 seconds of latency to any interaction. For features where users expect sub-100ms responses (autocomplete, search-as-you-type, button clicks), this latency is disqualifying without caching or a streaming architecture. Adding streaming and caching is additional engineering complexity that the deterministic version of the feature would not require.

5. Non-determinism management. The same input can produce different outputs. This means automated testing is harder (you cannot assert exact outputs), debugging is harder (reproducing a user-reported issue requires reproducing the exact model state), and user expectations are harder to set (users are accustomed to software that behaves identically every time).

6. Provider dependency. Your feature depends on an external model provider's availability, pricing, and API stability. Provider outages become your outages. Provider price increases become your cost increases. Provider deprecations require you to migrate. This dependency does not exist when you write your own deterministic code.

7. Maintenance trajectory. Deterministic features, once shipped, require maintenance proportional to code change frequency. AI features require maintenance proportional to the world's change frequency - because the model's relevance degrades as the world drifts from its training data, because user expectations evolve, and because new models with different capabilities and pricing continuously become available. The maintenance curve is steeper and less predictable.

graph TD
    subgraph Visible_Costs["Visible Costs (what teams plan for)"]
        IC["Inference Cost\n$X per 1K calls"]
        DT["Development Time\n(prompt engineering, integration)"]
    end
    
    subgraph Hidden_Costs["Hidden Costs (what teams discover later)"]
        QM["Quality Monitoring\n~0.5 FTE per feature per quarter"]
        HL["Hallucination Liability\nsupport burden + trust damage"]
        LO["Latency Overhead\nstreaming/caching infrastructure"]
        ND["Non-Determinism\ntesting + debugging complexity"]
        PD["Provider Dependency\noutages + price changes + deprecations"]
        MT["Maintenance Trajectory\nsteeper than deterministic features"]
    end
    
    subgraph Total["Total AI Tax"]
        T["Sum of all 7 components\nOften 3-5x the visible cost alone"]
    end
    
    Visible_Costs --> Total
    Hidden_Costs --> Total
    
    style Visible_Costs fill:#166534,stroke:#86efac,stroke-width:2px
    style Hidden_Costs fill:#991b1b,stroke:#fca5a5,stroke-width:2px
    style Total fill:#312e81,stroke:#a5b4fc,stroke-width:2px

This diagram divides the AI tax into visible costs and hidden costs. Visible costs include inference cost and development time - these are what teams typically plan for. Hidden costs include quality monitoring, hallucination liability, latency overhead, non-determinism management, provider dependency, and maintenance trajectory. All seven components sum to the total AI tax, which is often three to five times the visible cost alone.

The Decision Framework: Five Questions Before You Build

Before committing to an AI feature, ask these five questions in order. If the answer to any of them is unfavourable, the feature should be reconsidered or redesigned before proceeding.

Question 1: Is the Problem Tolerant of Errors?

This is the most important filter. Some problems tolerate occasional errors gracefully. Others do not. The distinction determines whether AI is even viable for the use case.

Error-tolerant problems: Content recommendations (a wrong recommendation is skipped, not catastrophic). Draft generation (the user reviews and edits the output). Search ranking (a slightly suboptimal ranking is rarely noticed). Classification for routing (a misrouted item gets rerouted by a human). Summarisation for quick review (the user can read the source if the summary seems off).

Error-intolerant problems: Financial calculations (a wrong number in an invoice is a legal and trust problem). Access control decisions (granting access incorrectly is a security breach). Medical dosage recommendations (an error can cause physical harm). Legal document generation (an incorrect clause can create liability). Data deletion (deleting the wrong records is irreversible).

The spectrum is not binary - it is a gradient. The key question is: what is the cost of the AI being wrong, and how quickly can the error be caught and corrected? If a wrong output costs $0.10 to fix (user clicks "regenerate"), the error tolerance is high. If a wrong output costs

For error-intolerant problems, AI can still participate - but as a suggestion layer, not a decision layer. The AI proposes, a human disposes. This changes the UX pattern (you need review and approval flows), the latency expectations (the human review adds time), and the value proposition (the AI saves time, not replaces judgment).

Question 2: Does the AI Version Substantially Outperform the Non-AI Version?

"Substantially" is the operative word. A marginal improvement does not justify the AI tax.

The comparison must be fair. Compare the AI version against the best non-AI alternative, not against doing nothing. If a regex-based email classifier achieves 88% accuracy and an LLM-based classifier achieves 93% accuracy, the AI adds 5 percentage points - but at 100x the cost per classification. Whether those 5 points are worth the cost depends entirely on the downstream impact of the errors.

Quantify the improvement. "The AI version is better" is not a decision-worthy statement. "The AI version reduces average customer support response time from 4.2 minutes to 1.8 minutes, saving $380,000 per year in agent time, at an inference cost of

The 10x rule of thumb. At Meta, we informally applied a 10x rule: the AI version should deliver at least 10x the value of the AI tax for the feature to be clearly justified. If the feature saves

Question 3: Is the Volume High Enough to Justify the Infrastructure?

AI features require infrastructure that deterministic features do not: prompt management, evaluation pipelines, caching layers, monitoring dashboards, failover systems. This infrastructure has a fixed cost regardless of feature usage volume.

The infrastructure break-even calculation: If the quality monitoring and infrastructure cost $5,000 per month and the feature handles 100 requests per day, each request carries

Low-volume features should share infrastructure with other AI features (through an AI gateway, shared evaluation pipeline, shared monitoring) or should not use AI at all. A feature that serves 50 users per day does not justify its own prompt evaluation pipeline.

Question 4: Can You Define Success Criteria Before Building?

This is the question that kills the most AI projects, because many teams start building before they know what "good enough" means.

Success criteria must be specific and measurable:

"The AI should generate good summaries" is not a success criterion. "The AI should generate summaries that are rated 4+ out of 5 by users in blind evaluation, with a factual accuracy rate above 95% as measured by an automated fact-checking pipeline" is a success criterion.

"The AI should be fast" is not a success criterion. "P95 latency for summary generation should be under 3 seconds for documents up to 10,000 words" is a success criterion.

"The AI should be cost-effective" is not a success criterion. "Total AI inference cost should remain under $0.02 per user interaction at the projected 12-month DAU" is a success criterion.

If you cannot define success criteria before building, you are conducting an experiment, not building a feature. Experiments are valuable - but fund and manage them as experiments, with explicit time-boxes and kill criteria, not as product features with indefinite investment.

Question 5: Do You Have the Evaluation Infrastructure?

You cannot ship what you cannot measure. Before building the feature, confirm that you have - or can build - the evaluation infrastructure to measure its quality continuously in production.

This means: a golden test set of at least 200 examples with ground-truth labels. An automated evaluation pipeline that runs against the golden test set on every prompt change. A production quality monitoring system that samples live traffic and flags degradation. A human review process for cases where automated evaluation is insufficient.

Without this infrastructure, you will ship an AI feature that you cannot monitor, cannot debug, and cannot improve. It will degrade silently, and you will not know until users complain - or leave.

graph TD
    Q1{"Q1: Error-tolerant\nproblem?"}
    Q2{"Q2: AI substantially\noutperforms non-AI?"}
    Q3{"Q3: Volume justifies\ninfrastructure?"}
    Q4{"Q4: Success criteria\ndefined?"}
    Q5{"Q5: Eval infrastructure\navailable?"}
    
    BUILD["✓ BUILD the AI feature"]
    DONT["✗ DO NOT build with AI"]
    SUGGEST["△ AI as suggestion layer\n(human-in-the-loop)"]
    EXPERIMENT["◇ Run as time-boxed\nexperiment first"]
    SHARED["□ Use shared AI\ninfrastructure"]
    
    Q1 -->|"Yes"| Q2
    Q1 -->|"No"| SUGGEST
    Q2 -->|"Yes (10x+ value)"| Q3
    Q2 -->|"No / marginal"| DONT
    Q3 -->|"Yes (>1K/day)"| Q4
    Q3 -->|"No"| SHARED
    Q4 -->|"Yes"| Q5
    Q4 -->|"No"| EXPERIMENT
    Q5 -->|"Yes"| BUILD
    Q5 -->|"No"| EXPERIMENT
    
    style BUILD fill:#166534,stroke:#86efac,stroke-width:2px
    style DONT fill:#991b1b,stroke:#fca5a5,stroke-width:2px
    style SUGGEST fill:#854d0e,stroke:#fde047,stroke-width:2px
    style EXPERIMENT fill:#1e3a5f,stroke:#7dd3fc,stroke-width:2px
    style SHARED fill:#312e81,stroke:#a5b4fc,stroke-width:2px

This decision tree shows five sequential questions for evaluating whether to build an AI feature. Question 1 asks if the problem is error-tolerant; if no, use AI as a suggestion layer with human-in-the-loop. Question 2 asks if AI substantially outperforms the non-AI alternative; if no, do not build with AI. Question 3 asks if volume justifies the infrastructure; if no, use shared AI infrastructure. Question 4 asks if success criteria are defined; if no, run as a time-boxed experiment first. Question 5 asks if evaluation infrastructure is available; if yes, build the AI feature; if no, run as a time-boxed experiment.

The Problems AI Is Actually Good At

After applying this framework to dozens of features at Meta, patterns emerge. Certain problem types consistently pass all five questions. Others consistently fail.

AI excels at these problem types:

Unstructured-to-structured transformation. Converting free-text into structured data - extracting entities from documents, classifying text into categories, parsing resumes into fields, converting natural language queries into SQL or API calls. These problems have clear evaluation criteria (did the extraction match the ground truth?), tolerate errors (a wrong extraction can be corrected), and the non-AI alternatives (regex, rule engines) are brittle and expensive to maintain.

Content generation from templates or constraints. Generating product descriptions from attribute tables, writing email drafts from bullet-point inputs, creating documentation from code annotations. The AI accelerates human work rather than replacing it - the human reviews and edits. Quality is measurable by comparing against human-written versions.

Similarity and relevance ranking. Search, recommendation, matching, deduplication. These problems are inherently fuzzy - there is no single correct answer - which makes error tolerance naturally high. The non-AI alternatives (keyword matching, TF-IDF) are measurably worse, and volume is typically high enough to justify infrastructure.

Summarisation and synthesis. Condensing long documents, extracting key points from meetings, generating digests from multiple sources. High error tolerance (the source is available for verification), measurable quality (ROUGE scores, human evaluation), and the non-AI alternative is human effort that scales linearly with content volume.

AI consistently fails at these problem types:

Deterministic computation. Anything requiring exact mathematical answers, precise string manipulation, or guaranteed logical deduction. Calculating a tax return, formatting a phone number, validating an email address. Zero error tolerance, and the non-AI solution (code) is both cheaper and perfectly reliable.

High-stakes single decisions. Approving a loan, diagnosing a disease, sentencing a defendant. Error cost is too high, accountability requirements are too strict, and the non-AI alternative (human expert judgment) carries legal and ethical standing that AI does not.

Stable, well-defined business logic. If the rules are clear, finite, and change infrequently, implement them as code. A pricing engine with 47 rules is better as a rules engine than an LLM prompt - faster, cheaper, deterministic, testable, and auditable. The LLM adds no value and introduces unnecessary risk.

Real-time safety-critical systems. Autonomous vehicle steering, industrial process control, flight navigation. Latency is disqualifying, error tolerance is zero, and non-determinism is unacceptable.

Applying the Framework: Three Real Decisions

Decision 1: AI-Powered Bug Report Triage

Proposal: Automatically classify incoming bug reports by severity, component, and team assignment using an LLM.

Q1 - Error-tolerant? Yes. A misrouted bug report adds a few hours of delay but is easily corrected by the receiving team. No downstream harm.

Q2 - Substantially better? Current system: keyword-based rules classify 60% of bug reports correctly. Proposed AI: pilot shows 89% accuracy. The 29-point improvement eliminates approximately 1,200 manual reclassifications per month (at 3 minutes each = 60 hours of engineering time saved per month).

Q3 - Volume? 800 bug reports per day. Comfortably above infrastructure break-even.

Q4 - Success criteria? Classification accuracy above 85%. Median triage time under 15 minutes (currently 2.3 hours). Cost under $500/month.

Q5 - Eval infrastructure? Yes - 6 months of historically classified bug reports provide a golden test set of 140,000+ examples.

Decision: Build. Passes all five questions cleanly. This is a textbook AI use case.

Decision 2: AI-Generated Performance Reviews

Proposal: Generate first-draft performance reviews from an employee's commit history, peer feedback, and self-assessment using an LLM.

Q1 - Error-tolerant? Partially. The manager reviews and edits the draft, so factual errors can be caught. But tone errors (overly negative, damning with faint praise) may subtly influence the manager's final review even after editing. The stakes are a person's career.

Q2 - Substantially better? The alternative is the manager writing from scratch (3-5 hours per report). The AI draft reduces this to 1-2 hours of editing. Valuable - but the quality bar is high and the failure mode is serious.

Q3 - Volume? Performance reviews happen twice per year. For a 500-person engineering org, that is 1,000 reviews per cycle - low volume.

Q4 - Success criteria? Hard to define. "The manager should not need to substantially rewrite the review" is subjective. "Factual accuracy above 95%" is measurable but insufficient - tone, emphasis, and omission matter as much as factual accuracy in a performance review.

Q5 - Eval infrastructure? No. Performance reviews are confidential. Building a golden test set requires access to historical reviews, which HR restrictions may prevent. Automated quality evaluation of review tone and impact does not exist.

Decision: Do not build - yet. Fails on Q4 (success criteria are fuzzy) and Q5 (no eval infrastructure). Run as a time-boxed experiment with 20 volunteering managers, manually evaluate the outputs, and revisit the decision with data.

Decision 3: AI-Powered Autocomplete for Search

Proposal: Replace the current prefix-match search autocomplete with LLM-powered suggestions that understand intent and context.

Q1 - Error-tolerant? Yes. A bad suggestion is simply not selected by the user. No downstream harm.

Q2 - Substantially better? Pilot shows 34% improvement in search success rate (user finds what they are looking for on the first query). Significant.

Q3 - Volume? 2 million search queries per day. High enough - but at 2 million calls per day, inference cost is the binding constraint. Current cost estimate:

Q4 - Success criteria? Search success rate above 75% (currently 56%). P95 latency under 200ms for autocomplete suggestions. Cost under $20,000/month.

Q5 - Eval infrastructure? Yes - search logs with click-through data provide continuous quality measurement.

Decision: Build - but with caching. Passes all five questions, but the 90x cost gap means caching is mandatory. Semantic caching on autocomplete queries should achieve 40-60% hit rates, bringing effective cost to $8,000-

Interview: System design interview connection: "Should we add AI to this feature?" is increasingly asked in product-focused system design interviews. Interviewers want to see that you evaluate the trade-off rigorously rather than defaulting to "yes, AI makes everything better."

What to emphasise: Present the five-question framework. Walk through the AI tax (all seven components, not just inference cost). Show that you can identify problems where AI is not the right tool - this demonstrates maturity. The strongest signal is when a candidate recommends against AI for a specific sub-problem within a larger AI system.

What separates strong from weak answers: Weak answers assume AI is always beneficial and focus only on the model's capability. Strong answers quantify the full AI tax, compare against non-AI alternatives, define measurable success criteria, and identify which parts of the system benefit from AI and which are better served by deterministic code.

Exercise: Starter: Your team proposes adding AI-powered code review comments to your CI/CD pipeline. Apply the five-question framework. What is your recommendation? What assumptions are you making about error tolerance, volume, and existing evaluation infrastructure?

Intermediate: A fintech company wants to use an LLM to generate investment portfolio summaries for retail customers. The summaries include performance numbers, risk assessments, and forward-looking statements. Apply the framework. Where does this proposal fail? How would you redesign it to pass?

Advanced: You are the AI engineering lead at an e-commerce company. The product team proposes six AI features: (1) AI-generated product descriptions, (2) AI-powered customer support chat, (3) AI-driven dynamic pricing, (4) AI-based fraud detection, (5) AI-powered search, (6) AI-generated return shipping labels. Evaluate each against the framework. Rank them in the order you would build them and justify the ranking. Which features would you reject entirely?

Key Takeaways

The question is never "can AI do this?" - the answer is almost always yes. The question is "does AI do this well enough, cheaply enough, and reliably enough to justify the AI tax?" The AI tax includes seven components - inference cost, quality monitoring, hallucination liability, latency overhead, non-determinism management, provider dependency, and maintenance trajectory - and most teams underestimate it by 3-5x because they plan only for inference cost.

The five-question framework provides a structured evaluation: error tolerance, substantial improvement over non-AI alternatives, volume justification, defined success criteria, and available evaluation infrastructure. Any "no" is a reason to reconsider - not necessarily to reject, but to modify the approach (human-in-the-loop, shared infrastructure, time-boxed experiment).

AI excels at unstructured-to-structured transformation, content generation from constraints, similarity ranking, and summarisation. AI consistently fails at deterministic computation, high-stakes single decisions, stable business logic, and real-time safety-critical systems. Knowing which category your problem falls into is the most important decision you make as an AI engineer.

Back to AI Engineering