Skip to content

AI architecture

Small, inspectable steps - each one testable.

AgroExpert separates the farmer’s app, the system of record and the AI service. Inside the AI service, every reasoning step is a focused node in a state graph, every model call goes through one routing layer, and every known limitation is written down. This page describes what exists in the code today.

Status legend

Implemented
Built and exercised by automated tests in its own service.
In validation
Built, but not yet validated end to end, on target hardware, or with real field data.
Partial
Some of the intended capability exists; documented gaps remain.
Designed
Specified or configured, but not yet integrated or running.
Planned
On the roadmap. No working implementation yet.
Research direction
An open question to investigate - not committed engineering work.

01System overview

Mobile, backend and AI service.

Select a layer to see its stack and responsibilities. Arrows show the direction of calls across each trust boundary.
  • HTTPS · JWT - App calls only the backend
  • Internal API · shared secret - Backend is the AI service’s only caller; AI service fetches crop-cycle context back

Backend API · stack

  • FastAPI
  • PostgreSQL + pgvector
  • Redis
  • Celery workers
  • Alembic migrations

Owns

  • Identity and sessions
  • Farms, plots, crop cycles
  • Media and analysis jobs
  • Notifications, reports, audit

02End-to-end workflow

From farmer input to plot history, in twelve stages.

Stage names follow the Diagnosis graph in the AI service. Each stage shows its inputs, outputs, honest limitations and current status.

Not yet validated end to end

The stages are implemented and tested within their own services. The complete loop - app, backend and AI service running together with real models - has not yet been validated.

Stage 01 of 12

Implemented

Farmer input

Capture what the farmer sees, in the farmer’s own words and photos, tied to a specific plot and crop.

Inputs
  • → One or more photos
  • → Plot and crop cycle
  • → Optional note or voice message
Outputs
  • ← An analysis job with uploaded media

Limitations

A photo shows one moment and one angle. It cannot show roots, soil chemistry or recent weather.

Mobile scan flow → backend media upload and Analysis Job.

03Deep dive

Computer vision

The first job is to refuse bad evidence. Classical, deterministic checks - no model, identical in development and production - validate the image, score blur and lighting, detect photos of screens through moiré patterns, and correct lighting before anything else sees it.

A vision-language model then reasons over the photo together with context. It describes symptoms; it does not localize lesions, because no detection or segmentation model exists yet.

Diagnosis from an image alone is fundamentally limited: different causes look alike, and published models trained on controlled images lose much of their accuracy on photos from other sources.

SourcesMohanty (2016) (opens in a new tab)Barbedo (2018) (opens in a new tab)

  • Image format, size and integrity validationImplemented
  • Blur and lighting quality assessmentImplemented
  • Screen-photo (moiré) manipulation detectionImplemented
  • Lighting enhancementImplemented
  • Concurrent multi-photo processingImplemented
  • Vision-language symptom reasoning (CPU development model)In validation
  • Thresholds calibrated on real farmer photosPlanned
  • Object detection and lesion segmentationPlanned

04Deep dive

OCR and document understanding

PaddleOCR (PP-OCRv5) - the same engine in every environment - extracts text from product labels and scanned documents. QR codes and barcodes are decoded independently of OCR. Structured extraction turns label text into fields, and low-confidence OCR on product labels is cross-checked by a vision model.

Extracted text is only ever what the label says. It must be shown next to the original image and kept separate from any recommendation.

  • Text detection and recognition (English, Hindi)Implemented
  • QR and barcode decodingImplemented
  • Structured product-label extractionImplemented
  • Vision-model cross-check for low-confidence textImplemented
  • OCR fallback for scanned PDFs in knowledge ingestionImplemented
  • Called from the backend or the appPlanned
  • Table recognition for soil-test reportsPlanned

05Retrieval-augmented generation

A knowledge pipeline built for traceability.

Retrieval-augmented generation conditions a model on retrieved documents, so answers can be grounded in - and traced back to - real sources. Select a stage.

Stage 07

Retrieval

Implemented

Hybrid dense + sparse search in Qdrant with reciprocal-rank fusion and crop metadata pre-filters; multi-query expansion for ambiguous questions.

Region filtering is planned - no region signal exists yet.

Known gap: retrieval has no minimum-relevance cutoff, so a question with no good match still returns its top results. The evaluation harness measures this explicitly with negative-control queries rather than hiding it.

SourcesLewis et al. (2020) (opens in a new tab)Gao et al. (2023) (opens in a new tab)

06Agentic workflows

Specialised nodes, explicit state, bounded loops.

Instead of one enormous prompt, each concern - safety screening, vision, context, memory, diagnosis, retrieval, recommendation, verification, confidence, localization - is a small node that reads shared state and returns a partial update.

That makes each step unit-testable with fakes, lets a failure be traced to a specific node, and keeps retry loops explicit and bounded rather than hidden inside a model’s behaviour.

  1. safety_input
  2. vision
  3. context_enrichment
  4. memory
  5. diagnosis
  6. diagnosis_retrieval
  7. recommendation
  8. verification
  9. confidence_scoring
  10. safety_output
  11. localization
  12. → END

Conditional edge: verification → recommendation - regenerate if ungrounded (bounded).

07Model routing

Application logic never names a model.

Every model call goes through one ModelRouter, which wraps exactly one provider chosen by configuration at startup. There is no silent fallback between tiers: a missing capability raises an explicit error. When a development model stands in for a production one, every response is flagged and the confidence score is discounted.

Model provider tiers
TierPurposeLanguage modelVision-language model
devDeterministic placeholder provider with no network or model. Used for fast tests.PlaceholderPlaceholder
ollamaReal models that run on a CPU-only development machine. Every call is flagged as a hardware substitution.mistral:latestllava:7b
productionThe specified production models for a GPU-backed deployment. Configured, not yet deployed.qwen3:14bqwen2.5vl:7b

Same model on every tier

Embeddings
BAAI/bge-m3
Reranking
BAAI/bge-reranker-v2-m3
OCR
PaddleOCR PP-OCRv5
Speech-to-text
faster-whisper (small in development; large-v3 specified for production)

Development and production stay separate

Model identifiers live in configuration, never at call sites. An optional hosted-model escalation for low-confidence diagnoses exists but is disabled by default and requires its own key. Production models are configured but have not yet been deployed on GPU hardware.

08Deep dive

Memory and plot context

A diagnosis is better informed when it knows what happened before. The AI service fetches crop-cycle context from the backend - recent timeline entries, past diagnoses, cached weather - and reads per-farmer and per-plot memory before reasoning. After a diagnosis, memory is updated in the background.

Missing context is never guessed: it lowers the confidence score instead.

  • Crop-cycle context from the backend (timeline, weather, past diagnoses)Implemented
  • Farmer and plot memory - read before diagnosisImplemented
  • Farmer and plot memory - updated after diagnosisImplemented
  • Crop memory narrative summaries with embeddingsImplemented
  • Conversation windowing and summarization (chat)Implemented
  • Persisted conversation memory across sessionsPlanned
  • Structured treatment-outcome trackingPlanned

09Safety and explainability

Defence in depth, with the gaps written down.

  • Input screening. User text is delimited, then screened for prompt injection by seven pattern rules and a semantic model check.
  • Output screening. Fabricated dosages are detected and stripped; unsafe advice is flagged.
  • Verification. Claims are checked against retrieved sources, with a bounded retry when a draft is ungrounded.
  • Disclaimers. Localized (English, Hindi, Nepali) and escalated when confidence is low or a safety flag is raised.
  • Human escalation. Low confidence should lead to “ask an expert,” not a more confident-sounding answer.

Measured limitation

In live testing on the development model, the semantic injection check missed a deliberately worded jailbreak. The layer is present and exercised, not proven robust. Hindi and Nepali texts have not been reviewed by native speakers.

SourcesOWASP Foundation (web) (opens in a new tab)U.S. National Institute of Standards and Technology (2023) (opens in a new tab)

Confidence score composition

A 0–100 score from four weighted components. The weights are design choices, not calibrated probabilities - calibration is a research question in its own right.

Vision · 35%
The model’s stated certainty, discounted by image quality, suspected screen photos and development-model substitution. Two of four intended signals do not exist yet.
Grounding · 35%
Retrieval support averaged with the per-claim faithfulness rate from verification.
Context completeness · 15%
Reduced when weather, recent timeline entries or past diagnoses are missing.
Consistency · 15%
Agreement with a recent prior diagnosis on the same crop cycle, when one exists.

SourcesGuo et al. (2017) (opens in a new tab)

10Deep dive

Evaluation and MLOps

Offline, a golden dataset measures retrieval quality - including negative-control queries that should retrieve nothing. Online, each chat response produces faithfulness, hallucination-rate and approximated retrieval-precision signals. Drift is tested statistically.

Canary cohorts, non-regression gates and rollback decisions exist as tested logic. Nothing yet routes live traffic or executes a rollback, because there is no multi-instance deployment.

  • Golden-dataset retrieval evaluation (precision@k, recall hit-rate)Implemented
  • Online faithfulness and hallucination signals (chat)Implemented
  • Drift detection (two-proportion z-test)Implemented
  • Request tracing and experiment tracking (MLflow)Implemented
  • Canary and rollback decision logicDesigned
  • Automated traffic routing and rollback executionPlanned
  • Diagnosis-side production quality signalsPlanned
  • Human and expert evaluation workflowsPlanned

11Implementation inventory

Every component, with its evidence-based status.

Compiled by reading the source, tests, configuration and each service’s own status documents.

Mobile app

  • Mobile app (Expo / React Native, TypeScript)

    Implemented

    Onboarding, phone sign-in, plots, crop timeline, activity and harvest logging, scan capture, chat, weather, notifications, search and settings, localized into nine languages (English, Hindi, Bengali, Gujarati, Kannada, Marathi, Punjabi, Tamil and Telugu).

  • Nepali language in the app

    Planned

    The AI service already writes its disclaimers in Nepali, English and Hindi, but Nepali is not yet one of the mobile app’s languages.

Backend

  • Backend API (FastAPI modular monolith)

    Implemented

    Domain modules for identity, farms, plots, crop cycles, timeline, media, analysis, recommendations, chat, knowledge base, weather, notifications, reports and administration, with PostgreSQL + pgvector, Redis and Celery workers.

AI service

  • Image validation and quality assessment

    Implemented

    Classical, deterministic computer vision: format and size validation, blur and lighting checks, moiré-based screen-photo detection and lighting enhancement.

  • Vision-language diagnosis

    In validation

    A vision-language model reasons over the photo, context and memory. Exercised on CPU with a smaller development model; the configured production model has not been deployed on GPU hardware.

  • Object detection and segmentation

    Planned

    Not implemented. The model router raises a capability-unavailable error rather than pretending.

  • OCR and label extraction

    Partial

    PaddleOCR (PP-OCRv5) text extraction, QR/barcode decoding, structured label fields and a vision-model cross-check are implemented in the AI service. No backend or app caller exists yet.

  • Retrieval-augmented knowledge base

    Implemented

    PDF/DOCX ingestion with OCR fallback, cleaning, de-duplication, chunking, bge-m3 hybrid embeddings, Qdrant retrieval, cross-encoder reranking and Redis caching. The corpus is a small seed set, not a reviewed agronomic library.

  • Agent workflows (LangGraph)

    Implemented

    Two state graphs: Chat (nine nodes, with verification and reflection retry loops) and Diagnosis (eleven nodes, with a bounded verification retry).

  • Safety screening

    In validation

    Prompt-injection and fabricated-dosage detection (pattern rules plus an LLM check). The LLM layer missed a deliberate test jailbreak on the development model - documented as a known limitation.

  • Confidence scoring and disclaimers

    Implemented

    A four-component weighted score with localized disclaimers that escalate on low confidence. Two of the intended vision signals do not exist yet; thresholds are uncalibrated.

  • Farmer, plot and crop memory

    Implemented

    Plot and farmer memory is read before a diagnosis and updated afterwards; crop memory stores an LLM-written narrative summary.

  • Evaluation and drift detection

    Partial

    Offline retrieval evaluation on a golden dataset, online faithfulness and hallucination signals for chat, and statistical drift tests. No human or expert evaluation workflow yet.

  • Canary and rollback

    Designed

    Cohort assignment, non-regression gates and rollback decisions are implemented as tested logic, but nothing routes live traffic or executes a rollback.

  • Product and dosage recommendations

    Planned

    Deliberately absent. Recommendation steps never name a specific product or dose; a verified treatment catalogue would be required first.

Integration & validation

  • Mobile ↔ backend

    Implemented

    Phone OTP auth, plot creation, chat, weather and the scan flow call the real backend. Email sign-in and guest chat remain mocked; plot sync is one-directional.

  • Backend ↔ AI service

    In validation

    The backend’s AI client and the AI service’s endpoints match field for field, and the AI service’s call back into the backend for crop-cycle context was live-verified. The backend still defaults to its mock AI mode in development, so the full farmer-to-AI loop has not been validated end to end.

  • Field validation with agricultural experts

    Research direction

    No pilot, field study or expert review has taken place yet.