AI architecture
Small, inspectable steps - each one testable.
AgroExpert separates the farmer’s app, the system of record and the AI service. Inside the AI service, every reasoning step is a focused node in a state graph, every model call goes through one routing layer, and every known limitation is written down. This page describes what exists in the code today.
Status legend
- Implemented
- Built and exercised by automated tests in its own service.
- In validation
- Built, but not yet validated end to end, on target hardware, or with real field data.
- Partial
- Some of the intended capability exists; documented gaps remain.
- Designed
- Specified or configured, but not yet integrated or running.
- Planned
- On the roadmap. No working implementation yet.
- Research direction
- An open question to investigate - not committed engineering work.
01System overview
Mobile, backend and AI service.
- HTTPS · JWT - App calls only the backend
- Internal API · shared secret - Backend is the AI service’s only caller; AI service fetches crop-cycle context back
Backend API · stack
- FastAPI
- PostgreSQL + pgvector
- Redis
- Celery workers
- Alembic migrations
Owns
- Identity and sessions
- Farms, plots, crop cycles
- Media and analysis jobs
- Notifications, reports, audit
02End-to-end workflow
From farmer input to plot history, in twelve stages.
Not yet validated end to end
The stages are implemented and tested within their own services. The complete loop - app, backend and AI service running together with real models - has not yet been validated.Stage 01 of 12
ImplementedFarmer input
Capture what the farmer sees, in the farmer’s own words and photos, tied to a specific plot and crop.
- Inputs
- → One or more photos
- → Plot and crop cycle
- → Optional note or voice message
- Outputs
- ← An analysis job with uploaded media
Limitations
A photo shows one moment and one angle. It cannot show roots, soil chemistry or recent weather.
Mobile scan flow → backend media upload and Analysis Job.
03Deep dive
Computer vision
The first job is to refuse bad evidence. Classical, deterministic checks - no model, identical in development and production - validate the image, score blur and lighting, detect photos of screens through moiré patterns, and correct lighting before anything else sees it.
A vision-language model then reasons over the photo together with context. It describes symptoms; it does not localize lesions, because no detection or segmentation model exists yet.
Diagnosis from an image alone is fundamentally limited: different causes look alike, and published models trained on controlled images lose much of their accuracy on photos from other sources.
SourcesMohanty (2016) (opens in a new tab)Barbedo (2018) (opens in a new tab)
- Image format, size and integrity validationImplemented
- Blur and lighting quality assessmentImplemented
- Screen-photo (moiré) manipulation detectionImplemented
- Lighting enhancementImplemented
- Concurrent multi-photo processingImplemented
- Vision-language symptom reasoning (CPU development model)In validation
- Thresholds calibrated on real farmer photosPlanned
- Object detection and lesion segmentationPlanned
04Deep dive
OCR and document understanding
PaddleOCR (PP-OCRv5) - the same engine in every environment - extracts text from product labels and scanned documents. QR codes and barcodes are decoded independently of OCR. Structured extraction turns label text into fields, and low-confidence OCR on product labels is cross-checked by a vision model.
Extracted text is only ever what the label says. It must be shown next to the original image and kept separate from any recommendation.
- Text detection and recognition (English, Hindi)Implemented
- QR and barcode decodingImplemented
- Structured product-label extractionImplemented
- Vision-model cross-check for low-confidence textImplemented
- OCR fallback for scanned PDFs in knowledge ingestionImplemented
- Called from the backend or the appPlanned
- Table recognition for soil-test reportsPlanned
05Retrieval-augmented generation
A knowledge pipeline built for traceability.
Stage 07
Retrieval
ImplementedHybrid dense + sparse search in Qdrant with reciprocal-rank fusion and crop metadata pre-filters; multi-query expansion for ambiguous questions.
Region filtering is planned - no region signal exists yet.
Known gap: retrieval has no minimum-relevance cutoff, so a question with no good match still returns its top results. The evaluation harness measures this explicitly with negative-control queries rather than hiding it.
SourcesLewis et al. (2020) (opens in a new tab)Gao et al. (2023) (opens in a new tab)
06Agentic workflows
Specialised nodes, explicit state, bounded loops.
Instead of one enormous prompt, each concern - safety screening, vision, context, memory, diagnosis, retrieval, recommendation, verification, confidence, localization - is a small node that reads shared state and returns a partial update.
That makes each step unit-testable with fakes, lets a failure be traced to a specific node, and keeps retry loops explicit and bounded rather than hidden inside a model’s behaviour.
- safety_input
- vision
- context_enrichment
- memory
- diagnosis
- diagnosis_retrieval
- recommendation
- verification
- confidence_scoring
- safety_output
- localization
- → END
Conditional edge: verification → recommendation - regenerate if ungrounded (bounded).
07Model routing
Application logic never names a model.
Every model call goes through one ModelRouter, which wraps exactly one provider chosen by configuration at startup. There is no silent fallback between tiers: a missing capability raises an explicit error. When a development model stands in for a production one, every response is flagged and the confidence score is discounted.
| Tier | Purpose | Language model | Vision-language model |
|---|---|---|---|
| dev | Deterministic placeholder provider with no network or model. Used for fast tests. | Placeholder | Placeholder |
| ollama | Real models that run on a CPU-only development machine. Every call is flagged as a hardware substitution. | mistral:latest | llava:7b |
| production | The specified production models for a GPU-backed deployment. Configured, not yet deployed. | qwen3:14b | qwen2.5vl:7b |
Same model on every tier
- Embeddings
- BAAI/bge-m3
- Reranking
- BAAI/bge-reranker-v2-m3
- OCR
- PaddleOCR PP-OCRv5
- Speech-to-text
- faster-whisper (small in development; large-v3 specified for production)
Development and production stay separate
Model identifiers live in configuration, never at call sites. An optional hosted-model escalation for low-confidence diagnoses exists but is disabled by default and requires its own key. Production models are configured but have not yet been deployed on GPU hardware.08Deep dive
Memory and plot context
A diagnosis is better informed when it knows what happened before. The AI service fetches crop-cycle context from the backend - recent timeline entries, past diagnoses, cached weather - and reads per-farmer and per-plot memory before reasoning. After a diagnosis, memory is updated in the background.
Missing context is never guessed: it lowers the confidence score instead.
- Crop-cycle context from the backend (timeline, weather, past diagnoses)Implemented
- Farmer and plot memory - read before diagnosisImplemented
- Farmer and plot memory - updated after diagnosisImplemented
- Crop memory narrative summaries with embeddingsImplemented
- Conversation windowing and summarization (chat)Implemented
- Persisted conversation memory across sessionsPlanned
- Structured treatment-outcome trackingPlanned
09Safety and explainability
Defence in depth, with the gaps written down.
- Input screening. User text is delimited, then screened for prompt injection by seven pattern rules and a semantic model check.
- Output screening. Fabricated dosages are detected and stripped; unsafe advice is flagged.
- Verification. Claims are checked against retrieved sources, with a bounded retry when a draft is ungrounded.
- Disclaimers. Localized (English, Hindi, Nepali) and escalated when confidence is low or a safety flag is raised.
- Human escalation. Low confidence should lead to “ask an expert,” not a more confident-sounding answer.
Measured limitation
In live testing on the development model, the semantic injection check missed a deliberately worded jailbreak. The layer is present and exercised, not proven robust. Hindi and Nepali texts have not been reviewed by native speakers.SourcesOWASP Foundation (web) (opens in a new tab)U.S. National Institute of Standards and Technology (2023) (opens in a new tab)
Confidence score composition
A 0–100 score from four weighted components. The weights are design choices, not calibrated probabilities - calibration is a research question in its own right.
- Vision · 35%
- The model’s stated certainty, discounted by image quality, suspected screen photos and development-model substitution. Two of four intended signals do not exist yet.
- Grounding · 35%
- Retrieval support averaged with the per-claim faithfulness rate from verification.
- Context completeness · 15%
- Reduced when weather, recent timeline entries or past diagnoses are missing.
- Consistency · 15%
- Agreement with a recent prior diagnosis on the same crop cycle, when one exists.
10Deep dive
Evaluation and MLOps
Offline, a golden dataset measures retrieval quality - including negative-control queries that should retrieve nothing. Online, each chat response produces faithfulness, hallucination-rate and approximated retrieval-precision signals. Drift is tested statistically.
Canary cohorts, non-regression gates and rollback decisions exist as tested logic. Nothing yet routes live traffic or executes a rollback, because there is no multi-instance deployment.
- Golden-dataset retrieval evaluation (precision@k, recall hit-rate)Implemented
- Online faithfulness and hallucination signals (chat)Implemented
- Drift detection (two-proportion z-test)Implemented
- Request tracing and experiment tracking (MLflow)Implemented
- Canary and rollback decision logicDesigned
- Automated traffic routing and rollback executionPlanned
- Diagnosis-side production quality signalsPlanned
- Human and expert evaluation workflowsPlanned
11Implementation inventory
Every component, with its evidence-based status.
Mobile app
Mobile app (Expo / React Native, TypeScript)
ImplementedOnboarding, phone sign-in, plots, crop timeline, activity and harvest logging, scan capture, chat, weather, notifications, search and settings, localized into nine languages (English, Hindi, Bengali, Gujarati, Kannada, Marathi, Punjabi, Tamil and Telugu).
Nepali language in the app
PlannedThe AI service already writes its disclaimers in Nepali, English and Hindi, but Nepali is not yet one of the mobile app’s languages.
Backend
Backend API (FastAPI modular monolith)
ImplementedDomain modules for identity, farms, plots, crop cycles, timeline, media, analysis, recommendations, chat, knowledge base, weather, notifications, reports and administration, with PostgreSQL + pgvector, Redis and Celery workers.
AI service
Image validation and quality assessment
ImplementedClassical, deterministic computer vision: format and size validation, blur and lighting checks, moiré-based screen-photo detection and lighting enhancement.
Vision-language diagnosis
In validationA vision-language model reasons over the photo, context and memory. Exercised on CPU with a smaller development model; the configured production model has not been deployed on GPU hardware.
Object detection and segmentation
PlannedNot implemented. The model router raises a capability-unavailable error rather than pretending.
OCR and label extraction
PartialPaddleOCR (PP-OCRv5) text extraction, QR/barcode decoding, structured label fields and a vision-model cross-check are implemented in the AI service. No backend or app caller exists yet.
Retrieval-augmented knowledge base
ImplementedPDF/DOCX ingestion with OCR fallback, cleaning, de-duplication, chunking, bge-m3 hybrid embeddings, Qdrant retrieval, cross-encoder reranking and Redis caching. The corpus is a small seed set, not a reviewed agronomic library.
Agent workflows (LangGraph)
ImplementedTwo state graphs: Chat (nine nodes, with verification and reflection retry loops) and Diagnosis (eleven nodes, with a bounded verification retry).
Safety screening
In validationPrompt-injection and fabricated-dosage detection (pattern rules plus an LLM check). The LLM layer missed a deliberate test jailbreak on the development model - documented as a known limitation.
Confidence scoring and disclaimers
ImplementedA four-component weighted score with localized disclaimers that escalate on low confidence. Two of the intended vision signals do not exist yet; thresholds are uncalibrated.
Farmer, plot and crop memory
ImplementedPlot and farmer memory is read before a diagnosis and updated afterwards; crop memory stores an LLM-written narrative summary.
Evaluation and drift detection
PartialOffline retrieval evaluation on a golden dataset, online faithfulness and hallucination signals for chat, and statistical drift tests. No human or expert evaluation workflow yet.
Canary and rollback
DesignedCohort assignment, non-regression gates and rollback decisions are implemented as tested logic, but nothing routes live traffic or executes a rollback.
Product and dosage recommendations
PlannedDeliberately absent. Recommendation steps never name a specific product or dose; a verified treatment catalogue would be required first.
Integration & validation
Mobile ↔ backend
ImplementedPhone OTP auth, plot creation, chat, weather and the scan flow call the real backend. Email sign-in and guest chat remain mocked; plot sync is one-directional.
Backend ↔ AI service
In validationThe backend’s AI client and the AI service’s endpoints match field for field, and the AI service’s call back into the backend for crop-cycle context was live-verified. The backend still defaults to its mock AI mode in development, so the full farmer-to-AI loop has not been validated end to end.
Field validation with agricultural experts
Research directionNo pilot, field study or expert review has taken place yet.