Skip to content

FEAT: Enable Local LLM Inference on Android #696

@iberi22

Description

@iberi22

Context

The current Android build uses a "Logic Engine" (mock) because ort (ONNX Runtime) failed to compile for aarch64-linux-android due to missing precompiled binaries/pkg-config issues.

Task

Enable real local inference on Android.

Options

  1. Fix ORT: Configure ort to download/compile correctly for Android NDK.
  2. Switch to Candle: Use Hugging Face's candle crate which is pure Rust and easier to compile for Android.
  3. Switch to Llama.cpp: Use llama.cpp bindings.

Requirements

  • Must run Phi-3-mini or Gemma-2b (quantized).
  • Must not crash the app on load.
  • Must stream tokens to the UI.

Reference

  • crates/synapse-cognition/src/lib.rs
  • apps/desktop/src-tauri/Cargo.toml

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestjulesAssigned to Google Jules

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions