Overview
- Built a unified abstraction across five model providers with asynchronous inference, session and context management, and multimodal image, video, and audio requests.
- Designed PostgreSQL schemas, indexing, GraphQL query patterns, and vector retrieval for sub-100ms local cross-session lookups.
- Implemented cost-aware provider routing with fallback logic, reducing inference costs by 40–60% in testing.
Technology
- LLMs
- Node.js
- FastAPI
- GraphQL
- PostgreSQL
- Redis
- Docker