Clerki - Your office AI coworker

Inspiration The idea for Clerki came from watching my non-technical colleagues struggle with everyday office tasks. I saw administrative assistants manually copy-pasting data from emails into spreadsheets, project managers drowning in scattered documents across different folders, and team members spending hours creating reports that could be automated. What it does Clerki is an AI-powered workspace assistant designed for non-technical office workers. Unlike traditional chatbots that only provide suggestions, Clerki: Connects to Gmail and automatically imports emails with attachments Organizes files into isolated workspaces Understands and create documents (PDF, Word, Excel, PowerPoint, images, text) Takes action through natural conversation—summarize reports, create documents, analyze data Learn and create playbooks Learn user usage through the seesion and do reproducable tasks Example conversations: "Analyze the emails from XXX and do the task" "Summarize all the sales reports in this workspace" "Create a PPT presentation from this data" "Find all mentions of 'budget' across my documents" How we built it We integrated a mini-agent system that can: Execute Python code safely in a sandboxed environment Use specialized "skills" (document processing, web browsing, etc.) Stream tool usage back to the frontend in real-time The challenge was making the agent actually useful . Early versions would just talk about doing things. We had to build proper tool integrations so it could read files, write files, and execute system commands, all while maintaining security. Implemented OAuth 2.0 flow for Gmail: User clicks "Connect Gmail" → redirects to Google consent screen Backend receives authorization code → exchanges for access token Token encrypted with Fernet and stored securely Backend fetches emails and attachments via Gmail API UI/UX Polish Challenges we ran into AI Agent Reliability Problem : Early versions of the AI agent would sometimes fail silently or produce incorrect results. Users would ask "Summarize this document" and get a response about a different file. Solution : Implemented rigorous tool validation: Log every tool call and result Verify file paths exist before reading Return structured error messages when tools fail Show users exactly what tools are being used in real-time Accomplishments that we're proud of Now Clerki is able to do complex documentation task and able to learn through out the session and can do repetitative tasks by just one click which saves office works a lot of time. What we learned Technical Learnings WebSocket Architecture : Learned to properly manage WebSocket connections with connection managers, handle disconnects gracefully, and stream chunked responses from AI models. OAuth 2.0 Flow : Implemented Google OAuth from scratch—understanding authorization codes, access tokens, refresh tokens, and secure token storage. Async Python : FastAPI's async/await patterns were new territory. Learned to use asyncio properly to handle concurrent file operations and API calls without blocking. AI Agent Design : Building a reliable AI agent is harder than it looks. Early versions would hallucinate file paths or claim to do things they didn't. We learned to: Validate all tool outputs before proceeding Implement proper error handling and retries Stream intermediate results for transparency