Skip to content

Code Lingua

Code with voice(and not in english).

Code Video Cerebral Valley

Core trick

Fine-tunes Qwen2-Audio with LoRA to map Indic-language speech directly to structured JSON with an English instruction field.

Tags

voicedev tools

Description

Code with voice(and not in english). We developed a customized model to code in Indic languages (Tamil, Hindi and Bengali). The pipeline starts from IntelliJ plugin Recording flow: Alt+Shift+R starts recording Alt+Shift+S stops recording and auto-submits the WAV file the backend generates a prompt and runs codex exec against the project root
KaushikSiva/lingua-code★ 00 forksPythonCreated Apr 19, 2026 · last push Apr 19, 2026

README excerpt

# Linguacode

Linguacode turns Hindi/Hinglish, Tamil/Tanglish, or Bengali/Banglish speech into strict JSON:

```json
{
  "lang": "",
  "raw_mixed": "",
  "clean_native": "",
  "clean_english": ""
}
```

`clean_english` is the downstream-friendly field. It is intended to feed coding tools, agents, or automation that work best on concise English instructions while preserving technical intent from the original utterance.

## Purpose

The project builds a practical end-to-end pipeline for:

1. ingesting speech data from public AI4Bharat datasets
2. converting transcripts into supervised audio-to-JSON targets
3. fine-tuning `Qwen/Qwen2-Audio-7B-Instruct` with LoRA or QLoRA
4. running inference either through a direct audio-to-JSON model or a baseline ASR-to-label pipeline

## Architecture

- `app/datasets.py` loads AI4Bharat datasets and materializes local audio/transcript examples
- `app/transliteration.py` uses Aksharantar lexicons plus fallback transliteration helpers
- `app/label_transcript.py` converts transcripts into strict JSON labels with validation, retries, and OpenAI batching
- `app/dataset_builder.py` builds intermediate, labeled, curated, stratified train/valid JSONL exports and stages Hub uploads
- `app/train_utils.py` loads `Qwen2-Audio-7B-Instruct`, applies LoRA/QLoRA, and trains on exported JSONL
- `app/transcribe.py` provides the baseline ASR path
- `app/infer.py` runs either baseline inference or direct model inference and writes JSON outputs

## Data Sources

Team