Multimodal Chat

Video, audio, pictures and documents go into the same chat call as text. Every model takes every modality, and a media turn is routed to a model that can actually read it.

The chat endpoint is not text-only. A message can carry text, an image, a recording, a video or a document, in any combination — and every ZYRAXON model accepts every one of them. Send the media, and the turn is routed server-side to a model that can actually read it.

The shape of a multimodal message

Instead of content being a string, it is an array of parts. This is the same shape OpenAI defined, so a client that already sends vision turns to OpenAI sends them here unchanged.

{
  "model": "zyraxon-cyber",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "Transcribe this, then summarise it in one line." },
      { "type": "input_audio",
        "input_audio": { "data": "<base64>", "format": "wav" } },
      { "type": "image_url",
        "image_url": { "url": "data:image/png;base64,..." } },
      { "type": "video_url",
        "video_url": { "url": "https://example.com/clip.mp4" } },
      { "type": "file",
        "file": { "filename": "report.pdf",
                  "file_data": "data:application/pdf;base64,..." } }
    ]
  }]
}

Inline or by URL

Every media part takes either base64 or an https:// URL. Send a URL and the server fetches it for you, so a large file never has to pass through your request body — which is what keeps a big video from becoming a 413.

# Inline base64 — fine for a small image.
curl -s https://zyraxonai.lovable.app/api/public/zyraxon/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "zyraxon-matrix",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is in this chart?" },
      { "type": "image_url",
        "image_url": { "url": "data:image/png;base64,'"$(base64 -w0 chart.png)"'" } }
    ]
  }]
}' | jq -r '.choices[0].message.content'

# By URL — the server fetches it. Use this for anything large.
curl -s https://zyraxonai.lovable.app/api/public/zyraxon/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "zyraxon-cyber",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "Summarise this recording." },
      { "type": "input_audio",
        "input_audio": { "data": "https://example.com/interview.mp3", "format": "mp3" } }
    ]
  }]
}' | jq -r '.choices[0].message.content'

Routing: which model answers a media turn

Every ZYRAXON model is declared to take every modality, but the upstream models behind them are not identical. So the turn is matched on the whole set of modalities it carries, and sent to a model that can read all of them.

Practical rules for large media

• ▸Prefer a URL over base64 for anything over a few hundred kilobytes — base64 inflates the bytes by a third and puts them in your request body.
• ▸Never send a long recording as WAV. A 50 MB WAV is minutes of audio; MP3 at 64–96 kbps is a fraction of that and transcribes identically.
• ▸Slice a long video. A multi-gigabyte file becomes a handful of model-sized parts; a client that slices before upload turns a failed request into a working one.
• ▸One modality per turn is easiest to reason about. A turn that carries four media types needs a model that supports all four; splitting it into two turns is often more robust.
• ▸Check `truncated` and the modality header before assuming the whole file was read.

Transcribe, then ask

When you only need the words from a recording, the speech endpoint is the better tool — it is built for long audio and streams very large files. Send the transcript to the chat endpoint as text. Use the audio part above when the model needs to hear the audio itself, not only read it.

# The reliable path for a long recording: transcribe, then chat.
TEXT=$(curl -s https://zyraxonai.lovable.app/api/public/stt -F "file=@interview.mp3" -F "language=en" | jq -r '.text')
curl -s https://zyraxonai.lovable.app/api/public/zyraxon/v1/chat/completions -H "Content-Type: application/json" \
  -d "$(jq -nc --arg t "$TEXT" \
        '{model:"zyraxon-apex", messages:[{role:"user", content:("Summarise: " + $t)}]}')" \
| jq -r '.choices[0].message.content'