AI Inference Guide

Code Examples

Getting Started Examples

Starting points for specific SDK shapes, not versionless production recipes. Install the shown package, choose a model listed by the current provider catalog, pin dependency versions, and test error, timeout, cancellation, and resource-limit paths before deployment.

WebLLM Browser Example

// npm install @mlc-ai/web-llm
import { CreateMLCEngine } from "@mlc-ai/web-llm";

// Initialize the engine
const engine = await CreateMLCEngine(
  "Llama-3.2-1B-Instruct-q4f32_1-MLC",
  {
    initProgressCallback: (progress) => {
      console.log('Loading:', Math.round(progress.progress * 100) + '%');
    }
  }
);

// Generate text
const response = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Hello! How are you?" }
  ],
  temperature: 0.8,
  max_tokens: 100
});

console.log(response.choices[0].message.content);

BrowserAI Example

// npm install @browserai/browserai
import { BrowserAI } from '@browserai/browserai';

const browserAI = new BrowserAI();

// Load model with progress tracking
await browserAI.loadModel('llama-3.2-1b-instruct', {
  quantization: 'q4f16_1',
  onProgress: (progress) => console.log('Loading:', progress.progress + '%')
});

// Generate text
const response = await browserAI.generateText('Hello, how are you?');
console.log(response.choices[0].message.content);

// Streaming example
const chunks = await browserAI.generateText('Write a story', {
  stream: true,
  temperature: 0.8
});

for await (const chunk of chunks) {
  console.log(chunk.choices[0]?.delta.content || '');
}

Ollama Local Server

Install Ollama using its official platform instructions and ensure the desktop app or service is running.

# Terminal
ollama pull llama3.2:1b
ollama run llama3.2:1b "Hello, world!"

// JavaScript (browser/devtools while the local service is running)
const response = await fetch('http://localhost:11434/api/generate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'llama3.2:1b',
    prompt: 'Hello!',
    stream: false
  })
});
if (!response.ok) throw new Error('Ollama request failed: ' + response.status);
console.log(await response.json());

LM Studio Desktop App

Load a model in LM Studio, then start its local server with lms server start. The example discovers the actual local model identifier instead of assuming one.

// Discover the identifiers exposed by this LM Studio server
const modelsResponse = await fetch('http://localhost:1234/v1/models');
if (!modelsResponse.ok) throw new Error('Model list failed: ' + modelsResponse.status);
const models = await modelsResponse.json();
const model = models.data?.[0]?.id;
if (!model) throw new Error('Load at least one model in LM Studio');

const response = await fetch('http://localhost:1234/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model,
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: 'Hello!' }
    ],
    temperature: 0.7,
    max_tokens: 100
  })
});

if (!response.ok) throw new Error('Completion failed: ' + response.status);
const data = await response.json();
console.log(data.choices[0].message.content);

Provider API Example (Together AI)

// npm install together-ai
import Together from "together-ai";

const together = new Together({
  apiKey: process.env.TOGETHER_API_KEY,
});

const model = process.env.TOGETHER_MODEL;
if (!model) throw new Error("Set TOGETHER_MODEL to an ID from the current serverless catalog");

const response = await together.chat.completions.create({
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Explain quantum computing in simple terms." }
  ],
  model,
  max_tokens: 500,
  temperature: 0.7,
  stream: true,
});

// Handle streaming response
for await (const chunk of response) {
  process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

Choose a model from the current serverless catalog

Vision model integration

Vision input schemas and supported model IDs differ across runtimes. Select a currently supported image-to-text or multimodal model from the runtime catalog, then validate image size/type limits, orientation, accessibility, memory pressure, and untrusted-file handling.

Use the current Transformers.js documentation and examples

Best Practices

Performance Optimization

• Benchmark quantized variants against quality and latency goals
• Implement proper caching strategies
• Optimize batch sizes for throughput
• Monitor memory usage and cleanup

User Experience

• Show loading progress for model downloads
• Implement streaming for long responses
• Provide fallback options
• Handle errors gracefully

AI Inference Guide

// npm install @mlc-ai/web-llm import { CreateMLCEngine } from "@mlc-ai/web-llm"; // Initialize the engine const engine = await CreateMLCEngine( "Llama-3.2-1B-Instruct-q4f32_1-MLC", { initProgressCallback: (progress) => { console.log('Loading:', Math.round(progress.progress * 100) + '%'); } } ); // Generate text const response = await engine.chat.completions.create({ messages: [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "Hello! How are you?" } ], temperature: 0.8, max_tokens: 100 }); console.log(response.choices[0].message.content);

// npm install @browserai/browserai import { BrowserAI } from '@browserai/browserai'; const browserAI = new BrowserAI(); // Load model with progress tracking await browserAI.loadModel('llama-3.2-1b-instruct', { quantization: 'q4f16_1', onProgress: (progress) => console.log('Loading:', progress.progress + '%') }); // Generate text const response = await browserAI.generateText('Hello, how are you?'); console.log(response.choices[0].message.content); // Streaming example const chunks = await browserAI.generateText('Write a story', { stream: true, temperature: 0.8 }); for await (const chunk of chunks) { console.log(chunk.choices[0]?.delta.content || ''); }

// JavaScript (browser/devtools while the local service is running) const response = await fetch('http://localhost:11434/api/generate', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model: 'llama3.2:1b', prompt: 'Hello!', stream: false }) }); if (!response.ok) throw new Error('Ollama request failed: ' + response.status); console.log(await response.json());

// Discover the identifiers exposed by this LM Studio server const modelsResponse = await fetch('http://localhost:1234/v1/models'); if (!modelsResponse.ok) throw new Error('Model list failed: ' + modelsResponse.status); const models = await modelsResponse.json(); const model = models.data?.[0]?.id; if (!model) throw new Error('Load at least one model in LM Studio'); const response = await fetch('http://localhost:1234/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model, messages: [ { role: 'system', content: 'You are a helpful assistant.' }, { role: 'user', content: 'Hello!' } ], temperature: 0.7, max_tokens: 100 }) }); if (!response.ok) throw new Error('Completion failed: ' + response.status); const data = await response.json(); console.log(data.choices[0].message.content);

// npm install together-ai import Together from "together-ai"; const together = new Together({ apiKey: process.env.TOGETHER_API_KEY, }); const model = process.env.TOGETHER_MODEL; if (!model) throw new Error("Set TOGETHER_MODEL to an ID from the current serverless catalog"); const response = await together.chat.completions.create({ messages: [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "Explain quantum computing in simple terms." } ], model, max_tokens: 500, temperature: 0.7, stream: true, }); // Handle streaming response for await (const chunk of response) { process.stdout.write(chunk.choices[0]?.delta?.content || ''); }