AI推理指南
代码示例
入门示例
这些是特定 SDK 写法的起点，而不是不区分版本的生产配方。请安装示例中的软件包，从服务商当前的目录中挑选模型，锁定依赖版本，并在部署前测试报错、超时、取消和资源上限等各条路径。
WebLLM 浏览器示例
// npm install @mlc-ai/web-llm
import { CreateMLCEngine } from "@mlc-ai/web-llm";
// Initialize the engine
const engine = await CreateMLCEngine(
"Llama-3.2-1B-Instruct-q4f32_1-MLC",
{
initProgressCallback: (progress) => {
console.log('Loading:', Math.round(progress.progress * 100) + '%');
}
}
);
// Generate text
const response = await engine.chat.completions.create({
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "Hello! How are you?" }
],
temperature: 0.8,
max_tokens: 100
});
console.log(response.choices[0].message.content);BrowserAI 示例
// npm install @browserai/browserai
import { BrowserAI } from '@browserai/browserai';
const browserAI = new BrowserAI();
// Load model with progress tracking
await browserAI.loadModel('llama-3.2-1b-instruct', {
quantization: 'q4f16_1',
onProgress: (progress) => console.log('Loading:', progress.progress + '%')
});
// Generate text
const response = await browserAI.generateText('Hello, how are you?');
console.log(response.choices[0].message.content);
// Streaming example
const chunks = await browserAI.generateText('Write a story', {
stream: true,
temperature: 0.8
});
for await (const chunk of chunks) {
console.log(chunk.choices[0]?.delta.content || '');
}Ollama 本地服务
请按照 Ollama 的各平台官方安装说明进行安装，并确认桌面应用或服务已在运行。
# Terminal
ollama pull llama3.2:1b
ollama run llama3.2:1b "Hello, world!"// JavaScript (browser/devtools while the local service is running)
const response = await fetch('http://localhost:11434/api/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: 'llama3.2:1b',
prompt: 'Hello!',
stream: false
})
});
if (!response.ok) throw new Error('Ollama request failed: ' + response.status);
console.log(await response.json());LM Studio 桌面应用
在 LM Studio 中加载模型，然后用 lms server start 启动它的本地服务。该示例会去查询实际的本地模型标识符，而不是直接假定一个。
// Discover the identifiers exposed by this LM Studio server
const modelsResponse = await fetch('http://localhost:1234/v1/models');
if (!modelsResponse.ok) throw new Error('Model list failed: ' + modelsResponse.status);
const models = await modelsResponse.json();
const model = models.data?.[0]?.id;
if (!model) throw new Error('Load at least one model in LM Studio');
const response = await fetch('http://localhost:1234/v1/chat/completions', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model,
messages: [
{ role: 'system', content: 'You are a helpful assistant.' },
{ role: 'user', content: 'Hello!' }
],
temperature: 0.7,
max_tokens: 100
})
});
if (!response.ok) throw new Error('Completion failed: ' + response.status);
const data = await response.json();
console.log(data.choices[0].message.content);服务商 API 示例（Together AI）
// npm install together-ai
import Together from "together-ai";
const together = new Together({
apiKey: process.env.TOGETHER_API_KEY,
});
const model = process.env.TOGETHER_MODEL;
if (!model) throw new Error("Set TOGETHER_MODEL to an ID from the current serverless catalog");
const response = await together.chat.completions.create({
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "Explain quantum computing in simple terms." }
],
model,
max_tokens: 500,
temperature: 0.7,
stream: true,
});
// Handle streaming response
for await (const chunk of response) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}视觉模型集成
不同运行时的视觉输入结构和受支持的模型 ID 各不相同。请从运行时目录中选择一个当前受支持的图生文或多模态模型，然后验证图片尺寸与类型限制、方向、无障碍性、内存压力，以及不可信文件的处理方式。
参考最新的 Transformers.js 文档与示例最佳实践
性能优化
- • 对照质量与延迟目标，实测各量化版本
- • 落实合适的缓存策略
- • 为吞吐量调优批大小
- • 监控内存占用与资源释放
用户体验
- • 展示模型下载的进度
- • 为长响应实现流式输出
- • 提供降级方案
- • 妥善处理错误