Skip to content

refactor: improve Ollama and LM Studio configuration with environment… #10243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
PineArt wants to merge 1 commit into
base: main
Choose a base branch
from
Open

refactor: improve Ollama and LM Studio configuration with environment… #10243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/api/providers/fetchers/lmstudio.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { LLM, LLMInfo, LLMInstanceInfo, LMStudioClient } from "@lmstudio/sdk"

import { type ModelInfo, lMStudioDefaultModelInfo } from "@roo-code/types"

import { flushModels, getModels } from "./modelCache"
import { refreshModels } from "./modelCache"

const modelsWithLoadedDetails = new Set<string>()

Expand All @@ -18,8 +18,8 @@ export const forceFullModelDetailsLoad = async (baseUrl: string, modelId: string

const client = new LMStudioClient({ baseUrl: lmsUrl })
await client.llm.model(modelId)
// Flush and refresh cache to get updated model details
await flushModels("lmstudio", true)
// Refresh cache to get updated model details using the provided baseUrl
await refreshModels({ provider: "lmstudio", baseUrl })

// Mark this model as having full details loaded.
modelsWithLoadedDetails.add(modelId)
Expand Down
47 changes: 35 additions & 12 deletions src/api/providers/native-ollama.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { BaseProvider } from "./base-provider"
import type { ApiHandlerOptions } from "../../shared/api"
import { getOllamaModels } from "./fetchers/ollama"
import { XmlMatcher } from "../../utils/xml-matcher"
import { normalizeToolSchema } from "../../utils/json-schema"
import type { SingleCompletionHandler, ApiHandlerCreateMessageMetadata } from "../index"

interface OllamaChatOptions {
Expand Down Expand Up @@ -158,15 +159,23 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
private ensureClient(): Ollama {
if (!this.client) {
try {
// Priority: environment variables > VS Code settings > default
const host =
process.env.OLLAMA_BASE_URL ||
process.env.OLLAMA_SERVER_URL ||
this.options.ollamaBaseUrl ||
"http://localhost:11434"
Comment on lines +162 to +167
Copy link
Contributor

@roomote roomote bot Dec 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this new priority chain, the error message in the ECONNREFUSED handler (line 334) becomes inconsistent. It still references this.options.ollamaBaseUrl || "http://localhost:11434" but the actual host used could differ when environment variables are set. Users debugging connection failures would see a misleading URL in the error message.

Consider storing the resolved host as an instance property (e.g., this.resolvedHost) so the error message can reference the actual URL being used.

Fix it with Roo Code or mention @roomote and request a fix.


const clientOptions: OllamaOptions = {
host: this.options.ollamaBaseUrl || "http://localhost:11434",
host,
// Note: The ollama npm package handles timeouts internally
}

// Add API key if provided (for Ollama cloud or authenticated instances)
if (this.options.ollamaApiKey) {
const apiKey = process.env.OLLAMA_API_KEY || this.options.ollamaApiKey
if (apiKey) {
clientOptions.headers = {
Authorization: `Bearer ${this.options.ollamaApiKey}`,
Authorization: `Bearer ${apiKey}`,
}
}

Expand All @@ -182,6 +191,9 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
* Converts OpenAI-format tools to Ollama's native tool format.
* This allows NativeOllamaHandler to use the same tool definitions
* that are passed to OpenAI-compatible providers.
*
* Normalizes JSON schemas to ensure compatibility with Ollama's Go unmarshaler,
* which expects `type` to be a string (not an array like ["string", "null"]).
*/
private convertToolsToOllama(tools: OpenAI.Chat.ChatCompletionTool[] | undefined): OllamaTool[] | undefined {
if (!tools || tools.length === 0) {
Expand All @@ -195,7 +207,9 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
function: {
name: tool.function.name,
description: tool.function.description,
parameters: tool.function.parameters as OllamaTool["function"]["parameters"],
parameters: normalizeToolSchema(
tool.function.parameters as Record<string, unknown>,
) as OllamaTool["function"]["parameters"],
},
}))
}
Expand Down Expand Up @@ -234,9 +248,12 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
temperature: this.options.modelTemperature ?? (useR1Format ? DEEP_SEEK_DEFAULT_TEMPERATURE : 0),
}

// Only include num_ctx if explicitly set via ollamaNumCtx
if (this.options.ollamaNumCtx !== undefined) {
chatOptions.num_ctx = this.options.ollamaNumCtx
// Only include num_ctx if explicitly set (env overrides settings)
const numCtx = process.env.OLLAMA_NUM_CTX
? parseInt(process.env.OLLAMA_NUM_CTX, 10)
: this.options.ollamaNumCtx
if (numCtx !== undefined && !isNaN(numCtx)) {
chatOptions.num_ctx = numCtx
}

// Create the actual API request promise
Expand Down Expand Up @@ -328,12 +345,15 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
}

async fetchModel() {
this.models = await getOllamaModels(this.options.ollamaBaseUrl, this.options.ollamaApiKey)
const baseUrl = process.env.OLLAMA_BASE_URL || process.env.OLLAMA_SERVER_URL || this.options.ollamaBaseUrl
const apiKey = process.env.OLLAMA_API_KEY || this.options.ollamaApiKey

this.models = await getOllamaModels(baseUrl, apiKey)
return this.getModel()
}

override getModel(): { id: string; info: ModelInfo } {
const modelId = this.options.ollamaModelId || ""
const modelId = process.env.OLLAMA_MODEL_ID || this.options.ollamaModelId || ""
return {
id: modelId,
info: this.models[modelId] || openAiModelInfoSaneDefaults,
Expand All @@ -351,9 +371,12 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
temperature: this.options.modelTemperature ?? (useR1Format ? DEEP_SEEK_DEFAULT_TEMPERATURE : 0),
}

// Only include num_ctx if explicitly set via ollamaNumCtx
if (this.options.ollamaNumCtx !== undefined) {
chatOptions.num_ctx = this.options.ollamaNumCtx
// Only include num_ctx if explicitly set (env overrides settings)
const numCtx = process.env.OLLAMA_NUM_CTX
? parseInt(process.env.OLLAMA_NUM_CTX, 10)
: this.options.ollamaNumCtx
if (numCtx !== undefined && !isNaN(numCtx)) {
chatOptions.num_ctx = numCtx
}

const response = await client.chat({
Expand Down
14 changes: 5 additions & 9 deletions src/core/webview/webviewMessageHandler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ import { getVsCodeLmModels } from "../../api/providers/vscode-lm"
import { openMention } from "../mentions"
import { getWorkspacePath } from "../../utils/path"
import { Mode, defaultModeSlug } from "../../shared/modes"
import { getModels, flushModels } from "../../api/providers/fetchers/modelCache"
import { getModels, flushModels, refreshModels } from "../../api/providers/fetchers/modelCache"
import { GetModelsOptions } from "../../shared/api"
import { generateSystemPrompt } from "./generateSystemPrompt"
import { getCommand } from "../../utils/commands"
Expand Down Expand Up @@ -923,10 +923,8 @@ export const webviewMessageHandler = async (
// Specific handler for Ollama models only.
const { apiConfiguration: ollamaApiConfig } = await provider.getState()
try {
// Flush cache and refresh to ensure fresh models.
await flushModels("ollama", true)

const ollamaModels = await getModels({
// Force-refresh with the user-provided baseUrl/apiKey so we don't hit stale localhost cache.
const ollamaModels = await refreshModels({
provider: "ollama",
baseUrl: ollamaApiConfig.ollamaBaseUrl,
apiKey: ollamaApiConfig.ollamaApiKey,
Expand All @@ -945,10 +943,8 @@ export const webviewMessageHandler = async (
// Specific handler for LM Studio models only.
const { apiConfiguration: lmStudioApiConfig } = await provider.getState()
try {
// Flush cache and refresh to ensure fresh models.
await flushModels("lmstudio", true)

const lmStudioModels = await getModels({
// Force-refresh with the user-provided baseUrl so we don't hit stale localhost cache.
const lmStudioModels = await refreshModels({
provider: "lmstudio",
baseUrl: lmStudioApiConfig.lmStudioBaseUrl,
})
Expand Down