Skip to content

Provider Setup (BYOK)

The AI Assistant runs against a built-in on-device model or one of six LLM providers: Anthropic, OpenAI, Google, Groq, OpenRouter, and a local Ollama server. For the hosted providers you bring your own API key; Flowfile encrypts it at rest with Fernet (using FLOWFILE_MASTER_KEY / master_key.txt), the same scheme that protects your other secrets. Two options work offline: the on-device model (no key or account) and Ollama (a server you run).

Not in Flowfile Lite

The AI Assistant (and BYOK provider setup) requires the full desktop/server build, not the browser-only Flowfile Lite edition.

Supported providers

Default models below reflect the provider classes as of 2026-10.

Provider Default model Tools Streaming Key env var Notes
Anthropic claude-sonnet-5-5 ✓ ✓ ANTHROPIC_API_KEY Sonnet 5.5 for explain / docgen / lineage; Haiku 4.5 for the fast surfaces (Cmd+K, ghost-node, autocomplete, agent_staged); Opus 4.7 for agent_complex.
OpenAI gpt-4.1-mini ✓ ✓ OPENAI_API_KEY Mini tier for the cheap surfaces; full gpt-4.1 for explain / agent_complex / docgen. Strict structured outputs supported via litellm.
Google (Gemini) gemini-2.5-flash ✓ ✓ GEMINI_API_KEY or GOOGLE_API_KEY Free tier ~250–1000 req/day, no card required. Pro for agent_complex.
Groq qwen/qwen3-32b ✓ ✓ GROQ_API_KEY Fast inference; free tier is ~30 RPM.
OpenRouter qwen/qwen3.6-35b-a3b ✓ ✓ OPENROUTER_API_KEY One key routing to many hosted models. agent_staged also defaults to qwen/qwen3.6-35b-a3b (billed, cheap); the Claude surfaces mirror the direct Anthropic defaults.
Ollama qwen3.5:9b ✓ (model-dependent) ✓ (none — local) Self-hosted; talks to your local Ollama server (default http://localhost:11434). qwen3.6:35b-a3b for agent_complex. Thinking is switched off on every request. Tool-use works on Qwen3.5+, Llama 3.1+ and most newer instruct models.

The "Tools" column means the provider can return structured tool-call arguments — required for the Agent surface. The Agent refuses to start against a model that lacks tool support.

Model IDs drift with releases

The default and per-surface model IDs are hardcoded in the provider classes at flowfile_core/flowfile_core/ai/providers/*.py (one file per provider) and are bumped as new models ship. The table above is a snapshot; check those files for the current values before relying on a specific model name.

Configuring keys in the UI

In the app:

  1. Open Settings → AI → Providers (the gear icon in the left sidebar).
  2. Pick a provider from the list. The panel shows class-level metadata (default model, supports tools, supports streaming) plus your current credential status: Configured (key saved), Env fallback (no key saved but a recognised env var is set on the server), or Unconfigured.
  3. Paste the API key into the API key field and click Save. For Ollama or self-hosted endpoints, set API base to the server URL.
  4. Click Test. Flowfile issues a 1-token ping and records the result on the credential (last_tested_at, last_test_status).

Providers tab on the AI settings page, with a status chip on each provider

Configure form for a single provider — masked API key field, optional default model, curated models list, and API base URL

To remove a key, click Delete. The credential row and the underlying encrypted secret are removed atomically.

Under the hood, these actions hit the BYOK routes:

Action HTTP
List providers + credentials GET /ai/providers
Save / update credential POST /ai/providers/{name}
Delete credential DELETE /ai/providers/{name}
Test credential POST /ai/providers/{name}/test

All BYOK endpoints require an authenticated user; credentials are scoped per user.

Env-var fallback

If no credential row exists for a user, Flowfile falls back to the standard provider env vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY / GOOGLE_API_KEY, GROQ_API_KEY, OPENROUTER_API_KEY) on the host process. The BYOK panel shows the provider as Env fallback in that case; saving a key in the UI takes precedence per user, and deleting it falls back to the env var again. Ollama needs no key — just an api_base (default http://localhost:11434).

Choosing models per surface

Each AI feature is a surface (explain, agent_staged, agent_live, agent_complex, cmd_k, ghost_node, settings_autocomplete, lineage, docgen, intent_classifier, cron; the chat drawer has no surface of its own and falls back to explain). Each provider class ships per-surface defaults — Haiku for fast paths, Sonnet for thinking paths, etc. — so you usually don't need to override anything.

When you do want to override:

  1. Per-credential default. In the BYOK panel, set Default model on the provider row. This wins over the class-level default on every surface except the Agent, which prefers the provider class' own tool-capable model where one is defined (see below).
  2. Curated model list. Some providers (notably OpenRouter) let you pin a list of models you've opted into. Flowfile will use the surface's preferred model if it's in your curated list; otherwise it falls back to the first model in the list. Useful when you want to limit yourself to free-tier models without losing the per-surface defaults.
  3. Per-request override. The API accepts an explicit model=... per call. This always wins.

The full resolution order, in priority:

  1. Explicit model= argument on the request.
  2. Credential row's default_model.
  3. Provider class' surface_models[surface], if it appears in your curated models list.
  4. First entry of your curated models list.
  5. Provider class' surface_models[surface].
  6. Provider class' default_model.

A note on the Agent surface: it requires a tool-capable model. Unless you pass an explicit model=, it routes to the provider class' surface_models entry for the agent surface where the class defines one, ahead of your credential default and curated list. A provider that cannot tool-call at all — the on-device model — is refused with a 422.

Rate limits

Cap per-provider request volume via env vars on the host: FLOWFILE_AI_<PROVIDER>_RPM (per minute) and FLOWFILE_AI_<PROVIDER>_RPD (per day). Unset means no enforcement. When a bucket fills, the scheduler delays the call and surfaces a "rate-limited, retrying in Ns" hint rather than 5xx-ing. Server Retry-After headers on 429 responses are always honoured. Limits are in-memory and per-provider (not per-(provider, model)); not persisted across restarts.

On-device model (no key required)

Flowfile can run a small model on the machine itself. It downloads a llama.cpp llama-server build plus a quantized model into the Flowfile data directory and drives it over its OpenAI-compatible API; it needs no key or account and works offline once installed. The provider id is local; it is deliberately kept out of the BYOK provider set above, so it never appears in the credential list and has no /ai/providers/* routes. The server boots lazily on the first AI call.

Set it up under Settings → AI → Providers, on the On-device AI row at the top of the list:

  1. With nothing installed, the row offers a single Set up button (labelled with the download size) that fetches the recommended model. Nothing downloads until you click it.
  2. Once installed, the row shows the active model with Start / Stop.
  3. Manage expands the row: a plain-language note on what a model this size can and can't do, the full catalog (install another size, switch the active one with Use, or delete one), and the context-window setting.

What a model this size can and can't do

The on-device models are small (2B to 9B parameters). They are good at chatting about your flow, explaining nodes, writing descriptions, and building a simple flow from one sentence (Simple build), and nothing leaves your machine. They cannot run the Agent: Auto-agent and Agent build step by step through tool calls, which these models don't support, so the chat drawer offers only Chat and Simple build while On-device AI is selected. Expect simpler answers than a cloud model, the occasional wrong column name, and slower replies on CPU; Flowfile sends the local model a trimmed view of the flow (at most 12 columns per node) to fit its context, and ⌘K / next-node suggestions miss more often. The same note appears in the app under Manage on the row.

On-device AI row on the Providers tab, expanded with Manage — what to expect from the model, the catalog, and the context window

The catalog is three Q4_K_M GGUF builds of Qwen3.5, pulled from Hugging Face on demand:

Model Notes
Qwen3.5 2B ~1.4 GB download. For low-RAM machines.
Qwen3.5 4B ~3.0 GB download. The default.
Qwen3.5 9B ~6.2 GB download. Needs ~8 GB free RAM; slow on CPU.

The server runs them with thinking switched off, so answers start at the first token. The earlier Qwen2.5-Coder 1.5B / 3B and Qwen2.5 7B downloads are older models: no longer offered for install, but one that is already on disk stays listed under Manage (marked older model), keeps working, and can be deleted. The context-window ceiling is per model (32k for the older ones, 128k for Qwen3.5).

The on-device model gets its own system prompt: a short assist prompt without the cloud footer that asks you to switch to agent mode (there is no agent mode on-device), and a one-line-per-node reference instead of the full one, so the whole prompt is about 3k tokens. If you ask it to build or change something, it describes the steps with the palette labels and points to Simple build for generating a whole flow.

Simple build asks the model for FlowFrame code rather than node settings, which the 4B handles well: a four-step flow comes back in a few seconds on an Apple Silicon laptop, and the code is shown in the chat bubble so you can see what it understood. Flowfile reads the script into nodes without running it.

Prebuilt runtimes exist for macOS, Linux, and Windows on x64 and arm64; elsewhere the card shows "Not available on this platform".

The on-device model streams but does not do tool calls, so it backs the read-only and text surfaces — Chat, Fix With AI, Generate Documentation, Lineage Q&A, the inline ✨ actions — plus one-shot flow generation. The Agent rejects it with a 422. For heavier work use a provider such as Ollama or OpenRouter.

Running Flowfile in Docker

The bundled llama-server needs libgomp1 in the image. If it exits on startup, see Troubleshooting.

Self-hosted (Ollama) setup

Ollama is the other offline path — a server you install and manage yourself. Quick start on macOS / Linux:

  1. Install and start Ollama (see ollama.com).

  2. Pull a tool-capable instruct model:

    ollama pull qwen3.5:9b
# or for the agent_complex surface:
ollama pull qwen3.6:35b-a3b

  3. In Flowfile, open Settings → AI → Providers and select Ollama:

    • Leave API key empty.
    • Set API base to http://localhost:11434 (the default; only override if your Ollama server is elsewhere).
    • Optionally set Default model to the tag you pulled.
    • Click Save, then Test.

Tool-call quality varies by model. Qwen3.5+ and Llama 3.1+ instruct models do tool calls correctly; older or non-instruct models sometimes return tool calls as text in the assistant content. The Agent surface compensates with Pydantic-repair on the tool-call shape.

Troubleshooting

503 Service Unavailable from any /ai/* endpoint. The feature flag is off. AI is on by default — this only happens if someone has explicitly set FEATURE_FLAG_AI=false in the env, or toggled it off at runtime via POST /system/feature_flags/ai. Re-enable by unsetting the env var (or setting it to true) and restarting, or toggle it back on via the admin endpoint.

422 Unprocessable Entity when starting an Agent session. The picked provider can't do tool calls — the on-device model never can. Check the Tools column in the provider table and switch to a tool-capable provider, or pass an explicit tool-capable model= on the request.

404 from POST /ai/providers/{name}. Provider name typo. The supported names are exactly: anthropic, openai, google, groq, openrouter, ollama (lowercase, no dashes).

On-device (local) model exits on startup in Docker / "no CPU backend found". The bundled llama-server loads CPU compute backends (libggml-cpu-*.so) that need the OpenMP runtime, libgomp1. The official flowfile-core image bundles it; a custom or older image may not — add it (Debian/Ubuntu: apt-get install -y libgomp1) and restart. The startup error now names the cause: no CPU backend found / exit code 127 → missing libgomp1; killed by SIGKILL → out of memory (give the container more RAM, or pick the 2B model / a smaller context under Manage on the On-device AI row, Settings → AI → Providers); killed by SIGILL → the image's architecture doesn't match the host CPU.

Credential Test returns ok=false with an authentication error. Key is wrong, expired, or missing required scopes. The error message from the upstream provider is surfaced in the error field of the test result.

See also