Skip to content
flui.cloud

Inference and BYO LLM

Flui has features that need a large language model: the assistant that drives the platform in natural language, and the AI assist in the database console. Those features should not care where the model comes from. So Flui draws a clean line: anything that wants inference asks for a single, normalized endpoint, and a resolver behind that line works out how to produce it.

That endpoint is always the same shape — an OpenAI-compatible { baseUrl, apiKey }. No Flui abstraction sits over the model itself; the model ids, the request format, the streaming are all plain OpenAI-compatible. The only thing Flui owns is how the endpoint is sourced and credentialed, and that is where the two ways below differ.

Two ways to get an endpoint

There are exactly two sources of inference, and they are deliberately different things.

Native — inference as a provider capability

Some cloud providers also run an OpenAI-compatible inference service. When they do, Flui models that as a capability the provider declares, just like firewalls or DNS — an InferenceCapability carrying a baseUrl, an optional curated models list for display, a euDataResidency flag, and a sharesComputeCredentials flag.

The interesting flag is sharesComputeCredentials. For Scaleway it is true: the provider’s inference service authenticates with the same IAM secret Flui already holds to provision your infrastructure. So when you point Flui at a Scaleway account, you get inference for free — there is no separate AI key to create, paste or rotate. Flui reuses the existing compute credential as the inference bearer token.

A provider that only did inference — that you would never use for compute — would not have a compute credential to borrow. For that case Flui supports a dedicated credential whose purpose is inference rather than compute, kept under the same credential model as everything else. Scaleway just doesn’t need one, because its one credential already covers both.

The curated models list on the capability is only a placeholder for display before a key exists. Once Flui has a working credential, the live model list is read straight from {baseUrl}/models — so the models you can pick are always the ones the provider actually offers right now, not a list baked into Flui.

BYO connection — bring your own endpoint

The second source is a connection you configure: any OpenAI-compatible endpoint — OpenAI, Mistral, a Claude-compatible gateway, or a server you run yourself. This is not a provider. It declares no capabilities and provisions no infrastructure. It is pure connection config: a base_url, an API key (stored encrypted), an optional list of models, and an is_default flag.

These live behind their own small API under inference/connections — you can list them, create one, validate it against its endpoint, and delete it. Validation actually calls the endpoint, so a bad key or wrong URL is caught when you save it, not the first time the assistant tries to answer a question.

The split matters: native inference is something a provider has, and you get it by virtue of connecting that provider; a BYO connection is something you bring, independent of any provider, and it works the same whether or not you run any Flui-managed infrastructure at all.

How the default endpoint is chosen

Most of the time the assistant doesn’t name a specific source — it just asks for “the” inference endpoint. The resolver answers that with a fixed preference order:

  1. A configured native provider, EU-first by the order providers are declared. If an inference-capable provider has a usable credential, that wins.
  2. Otherwise, the BYO connection flagged as default.
  3. Otherwise, any single configured connection — when there is exactly one, it is implicitly the default, so the resolver uses it rather than failing for lack of an explicit flag.

If none of those produce an endpoint, resolution fails cleanly with “no inference endpoint configured” rather than guessing. The ordering favours native providers first because, when one is present, it usually means inference is already paid for and credentialed through the same account you provision with — the lowest-friction path.

EU data residency is first-class

euDataResidency is a property of the capability, surfaced alongside each native provider, not a footnote. For a self-hosted platform that may be running under European data rules, where the prompt goes is part of the decision, so Flui makes it visible at the point you choose a provider — Scaleway’s inference, for example, declares EU residency. BYO connections point wherever you configure them, so residency there is your call by construction.

Picking a model id from a raw /models list is not obvious — names are version-pinned and not portable across providers, and some are a bad fit for tool use or cost more than you’d want for a first run. Flui ships an opinionated, overridable recommendation policy to smooth that.

The policy is a config file at assistant/config/recommended-models.json. It names a recommendedProvider to get started with — currently Mistral, with the model mistral-small-2506, called out for its generous free tier — and then, per provider, a curated list of version-pinned model ids with short descriptions. Some entries carry a caveat note (for instance, a stronger model marked “not recommended on free plan”) so the trade-off is visible before you switch. The whole file can be replaced via an environment override, so an operator can re-point the recommendations without a code change.

These recommendations are read at startup, validated, and exposed through the assistant API so the dashboard can preselect a sensible default instead of dropping you into an unfiltered model list.

Who consumes inference

The resolved endpoint is the single seam every AI feature sits on:

  • The Flui Assistant — the natural-language layer over the platform — runs against whatever the resolver returns. See The Flui assistant and MCP.
  • The database console’s AI assist — the helper that turns a question into a query — uses the same resolved endpoint. See The database console.

Because both go through one resolver, configuring inference once — native or BYO — lights up every AI feature at the same time, with the same data-residency and credential story.