Skip to content
Flui Documentation

Inference and bring-your-own-model

Some Flui features need a large language model: the assistant that drives the platform in natural language, and the AI assist in the database console. Those features do not care where the model comes from. Anything that wants inference asks for a single, normalized endpoint, and a resolver works out how to produce it.

The endpoint always has the same shape: an OpenAI-compatible { baseUrl, apiKey }. Flui adds no abstraction over the model itself; model ids, request format and streaming are plain OpenAI-compatible. Flui only decides how the endpoint is sourced and credentialed, and that is where the two sources below differ.

Two ways to get an endpoint

Native: inference as a provider capability

Some cloud providers also run an OpenAI-compatible inference service. When they do, Flui models it as a capability the provider declares, like firewalls or DNS: an InferenceCapability carrying a baseUrl, an optional curated models list for display, a euDataResidency flag and a sharesComputeCredentials flag.

For Scaleway, sharesComputeCredentials is true: the provider’s inference service authenticates with the same IAM secret Flui already holds to provision your infrastructure. When you point Flui at a Scaleway account, you get inference with no separate AI key to create, paste or rotate. Flui reuses the compute credential as the inference bearer token.

A provider used only for inference, never for compute, would have no compute credential to borrow. For that case Flui supports a dedicated credential for inference, kept under the same credential model as everything else. Scaleway does not need one, because its single credential covers both.

The curated models list on the capability is only a placeholder shown before a key exists. Once Flui has a working credential, it reads the live model list from {baseUrl}/models, so the models you can pick are the ones the provider offers right now.

A connection you bring yourself

The second source is a connection you configure: any OpenAI-compatible endpoint, such as OpenAI, Mistral, a Claude-compatible gateway, or a server you run yourself. Such a connection declares no capabilities and provisions no infrastructure, so Flui does not treat it as a provider. It holds only connection config: a base_url, an API key (stored encrypted), an optional list of models, and an is_default flag.

These connections have their own small API under inference/connections: you can list them, create one, validate it against its endpoint, and delete it. Validation calls the endpoint. The dashboard validates as soon as you save, so a bad key or wrong URL shows up then, instead of the first time the assistant tries to answer a question. Over the API, saving does not check the key; call validate yourself.

Native inference comes with a provider you connect. A connection you bring is independent of any provider, and works the same whether or not you run any Flui-managed infrastructure.

Two levels: the installation’s, and your own

A connection you bring sits on one of two levels, and the level alone decides who can see and use it.

The installation’s connection is visible to and usable by everyone. It is the instance’s own credential to a model provider: one key, spent by every assistant on the installation. Whoever manages integrations connects and disconnects it.

Your own connection belongs to one person. Nobody else sees it in a list, and, more importantly, nobody else can spend it. You can connect one and disconnect it again without any permission, because a person who may add a key but not remove it is in the worst state a credential model can produce.

The dashboard’s settings screen manages the installation’s connections only, and hides those controls from anyone who may not manage integrations, instead of showing a button that answers with a refusal. You create a private connection through the API: POST to inference/connections/mine, and DELETE on inference/connections/mine/<id> to remove it.

Privacy is enforced where the key is spent

Hiding a private connection from other people’s lists protects nothing on its own. Almost nothing that uses a connection reads the list: the assistant’s chat, the agent loop and each console assistant take a connection id from a request and turn it straight into a live key. So Flui checks whose connection it is at that point, the one place all of those paths pass through.

As a result, a colleague who learns the id of your connection still cannot use it. Their attempt gets the same answer, in the same words, as an id that never existed, so guessing ids reveals nothing.

Validating a connection goes through the same check, because validating spends it: the check calls the model provider with the key. A colleague cannot even test your private connection, let alone chat with it.

Nothing is promoted quietly

A private connection stays private. Flui has no path that turns one into the installation’s connection: not when it is created, not later, and not when the person who brought it stops using Flui. It goes with that person, and nobody inherits it. If the installation should have a shared model, somebody connects one at the installation’s level.

Defaults do not cross levels

Marking your own connection as the default makes it the default for you. It does not replace the installation’s default for anyone else, and it is never handed to a colleague who asked for no connection in particular. That would leak the connection without anyone needing its id.

One person can see everything, and still cannot spend it

Someone who may manage users (the owner rung) sees every connection in the list, private ones included, because somebody has to be able to answer “what is plugged into this instance?”. They can also remove one. They cannot use it: seeing a colleague’s row does not let them charge that colleague’s model account, and the resolver answers them as if the connection did not exist, as it does for anyone else.

An empty list is not proof

Because private connections are listed to nobody else, an empty list does not mean nothing is connected. It means nothing is connected that you can see. This matters most for an agent reporting on the installation, and Flui tells the agent so explicitly, so it does not draw the wrong conclusion.

How the default endpoint is chosen

Most of the time the assistant does not name a specific source; it asks for the inference endpoint. The resolver answers in a fixed order of preference:

  1. A configured native provider, EU-first by the order providers are declared. If an inference-capable provider has a usable credential, it wins.
  2. Otherwise, a connection flagged as default that you can use: your own first, then the installation’s.
  3. Otherwise, the most recent connection you can use. When nothing is flagged, a usable connection is better than failing for lack of a flag.

If none of these produce an endpoint, resolution fails with “no inference endpoint configured” instead of guessing. Native providers come first because, when one is present, inference is usually already paid for and credentialed through the account you provision with, which is the simplest path.

What an agent may do with a model connection

An agent connected to Flui speaks through the model, so giving it authority over the model connection needs care. Two rules apply.

An agent cannot unplug the installation’s model. Deleting that connection requires the permission to manage integrations, and no agent scope carries it. An agent that could remove the endpoint every assistant on the installation runs on would be one bad turn away from a silent outage.

An agent can ask a person to connect one. The tool inference_connection_request does this. It has no argument that could carry an API key, and it never sees one. It reports the connections it can see, states that private connections are not counted, and returns the settings screen where a person picks a provider and pastes the key.

EU data residency is first-class

euDataResidency is a property of the capability, shown alongside each native provider. For a self-hosted platform that may run under European data rules, where the prompt goes is part of the decision, so Flui shows it when you choose a provider. Scaleway’s inference, for example, declares EU residency. A connection you bring points wherever you configure it, so residency there is up to you.

Picking a model id from a raw /models list is not obvious. Names are version-pinned and not portable across providers, and some models are a poor fit for tool use or cost more than you want for a first run. Flui ships a recommendation policy that you can override.

The policy is a config file at assistant/config/recommended-models.json. It names a recommendedProvider to start with, Scaleway, with the model deepseek-v4-flash-0731: it runs in the EU on the credential you already use for your servers, and it gave the most accurate answers in Flui’s own test questions. Then, per provider, it lists version-pinned model ids with short descriptions. For Scaleway that is deepseek-v4-flash-0731 as the default and gpt-oss-120b as a cheaper alternative. An entry can carry a caveat note so a trade-off is visible before you switch. A connection you bring yourself is not in the list, and its models stay selectable in the picker under the other models. An environment override can replace the whole file, so an operator can change the recommendations without a code change.

The recommendations are read at startup, validated, and exposed through the assistant API, so the dashboard can preselect a sensible default instead of showing an unfiltered model list.

Who consumes inference

Every AI feature uses the resolved endpoint:

  • The Flui Assistant, the natural-language layer over the platform, runs against whatever the resolver returns. See The Flui assistant and MCP.
  • The database console’s AI assist, which turns a question into a query, uses the same resolved endpoint in every console family that has one. See The database console.

Each asks for the endpoint as the person on the other end of the conversation, which keeps a private connection private in a chat as well as in a list.

Because they all go through one resolver, configuring inference once enables every AI feature at the same time, with the same data residency and credentials.