Deterministic Computations Inside Probabilistic AI Systems

An AI system is often called probabilistic because the language model estimates the probability distribution of the next token. This does not mean, however, that every stage of the application is necessarily random. The method of token selection, software code execution, the order of operations on the CPU or GPU, and reproducibility requirements are different aspects of the system.

Therefore, a composite architecture is practically useful: the model is responsible for interpreting natural language and deciding what to compute, while a standard software module handles how to perform the calculation. This approach combines the flexibility of a language model with controlled arithmetic, but it does not guarantee a correct answer on its own. The model might choose the wrong formula, mix up units, or pass incorrect initial data to the tool.

Four Distinct Properties That Must Not Be Conflated

In engineering discussions, it is useful to break down the word "determinism" into at least four concepts.

Property	Practical Question	What It Does Not Guarantee
Execution determinism	Will the same result be obtained under given conditions?	That the input data and chosen formula are correct
Bitwise reproducibility	Will all bits of the result match between runs?	That the result is mathematically accurate
Numerical accuracy	How close is the computed value to the required mathematical result?	That a rerun will yield the same bits
Interpretation correctness	Did the system correctly understand the task, units, and constraints?	That the computational engine is reproducible

Thus, a reproducible answer can be consistently wrong, and a numerically acceptable answer might not match bitwise with the result of another run. The required guarantee must be defined for the specific task, operation, device, and software environment.

Token Probability and Choice Randomness Are Not the Same Thing

At each step, a language model estimates the probabilities of possible next tokens. Subsequent behavior depends on the decoding strategy.

With greedy decoding, the token with the highest probability is selected. This choice itself does not require random sampling. Given unchanged input data, parameters, and computational environment, the strategy sets an unambiguous selection rule, provided the implementation also fixes the tie-breaking rule for maximum scores, although this still does not prove the determinism of the entire system.

With sampling, the next token is randomly selected from the distribution. It is the sampling procedure that adds randomness to the generation. Parameters that transform the distribution before sampling can change the diversity of responses, but the fundamental difference remains the same: greedy selects the maximum, sampling performs a random choice. This separation of strategies is described in the Hugging Face documentation. [web:32]

The caveat is essential: "probabilistic model" does not mean that every execution stage is necessarily random. Conversely, greedy decoding does not provide a universal guarantee of the same answer across all environments. The result can be influenced by the model version, operator implementation, hardware, parallel execution, and changes in the serving infrastructure.

Architecture: The Model Chooses the Operation, the Application Computes

A practical way to introduce deterministic calculations into an AI system is to offload them to an external tool: a calculator, a software function, an SQL query, a symbolic engine, or a specialized library.

The documented function calling flow looks like this:

1. The application sends the model a prompt and a description of available tools.
2. The model returns the function name and prepared arguments.
3. The application validates the call request.
4. The application code executes the corresponding function.
5. The result or a structured error is returned to the model.
6. The model generates a response for the user. [web:26]

The key boundary lies between steps 2 and 4. The model's request to call a tool does not perform the computation itself. The application is responsible for validating arguments, access rights, launching the code, resource limits, and error handling.

The simplified flow can be represented as follows:

User
    ↓
Language model: interprets the prompt
    ↓
Structured call: function name + arguments
    ↓
Application layer: validates and normalizes data
    ↓
Computational tool: performs the operation
    ↓
Result validation and error handling
    ↓
Language model: explains the result

This does not turn the entire AI system into a deterministic one. Prompt interpretation, tool selection, and final phrasing can remain probabilistic. Only the isolated computational stage can be deterministic under given conditions.

What to Check Before Running the Tool

The following list is an engineering recommendation, not a guarantee of a specific API. Before executing the call, the application layer must determine:

• whether the model is allowed to call the selected function;
• whether the argument structure matches the expected schema;
• whether the values have valid types and ranges;
• whether the units of measurement are consistent;
• whether mandatory parameters are present;
• whether the passed identifiers, paths, and expressions can be used safely;
• whether the user has the right to the requested operation and data;
• how timeouts, overflows, division by zero, and dependency unavailability are handled;
• whether confirmation is needed before an operation with external consequences.

After execution, it is also useful to check the type, range, and status of the result. A deterministic function will reliably repeat an error if the model consistently passes it incorrect data.

Deterministic Does Not Mean Mathematically Exact

Even a fixed formula can behave differently depending on the type of arithmetic and the order of operations. This is especially important for parallel floating-point computations.

In ordinary mathematics, addition is associative. In floating-point arithmetic, due to rounding, the equality

(a + b) + c = a + (b + c)

is not guaranteed in the general case. In other words,

(a + b) + c is not necessarily equal to a + (b + c).

In parallel floating-point computations, changing the grouping order of terms can change the result: finite arithmetic is not strictly associative. But such an order does not necessarily have to change between runs; it depends on the API, the algorithm, and the execution conditions. [web:51]

This requires different formulations of the guarantee:

• for integer arithmetic, the type range and overflow rules must be considered;
• for decimal arithmetic, scale, rounding, and implementation are important;
• for floating-point arithmetic, the permissible deviation and comparison order should be explicitly set;
• for symbolic computations, permissible expression transformations and the equivalence criterion must be defined.

The choice of engine depends on the task. Monetary values often require controlled decimal or integer arithmetic, and identity verification may require a symbolic engine. Ordinary floating-point arithmetic is suitable for many scientific and ML computations, but its error and reproducibility requirements should be defined separately.

Reproducibility Levels on CPU and GPU

The statement "the operation is deterministic" is incomplete without describing the conditions. NVIDIA CCCL distinguishes several levels of guarantee:

1. no reproducibility guarantee;
2. repeatability between runs on the same GPU with identical input data and settings;
3. repeatability between GPUs. [web:51]

This is visible in the example of cub::DeviceReduce: the current online NVIDIA documentation indicates run_to_run as the default mode. On the same GPU, identical input data, build, and launch settings yield the same reduction tree. This guarantee does not promise bitwise matching for pseudo-associative floating-point addition between GPUs. gpu_to_gpu is a separate mode for specified combinations of types and operators: for example, for float/double with cuda::std::plus, the documentation describes a reproducible accumulator. Unsupported combinations of types and operators are rejected at compile time when this mode is requested. The link in the article points to /unstable/, so it describes the current development page, not a pinned release. For a specific system, the CCCL/CUDA Toolkit version, actual type and operator, launch settings, and requested reproducibility level must be checked. [web:51]

For an applied system, it is useful to choose the required level in advance:

• Functional stability: the answer passes given checks, even if the lower bits differ.
• Numerical reproducibility: results match within a set tolerance.
• Bitwise reproducibility: the result representation matches completely under specified conditions.
• Cross-platform reproducibility: the stated guarantee extends to a specific set of devices and versions.

Bitwise identity can be important for regression testing, auditing, or comparing implementations. For many analytical and ML tasks, a numerical tolerance is sufficient. This decision cannot be made abstractly: it depends on the cost of an error and the purpose of the result.

Seed Controls Randomness, Not the Entire System

A fixed seed helps reproduce the sequence of values from a pseudo-random number generator. It does not automatically fix the order of parallel operations, library version, hardware platform, or the implementation of each computational core.

The PyTorch documentation recommends managing sources of randomness and allows enabling torch.use_deterministic_algorithms(...). In this mode, the library chooses a deterministic implementation where available, or reports an operation without such an implementation according to the call setting. At the same time, PyTorch explicitly limits the expected reproducibility to specific versions, platforms, and devices: identical results are not guaranteed between releases, different platforms, or CPU and GPU. Deterministic operations are also often slower. [web:2]

Consequently, the local configuration of a reproducible experiment usually includes not only the seed, but also:

• the version of PyTorch and dependencies;
• the model and its exact revision;
• decoding settings;
• device type and version;
• drivers and computational libraries;
• input data and the order of their processing;
• parameters of deterministic algorithms;
• the criterion for comparing results.

This list increases the controllability of the experiment but does not create an unconditional guarantee between any environments.

seed and system_fingerprint in Language Model APIs

For the specified OpenAI APIs, the documentation describes generation as non-deterministic by default. Identical seed, request parameters, and system_fingerprint should yield results that are "mostly identical", however, the seed does not guarantee determinism. system_fingerprint helps track changes in the model configuration and serving infrastructure. [web:1]

Two caveats are important here.

First, this is a description of specific APIs, not a universal property of all language models and services. Second, "mostly identical" results are not equivalent to strict bitwise reproducibility. If the application needs exact repeatability of a critical number, it is more reliable to obtain this number from a controlled computational module rather than extracting it from freely generated text.

How to Design a Verifiable Calculation

A practical template can be divided into several contracts.

1. Interpretation Contract

It is necessary to define which intents the model recognizes, which tools it can choose, and which data it must request from the user. If units or initial parameters are ambiguous, the system must not silently guess them.

2. Argument Contract

Function arguments must have a formal schema: types, mandatory fields, ranges, units, and constraints. The application layer validates this schema independently of the model's confidence.

3. Computation Contract

For the function, the following should be specified:

• the numeric type used;
• rounding rules;
• behavior on overflow and invalid values;
• the required reproducibility level;
• permissible deviation;
• supported devices and versions.

4. Result Contract

The tool must return not only the value but, if necessary, the unit of measurement, status, error details, and tracing data. The model presents this structured result but must not imperceptibly replace it with a new independent calculation.

5. Testing Contract

Tests of at least three classes are useful:

• verifying the correctness of the formula on known cases;
• repeated runs in the same environment;
• comparing supported devices and versions with the chosen tolerance.

If bitwise identity is required, the test must compare exactly the result representation. If numerical closeness is sufficient, the tolerance should be explicitly fixed, not replaced by a vague requirement of "approximately the same".

Typical Erroneous Conclusions

"Temperature is at minimum, so the entire system is deterministic"

No. The decoding strategy regulates token selection but does not fix model versions, infrastructure, the order of parallel computations, and external tools.

"Seed is set, so the result is guaranteed"

No. The seed controls the pseudo-random number generator but does not eliminate all hardware, algorithmic, and infrastructural sources of differences. The PyTorch and OpenAI documentation formulates guarantees with caveats. [web:2][web:1]

"The function is deterministic, so the model's answer is correct"

No. The code can accurately execute an incorrectly chosen operation with incorrect arguments. Interpretation correctness and computation correctness are verified separately. [web:26]

"The same number means an exact calculation"

No. Repeatability of a result does not prove its mathematical accuracy. Stable rounding, systematic error, or an incorrect formula can also be reproducible.

"A small difference between GPUs always means an error"

No. For floating-point operations, the difference can be a consequence of the order of parallel summation. The acceptability of such a difference is determined by a predefined tolerance and task requirements. [web:51]

Engineering Checklist

Before releasing an AI function that performs calculations, it is worth answering the following questions:

1. Where is the probabilistic decision made, and where does ordinary code execution begin?
2. Is greedy token selection or sampling from a distribution used? [web:32]
3. Who validates the tool name and its arguments?
4. What access rights are needed to perform the operation?
5. How are errors, timeouts, and invalid values handled?
6. Is bitwise identity needed, or is a numerical tolerance sufficient?
7. Which numeric type suits the task: integer, decimal, symbolic, or floating-point?
8. For which devices, libraries, and versions is reproducibility claimed?
9. Are the seed, decoding parameters, and software environment fixed?
10. Is the tool's result validated before passing it to the user?
11. Can the model alter or misrepresent the computed value?
12. Are there tests for formula correctness and reproducibility in supported environments?

Conclusion

Probabilistic control and deterministic computation can coexist in a single AI system. The language model interprets the prompt, selects the action, and prepares a structured call; the application validates the arguments and launches the computational module; the model then explains the obtained result. [web:26]

But determinism must always be formulated with conditions. Repeatability on one GPU does not promise bitwise matching on another device. A fixed seed does not eliminate all sources of discrepancies. The same result does not prove mathematical accuracy, and a deterministic tool does not fix an incorrect problem formulation. [web:51][web:2][web:1]

The main engineering rule: entrust the model with interpretation and operation selection, and critical computation to a verifiable implementation with explicitly defined types, tolerances, environment, and reproducibility level.

The sources describe primarily specific libraries and APIs, not a single standard of determinism for all AI systems. Their guarantees cannot be automatically transferred to other services and environments. When working with floating-point numbers, the permissible deviation and the required reproducibility level must be defined separately.

Sources

• [web:1] OpenAI, "How to make your completions outputs consistent with the new seed parameter". https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter
• [web:2] PyTorch, "Reproducibility". https://docs.pytorch.org/docs/stable/notes/randomness.html
• [web:26] OpenAI, "Function calling". https://developers.openai.com/api/docs/guides/function-calling
• [web:32] Hugging Face, "Generation strategies". https://huggingface.co/docs/transformers/generation_strategies
• [web:51] NVIDIA, "CUB cub::DeviceReduce API reference (unstable documentation)". https://nvidia.github.io/cccl/unstable/cub/api/structcub_1_1DeviceReduce.html