TotalFusionFOR TOTAL.JS AND NODE.JS

Errors and cancellation

Every error is a FusionError with a stable code.

CodeMeaning
REMOTE_ERRORThe function in the worker raised (Python) or returned an error (Rust and Go). err.remote has name, message and, in Python, traceback, which is also appended to err.stack. A function can give its error a code of its own.
TIMEOUT / ABORTEDThe call exceeded its timeout or its AbortSignal fired.
STREAMING_FUNCTIONA generator was called with fusion.python(): use fusion.stream().
CANCELLEDThe stream was cancelled by its consumer.
WORKER_CRASHEDThe worker died during the call. The call is not retried, because it may have had side effects, unless it was declared idempotent. The worker is replaced.
QUEUE_FULLThe bounded queue is full: backpressure instead of unbounded memory growth.
NOT_FOUNDUnknown module or function (or not exposed).
NOT_SERIALIZABLE / ENCODE_ERRORThe result / the arguments are not valid JSON.
SPAWN_FAILED / START_FAILED / START_TIMEOUTThe interpreter or the worker was not found, cannot start, or did not become ready in startTimeout. err.hint says how to install a missing library.
PROTOCOL_MISMATCHThe worker speaks another version of the protocol (a Rust or Go worker built with another version of the crate or the module).
WASM_TRAP / WASM_EXITA WebAssembly module trapped (an out of bounds access, unreachable) or ended its own process while it ran the call. The call fails, and the module starts again with a new instance.
INVALID_CONFIG / ALREADY_STARTEDAn option is wrong, or configure() was called after the pool started.
CLOSEDThe pool was shut down while the call was pending.

Cancellation#

async def functions are cancelled for real (asyncio cancellation). Synchronous functions cannot be interrupted: after a timeout the caller is rejected immediately, and if the function does not finish within cancelGrace (2 s) the worker is killed and replaced.

Rust workers cancel cooperatively: the function looks at call.is_cancelled() from time to time and returns. One that never looks is killed after cancelGrace, like a synchronous function in Python.

Go workers do the same with call.Cancelled(), and with call.Context(), a context.Context that is cancelled at the same time and that can be passed to whatever the function calls.

Edit this page on GitHub