StageFlow¶
A framework for describing and running JSON-defined pipelines: a graph of
nodes, user-defined stages, an immutable data frame, CEL expressions, retry
and block-scoped try/except, parallel branches, loops over a list,
nested pipelines, and per-tenant limits on all of it.
Requires Python 3.11+ (the common-expression-language CEL binding does).
The picture is the editor: a separate web page that draws and debugs a graph, while the core executes it.
Where to start¶
New here? The tutorial builds a working pipeline step by step, with screenshots from the editor. Once the pipelines are somebody else's, Building a backend is the other half: your stages behind an HTTP contract, with a policy deciding what may be composed on them and how much it may spend.
| Page | What it covers |
|---|---|
| Quick start | installing, writing a stage, running a pipeline |
| Node types | the nine node types and their fields |
| Data model | the frame, argument buckets, outputs |
| Expressions | CEL, the .$ suffix, the vars namespace |
Building a backend¶
The editor executes nothing: your stages live on a backend of your own. Five steps, taking the example backend apart in the order it was written.
| Page | What it covers |
|---|---|
| What a backend is | the division of labour, and what the track builds |
| 1. Your stages | register_stage, specs, arguments, events, streaming |
| 2. The endpoints | the seven endpoints, a real Session, SSE, CORS |
| 3. What may be composed | a Policy per caller, and telling the editor |
| 4. What a run costs | reserve:, charge(), meters of your own |
| 5. Who is calling | plans, a credential, and where the check goes |
Reference¶
| Page | What it covers |
|---|---|
| The entry node | graph start and initial variables |
| The stage node | arguments, outputs, consume |
| The condition node | a fork on a CEL expression |
| The switch node | many roads, first matching case |
| The parallel node | concurrency, merge rules, cancellation |
| The try node | a region of the graph under except |
| The map node | a region of the graph run once per element |
| The subpipeline node | a nested graph and its boundary |
| The terminal node | the end of a run, result and artifacts |
| Errors | retry, how an error travels, the exceptions |
| Policy | restricting the stages and node types a pipeline may use |
| Limits | meters, budgets, what a stage spends |
| Variable typing | gradual typing, static and runtime checks |
| Localization | answering in the caller's language, stage prose per locale |
| Session control | stop/pause/resume, user input, snapshots |
| Step debugging | stopping between nodes, editing the frame |
| Built-in stages | what ships with the package |
| Stage specification | the YAML docstring contract |
| Schema and stage specs | JSON Schema and editor metadata |
Development¶
| Page | What it covers |
|---|---|
| Releasing | tag-driven publishing to PyPI |
Source and issues: github.com/leo-need-more-coffee/stageflow