Execution Model

Execution Model

Queue/Stack Architecture - Understanding how ION Flow manages execution order and data flow between nodes.

---

Overview

ION Flow uses a Queue/Stack execution model inspired by traditional database cursor patterns. This architecture provides:

• Deterministic execution order via FIFO queue
• Data reference tracking via stack history
• Stateless external orchestration - each API call is independent
• Persistence - execution can be paused and resumed

---

Core Concepts

Queue (FIFO)

The queue determines what nodes to execute next:

graph LR
    subgraph Queue [Queue - FIFO]
        Q1[Entry 1] --> Q2[Entry 2] --> Q3[Entry 3]
    end

    Push[PushNode] -->|adds entries| Queue
    Queue -->|next entry| Continue[ContinueNode]

Queue Entry Structure:

type Queue struct {
    Data         interface{} // Data to pass to node
    Source       *string     // Source node ID
    Target       *string     // Target node ID
    SourceHandle *string     // Output handle (e.g., "output")
    TargetHandle *string     // Input handle (e.g., "input")
    EdgeId       *string     // Edge that created this entry
}

Stack (Execution History)

The stack tracks what has been executed for data references:

graph TB
    subgraph Stack [Stack - History]
        S3[Latest: Node 3]
        S2[Node 2]
        S1[Oldest: Node 1]
    end

    S3 --> S2 --> S1

    Ref["{{$1.data}}"] -->|lookup| Stack

Stack Entry Structure:

type Stack struct {
    Data         interface{} // Node output data
    Source       *string     // Source node ID
    Target       *string     // Target node ID
    SourceHandle *string     // Output handle
    TargetHandle *string     // Input handle
    EdgeId       *string     // Edge ID
}

---

Execution Flow

Step-by-Step Process

sequenceDiagram
    participant Client
    participant Queue
    participant Stack
    participant Node

    Note over Client,Node: 1. RunNode creates initial queue entry
    Client->>Queue: Push initial entry

    Note over Client,Node: 2. PushNode processes and creates new entries
    Client->>Queue: Pop entry
    Queue-->>Client: Current entry
    Client->>Node: Execute node
    Node-->>Client: Outputs
    Client->>Queue: Push new entries (from edges)

    Note over Client,Node: 3. ContinueNode moves to next
    Client->>Queue: Get next entry
    Queue-->>Client: Next entry
    Client->>Stack: Push current to stack

    Note over Client,Node: 4. Repeat until queue empty

Data Reference Resolution

When a node references previous data using {{$1.field}}:

1. The expression parser finds the pattern
2. Looks up position 1 in the stack
3. Retrieves the data from that stack entry
4. Replaces the expression with the actual value

Example:

// Node n_3 references data from previous execution
nodeData := map[string]any{
    "user_id": "{{$1.response.id}}",      // From stack position 1
    "order_total": "{{$2.total}}",         // From stack position 2
}

// After resolution
nodeData := map[string]any{
    "user_id": "12345",
    "order_total": 99.99,
}

---

Queue Operations

Push to Queue

When PushNode executes, new queue entries are created for connected nodes:

// Node n_1 outputs to "output" handle
// Edge connects n_1:output -> n_2:input
// Edge connects n_1:output -> n_3:input

outputs := []models.Ot{
    {TargetHandle: helpers.String("output"), Data: result},
}

// Creates queue entries:
// 1. {Source: "n_1", Target: "n_2", Data: result}
// 2. {Source: "n_1", Target: "n_3", Data: result}

Pop from Queue

ContinueNode retrieves the next entry (FIFO order):

// Queue state: [n_2, n_3, n_4]
response := flow.ContinueNode(flowId)
// Returns n_2, Queue state: [n_3, n_4]

response = flow.ContinueNode(flowId)
// Returns n_3, Queue state: [n_4]

---

Stack Operations

Push to Stack

When moving to the next node, the current entry is pushed to the stack:

// In ContinueNode:
err = services.PushStack(db, models.Stack{
    Source:       queue.Source,
    Target:       queue.Target,
    SourceHandle: queue.SourceHandle,
    TargetHandle: queue.TargetHandle,
    Data:         queue.Data,
    EdgeId:       queue.EdgeId,
})

Reference Lookup

Stack lookups use the source node ID:

// Reference pattern: {{$1}} or {{n_1}}
// Looks up stack entry where Source matches

stack, err := services.GetStackBySource(db, "$1")
// Returns stack entry with Data field

---

Branching Behavior

Parallel Branches

When a node has multiple output edges, all branches execute:

graph TD
    A[Node A] -->|output| B[Node B]
    A -->|output| C[Node C]
    A -->|output| D[Node D]

Queue after A completes: [B, C, D]

All three nodes will be processed in order.

Conditional Branches

Condition nodes only output to one branch:

graph TD
    A[Condition] -->|then| B[True Branch]
    A -->|false| C[False Branch]

If condition is true: Queue gets [B] If condition is false: Queue gets [C]

---

Merge Points

When multiple branches converge on a single node:

graph TD
    A[Node A] -->|output| D[Merge Node]
    B[Node B] -->|output| D
    C[Node C] -->|output| D

The merge node executes once per incoming edge. Each execution has access to different stack data.

---

Expression Language Integration

Stack-Based References

Pattern	Description	Example
{{$1}}	Full data from stack position 1	{{$1}} → {"id": 123}
{{$1.field}}	Specific field	{{$1.id}} → 123
{{$1.nested.field}}	Nested field	{{$1.user.name}} → "John"
{{n_1.data}}	Named node reference	{{n_1.response}} → {...}

Built-in Functions

// Base64 encoding
"{{toBase64(password)}}"

// Array merging by key
"{{fuseByKey(orders, items, 'order_id')}}"

---

State Diagram

stateDiagram-v2
    [*] --> Empty: LoadFlow
    Empty --> HasItems: RunNode
    HasItems --> Processing: ContinueNode
    Processing --> HasItems: PushNode (with new edges)
    Processing --> Empty: PushNode (no more edges)
    Empty --> [*]: Flow Complete

Queue States

State	Description
pending	Entry waiting to be processed
processing	Currently being executed
completed	Successfully processed

---

Database Schema

Queue Table

CREATE TABLE queue (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    source TEXT,
    target TEXT,
    source_handle TEXT,
    target_handle TEXT,
    data TEXT,
    data_type TEXT,
    edge_id TEXT,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP,
    status TEXT DEFAULT 'pending'
);

Stack Table

CREATE TABLE stack (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    source TEXT,
    target TEXT,
    source_handle TEXT,
    target_handle TEXT,
    data TEXT,
    data_type TEXT,
    edge_id TEXT,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP,
    status TEXT
);

---

Performance Considerations

Queue Size

Large flows may accumulate many queue entries:

• Iterator nodes with large arrays create many entries
• Multiple parallel branches multiply entries

Stack Growth

The stack grows with each processed node:

• Used for data reference resolution
• Consider cleanup for long-running flows

SQLite Efficiency

• Indexes on id, source, target columns
• Batch inserts for bulk operations
• Connection pooling handled by Go’s database/sql

---

Related Documentation

• Flow Execution - Execution lifecycle
• Models - Queue model details
• Models - Stack model details
• Expression Library - Reference resolution