Core Module

Core Module

The heart of ION Flow - The flow execution engine responsible for processing workflow nodes, managing state, and orchestrating data flow.

---

Overview

The Core module (core/) is the foundational engine of ION Flow. It provides a generic, portable flow execution system that can be used independently or integrated into larger applications like the Backend.

Key Responsibilities

• Flow Execution - Load, run, and manage workflow instances
• Node Processing - Execute individual nodes and handle their outputs
• State Management - Persist execution state in SQLite databases
• Queue/Stack Management - Control flow order and data passing
• Expression Evaluation - Dynamic parameter resolution using expression language

---

Module Structure

core/
├── actions/            # Node type implementations
│   ├── flow/           # Main flow control API (LoadFlow, RunNode, etc.)
│   ├── condition/      # Conditional logic nodes
│   ├── switchnode/     # Switch/case decision nodes
│   ├── mapper/         # Data mapping/transformation nodes
│   ├── transformer/    # Advanced data transformation
│   ├── iterator/       # Loop/iteration nodes
│   ├── timer/          # Delay/timer nodes
│   └── store/          # Data storage nodes
├── models/             # Data structures
├── services/           # Business logic services
├── helpers/            # Utility functions
├── libs/               # External library wrappers
├── db/                 # SQLite database operations
└── global/             # Global constants and variables

---

Core API

The primary API is located in core/actions/flow/flow.go:

Function	Purpose
LoadFlow	Creates a new SQLite workspace and initializes the flow
RunNode	Starts execution of a specific node
PushNode	Pushes results to connected nodes via edges
ContinueNode	Gets the next node from the queue for processing

Basic Flow Execution

import "gitlab.com/altacrest/flow_binaries/core/actions/flow"

// 1. Load flow into workspace
err := flow.LoadFlow(flowId, flowData)

// 2. Run the initial node
response, err := flow.RunNode(flowId, startNodeId, "output")

// 3. Execute flow loop
for response.Queue != nil {
    // Push results and continue to next nodes
    result, err := flow.PushNode(flowId, outputs, message, status, clientPath)

    // Get next node from queue
    response, err = flow.ContinueNode(flowId)
    if response == nil || response.Queue == nil {
        break // Flow completed
    }
}

---

Execution Model

The core uses a pointer-based execution model similar to database cursors:

sequenceDiagram
    participant Client
    participant Flow as Flow API
    participant DB as SQLite
    participant Node as Node Executor

    Client->>Flow: LoadFlow(flowId, flow)
    Flow->>DB: Create workspace tables
    Flow-->>Client: Success

    Client->>Flow: RunNode(flowId, nodeId, targetId)
    Flow->>DB: Get node, push to queue
    Flow-->>Client: Response{Queue, Node, Params}

    loop While Queue has items
        Client->>Flow: PushNode(flowId, ots, ...)
        Flow->>DB: Pop queue, get node
        Flow->>Node: Execute node
        Node-->>Flow: Outputs
        Flow->>DB: Push outputs to queue
        Flow-->>Client: Result{Queues, Outputs}

        Client->>Flow: ContinueNode(flowId)
        Flow->>DB: Get next from queue
        Flow-->>Client: Response or nil
    end

Key Concepts

1. Queue - FIFO structure holding pending node executions
2. Stack - Tracks execution history for data references
3. Outputs (Ot) - Node output targets with associated data
4. Edges - Connections between node output and input handles

---

Available Node Types

Node Type	Directory	Purpose
Flow	actions/flow/	Main execution control
Condition	actions/condition/	If/else logic
Switch	actions/switchnode/	Multi-way branching
Mapper	actions/mapper/	Data mapping/transformation
Transformer	actions/transformer/	Advanced data manipulation
Iterator	actions/iterator/	Loop over collections
Timer	actions/timer/	Delay execution
Store	actions/store/	Data storage operations

---

Data Persistence

All flow state is persisted in SQLite databases:

• Each flow execution creates its own SQLite file
• Database path is configurable via db.DB_PATH
• Tables: nodes, edges, queue, stack, outputs

import "gitlab.com/altacrest/flow_binaries/core/db"

// Set storage path
db.DB_PATH = "./storage/flows"

---

Expression Language

The core includes a powerful expression language for dynamic parameter evaluation:

// Reference previous node outputs
"{{$1.response.data}}"

// Use built-in functions
"{{toBase64(password)}}"

// Merge arrays by key
"{{fuseByKey(orders, items, 'order_id')}}"

See Expression Library for complete documentation.

---

Dependencies

Library	Version	Purpose
github.com/expr-lang/expr	v1.17.2	Expression evaluation
github.com/google/uuid	v1.6.0	UUID generation
github.com/mattn/go-sqlite3	v1.14.24	SQLite driver

---

Related Documentation

• Flow Execution - Detailed execution process
• Execution Model - Queue/Stack architecture
• Models - Data structures
• Actions - Node implementations
• Services - Service layer
• Helpers - Utility functions
• Expression Library - Expression language