ION Flow - System Architecture

ION Flow - System Architecture

Comprehensive documentation of the ION Flow workflow automation system architecture, module relationships, and external library dependencies.

---

Overview

ION Flow is a modular workflow automation system built in Go with a layered architecture designed for:

• Modularity - Independent, self-contained modules
• Portability - Reusable packages across projects
• Scalability - Multi-tenant support with isolated data
• Real-time Execution - WebSocket-based flow monitoring

The system uses a Go workspace monorepo pattern with module replace directives, allowing independent development while maintaining cross-module dependencies.

---

System Architecture

High-Level Module Structure

ion-binaries/
├── core/           # Flow execution engine (the heart)
├── backend/        # ION Flow backend with auth & real-time features
├── server/         # REST microservice for flow APIs
├── packages/       # Reusable generic modules
│   ├── channel/    # Authentication engine (OAuth, API Key)
│   └── helpers/    # Generic utility functions
└── cli/            # DEPRECATED - Multi-language CLI experiment

Module Status

Module	Purpose	Status
core/	Flow execution engine	Active
backend/	ION Flow backend with WebSocket support	Active
server/	REST microservice for flow execution	Active
packages/	Reusable portable modules	Active
cli/	Multi-language CLI	Deprecated

---

Module Dependency Graph

graph TB
    subgraph packages [packages - Portable Modules]
        helpers[helpers]
        channel[channel]
    end

    subgraph core [core - Flow Engine]
        coreActions[actions]
        coreModels[models]
        coreServices[services]
        coreDb[db - SQLite]
        coreLibs[libs]
    end

    subgraph backend [backend - ION Backend]
        board[board - WebSocket Engine]
        controllers[controllers]
        backendServices[services]
        nodes[nodes]
    end

    subgraph server [server - REST Microservice]
        serverControllers[controllers]
    end

    channel --> helpers
    channel --> core
    backend --> core
    backend --> channel
    backend --> helpers
    server --> core

Dependency Rules

1. Core - Has no dependencies on other internal modules (only external libraries)
2. Packages - Self-contained, can depend on each other but not on core/backend
3. Channel - Depends on helpers and core models
4. Backend - Depends on core, channel, and helpers
5. Server - Depends only on core

---

Core Module (core/)

The heart of ION Flow - responsible for flow execution, state management, and node processing.

Internal Structure

core/
├── actions/        # Node type implementations
│   ├── flow/       # Main flow control API
│   ├── condition/  # Conditional logic nodes
│   ├── switchnode/ # Switch/case nodes
│   ├── mapper/     # Data mapping nodes
│   ├── transformer/# Data transformation nodes
│   ├── iterator/   # Loop/iteration nodes
│   ├── timer/      # Timer/delay nodes
│   └── store/      # Data storage nodes
├── models/         # Data structures (Node, Queue, Stack, Edge)
├── services/       # Business logic (node, queue, stack, edge services)
├── helpers/        # Core-specific utilities
├── libs/           # External library wrappers (expression language)
├── db/             # SQLite database operations
└── global/         # Global constants and variables

Core API (core/actions/flow/flow.go)

Function	Purpose	Signature
LoadFlow	Creates new SQLite workspace and initializes flow	LoadFlow(flowId string, flow models.Flow) error
RunNode	Starts execution of a specific node	RunNode(flowId string, nodeId string, targetId string) (*models.Response, error)
PushNode	Pushes results and continues to connected nodes	PushNode(flowId string, ots []models.Ot, message string, status string, clientPath string) (*models.Result, error)
ContinueNode	Gets next node from queue and prepares execution	ContinueNode(flowId string) (*models.Response, error)

Core Dependencies

Library	Version	Purpose
github.com/expr-lang/expr	v1.17.2	Expression language for dynamic parameter evaluation in node data
github.com/google/uuid	v1.6.0	UUID generation for flow and node identifiers
github.com/mattn/go-sqlite3	v1.14.24	SQLite driver for flow state persistence

---

Backend Module (backend/)

The main ION Flow backend - extends core with authentication, multi-tenancy, and real-time features.

Internal Structure

backend/
├── ion/
│   ├── board/          # WebSocket flow execution engine
│   ├── controllers/    # REST API controllers
│   ├── services/       # Business logic services
│   ├── models/         # Backend-specific models
│   ├── nodes/          # Platform-specific node implementations
│   ├── params/         # DTOs for controller parameters
│   ├── helpers/        # Backend-specific utilities
│   ├── middleware/     # HTTP middleware (auth, logging)
│   ├── database/       # Database connection management
│   ├── scheduler/      # Cron job scheduling
│   └── workspace/      # Flow workspace management
└── routes/
    ├── api.go          # REST API routes
    └── channels.go     # WebSocket routes

Board Engine (backend/ion/board/)

The WebSocket-based execution engine for real-time flow monitoring:

Component	File	Purpose
ExecutionEngine	engine.go	Manages flow execution lifecycle
Hub	hub.go	WebSocket connection management
CompanyDevFlow	company_dev_flow.go	Development mode execution
CompanyLiveFlow	company_live_flow.go	Production mode execution
AccountDevFlow	account_dev_flow.go	Account development mode
AccountLiveFlow	account_live_flow.go	Account live mode

Execution States

Status	Description
pending	Waiting to start
running	Currently executing
paused	Execution paused
waiting_input	Waiting for user input via WebSocket
stopped	Manually stopped
completed	Successfully finished
error	Failed with error

Backend Dependencies

Library	Version	Purpose
github.com/gorilla/mux	v1.8.1	HTTP router for REST API endpoints
github.com/gorilla/websocket	v1.5.3	WebSocket support for real-time flow execution
github.com/golang-jwt/jwt/v5	v5.2.1	JWT authentication and token validation
github.com/robfig/cron/v3	v3.0.1	Cron job scheduling for scheduled flows
github.com/rs/cors	v1.11.1	CORS middleware for API access control
github.com/joho/godotenv	v1.5.1	Environment variable loading from .env files
github.com/go-git/go-git/v6	v6.0.0	Git operations for flow version control
github.com/stretchr/testify	v1.11.1	Testing utilities and assertions
gorm.io/gorm	v1.30.0	ORM for database operations
gorm.io/driver/postgres	v1.5.11	PostgreSQL driver for multi-tenant data
gorm.io/driver/sqlite	v1.4.3	SQLite driver for local storage
gorm.io/datatypes	v1.2.6	GORM JSON and custom data types

---

Server Module (server/)

A lightweight REST microservice that exposes core flow APIs for external consumption.

Endpoints

Endpoint	Method	Purpose
/load	POST	Load flow into workspace
/run	POST	Run initial node
/continue	POST	Continue to next node
/push	POST	Push node results
/interrupt	POST	Interrupt execution
/mapper	POST	Execute mapper
/data-store-*	Various	Data store CRUD operations
/data-structure-*	Various	Data structure CRUD operations

Server Dependencies

Server inherits dependencies from core and adds minimal HTTP handling:

Library	Version	Purpose
github.com/mattn/go-sqlite3	v1.14.24	SQLite driver (inherited from core)

---

Packages (packages/)

Reusable, portable modules designed to work independently.

Channel Package (packages/channel/)

Authentication engine supporting multiple auth types:

packages/channel/
├── core/           # Core authentication service
├── services/       # Auth flow services
│   ├── authorize.go   # Authorization flow handling
│   ├── token.go       # Token retrieval
│   ├── refresh.go     # Token refresh logic
│   ├── callback.go    # OAuth callback handling
│   ├── info.go        # Connection info retrieval
│   └── response.go    # Response processing
└── models/         # Authentication models
    ├── app.go          # App configuration
    ├── app_action.go   # App action definitions
    ├── app_connection.go # Connection configuration
    ├── app_webhook.go  # Webhook configuration
    ├── connection.go   # Active connections
    └── field.go        # Field definitions

Supported Authentication Types

Type	Description
oauth2_code	OAuth 2.0 Authorization Code flow
oauth2_code_refresh_token	OAuth 2.0 with automatic token refresh
api_key	API Key authentication
basic	HTTP Basic authentication
client_credentials	OAuth 2.0 Client Credentials grant
owner_credentials	OAuth 2.0 Resource Owner Password grant

Channel Dependencies

Library	Version	Purpose
gorm.io/datatypes	v1.2.6	JSON field support for OAuth tokens and credentials

Helpers Package (packages/helpers/)

Generic utility functions:

packages/helpers/
├── config.go       # Configuration parsing utilities
├── sanitizer.go    # JSON/JSONC sanitization
└── pointers.go     # Pointer helper functions

Key Functions

Function	File	Purpose
ParseStringWithDefault	config.go	Parse string with fallback
ParseIntWithDefault	config.go	Parse integer with fallback
ParseBoolWithDefault	config.go	Parse boolean with fallback
SanitizeJSONC	sanitizer.go	Remove comments from JSONC
ParseJSONC	sanitizer.go	Parse JSONC to map
ParseJSONCArray	sanitizer.go	Parse JSONC to array
ParseJSONCToType	sanitizer.go	Generic JSONC to type parsing

---

Data Flow Architecture

Flow Execution Model

The execution model follows a pointer-based pattern similar to database cursors:

sequenceDiagram
    participant Client
    participant Engine as Board Engine
    participant Core as Core Flow
    participant DB as SQLite

    Client->>Engine: Start Flow
    Engine->>Core: LoadFlow(flowId, flow)
    Core->>DB: Create workspace

    Engine->>Core: RunNode(flowId, nodeId)
    Core->>DB: Get node, update status
    Core-->>Engine: Response with queue

    loop While queue has items
        Engine->>Core: PushNode(flowId, ots)
        Core->>DB: Push to queue
        Core-->>Engine: Result

        Engine->>Core: ContinueNode(flowId)
        Core->>DB: Pop queue, get next node
        Core-->>Engine: Response or null
    end

    Engine-->>Client: Execution complete

Data Persistence

Module	Storage	Purpose
Core	SQLite files	Flow execution state, node data, queues
Backend	PostgreSQL	Multi-tenant data, accounts, companies
Backend	SQLite	Local workspace storage

---

Multi-Tenancy Model

The backend supports two database connection patterns:

Global Connection (Account Level)

For data shared across tenants:

// Uses global database connection
if err := database.GetConnection().
    Model(&model).
    Where("id = ?", id).
    First(&result).Error; err != nil {
    return nil, err
}

Tenant Connection (Company Level)

For company-isolated data:

// Uses company schema for tenant isolation
if err := company.CompanySchema(models.TableName()).
    Where("id = ?", id).
    First(&result).Error; err != nil {
    return nil, err
}

---

Authentication Flow

flowchart TD
    A[Start Auth] --> B{Auth Type?}

    B -->|oauth2_code| C[Generate Auth URL]
    C --> D[Redirect to Provider]
    D --> E[Callback with Code]
    E --> F[Exchange for Token]
    F --> G[Get User Info]

    B -->|api_key| H[Make Test Request]
    H --> I[Process Response]

    B -->|client_credentials| J[Get Token]
    J --> K[Get Info]

    B -->|basic| L[Make Auth Request]
    L --> M[Process Response]

    G --> N[Return Connection]
    I --> N
    K --> N
    M --> N

---

Technology Stack Summary

Category	Technology	Version
Language	Go	1.24.3
HTTP Router	Gorilla Mux	1.8.1
WebSocket	Gorilla WebSocket	1.5.3
ORM	GORM	1.30.0
Primary Database	PostgreSQL	-
State Storage	SQLite	3.x
Expression Language	expr-lang/expr	1.17.2
Authentication	JWT	5.2.1
Job Scheduling	robfig/cron	3.0.1
Version Control	go-git	6.0.0

---

Related Documentation

• Core Module - Flow engine details
• Backend Module - Backend specifics
• Channel Package - Authentication details
• Server Module - REST microservice